It's a mitigation for a software exploitation technique called Return Oriented Programming (ROP). The mitigation is referred to as 'Control Flow Integrity' (CFI).
Essentially an attacker who has the ability to exploit the first stage of a vulnerability will be able to stitch together "gadgets" from the program to build up a second stage of the exploit.
Control flow integrity, to my understanding, applies a validation or restriction of the program's call graph. This limits the attackers ability to just stitch up their own arbitrary call graph. There are 'forward edge' protections (calling a function) and 'reverse edge' protections (ret). But of course there are more ways to control the flow of a program, as this document discusses - like longjmp.
I won't try to get more detailed as I'm not an expert. Hopefully this will help you find more information.
I'll add on since this is the most informative post so far (and I've written a static binary re-writer to add shadow stack protection to an existing binary).
A shadow stack is a limited subset of the call stack that only stores return addresses. In normal operation, Every time your compiled program makes a function call, it stores the return address on the main call stack (modulo certain compiler optimizations) so that when the called function returns, your program can resume executing directly after the point at which it called the function.
With a shadow stack, when a function is called, the return address is copied to a separate "shadow" stack as well as the call stack. When the called function returns, the return address on the two stacks are compared and the program fails if they are different.
In new Intel microprocessors, the shadow stack is implemented in hardware. The numerous corner cases require software support that the article describes.
Can you provide some details on your binary rewriter to add shadow stack support? Was this a pure software approach, or was it designed to take advantage of the support in new intel microprocessors? Do you have a write up of or can you give a quick overview of your methodology? Is the source code published somewhere?
No, it was proprietary code, and it wasn't for an Intel processor. It was a pure software approach, but the particular (embedded) environment made it harder to attack the shadow stack itself.
I had a pretty cool optimization that I don't think anyone's figured out yet. Oh well. That's the downside of software-as-trade-secrets.
https://software.intel.com/content/www/us/en/develop/article...
Essentially an attacker who has the ability to exploit the first stage of a vulnerability will be able to stitch together "gadgets" from the program to build up a second stage of the exploit.
Control flow integrity, to my understanding, applies a validation or restriction of the program's call graph. This limits the attackers ability to just stitch up their own arbitrary call graph. There are 'forward edge' protections (calling a function) and 'reverse edge' protections (ret). But of course there are more ways to control the flow of a program, as this document discusses - like longjmp.
I won't try to get more detailed as I'm not an expert. Hopefully this will help you find more information.