In the previous article on Access Violations, we briefly mentioned Data Execution Prevention (DEP). To begin with, some quick background on DEP. Data Execution Prevention, or DEP, is Microsoft’s software implementation that takes advantage of hardware NX or XD support. NX stands for No Execute and XD stands for Execute Disabled and are the ability for the processor to mark physical memory locations with a flag indicating whether or not the data in that location should be executable or not. NX is AMD’s implementation and XD is Intel’s, but they are basically the same thing. This software support requires the Windows PAE kernel be installed, but this should happen automatically, so you don’t have to set the /PAE switch in your Boot.ini. What all of this means is that with DEP, the operating system has the ability to block certain code from executing on the system. DEP was first introduced with Windows XP Service Pack 2 and has been included in every Microsoft OS and service pack since then.
With hardware enforced DEP, all memory spaces are automatically marked as non-executable unless they are explicitly told they are being allocated for executable code. This flag is set on a per-page basis and is set via a bit in the page table entry (PTE) for that page. If something tries to execute code from a memory region that is marked as non-executable, the hardware feature passes and exception to DEP within Windows and lets it know that this is happening. DEP then causes an assert within the code stack that is executing, which causes it to fail with an access violation, which should look pretty much like the following: