In the interests of helping secure the platform, we want more people to opt-in to using Data Execution Prevention (aka DEP aka NX), and we have lowered the barrier to entry for application developers in Windows Vista SP1, Windows XP SP3 and Windows Server 2008.
We’ve added some new APIs that allow a developer to set DEP on their process at runtime rather than using linker options. The new APIs also give developers some more flexibility if your application uses an older version of the Abstract Type Library (ATL.) Before I explain the new APIs, let me give you a little history behind ATL and NX.
Some ATL History: ATL has been around for a long time; it’s reasonably light-weight and allows developers to build COM components rapidly. It also includes classes for manipulating security descriptors and such; to be honest, it makes working with Windows security objects open to mere mortals.
Older versions of ATL, and by older I mean pre-Visual C++ 2005, used dynamically generated code in small isolated cases. Obviously, without the appropriate APIs this is going to cause problems on a DEP-enabled computer, because you can’t execute data. This code is referred to as a “thunk” and versions of ATL in VC++ 2005 and later work correctly with DEP.
The APIs: The most important API added is SetProcessDEPPolicy, which sets the DEP policy for the running process. You would normally use this function pretty early in main.
The function takes only one argument: the policy setting. The possible values are:
Microsoft, Windows Vista, Windows XP, Windows Server 2008, Service Pack, SP1, SP3, Vista SP1, XP SP3, WS2008, NX APIs, API