I've been reading over the details for Microsoft's own secure computing effort - the Next-Generation Secure Computing Base (or NGSCB for short). It presents a lot of radical ideas as far as application security is concerned, including:

• CPU and memory controller changes to enable strong process isolation (each application has its own execution memory space)
• Changes to secure user input (keyboard and mouse) and output (display integrity and confidentiality) to enable a secure path to and from the user

The first is not so bad. But the second has some pretty scary consequences.

On windows, most applications use windows hooks (e.g. SetWindowsHookEx) to extend windows. SetWindowsHookEx hooks come in various types - you can use them to monitor messages to windows, or pick up all keyboard or mouse input, or use them to record and play back user input(journaling).

Essentially the way the system works is you write a dll that implements a function that will be called whenever a hook event happens. Then you give the OS a handle to the dll, the hook type you want, and the address of the function in the dll. Then you identify the target thread, or pass 0 for all threads. In most cases, 0 is passed - to effect all threads. Then whenever an event happens for the hook type you registered, the hook framework will:
a) check that the security context of the thread that registered the hook against the security context of the process causing the event, and if allowed, load the hook into the process. This is why some hook apps will not load into programs run as other users with the runas.exe utility. If it passes this it will then
load the dll into the process if it has not already been loaded
c) and call the event function in the dll with information about the event.

So this is a pretty useful way to extend a lot of things. But it's definately not for the squeamish. This probably seems like a lot of work just to extend a single functionality in windows, but there isn't much of an alternative. With the window manager at least - all the important code runs in kernel mode in win32k.sys under the context of the calling thread. The kernel side functions are not exported, and are in a hidden table which is not accessible without using a hack. And even then, there is no guarantee that the function you want to hook will reside at the same place in the table when win32k.sys is updated via windows update, a service pack, or a newer version of the OS. - The functions are assigned to various sections of the table using macros which assign them in alphabetical order. Hooks are easier and more reliable than this.

Getting to the point though, NGSCB will undoubtedly shield processes from hooks. The result will be that developers who rely on windows hooks to extend or modify default OS behaviors will not be able to do so in those processes, resulting in inconsistent application of the new behavior, annoying the user and ultimately devaluing the software.

It's unfortunate that the same entry points used to extend default OS behaviors in a lot of interesting ways can also be used in such detrimental ways - viruses and trojans.

I am not against secure processes. There's definately a need for it. But when the move blocks ISV's from extending os behavior, it hurts them and stands in the way of OS innovation. Windows XP includes a theme system that allows applying any theme - they implemented this a good 5 years after we started doing it. Their design is well thought out, and elegant. But would it have happened if we hadn't done it first? (Please note that I am not accusing the developers themselves of anything here.)

ISV's that extend the OS have to be quite nimble with their software - the OS vendor could implement a feature they are implementing at any time - so they are constantly having to improve upon the original idea. WindowBlinds can do a lot more things than XP Visual Styles, for instance, and mostly faster, thanks to countless hours of optimization work from the lead developer.

But I digress. I propose that we should be able to modify the default behavior of secure apps, in a controlled way. Ideally there would be some sort of signing system, where either the OS vendor, or the user, approves an application for extending these secure programs.

At the least, it could be an additional hook framework check, and at the most, for example, should Aero be 3d polys with a whole new API, it could be a set of interfaces. Locking everyone out is not a good solution - it's akin to a man being a prison in his own home.