The Sad History of the Microsoft POSIX Subsystem

When Windows NT was first being developed, one of the goals was to make the kernel separate from the programming interface. NT was originally intended to be the successor to OS/2 but Microsoft also wanted to include compatibility to run Windows 3.x applications and to meet 1980s era DoD Orange Book and FIPS specifications to sell to the defense market.  As a result, Windows NT was developed as a multiuser platform with sophisticated discretionary access controls capabilities and it was implemented as a hybrid microkernel 3 userland environments:

  • Windows (Win32)
  • OS/2
  • POSIX

Microsoft had a falling out with IBM over Win32 and the NT project split from OS/2. The team focus shifted to Win32 so much that the Client-Server Runtime Subsystem (CSRSS) that hosts the Win32 API became mandatory and OS/2 and POSIX subsystems were never really completed but they were shipped with the first five versions of Windows NT through Windows 2000. The OS/2 subsystem could only run OS/2 1.0 command-line programs and had no presentation manager support. The POSIX subsystem supported POSIX.1 spec but provided no shells or UNIX-like environment of any kind. With the success of Win32 in the form of Windows 95, the development of the OS/2 and POSIX subsystems ceased. They  were entirely dead and gone from Windows XP and Windows Server 2003.

Meanwhile, around 1996,  Softway Systems developed a UNIX-to-Windows NT porting system called OpenNT. OpenNT was built on the NT POSIX subsystem but fleshed it out into a usable UNIX environment. This was at a time when UNIX systems where hugely expensive. Softway used OpenNT to re-target a number of UNIX applications for US Federal agencies onto Windows NT. In 1998, OpenNT was re-named Interix. Softway Systems also eventually built a full replacement for the NT POSIX subsystem in order to implement system calls that the Microsoft POSIX subsystem didn’t support and to develop a richer libc, single-rooted view of the file system and a functional gcc.

Microsoft acquired Softway and the Interix platform in 1999. Initially Interix 2.2 was made available as a fairly expensive paid add-on to Windows NT 4 and Windows 2000. Later it was incorporated as a component of Services for UNIX 3.0 and 3.5 (SFU) and SFU was made free-of-charge. When Interix became free, Microsoft removed the X11 server component that was previously bundled with Interix because in the wake of U.S. vs Microsoft, they did not want to defend law suits from the entrenched and expensive PC X Server industry but the X11 client libraries remained.

SFU/Interix 3.0 was released in early 2002 followed up with SFU 3.5 less than two years later and cool stuff got implemented like fast pthreads, fork(), setuid, ptys, deaemons with RC scripts including inetd and sendmail among other things. InteropSystems ported OpenSSH and developed a high-performance port of Apache using pthreads and other proof-of-concept ports like GTK and GIMP among many other things. Hotmail even ran on Interix. And enterprising people did cool things like a Linux ELF binary loader on top of Interix.

I got into this stuff and built and donated ports to the SFU/SUA community, including cadaver, ClamAV, GnuMP, libtool, NcFTP, neon, rxvt and gnu whois. My company sponsored the port of OpenSSH to Interix 6.0 for Vista SUA (because it broke backwards compatibility with Interix 3.5 binaries). We ran Interix on all of our workstations and servers. We used it for management, remote access and to interop with clients who used Solaris, Linux and OS X on various projects.

Slowly Going Off the Rails

With Windows Server 2003 R2 (and only R2), Interix became a core operating system component, rebranded as “Subsystem for UNIX Applications” (SUA). Around this time, the core development team was reformed in India rather than Redmond and some of the key Softway developers moved on to other projects like Monad (PowerShell) or left Microsoft. Interix for Windows Server 2003 R2 (aka Interix 5.2) was broken. It shipped with corrupt libraries and a number of new but flawed APIs and broke some previously stable APIs like select(). Also, related to the inclusion of Interix as an OS component, SP2 for Windows Server 2003 clobbers Interix 3.5 installations.

Things have been downhill from there. It’s not just that obvious things didn’t get implemented like a fully-functional poll() or updating binutils and gcc to something reasonably modern. The software suffered from severe bitrot.

One of the consequences of including SUA as an OS component has been that a bifurcation of the “subsystem” from the “tools”. The subsystem consists of just a few files: psxss.exe, psxss.dll, posix.exe and psxrun.exe. This implements the runtime and a terminal environment but nothing else, not even libc. In order to get shells, PTYs and usable programs, you have to install the “Utilities and SDK for UNIX-based Applications”  (aka tools) which is sizable download. Apparently Microsoft has concern about bundling GPL code onto the actual Windows media.

OK. This is a little weird but not a big deal except that the development timeline of the tools is now completely out of whack with Windows releases. The tools for Vista were only available in beta when Vista went gold and the version for Windows Server 2008 and Vista SP1 was not available until about a month after Vista SP1/Win2k8 was released. When Windows 7 was released no tools were available at all in July 2009 when Windows 7 was released. They didn’t become available until 8 months later in March 2010 and contain no new features.

To top things off, while SFU 3.5 ran on all versions of NT 5.x, SUA only runs on Windows Server and the Enterpise and Ultimate client editions. SUA is not available on Vista Business or Home and Windows 7 Professional and Home editions.

Is Interix Dead?

For some reason Microsoft seems to be ambivalent about this technology. On the one hand they bring it into the core of the OS and make it a “premium” feature that only Enterprise and Ultimate customers get to use and on the other they pare back development to almost nothing.

Interix has been supported with support forms and a ports tree maintained by InteropSystems collectively known as SUA Community which operates with supplemental funding from Microsoft. The /Tools ports tree is the source for key packages not provided by Microsoft such as Bash, OpenSSH, BIND, cpio and a ton of libraries that Microsoft does not bundle.  Microsoft has been increasingly reluctant to fund the SUA Community and has survey users on a number of occasions. The latest survey was very pointed and culminated with Microsoft cutting off funding and shuttering the SUA Community site on July 6th, 2010 but a few days later it was back online. I’m not sure how or why.

I have no inside knowledge but my gut says that Interix has lost internal support at Microsoft. It is being kept on life support because of loud complaints from important customers but it is going nowhere. I will be surprised if there is a Subsystem for UNIX-based Applications in Windows 8. I think the ambivalence is ultimately about an API war. At some level, the strategerizers have decided it is better to not dignify UNIX API with support. I think the calculus is that people will still use Windows but it chokes off oxygen for UNIX-like systems if it takes a lot of extra work to write cross-platform code for Windows and UNIX—the premise being that you write for Windows first because that’s where the market is. Furthermore, in a lot of business cases what is needed is Linux support or Red Hat Linux version X support in order to run something. I think Microsoft realizes that it is hard for Interix to beat Linux which is why SUSE and Red Hat Linux can be virtualized under Hyper-V.

I also believe that Microsoft sees C/C++ APIs as “legacy”. I think they want to build an OS that is verifiably secure and more reliable by being based on fully managed code. The enormous library of software built for the Windows API is a huge legacy problem to manage in migrating to such a system. Layering POSIX/UNIX on top of that makes it worse.

Whatever the reason, it seems pretty clear that Interix is dying.

Advertisements

Upgrade and Zenburn the Console Window

Background

In Windows, console windows (aka command line or “DOS” windows) are special. In a UNIX-like environment a terminal emulator talks to a local virtual teletype and connects to three text streams: standard in, standard out and standard error. From the first version of Windows NT to now, the client-server runtime subsystem automatically creates a special window environment for any command-line program or shell. In fact, up until Windows 7, the window was actually owned by csrss.exe which runs with system privileges. These windows mostly look like the rest of Windows but they are different in deep ways. Even if you remove the entire graphics and windowing infrastructure from Windows as in Windows Server Core, you are left with the GINA logon screen and a console window which looks exactly the same because the code to draw them is in csrss.exe rather than the normal window manager. That’s also the reason why they aren’t themed in Windows XP. Console Windows also can’t be resized in the usual way. On the other hand, Console windows offer a richer programming environment than sdtin,stdout and sderror because they always have a title bar and exist within a window station which means that services like the clipboard can be assumed. The gist is that for reasons of backwards compatibility and security, it is difficult for Microsoft to change very much about how console windows work. And besides, it’s a nerd feature that normal people never use.

powershell-standard-console-scrollbars

For me, there are some minor annoyances with console windows that I mostly try to ignore:

  1. Console Windows aren’t easily resizable. You have to change the definition of the rows and columns. Dragging the window borders either creates scrollbars or does nothing.
  2. Copy operates in a block mode rather than a line mode. That means if you copy text, you end up having to fix it where line breaks were added or you can accidentally leave out columns and end up with garbage.
  3. Marking for copy blocks the execution of anything trying to write to the window. This makes “Quick Edit” mode dangerous because clicking on the window tends to freeze it. Because of item 1 and 3, I end up creating giant console windows and leaving them that way. The shortcuts that PowerShell creates for itself do this by default.

Enter Console.exe

Its not easy to replace console windows with something else. If you use a terminal emulator with VTYs in the POSIX subsystem (like xterm in the POSIX/SUA subsystem) some console applications that expect to interact with the console window or the window station service won’t run at all or only work in a special mode, like PowerShell. Also, the VTYs exist in the POSIX subsystem and don’t have access to your current window station, so you can’t start windowing applications from a terminal on a VTY (like “explorer .” to open Explorer in the current command shell directory).

Console (aka Console2) is a modern terminal environment like a Gnome Terminal or Konsole  for Windows that is 100% compatible with applications that expect a console window running in a window station. That’s because it works by hooking and hiding the console window created by CSRSS while providing a richer user environment. I gave up on this thing when I started using Vista x64 because it was broken. It has matured a lot since then and the latest beta works great with Windows 7 x64.

  • Arbitrary window resizing by dragging the borders!
  • Copy selects lines and keeps line breaks intact like xterm et al. (Hint: the default is SHIFT+{mouse-select} to select, {mouse-click} to copy and {mouse-middle-click} to paste.)
  • Available tabbed environment.
  • Easily configure fonts without editing the registry.
  • Save multiple shell environments (like visual studio command prompt, cygwin, powershell, etc.)
  • Also, there are toys like transparency and background images.
  • All the key bindings and window layout stuff is configurable.
  • The windows console function key bindings (like F7 for history) still work.
    Unfortunately, selecting text still blocks the underlying windows console and therefore the execution of any script or application that might be generating text, but we get a configurable quick edit behavior where the defaults won’t have you accidentally selecting and blocking a window.

It Just Needs Zenburn

Zenburn is a low contrast color scheme originally developed for Vim and subsequently ported to almost everything. It’s a dark, low eye strain theme that is very addictive.

Here are Zenburn colors for console.xml:

 

<colors>
        <color id="0" r="62" g="62" b="62"/>
        <color id="1" r="100" g="100" b="175"/>
        <color id="2" r="0" g="128" b="0"/>
        <color id="3" r="0" g="128" b="128"/>
        <color id="4" r="51" g="35" b="35"/>
        <color id="5" r="170" g="80" b="170"/>
        <color id="6" r="220" g="220" b="0"/>
        <color id="7" r="220" g="220" b="204"/>
        <color id="8" r="192" g="128" b="128"/>
        <color id="9" r="175" g="175" b="255"/>
        <color id="10" r="127" g="159" b="127"/>
        <color id="11" r="140" g="208" b="211"/>
        <color id="12" r="227" g="113" b="113"/>
        <color id="13" r="200" g="128" b="200"/>
        <color id="14" r="240" g="223" b="175"/>
        <color id="15" r="255" g="255" b="255"/>
</colors>

Console.sf.net with Zenburn colors showing muted red error.zenburn-console

Console.sf.net with muted colors showing F7 history popup.zenburn-console-popup

Console.sf.net with Zenburn colors, multiple tabs and transparency. (The elephant is my desktop wallpaper.)

console-tabs-transparency

Use Image Hijacking to Globally Replace Notepad.exe

I like to use a progammer’s text editor called EmEditor. Some people like Notepad++, vim, etc. You can change file associations but some things are just hard-coded to call notepad.exe. Notepad.exe is a protected system file which makes it hard/unsafe to replace.

Instead of replacing it, you can hijack calls for notepad.exe and redirect them to another text editor by registering a fake debugger.

You just need a tiny script to capture the arguments being sent to notepad and send them to your text editor of choice, instead. Replace the paths in the example with the paths and text editor for your system.

Script

 var shell, args="",
    editor = "C:\\Program Files\\EmEditor\\EmEditor.exe";
if( WScript.Arguments.length > 0 ){
	for( i=1; i<WScript.Arguments.Length; i++ ){
		args += WScript.Arguments(i) + " ";
	}
	shell = new ActiveXObject("WScript.Shell");
	shell.Exec("\"" + editor + "\" " + args);
}

Registry

new-item 'HKLM:\software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe'
Set-ItemProperty 'HKLM:\software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe' -name  Debugger -value 'wscript "C:\Program Files (x86)\Utilities\replace-notepad.js"'

Update

It’s been pointed out by commenter “MM” that files with multiple spaces together in the name break the WScript hijack. In this scenario, the Dubugger redirect splits the file name on whitespace and discards the extra whitespace. I put together a little hijack in C# that fixes this problem as long as there aren’t two or more files with names that differ only by the amount of whitespace between words in the file name. If that’s the case, you’re out of luck and should probably think about why you are naming files this way. Otherwise, this slightly fancier hijack will handle files with multiple spaces separating words in the name.

using System;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Text.RegularExpressions;
using System.Windows.Forms;

class Program
{
	public static void Main(string[] args)
	{
		string handler = args[0];
		
		if(args.Length == 3) Process.Start(handler, string.Format("\"{0}\"", args[2]));
		else if(args.Length > 3)
		{
			string dirplus = args[2];
			int index = dirplus.LastIndexOf("\\");
			var dir = new DirectoryInfo(dirplus.Substring(0,index));
			var filepart = dirplus.Substring(index + 1);
			var expression = filepart + "\\s+" + string.Join("\\s+", args.Skip(3));

			var names = dir.GetFiles(filepart + "*").Select(x => x.Name);
			var files = names.Where(x => Regex.IsMatch(x, expression));
			if(files.Count() > 1)
			{
				MessageBox.Show(
					string.Format("{0} possible matches found with file names that differ only by spaces between words.\r\nChoosing the first match.", files.Count()), 
					"Warning", 
					MessageBoxButtons.OK, 
					MessageBoxIcon.Warning);
			}	
				 
			Process.Start(handler, string.Format("\"{0}\"", files.First()));
		}
	}
}

> csc /target:winexe .\hijack.cs

In the regsitry, set the Debugger like this:

Set-ItemProperty 'HKLM:\software\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\notepad.exe' -name  Debugger -value '"C:\Program Files (x86)\Utilities\hijack.exe" "C:\Program Files\EmEditor\EmEditor.exe"'

Run Process Explorer x64 LUA from Program Files without UAC prompts

procexpProcess Explorer by Mark Russinovich is a great improvement over the Task Manager program that ships with Windows. It give a ton of information about processes running on your computer. It keeps presents a full range of stats on every process including memory consumption and CPU time, loaded DLLs and open handle, strings embedded in the binary, environmental variables defined for the process, the full arguments used to start the process and nifty tools to find a process or a handle and a handy restart process. It is a great aid to debugging. ProcExp has some quirks when running on x64 Windows, though.

I have run my workstation as a “User” without admin rights for over 12 years since the days when I started running NT4 on my laptop. I used to have to log out and log in as an “Administrator” to install software or make system changes. There were tweaks you could make to dial back the security for some little things like creating a security role that could change the system date and time (which allows you to open the old-style date and time applet by clicking on the taskbar clock). With Windows 2000, things got a lot better with the runas service (like su(1) on UNIX)  but there were still some painful quirks because, for example, some software expects to be installed by the same admin user account that is using it. That’s where Aaron Margolis’s excellent makemeadmin script comes in. And finally with Windows Vista, we get UAC which, is nothing more than a speedbump warning system if you are an Administrator. However, if you are a non-Admin user it is a graphical just-in-time way to change the security of a running process by giving it Administrator credentials.

One of the main advantages of running with a limited user account (LUA) is that binaries in Program Files where they are protected from tampering by something malicious that you might accidentally invoke. For example, if you got hit with a zero-day browser flaw the worst (and this is very bad) thing that can happen is to have your personal data stolen or corrupted. The system itself cannot be subverted. Rogue usermode binaries cannot be installed and neither can drivers be installed. Hence, you cannot become part of a botnet and this is a very good thing.

Furthermore, most commodity attacks for Windows go straight for installing a rootkit without passing Go. That means instead of doing something bad to you that could succeed they try to do something worse that can’t and they crash doing nothing. That’s not a promise, just a generalization. AppLocker is what you need to take LUA to the next level to prevent unauthorized code from executing at all.

Anyway… Enough back story. Suffice it to say that I run my system without administrative rights and I don’t want to be typing in my admin credentials unless actually necessary.

The problem is that the x64 version of Process Explorer is embedded inside of the 32-bit version. When you invoke the 32-bit version of ProcExp on x64 Windows, procexp.exe extracts procexp64.exe into the same directory where it is currently running and starts procexp64.exe. If you have your Sysinternals tools in Program Files then in Vista or later with UAC turned on this generates a UAC prompt because procexp.exe is trying to write to the protected Program Files directory tree. (On Windows Server 2003 x64 or Windows XP x64 you get an access denied.) After procex64.exe exits the file is deleted.

ProcExp can actually run just fine without elevated permissions unless you need process details for a service or some other process running with another user’s credentials and procexp has a way to elevate itself to deal with that scenario, just like Task Manager.

Here is the trick to get procexp working LUA in Program Files on x64:

  • Download the procexp ZIP archive from Technet
  • Extract the ZIP file somewhere and run procexp.exe
  • Accept the license prompt
  • Make a copy of procexp64.exe (CTRL+C, CTRL+V will suffice)
  • Exit procexp.exe.
  • Delete procexp.exe.
  • Rename your copy of procexp64.exe to procexp.exe
  • Copy procexp.exe to your Sysinternals folder in “C:\Program Files”

Mark Russinovich could also solve this issue to either releasing a standalone x64 binary of procexp or changing the behavior of the extraction so that procexp64.exe doesn’t get deleted on exit (meaning you would just elevate once). In the mean time, the workaround isn’t too painful.

Boot Camp 3.1, from Apple this time

Boot Camp now officially supports Windows 7. As I expected, it is largely a repackaging of drivers previously released for Boot Camp 2.2. Here’s the flyby of what is in the update (I looked at the x64 version).

  • NVidia  display driver 8.16.11.8861, 01/05/2010
  • Binary.aapltp_Bin
  • Binary.AppleBTBroadcom_Bin
  • Binary.AppleBTE_Bin
  • Binary.AppleBT_Bin
  • Binary.AppleDisplay_Bin
  • Binary.AppleiSight_Bin
  • Binary.AppleODD_Bin
  • Binary.asix_ethernet_Bin
  • Binary.AtherosWin7_Bin
  • Binary.Atheros_Bin
  • Binary.Ati_GraphicsWin7_Bin
  • Binary.Ati_Graphics_Bin
  • Binary.BroadcomEthernet_Bin
  • Binary.BroadcomWireless_Bin
  • Binary.Cirrus_Audio_Bin 6.6001.1.21 (all new!)
  • Binary.crystal_beach_Bin
  • Binary.intel_ethernet_Bin
  • Binary.IRFilter_Bin
  • Binary.Keyboard_Bin 3.0.0.0 (same version number as was in Boot Camp 3.0 but I can definitely dim the keyboard more, now)
  • Binary.marvell_ethernet_Bin
  • Binary.MultiTouchMouse_Bin
  • Binary.MultiTP_Bin 3.0.0.0 (same as in 2.2)
  • Binary.null_driver_Bin
  • Binary.Realtek_Bin
  • Binary.Sigmatel_Bin

Available from Apple in x86 and x64 flavors.

Raymond Chen answers my question

Nearly four years ago, I asked Raymond Chen why Microsoft has continued to use cryptic 8.3 filenames in Windows even though long filenames have been supported for many years. I wasn’t paying attention when Raymond answered me a year later. I just stumbled across this today.

Commenter Brian Reiter asks a duplicate of a question that was already submitted to the Suggestion Box: Darren asks why operating system† files still (for the most part) adhere to the old 8.3 naming convention.

It comes down to a handful of interesting reasons:

  1. Once a name is chosen, it can’t be changed for reasons of backwards compatibility with applications that may load a system DLL or invoke a system executable.
  2. By default all long file names are given a short file name. Loading a DLL by reference to a short and long file name into a program will yield two separate references of the DLL in memory.
  3. 8.3 filenames are know to work. If it ain’t broke, don’t fix it. Also, a related point here is that the system component installer technology in Windows XP and prior could only support 8.3 filenames. Pretty much any system component developed prior to Vista had to have 8.3 filenames. That includes some components developed for Vista while the new installer system was under development. (There a

My original question was why does the Framework Design Guidelines for .NET applications specify a totally different naming. The gist is that while native code and legacy components have the gotchas above, .NET assemblies are different because they have additional metadata including version number and public key token. In order to dynamically load an assembly, you either need to know its strong name or its full path and filename. There is very little chance of accidentally loading the wrong assembly unless you are the one building and signing the assemblies in question.

Make your own Apple Boot Camp “3.1” update

Apple loves to tout compatibility with Windows.

Have a Windows application you need to use once in a while? No problem. Every new Mac lets you install Windows XP and Vista and run them at native speeds, using a built-in utility called Boot Camp.

There is something to this and the Apple hardware is awesome but Apple is a bit heel-dragging and sloppy about the way they release drivers for Boot Camp. For example, Windows 7 was finished July 22, 2009 but as of December 15, 2009 it is still officially unsupported by Apple. Windows 7 uses the same driver model as Vista, so that policy from Apple is just recalcitrant and disingenuous. Apple really does provide all the drivers you need to get Windows 7 x86 or x64 to run natively on Mac hardware.

On the other hand, some of the drivers in Boot Camp 3.0 are really flaky. For the latest Macbook Pros that use Cirrus Logic audio controllers, the volume is messed up so the internal speakers are un-hearable and the built-in microphone doesn’t work at all with some applications—notably Skype.

Apple actually does have updated drivers available. They are packaged as Boot Camp Drivers Update 2.2.

Setup is simple and straightforward — just as you’d expect with a Mac.

Awesome.

Except that if you are running Boot Camp 3.0, you are screwed and the setup is very not straightforward and there is no one-click installer from Apple. It is doable, though, and worthwhile.

This update addresses issues with the Apple trackpad and turns off the red digital audio port LED on laptop computers when it is not being used. It also includes support for the Apple Magic mouse and wireless keyboard. It is intended only for use with Microsoft Windows XP and Microsoft Windows Vista running on a Mac computer using Boot Camp.

If, like me, you are running Boot Camp 3.0 with Windows 7 on your MacBook Pro and you want these updates here’s what you have to do.

  • If you don’t already have it, install 7-zip.
  • Download the Boot Camp Drivers Update 2.2 for Windows from http://support.apple.com/kb/DL967
  • After you have BootCamp_Update_2.2.exe, right-click on it and select 7-zip | Extract to “BootCamp_Update_2.2\”.
  • Now you have a directory of files that includes BootCampUpdate32.msp and BootCampUpdate64.msp. The 64 version will work with Windows Vista or 7 x64.
  • Right-click on the appropriate msp file and extract it with 7-zip.
  • Now you have a directory full of weird file and folder names. One of the folders should be named “BootCamp24ToBootCamp223”. Inside there are some files that are named with the pattern Binary.*_Bin:

Binary.Cirrus_Audio_Bin –> Fixes audio levels and microphone
Binary.Keyboard_Bin –> Same version that shipped with Boot Camp 3.0
Binary.MultiTouchMouse_Bin –> Magic Mouse driver
Binary.MultiTP_Bin –>  Fixes accidental select while dragging
Binary.TrackPad_Bin –> I don’t have the older touchpad so I don’t know

    • Each of these files is an archive that you can extract with 7-zip. Once you extract them, you can install the drivers by running DPInst.exe or by pointing the Device Manager at the extracted drivers in the usual way.

Happy updating.

Now that that’s out of the way, can someone explain to me why this had to be so hard?

Why bundle these updates with a smug “Setup is simple and straightforward — just as you’d expect with a Mac” tagline but make sure that the customers who bought the latest hardware and latest OS X cannot install them? It makes no sense.

Why did Boot Camp 3.0 ask me to configure automatic driver updates from Apple but Apple doesn’t actually publish any driver updates though that channel? It makes no sense.

%d bloggers like this: