Solving Very Slow Visual Studio Build Times in VMWare

ClockIn 2009, we chose 15″ MacBook Pro BTO over Dell, HP and Lenovo offerings. Apple offered us the best hardware and equivalent lease terms but with much simpler servicing done by ETF rather than (often incorrect) paper statements and checks. 99% of the time we ran this MacBooks with Windows 7 under Boot Camp. When I started doing some iOS development, I ran Snow Leopard and Xcode under VirtualBox and when I got fed up with the flakiness, I used VMware Workstation. OS X didn’t run very well under virtualization mostly because accelerated Quartz Extreme drivers don’t exist for VMware Workstation. Still, it actually worked well enough and was much more convenient than dual booting — which is such a huge time suck. When the lease period ended, we renewed with Apple and decided to just use OS X as the host environment for a variety of reasons:

The transition went very smoothly. I was using a VM with Windows Server 2008 and Visual Studio 2010 for primary .NET web development. Configuring IIS Express to serve outside of localhost bound to a host-only adapter is great for cross-browser testing, but it can be even more useful to enable remote access to Fiddler and point external browsers to the Fiddler proxy running in the VM to get both client debugging and HTTP sniffing at the same time. All of this was working out great.
 
At the start of October I downloaded Windows Server 2012 and Visual Studio 2012 and created a new development VM. In order to minimize disk storage, I moved my source tree to a folder in the host OS X environment and exposed it via the “Shared Folders” feature of VMWare Fusion 5.x to Windows. At the same time I started working on a new project.

 

The Slowening

Once the project grew to 20, 30 and 50k lines of C# code, the build times started to become horrifically slow. When combined with running unit tests, build became a big time for a bathroom break or cup of coffee event like building a project in C 15 years ago. Builds would show cdc.exe running at ~50% CPU (e.g. 1 core) and some other stuff totaling ~70% CPU. The VM was not memory bound and network IO was minimal. This was my first substantial project using Code First EF, so I thought maybe the complex object graph is just hard for the C# compiler to deal with.
 
After a few weeks of increasingly painful build times, I was looking at breaking my solution up so that I could build against pre-compiled DLLs — anything to make it go faster. I ran across a post on SuperUser:

… (Full disclosure: I work on VMware Fusion.)

I have heard that storing the code on a “network” drive (either an HGFS share or an NFS/CIFS share on the host, accessed via a virtual ethernet device) is a bad idea. Apparently the build performance is pretty bad in this configuration.

Oh really? Hmmm. Maybe it isn’t that my class libraries are so complex but something else is going on. Here are some empirical measurements of rebuild time of an actual solution:

VMWare shared folder: 	50 sec
OS X SMB share: 	18 sec
within virtual disk:	 9 sec

Wow. Problem solved. Incremental builds are basically instantaneous and a full rebuild takes 9 seconds when the code is hosted inside the VM image. Not only does hosting the source code within the VM virtual disk make the build go 5.5x faster, the CPU time of csc.exe goes way down. I don’t know how the VMWare shared folder is implemented. It appears as a mapped drive to a UNC name to Windows but it is very slow. Moral of the story is just don’t host your source code on the host machine with VMWare. The performance penalty is just not worth it. If you need to share the source code tree inside the VM with the host OS, create a file share from the VM to the host over a host-only adapter.

 

Update

I can confirm this is still a problem in VMWare Fusion 6. I’m hoping maybe the new SMB implementation in Mavericks might greatly improve the performance of sharing source code from the host OS to the VM.

Update

I just updated my virtual machine to Windows Server 2012 R2 (aka Windows 8.1 server). It is running on VMWare Fusion 6. The build time of this large project is now 15 seconds over VMWare Shared Folders. Significant remaining issue is that Visual Studio uses a whole core of CPU to do nothing — just having a large solution open, not editing anything.

Advertisements

Really Least-Privilege Development: AppLocker and Visual Studio

AppLocker is a software execution policy tool in Windows 7 Enterprise and Ultimate and Windows Server 2008 R2. An AppLocker policy can be used to shift Windows from a  model where execution of code is permitted by default to a model where execution is denied by default. AppLocker is aware of binary exe, DLL/OCX, the scripting engines that ship with Windows and Windows Installer packages. The default rule sets for these categories will only allow code that is installed into the system or program files directories to execute. Once AppLocker is turned on, execution of code is denied by default and an unprivileged user cannot add executable code to the system.

If untrusted users can’t execute new code, then how can Visual Studio possibly work without making developers admin?

Grant Execute on Source/Build Tree: Fail

I thought this would be super simple and that all I would have to do was create Executable, DLL and Script path rules to grant execute on my source tree. At first, it seemed to work and I was off and running. Then I tried to build a big complicated project and the build failed all of a sudden. This solution had post-build rules but so what? I had script enabled for the build tree.

Build Actions Fail

Procmon shows that the pre- and post-build events are implemented as temporary batch scripts in %TEMP%. They are named <cryptic-number>.exec.cmd.

procmon-postbuild

Unfortunately %TEMP% is not a usable macro in AppLocker. You have to either create a generic Script rule to allow all *.exec.cmd scripts to execute or create rules for each Visual Studio user like C:\Users\<username>\AppData\Local\Temp\*.exec.cmd. Either way, post-build actions will start to work.

Web Apps Crash with Yellow Screen of Death

Another issue is that running Web apps with the built-in Visual Studio Development Web Server (aka Cassini) fails miserably. The obvious clue that this is AppLocker is “The program was blocked by group policy.”

webdev.webserver20-yellowdeath

The problem is that Cassini copies the DLLs and runs them from a subdirectory of %TEMP%\Temporary ASP.NET Files\.

webdev.webserver20-temp

 

In order for Cassini to work, you have to disable DLL rules or  create a DLL allow path rule for every developer in the form C:\Users\<username>\AppData\Local\Temp\Temporary ASP.NET Files\*.dll.

Visual Studio’s HelpLibAgent.exe Crashes

This one is a bit weirder and more surprising. Visual Studio 2010 has a new help system that operates as a local HTTP server. Invoking help with AppLocker DLL rules enabled generates a serious crash.

helplibagent-crash

I’m not sure why it does this but HelpLibAgent.exe generates a random string and then two .cs files based on that string and invokes the C# compiler to generate a DLL based on the random string which is dynamically loaded by HelpLibAgent. This seems weird on the face of it that and there’s nothing in that code that looks like it has to be generated on a per-user basis at all. Weird, weird, weird.

helplibagent-dll-compile

In order for this to work you have to allow any randomly named dll to load out of %TEMP% which means disabling DLL rules or modifying the rule that was necessary for Cassini:

C:\Users\<username>\AppData\Local\Temp\*.dll.

Summary

In order to run Visual Studio with AppLocker a user needs the following rules:

  • DLL, EXE and Script: Allow path on source tree / build directory structure
  • Script: Allow path on %TEMP%\*.cmd.exec
  • DLL: Allow path on %TEMP%\*.dll
  • Unfortunately %TEMP% is not available in AppLocker so a C:\Users\<username>\AppData\Local\Temp\* for every <username> needed has to exist. These are probably best implemented as local policies.
  • Optional: Allow script *.ps1. (This is pretty safe because PowerShell has its own tight script execution security model.)
  • It’s unfortunate that DLL rules have to be enabled for a well-known location like %TEMP% but that still doesn’t make the DLL rule useless.

    • OCX is still not permitted from %TEMP%
    • AppLocker DLL rules are complementary to CWDIllegalInDllSearch for mitigating DLL Hijacking because it provides a more granular options. This is particularly important if you need to use a global CWDIllegalInDllSearch setting of 1 or 2 for compatibility reasons.
    Once these rules are in place, the experience is seamless. The rules don’t get in the way of anything.
    Note that AppLocker script rules only apply to the scripting hosts that ship with Windows: CMD, Windows Scripting Host (.vbs and .js) and PowerShell). Perl, Python, Ruby, etc interpreters are not affected by AppLocker policy. Similarly, execution of Java jar files are not affected by AppLocker.
    It would be nice if DLL rules were a little smarter. For instance, I would like to be able to allow managed DLLs on some path but not native code.

MSFT Help Viewer Duplicate Entries

update-helpMicrosoft Help Viewer 1.0 is a new document database that ships with Visual Studio 2010. It is basically an offline version of the “lightweight” view of the MSDN library online. It even runs in its own little web server and is accessed through a browser.

It ships with a number of categories of documentation including documentation for the .NET Framework version 4.0. You can install additional documentation from online or offline sources, including the .NET Framework 3.5 documentation.

Unfortunately, it doesn’t make much sense if you install both the v3.5 and 4.0 documentation. Firstly, the 4.0 documentation seems to be a superset of the entire 3.5 library. Secondly, installing both inserts two links for every article into all the navigation but both links resolve to the same document and that document specifies which version of the Framework the API is supported in.

doubled-navigation

I don’t see much point in installing more than one version of the .NET Framework documentation. Just stick with the .NET Framework 4 documentation that ships as part of the default options. It will update from online sources when changes are published.

Visual Studio 2010 Professional Should be Free

Microsoft has created yet another SKU for Visual Studio 2010, Ultimate Edition.

This is out of hand.

  • Visual Studio Express editions Basic CMYK
  • Visual Studio Professional
  • Visual Studio Premium
  • Visual Studio Ultimate
  • Visual Studio Test Professional
  • Visual Studio Team Foundation Server
  • Visual Studio Lab Management

The express editions are free of charge but weirdly crippled:

  1. Rather than being features extending the base IDE, there are entirely separate Express IDEs for each language.
  2. The source control plugin API is missing
  3. Extremely limited refactoring (at a time when the refactorings in the full edition don’t compare well to Eclipse or Netbeans)
  4. No conditional breakpoints
  5. No remote debugging
  6. No thread debugging
  7. No support for compiling 64-bit native images
  8. No support for setup projects
  9. No support for solutions which contain projects written in different languages (because of item #1).
  10.   No MS Office development support.
  11.   No VSIX extensions (like this spell checker).

And apparently, you don’t have access to F# and IronPython languages with any Express edition. What?

Visual Studio Professional is the vanilla full-featured version of Visuals Studio 2010.

Visual Studio is really the mechanism by which developers add value to Microsoft’s platforms. It is used to build applications that people actually use. We are not living in the gay 90s anymore when compilers were generally very expensive and IDEs were new and a huge value-add. Now, every platform vendor I can think of except for Microsoft gives away the best development tools it can in order to draw developers to it.

Here are some examples:

  • Apple gives away XCode and all its developer tools and documentations to anyone that registers.
  • Eclipse is free and open source.
  • Netbeans is free and open source

Visual Studio Express editions do not have parity with the features of XCode, Netbeans and Eclipse. Visual Studio Professional is much closer.

But to get Visual Studio Professional, you have to be student or faculty at an institution participating in the Microsoft Academic Alliance program, an employee of a Microsoft Certified Partner or you or your employer have to buy an MSDN subscription every year. There are now 6 MSDN subscription SKUs.

  • MSDN Operating Systems
  • MSDN Embedded
  • Visual Studio Professional with MSDN
  • Visual Studio Test Professional with MSDN
  • Visuals Studio Premium with MSDN
  • Visual Studio Ultimate with MSDN

These range in price from $699 to $11,899 retail with the “Professional” version weighing in at $1,199 ($799 for a renewal). The Operating Systems one doesn’t even come with Visual Studio which makes no sense at all. Why offer developers a subscription to your operating systems without giving them the tools to develop applications on the operating systems?

This state of affairs is out of control.

I don’t have any issue with Microsoft selling value-adds over and above of Visual Studio Professional (e.g. Premium, Ultimate, Professional Tester, Team Server, etc.) to compete with IBM Rational and Perforce et al in the application lifecycle management and enterprise architecture modeling stuff and build management and testing.

But rather than trying to squeeze 800 bucks a year out of developers, Microsoft should discard the Express editions of Visual Studio and make Visual Studio 2010 Professional available at no cost to anyone with a valid copy of Windows.

Otherwise, Microsoft is literally driving startups and young developers to other platforms which offer fully functional free tools from vendors like Apple, IBM, Oracle (Sun), Novell, Red Hat and Canonical.

And when I say free I don’t mean crippled or ad supported. In order to keep the Windows platform relevant, Microsoft needs to make credible modern tools available to anyone that might be interested. That means Visual Studio 2010 Professional should be a free download.

Seriously.

Office 2010 Beta causes Visual Studio 2008 ping-of-death

After installing Office 2010 Beta, I discovered that the ASPX editor in Visual Studio 2008 will hang. It doesn’t hang instantly but it does hang before you can get much done, usually after doing a copy/paste operation. Once the hang happens, clicking anywhere the Visual Studio window causes a “ping” sound and the window is frozen.

I had this issue despite completely uninstalling all Office 2007 products before installing Office 2010 Professional Plus Beta (as well as Visio 2010 and Project 2010).

Visual Studio 2008 uses a component that is shared with Office SharePoint Designer 2007 (formerly known as FrontPage) and Expression Web 2 for authoring web pages. The Office 2010 Beta installation appears to corrupt this component or replace it with one that is not backwards compatible with Visual Studio 2008.

Fortunately this issue is easy to fix. This component shows up in “Programs and Settings” as “Microsoft Visual Studio Web Authoring Component”.

appwiz

Righ-click and choose “Change” from the context menu. Choose the “Repair” option in the dialog that pops up.

repair

This only takes a minute or so and should resolve the issue.

via Martin Hinselwood.

Multiple Versions of IE with the Visual Studio Built-In Web Server: The Solution

The Problem

Last time I discussed the the unfortunate crippling of the Visual Studio built-in web  server, webdev.webserver, so that it can only process requests that originate from localhost and the side-effect that this creates a big impedance barrier to testing multiple versions of Internet Explorer with your web apps. I promised a solution to the dilemma.

The key is to run your different versions of IE in virtualization software and use a personal proxy server to forward their requests. If the proxy is running on your host OS and the browsers in the client VMs use the host OS proxy then, from the perspective of webdev.webserver in your host OS, all of the requests will appear to originate from localhost and it will serve them.

There are a few gotchas.

Step 1: loopback adapter

The loopback adapter is a virtual network interface device. It provides a way for us to create a shared network between the virtual machines and the host machines without altering the configuration of any real network interfaces.

Install the loopback adapter via Device Manager (devmgmt.msc) by right-clicking on the root “computer” node and selecting “Add legacy hardware”. This should bring up the Add Hardware wizard. Choose the manual, advanced install. Next you should see a list of common hardware types. Select “Network adapters”:

add-hardware_common-hardware-types

Select Microsoft as the manufacturer and “Microsoft Loopback Adapter” as the network adapter.

add-hardware_select-network-adapter

Finish out the wizard and it will create a new network device which will appear in your “Network Connections” control panel (ncpa.cpl). It will probably be called something like “Local Area Connection (2)”. I like to rename this to something more descriptive like “loopback” or “Internal Connection”.

netowork-connections-arg

Now you can manually assign a static IP address to this connection. Choose something from one of the ranges defined by the IETF as private: 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16.

For my example, I’m going to subnet the 10 network and use 10.237.0.1 mask 255.255.0.0 with no default gateway or DNS servers defined.

At this point, if you are running Windows Vista or 7, you may notice a small problem. The “Internal Connection” device says it is on the “Unidentified Network” which means that Windows thinks you are connected to a “Public Network” which means that Windows Firewall will block Windows File and Printer sharing.

network-sharing-arg

In order to calm Windows down, we need to make a registry change to mark our loopback adapter as an endpoint device. This indicates to Windows that it is not a true network device that connects to an external network. In my opinion, this should be the default setting for the loopback driver, but it isn’t. In order to make this change we need to create the *NdisDeviceType  value as DWORD of 1 in the key for our loopback adapter. (See MSDN documentation.)

Network adapters are configured under the following registry key:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}

The default value on this key is “Network Adapters”. There will be several four-digit number sub-keys (such as 0016) depending on how many network interfaces are installed on your machine. One of these will have a DriverDesc value of “Microsoft Loopback Adapter”.

Once you have found the key for the loopback adapter, add a DWORD value to it called *NdisDeviceType with a DWORD value of 1.  Note: common mistake is to leave off the asterisk, which should be included as part of the value name.

loopback-registry-fix

Once you have added this value, you must bounce the driver by disabling and enabling your loopback device or reboot for the change to take effect.

netowork-connections-fixed

network-sharing-fixed

The last loopback adapter-related activity is to tell Windows Firewall not to monitor the loopback interface.

In Windows 7, you do this through the advanced settings link of the Firewall control panel applet. Right-click on the root “Windows Firewal with Advanced Security” node and choose properties. You can then set the “Protected Network Connections” for the Domain Profile, Private Profile and Public profile.

win7-firewall-binding

In Windows Vista, it is a little easier to get to the dialog. In the Windows Firewall control panel applet, click “Change Settings”.

winvista-firewall-binding

Step 2: Install a Personal Proxy

Pretty much any lightweight personal proxy server will do for this. I like privoxy which I also use for general ad-blocking across all of my browsers. You can download the latest stable release from privoxy.org.

The privoxy installer is straightforward. Just run it.

Privoxy is essentially a unix-style daemon. It is configured through unixy text files. We need to edit Config.txt located in the Privoxy install directory to tell privoxy to listen on the IP address we bound to our loopback adapter. Privoxy will install into C:\Program Files\Privoxy on x86 Windows or C:\Program Files (x86)\Privoxy on x64 Window.

Look for the listen-address in Config.txt. Set it to the IP address you bound to your loopback adapter and also set the TCP port number to listen on. Privoxy is normally configured to listen on 8118. In my setup, the listen-address is 10.235.0.1:8118.

privoxy-listen-address

Finally, we want to configure Privoxy to run as a service so that it will just be there all the time without having to start up its rudimentary GUI. This is how to do it in PowerShell:

PS> ./privoxy --install
PS> set-service privoxy -StartupType Automatic
PS> start-service privoxy

Step 3: Virtual Machines

The hard part is behind us. The rest is pretty easy. We just need to set up Virtual  Machines using NAT (shared) networking.

vpc-shared-networking

You can get pre-configured virtual machines from Microsoft. These are set up for Virtual PC 2007 but can be run under Sun VirtualBox by just uninstalling the Virtual PC extensions and installing the VirtualBox drivers. On Windows Virtual PC with Windows 7, the XP image is a pain to deal with because Microsoft removed files to compress the image including USB drivers that are useless to VirtualPC 2007 but are essential to Windows Virtual PC. You have to get the drivers back onto the VHD in order to install the Windows Virtual PC extensions. Windows XP Mode is probably a better bet or just install Windows XP from scratch.

A couple of useful items are the IE7 Blocker and the IE8 blocker. These will prevent Automatic Updates from upgrading your browser and defeating the whole purpose of this exercise.

Inside of the client OS (probably Windows XP) you just the proxy settings in Internet Options so that HTTP is proxied to 10.237.0.1 port 8118 (or whatever you configured). Uncheck the bypass proxy for local addresses option.

xp-vm-proxy xp-vm-proxy-detail

With IE6 in the VM, you can now just go to the same localhost URLs that you use when you launch browsers on your host OS. The one remaining gotcha is that IE7 and IE8 are hard-coded to bypass a proxy server for localhost or 127.0.0.1. They will always bypass a proxy for localhost. The simplest workaround is to put a trailing period after the host name but before the port number so that http://localhost:8080 becomes http://localhost.:8080.

Here’s the whole shebang working with integration features enabled on Windows Virtual PC on Windows 7 x64. This is IE8 on Windows 7 (host OS), IE7 on Windows Vista and IE6 on Windows XP. All three are pointed at the same web project started from Visual Studio 2008. If you look very closely, you just might be able to see the extra period in the URL on IE7.

ie6-7-8

Multiple Versions of IE with the Visual Studio Built-In Web Server

 

For years, setting up a web project to run locally on your development machine with Visual Studio (and before that Visual InterDev) required a ton of prerequisites. You had to configure IIS and FrontPage Extensions. You had to have permissions set correctly in order to publish and debug. The setup did not play very well with source control systems and was generally a big nightmare time suck.

ie6-7-8-small

In the Java world things were much better much sooner, particularly if all you wanted was a servlet container to run your simple JSP site or to host your Spring POJO application. Back around 2002 or 2003 Netbeans 3 would magically publish your code into Apache Tomcat and let you debug it. You could do something similar with Eclipse and other Java IDEs of the day.

With Visual Studio 2005, Microsoft adopted something very similar to the Java IDE with Tomcat model. Starting with Visual Studio 2005, by default, you get magical publishing to a lightweight web server called webdev.webserver.exe. Webdev.webserver is based on the Cassini sample web server and shares a quirk of Cassini: it only accepts requests from localhost.

Microsoft says that this is for security reasons. They wanted to bundle a web server with .NET 2.0 to make it easier to get started programming but on the other hand they were licking their wounds from the constant successful attacks on Windows XP that had only started to abate with the rollout of SP2. So, Cassini cum webdev.webserver is hardcoded to refuse connections unless they originate from your own computer.

On the face of it that doesn’t seem like much of a problem but here’s the thing. Microsoft has 3 supported web browsers—Internet Explorer 6, Internet Explorer 7 and Internet Explorer 8—and you can only run one of them on a given Windows installation. Furthermore, you cannot run IE6 on Windows Vista and you cannot run IE6 or IE7 on Windows 7. Microsoft’s solution to help out developers is to pass out free copies of Virtual PC and to provide free virtual machine images of Windows running various browser configurations.

Gotcha. You can’t use these virtual machines with Visual Studio’s webdev.webserver. You have to publish to IIS. Ugh!

As the screenshot above may have given away, there is a solution. The key is that all webdev.webserver cares about is that requests originate from localhost. I’ll post all the gory details next time.

%d bloggers like this: