Backup Windows Server on EC2 with PowerShell and AWS CLI

When we first started working with Windows Server or Amazon EC2, we ran into the problem of how to back up SQL Server databases. My initial thought was to somehow mount S3 as a volume and have SQL Server write backup jobs to it. That turns out to be kind of hard.

After a little head-scratching, I realized that EBS volume snapshots are stored in S3. Therefore, all we need to do is mount an EBS volume and have SQL Server write backup jobs to that and then snapshot that volume. Oh, and unmount it from the running Windows server before snapshot to make sure that we are taking clean snapshots.

It turns out to not be too hard to make this happen on Windows Server 2012 Core using a PowerShell script that drives the AWS CLI and does the hoodoo with the Windows volume manager to mount and dismount the EBS backup volume.

This solution requires PowerShell Storage Cmdlets which ship with Windows Sever 2012/ and Amazon Web Services python-based command-line interface. Works great on Windows Server Core.

The biggest problem that I ran into was that the script would run just fine when invoked manually or when I would trigger the job from schtasks while logged in, but it would hang and fail when run from scheduled tasks in the middle of the night. Long story short: when schtasks runs the job with nobody logged in it gives the .DEFAULT environment instead of the environment of the user context of the scheduled task. That meant that AWSCLI didn’t receive the correct %USERPROFILE% environmental variable and was not able to locate its config file with the user id and key.

The simplest solution was to wrap the invocation of the powershell script in a cmd batch script:

set USERPROFILE=C:\Users\Administrator\

powershell.exe -file “C:\Program Files\Invoke-BackupJob.ps1” -path E:\ -diskNumber 2 -ec2VolumeID vol-<id-number-here> -description “Important Backup” > “C:\Program Files\Utility\Log.txt”

Use the PowerShell Get-Disk cmdlet to figure out the disk number of the EBS volume in Windows and the EC2 console or AWSCLI to figure out the EBS volume ID in EC2.

[gist https://gist.github.com/breiter/7887782]

Cool PowerShell Script Replicates Telnet

Lee Holmes has a cool script to reproduce telnet-like functionality via the TcpClient object in PowerShell.

## Connect-Computer.ps1 
## Interact with a service on a remote TCP port 
param( 
    [string] $remoteHost = "localhost", 
    [int] $port = 23
     ) 

try
{
    ## Open the socket, and connect to the computer on the specified port 
    write-host "Connecting to $remoteHost on port $port" 
    $socket = new-object System.Net.Sockets.TcpClient($remoteHost, $port) 
    if($socket -eq $null) { return; } 

    $stream = $socket.GetStream() 
    $writer = new-object System.IO.StreamWriter($stream) 

    $buffer = new-object System.Byte[] 1024 
    $encoding = new-object System.Text.AsciiEncoding 

    while($true) 
    { 
       ## Allow data to buffer for a bit 
       start-sleep -m 500 

       ## Read all the data available from the stream, writing it to the 
       ## screen when done. 
       while($stream.DataAvailable)  
       {  
          $read = $stream.Read($buffer, 0, 1024)    
          write-host -n ($encoding.GetString($buffer, 0, $read))  
       } 

       ## Read the user's command, quitting if they hit ^D 
       $command = read-host 
      
       ## Write their command to the remote host      
       $writer.WriteLine($command) 
       $writer.Flush() 
    } 
}
finally
{
    ## Close the streams 
    $writer.Close() 
    $stream.Close()
}

This solves the problem of telnet.exe crashing conhost.exe when using Console2.

Update

I tweaked Lee’s code to use try/finally which obviates the need for a special escape sequence to clean up the TCP resources.

WorkAround: Console2 + Telnet.exe on Windows 7 Crashes ConHost.exe

conshost-croak

Entering the Windows telnet.exe prompt mode causes the shell process underneath Console2 to crash on Windows 7. You can reproduce this by just invoking telnet with no arguments in a Console2 session. An alternate route to hit this is exiting a telnet session with CTRL+].

The symptom is Console2 freezing because it is no longer getting text from its slaved native console followed up with a crash dialog for conhost.exe and whetever your shell was.

Unhandled exception at 0xffdd1df9 in conhost.exe: 0xC0000005: Access violation reading location 0x0000000003385460.>    conhost.exe!SB_StreamWriteToScreenBuffer()  + 0x61 bytes   

A workaround is to launch telnet into a separate conhost.exe window.

#force telnet.exe to start in its own console window. 
function telnet { start $env:SystemRoot\System32\telnet.exe $args }

Or use putty –P port –telnet host as an alternative to telnet.exe. (I can’t get plink.exe to be interactive with telnet for me.)

Update

Unless you need something nifty that telnet.exe is doing, a more elegant solution is to Lee Holmes’ Connect-Computer.ps1 PowerShell script as a telnet replacement.

FIX: VirtualBox Host-Only Network Adapter Creates a Virtual “Public Network” Connection That Causes Windows to Disable Services

vbox-host-only-net-net-n-sharing

vbox-host-only-net-connectletVirtualBox creates a “VirtualBox Host-Only Network” device which is essentially a loopback adapter for creating network connections between virtual machines and between the host and virutal machines. Unfortunately, it shows up to Windows as an unidentified public network. Connecting to a “public network” ratchets up your firewall and disables network discovery and SMB/CIFS network shares. This is kind of a big side effect of installing some VM software.Fortunately, Windows does have a way to mark a network device as virtual by creating a registry value.

The type of the device. The default value is zero, which indicates a standard networking device that connects to a network. Set *NdisDeviceType to NDIS_DEVICE_TYPE_ENDPOINT (1) if this device is an endpoint device and is not a true network interface that connects to a network. For example, you must specify NDIS_DEVICE_TYPE_ENDPOINT for devices such as smart phones that use a networking infrastructure to communicate to the local computer system but do not provide connectivity to an external network.

This powerhsell script will find the “VirtualBox Host-Only Ethernet Adapter device entry in the registry and adds the ‘*NdisDeviceType’ value of 1. After reboot, the “VirtualBox Host-Only Ethernet Adapter” will no longer be monitored by the Network and Sharing center.

# tell windows that VirtualBox Host-Only Network Adapter
# is not a true network interface that connects to a network
# see http://msdn.microsoft.com/en-us/library/ff557037(VS.85).aspx
pushd
echo 'Marking VirtualBox Host-Only Network Adapter as a virtual device.'
cd 'HKLM:\system\CurrentControlSet\control\class\{4D36E972-E325-11CE-BFC1-08002BE10318}'
ls ???? | where { ($_ | get-itemproperty -name driverdesc).driverdesc `
-eq 'VirtualBox Host-Only Ethernet Adapter' } |`
new-itemproperty -name '*NdisDeviceType' -PropertyType dword -value 1
echo 'After you reboot the VirtualBox Host-Only Network unidentified public network should be gone.'
popd


Copy and Paste with Clipboard from PowerShell

Many times I want to use PowerShell to manipulate some text that I have in something else like email or a text editor or some such or conversely, I do something at the command line and I want to paste the output into a document of some kind.

I’m not sure why there is no cmdlet for outputting to clipboard.

PS> get-command out-*

CommandType     Name             Definition                                         
-----------     ----             ----------                                         
Cmdlet          Out-Default      Out-Default [-InputObject <PSObject>] [-Verbose]...
Cmdlet          Out-File         Out-File [-FilePath] <String> [[-Encoding] <Stri...
Cmdlet          Out-GridView     Out-GridView [-InputObject <PSObject>] [-Title <...
Cmdlet          Out-Host         Out-Host [-Paging] [-InputObject <PSObject>] [-V...
Cmdlet          Out-Null         Out-Null [-InputObject <PSObject>] [-Verbose] [-...
Cmdlet          Out-Printer      Out-Printer [[-Name] <String>] [-InputObject <PS...
Cmdlet          Out-String       Out-String [-Stream] [-Width <Int32>] [-InputObj...

Fortunately, though Windows ships with a native clip.exe for piping text to clipboard. You can alias it to out-clipboard or just pipe to “clip”.

new-alias  Out-Clipboard $env:SystemRoot\system32\clip.exe

Now you can just pipe to Out-Clipboard or clip just like the other Out-* cmdlets.

The larger issue is that PowerShell doesn’t expose a way to paste text from the clipboard. However, the WinForms API of the .NET Framework has a GetText() static method on the System.Windows.Forms.Clipboard class. There is a catch, though:

The Clipboard class can only be used in threads set to single thread apartment (STA) mode. To use this class, ensure that your Main method is marked with the STAThreadAttribute attribute.

This is unfortunate because PowerShell runs in a multithreaded apartment by default which means that the workaround is to spin up a new PowerShell process which is slow.

function Get-ClipboardText()
{
	$command =
	{
	    add-type -an system.windows.forms
	    [System.Windows.Forms.Clipboard]::GetText()
	}
	powershell -sta -noprofile -command $command
}

A slightly less obvious approach is to paste the text from the clipboard into a non-visual instance of the System.Windows.Forms.Textbox control and then emit the text of the control to the console. This works fine in a standard PowerShell process and is substantially faster than spinning up a new process.

function Get-ClipboardText()
{
	Add-Type -AssemblyName System.Windows.Forms
	$tb = New-Object System.Windows.Forms.TextBox
	$tb.Multiline = $true
	$tb.Paste()
	$tb.Text
}

The TextBox method is about 500 times faster than the sub-process method. On my system, it takes 0.501 seconds to do spin up the sub-process and emit the text from the clipboard versus 0.001 seconds to use the TextBox control in the current process.

I also alias this function to “paste”. The commands I actually type and remember are “clip” and “paste”.

When using paste, I generally am setting a variable or inserting my pasted text into a pipeline within parens.

CAVEAT: Whenever you capture a value without any {CR}{LF}, the clipboard will add it. You may need to trim the newline to get the behavior you expect when pasting into a pipeline.

Here’s some one-liners to illustrate. I have dig from BIND in my path and I have Select-String aliased to grep. Notice how when I paste the IP address I captured to the clipboard from the one-liner gets pasted back with an extra blank line.

PS> ((dig google.com) | grep '^google\.com') -match '(\d+\.\d+\.\d+\.\d+)$' | out-null; $matches[0]
72.14.235.104
PS> ((dig google.com) | grep '^google\.com') -match '(\d+\.\d+\.\d+\.\d+)$' | out-null; $matches[0] | clip
PS> paste
72.14.235.104

PS> ping (paste)
Ping request could not find host 72.14.235.104
. Please check the name and try again.
PS> ping (paste).trim()

Pinging 72.14.235.104 with 32 bytes of data:
Reply from 72.14.235.104: bytes=32 time=192ms TTL=51
Reply from 72.14.235.104: bytes=32 time=208ms TTL=51
Reply from 72.14.235.104: bytes=32 time=212ms TTL=51
Reply from 72.14.235.104: bytes=32 time=210ms TTL=51

Ping statistics for 72.14.235.104:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 192ms, Maximum = 212ms, Average = 205ms

Command Prompt and PowerShell Here with Console.exe

command-hereIn antediluvian turn of the century days, there was an Open Command Window Here PowerToy. When Vista was released, this functionality was baked in to the default Explorer shell. Hold {SHIFT} and right-click on a drive, directory or the background of a directory and you will have an “Open command window here” option in the resulting context menu. Although PowerShell is installed by default on Windows 7, there is no parallel “Open PowerShell window here” option, but it can be easily added.

“Open PowerShell window here”

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\powershell]
@="Open PowerShell window here"
"Extended"=""
[HKEY_CLASSES_ROOT\Directory\shell\powershell\command]
@="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -NoExit -Command Set-Location -LiteralPath '%V'"

[HKEY_CLASSES_ROOT\Directory\Background\shell\powershell]
@="Open PowerShell window here"
"Extended"=""
[HKEY_CLASSES_ROOT\Directory\Background\shell\powershell\command]
@="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -NoExit -Command Set-Location -LiteralPath '%V'"

[HKEY_CLASSES_ROOT\Drive\shell\powershell]
@="Open PowerShell window here"
"Extended"=""
[HKEY_CLASSES_ROOT\Drive\shell\powershell\command]
@="C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -NoExit -Command Set-Location -LiteralPath '%V'"

Use Console.exe Windows

If you use command-line a lot in Windows, you will appreciate the advantages of using console.sf.net as the terminal window. The “Open … window here” context menus can be tweaked to open the new command prompts in Console rather than vanilla Windows CSRSS console windows.

"Open command window here” with Console

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\cmd]
@="@shell32.dll,-8506"
"Extended"=""
"NoWorkingDirectory"=""
[HKEY_CLASSES_ROOT\Directory\shell\cmd\command]
@="\"C:\\Program Files\\Console2\\Console.exe\" -r cmd.exe -d \"%V\" -w \"Command Prompt\""

[HKEY_CLASSES_ROOT\Directory\Background\shell\cmd]
@="@shell32.dll,-8506"
"Extended"=""
"NoWorkingDirectory"=""
[HKEY_CLASSES_ROOT\Directory\Background\shell\cmd\command]
@="\"C:\\Program Files\\Console2\\Console.exe\" -r cmd.exe -d \"%V\" -w \"Command Prompt\""

[HKEY_CLASSES_ROOT\Drive\shell\cmd]
@="@shell32.dll,-8506"
"Extended"=""
"NoWorkingDirectory"=""
[HKEY_CLASSES_ROOT\Drive\shell\cmd\command]
@="\"C:\\Program Files\\Console2\\Console.exe\" -r cmd.exe -d \"%V\" -w \"Command Prompt\""

“Open PowerShell window here” with Console

 

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\powershell]
@="Open PowerShell window here"
"Extended"=""
[HKEY_CLASSES_ROOT\Directory\shell\powershell\command]
@="\"C:\\Program Files\\Console2\\Console.exe\" -r C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -d \"%V\""

[HKEY_CLASSES_ROOT\Directory\Background\shell\powershell]
@="Open PowerShell window here"
"Extended"=""
[HKEY_CLASSES_ROOT\Directory\Background\shell\powershell\command]
@="\"C:\\Program Files\\Console2\\Console.exe\" -r C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -d \"%V\""

[HKEY_CLASSES_ROOT\Drive\shell\powershell]
@="Open PowerShell window here"
"Extended"=""
[HKEY_CLASSES_ROOT\Drive\shell\powershell\command]
@="\"C:\\Program Files\\Console2\\Console.exe\" -r C:\\Windows\\system32\\WindowsPowerShell\\v1.0\\powershell.exe -d \"%V\""

 

 

Restore Defaults

For PowerShell, remove the powershell keys from beneath HKCR\Directory\shell, HKCR\Directory\Background\shell and HKCR\Drive\shell.

“Open command window here” restore default

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\Directory\shell\cmd]
@="@shell32.dll,-8506"
"Extended"=""
"NoWorkingDirectory"=""
[HKEY_CLASSES_ROOT\Directory\shell\cmd\command]
@="cmd.exe /s /k pushd \"%V\""

[HKEY_CLASSES_ROOT\Directory\Background\shell\cmd]
@="@shell32.dll,-8506"
"Extended"=""
"NoWorkingDirectory"=""
[HKEY_CLASSES_ROOT\Directory\Background\shell\cmd\command]
@="cmd.exe /s /k pushd \"%V\""

[HKEY_CLASSES_ROOT\Drive\shell\cmd]
@="@shell32.dll,-8506"
"Extended"=""
"NoWorkingDirectory"=""
[HKEY_CLASSES_ROOT\Drive\shell\cmd\command]
@="cmd.exe /s /k pushd \"%V\""

MSE v2 Command-Line Scanning

Microsoft Security Essentials (MSE) 2.0 adds command-line file scanning to its command-line interface.

MpCmdRun.exe is the command-line interface to MSE.

(Note that MSE has moved from “C:\Program Files\Microsoft Security Essentials” to “C:\Program Files\Microsoft Security Client”. And MpCmdRun.exe has moved to a subdirectory called Antimalware.)

In MSE v1, MpCmdRun was conspicuously missing an option to scan a file from a command line but the problem is rectified in MSE v2 beta.

mpcmdrun-filescan 

The file scanning is a new sub-option of the –Scan argument.

-Scan [-ScanType value]
    0  Default, according to your configuration
    1  Quick scan
    2  Full system scan
    3  Single file custom scan

       [-File ]
            Indicates the file path to be scanned, only valid for custom scan

       [-DisableRemediation]
            This option is valid only for custom scan.
            When specified:
              - File exclusions are ignored.
              - Archive files are scanned.
              - Actions are not applied after detection.
              - Event log entries are not written after detection.
              - Detections from the custom scan are not displayed in the user
                interface.

The usage is slightly awkward. It requires specifying –Scan –ScanType 3 –File <filename>.

A simple powershell function or a batch file.

Powershell

function Scan-File( $file )
{
    $exe = Join-Path $env:ProgramFiles &quot;Microsoft Security Client/Antimalware/MpCmdRun.exe&quot;
	&amp; $exe -Scan -ScanType 3 -File $file
}

Batch

@echo off
setlocal
set path=%programfiles%\Microsoft Security Client\Antimalware;%path%
cmd /c MpCmdRun.exe -Scan -ScanType 3 -File %1

Niftier PowerShell integration is possible by extending the work of the Scripting Guy’s Invoke-SecurityEssentials.ps1 script for MSE v1.0.

PowerShell’s Object Pipeline Corrupts Piped Binary Data

Yesterday I used curl to download a huge database backup from a remote server. Curl is UNIX-ey. By default, it streams its output to sdout and you then redirect that stream to a pipe or file like this:

$ curl sftp://server.somehwere.com/somepath/file > file

The above is essentially what I did from inside of a PowerShell session. After a couple of hours, I had my huge download and discovered that the database backup was corrupt. Then I realized that the file I ended up with was a little over 2x the size of the original file.

What happened?

Long story, short. This is a consequence of the object pipeline in PowerShell and you should never pipe raw binary data in PowerShell because it will be corrupted.

The Gory Details

You don’t have to work with a giant file. A small binary file will also be corrupted. I took a deeper look at this using a small PNG image.

PS> curl sftp://ftp.noserver.priv/img.png > img1.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 46769  100 46769    0     0  19986      0  0:00:02  0:00:02 --:--:— 74950

(FYI. Curl prints the progress of the download to stderr, so you see something on the console even though the stdout is redirected to file.)

This is essentially what I did with my big download and yields a file that is more than 2x the size of the original. My theory at this point was that since String objects in .Net are always Unicode, the bytes were being doubled as a consequence of an implicit conversion to UTF-16.

Using the > operator in PowerShell is the same thing piping to the Out-File cmdlet. Out-File has some encoding options. The interesting one is OEM:

"OEM" uses the current original equipment manufacturer code page identifier for the operating system.

That is essentially writing raw bytes.

PS> curl sftp://ftp.noserver.priv/img.png | out-file -encoding oem img2.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 46769  100 46769    0     0  26304      0  0:00:01  0:00:01 --:--:-- 74950

I was clearly on to something because this almost works. The file is just slightly larger than the original. It almost worked.

Just to prove that my build of curl isn’t broken, I also used the –o (–output) option.

PS> curl sftp://ftp.noserver.priv/img.png -o img3.png
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 46769  100 46769    0     0  25839      0  0:00:01  0:00:01 --:--:-- 76796

Here’s the result. You can see by the file sizes and md5 hashes that img1.png and img2.png are corrupt but img3.png is the same as the original img.png.

PS> ls img*.png | select name, length | ft -AutoSize

Name     Length
----     ------
img.png   46769
img1.png  94168
img2.png  47083
img3.png  46769


PS> md5 img*.png
MD5 (img.png) = 21d5d61e15a0e86c61f5ab1910d1c0bf
MD5 (img1.png) = eb5a1421bcc4e3bea1063610b26e60f9
MD5 (img2.png) = 03b9b691f86404e9538a9c9c668c50ed
MD5 (img3.png) = 21d5d61e15a0e86c61f5ab1910d1c0bf

Hrm. What’s going on here?

Let’s look at a diff of img.png and img1.png, which was the result of using the > operator to redirect the stdout of curl to file.

diff-img-img1

The big thing to see here is that there are a lot of extra bytes. Crucially, the bytes are Unicode glyphs and FFFE have been added as the first two bytes. 0xFFEE is the byte order mark for a little-endian UTF-16. That confirms my theory that internally PowerShell converted the data to Unicode.

I can also create the same behavior by the Get-Content (aliased as cat) cmdlet to redirect binary data to a file.

PS> cat img1.png > img4.png
PS> ls img1.png, img4.png | select name, length | ft -AutoSize

Name     Length
----     ------
img1.png  94168
img4.png  94168

So what is going on inside that pipeline?

PS> (cat .\img.png).GetType() | select name

Name
----
Object[]


PS> cat .\img.png | %{ $_.GetType() } | group name | select count, name | ft -AutoSize

Count Name
----- ----
  315 String

The file is being converted into an Object array of 315 elements. Each element of the array contains a String object. Since the internal data type of String is Unicode, sometimes referred to loosely as “double-byte” characters, the total size of that data is roughly doubled.

Using the OEM text encoder converts the data back but not quite. What is going wrong? Time to look at a diff of img.png and img2.png, which was the OEM text encoder.

 diff-img-img2

What you see here is a lot of 0x0D bytes have been inserted in front of all of the 0x0A bytes.

PS> (ls .\img2.png ).Length - (ls .\img.png ).Length
314

There are actually 314 of these 0x0D bytes added. What the heck is 0x0D? It is Carriage Return (CR). 0x0A is Line Feed (LF). In a Windows text file each line is marked with the sequence CRLF. 314 is exactly the number of CRLF sequences you need to turn a 315 element array of strings into a text file with Windows line endings.

Here’s what is happengin. PowerShell is making some assumtions:

  1. Anything streaming in as raw bytes is assumed to be text
  2. The text is converted into an array by splitting on bytes that would indicate an end of line in a text file.
  3. The text is reconstituted by out-file using the standard Windows end of line characters.

While this will work just fine with any kind of text, it is virtually guaranteed to corrupt any binary data. With the default text encoding you get a doubling of the original bytes and a bunch of new 0x0D bytes, too. The corruption fundamentally happens when the data is split into a string array. Using a binary encoder at the end of the pipeline doesn’t put the data back correctly because it always puts CRLF at the end of every array element. Unfortunately since there is more than one possible end of line sequence, this is as good as anything. Using a Windows to Unix conversion will not fix the file. There is no way to put humpty dumpty back together again.

To Sum Up, Just Don’t Do It

The moral is that it is never safe to pipe raw binary data in PowerShell. Pipes in PowerShell are for objects and text that can safely be automagically converted to a string array. You need to be cognizant of this and use Stream objects to manipulate binary files.

When using curl with PowerShell, never, never redirect to file with >. Always use the –o or –out <file>switch. If you need to stream the output of curl to another utility (say gpg) then you need to sub-shell into cmd for the binary streaming or use temporary files.

%d bloggers like this: