Dreaming in PowerShell

Test-Driving PowerShell v3: How To Install And Run PSv3 BETA

2012-05-15T02:32:00Z

A new PowerShell version is coming soon! The language itself will stay pretty much the same, but there are tons of cool new features that enhance performance and ease of use. A complete list of all of the new features is available here: http://www.microsoft.com/en-us/download/details.aspx?id=28998 (see the lower part of this page for feature details).

Which raises the question: how do I install and get PSv3 up and running? And are there any compatibility issues?

Update: PowerShell V3 Beta is out. I updated all links accordingly.

Requirements for PowerShell V3 Beta

Let's first take a look at the prerequisites. Before you can install PowerShell V3 Beta, you must ensure that these prerequisites are met:

you run Windows Server 2008SP2, Windows7SP1/Server 2008R2SP1 (so with all of the latest service packs installed). Windows XP/Server 2003/Vista is not supported and will not be supported anymore apparently.
you must have installed .NET Framework 4.0. If you are unsure, download the .NET Framework 4.0 runtimes (http://www.microsoft.com/en-us/download/details.aspx?id=17718) and try and install them. If the installer offers to "repair" the installation, then you know .NET 4.0 is present, and you can cancel the installation.
you must use an en-US MUI, so CTP2 does not run on Windows if any other language is installed (see below for some workarounds)

Installing English MUI

Unfortunately, PowerShell V3 Beta only installs on US-English systems. If you are running Windows 7 Business, Enterprise or Ultimate, you can temporarily change your display language to En-US, install the Beta, then change back to whatever language you prefer. PowerShell V3 Beta runs on any locale, it just installs only on En-US.

But what if you do not have access to the En-US language pack? And what if you run Windows 7 Professional or a Vista version that does not support MUIs?

You can download the MUI packs separately for free. Just make sure you download the right MUI. It must match your operating system and your platform (32bit/64bit) and your service pack level. You can find all of the downloads on various sites including this one: http://www.mydigitallife.info/windows-7-sp1-mui-language-packs-official-direct-download-links/

On a windows system that supports MUIs, after installing the downloaded MUI, it will show up in your control panel regions applet when you select Change Display Language. If you are running a windows version that does not support MUIs, go to http://www.froggie.sk/ and download the free tool vistalizator. It allows you to open and install downloaded MUIs and then change your display language.

Installing PowerShell V3 Beta

Once the prerequisites are met and your display language is switched to En-US, you can download and install the Beta package that you downloaded from here:

http://www.microsoft.com/en-us/download/details.aspx?id=28998

If the installation package complains that it is not applicable to your system, then you did not meet all of the prerequisites mentioned, or you downloaded the wrong version (wrong OS or platform).

The installation requires a reboot, and after rebooting, PowerShell V3 Beta is up and running. You can now enjoy the many new features, including tabcompletion in the console that no longer erases everything to the right, and a drastically improved ISE editor. Just note that the first launch of powershell.exe and ISE may take some time.

Getting Updated Help

PowerShell V3 no longer ships with help files, so one of the first things you may want to do is launch a powershell with full admin privileges, and then download the help materials like so:

PS> update-help -UICulture en-us -force

Preserving Compatibility

PowerShell V3 Beta is mostly backwards compatible, so you can continue to run your old scripts. When you create new scripts, though, you may be lured into using new PSv3 features, and if you do, your scripts no longer run on PSv2. To check, you can always run a test environment in PowerShell V2 mode like this:

PS> powershell.exe -version 2.0

This will launch a new PowerShell session in 2.0 mode, so you can test-drive your scripts and ensure they do not use V3 features and fail in V2.

We'll begin covering new PowerShell V3 features soon so you'll be prepared when PowerShell V3 hits the floor later this year. With the installation guideline found here, you now have all the details you need to setup your own cozy PSv3 test lab. Just remember: the current V3 is a Beta, it is not a production environment. While I personally did not stumble across any buggy behavior, you may not want to toss this version onto your production servers yet.

Stay tuned and have fun with PSv3 now!

Tobias

Microsoft MVP PowerShell Germany

P.S.
If you live in Germany or other parts of Europe and your company would like to set up a truly great PowerShell training, just contact me! I regularly train mid- to large-size companies. Trainings are always a blast with tons of real-world-examples and solutions. Here's how to get in touch with me: tobias.weltner@scriptinternals.de

Managing Child Processes

2012-05-09T02:09:00Z

Today in my training, a student asked how to manage child processes. That's not a bad question at all, and as it turns out, it's a great way to show off the capabilities of PowerShell.

So why would you care about child processes in the first place? Well, for example to monitor and keep track of what installation packages are doing.

When you launch an application from powershell, like setup.exe for example, you can get back its process id and monitor it or kill it if it takes too long - that's cool. But what if this process launches additional processes in the background? How can you get a hold of all of these processes?

Launching A Process And Keeping Control

Let's start simple and try and launch a process from PowerShell. The objective is to get back its process id, so you can continue to check and monitor it. Start-Process can start processes for you, and with a little-known parameter called -PassThru, it will even pass the process object back to you. This way, you can store it in a variable and use it later to retrieve the process id, read its CPU consumption or kill it:

PS> $process = Start-Process -FilePath notepad -PassThru
PS> $process.CPU
0,0156001
PS> $process.ID
3992
PS> $process.hasExited
False
PS> $process.Kill()
PS> $process.hasExited
True

As you see, Start-Process delivered back an object that was able to read real-time statistics from the live process and was even able to kill it at your convenience.

Finding Child Processes

To find all the processes a given process has started, Get-Process is of no use because it does not return the parent process id. Fortunately, WMI can help. Its class Win32_Process does include the information. So here is a small function called Find-ChildProcess. It accepts a process id and returns all the child processes that were spawned from the parent process:

function Find-ChildProcess {
param($ID=$PID)

Get-WmiObject -Class Win32_Process -Filter "ParentProcessID=$ID" |
Select-Object -Property ProcessName, ProcessID, CommandLine
}

By default, Find-ChildProcess uses the process ID of your current PowerShell host. So if you run Find-ChildProcess without specifying a process ID, you get back all the processes that you launched from within your powershell session. Try it and launch a couple of windowed applications such as notepad! Then, call Find-ChildProcess. You get back all the child processes. Cool!

Making It Recursive

Of course, a child process can launch another process as well, so there can be a literal process tree. To get a hold of these child-child-processes, what we need as a final touch is recursion. Since we already have a function that gets child processes, all you need to do is call it again for all child processes to find the processes of a child process.

The final solution also renames and changes the column (property) ProcessID. It renames it to 'ID' and changes the type to an integer array. Why? Because Stop-Process binds to that property via pipeline. So by fitting the column to the needs of Stop-Process, you can then pipe the results of Find-ChildProcess directly to Stop-Process in case you need to stop and kill the whole enchilada (you can download this function here: http://powershell.com/cs/media/p/16455.aspx):

function Find-ChildProcess {
param($ID=$PID)

$CustomColumnID = @{
Name = 'Id'
Expression = { [Int[]]$_.ProcessID }
}

$result = Get-WmiObject -Class Win32_Process -Filter "ParentProcessID=$ID" |
Select-Object -Property ProcessName, $CustomColumnID, CommandLine

$result
$result | Where-Object { $_.ID -ne $null } | ForEach-Object {
Find-ChildProcess -id $_.Id
}
}

NOTE: I added the Where-Object in black and bold above after it occured to me that some process do not have parent process information and this could lead to unwanted WMI "query invalid" exceptions. Now all is fine.

To test drive this, in your powershell enter Start-Process powershell. This opens a new powershell in a new window - a child process. Inside the new powershell window, enter Start-Process cmd. This opens yet another nested child process, a cmd shell. Inside of it, enter notepad to launch an instance of notepad.

Then return to your initial powershell and display the child items:

PS> Find-ChildProcess

ProcessName Id CommandLine
----------- -- -----------
notepad.exe 3180 "C:\Windows\system32\notepad.exe"
powershell.exe 6112 "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe"
cmd.exe 5728 "C:\Windows\system32\cmd.exe"
notepad.exe 5636 notepad

As you see, all child processes - even the nested ones - are reported.

And since there is a fitting column ID with the proper type, you can then pipe the results on to Stop-Process to kill the entire process tree:

PS> Find-ChildProcess | Stop-Process

So Find-ChildProcess effectively became a new member of the PowerShell command family, playing by the rules. You can combine it with other related cmdlets. Great question! And a cool solution that illustrates the power of PowerShell and its extensibility.

Stay tuned...!

Tobias

Microsoft MVP PowerShell Germany

Don't Waste Speed - How To Activate The PowerShell Turbo Mode

2012-05-01T20:35:00Z

This afternoon, my buddy Alexandar (and fellow PowerShell MVP from Serbia) reviewed another batch of PowerTips when he stumbled across some lines of code that looked a bit unintuitive to him and asked to rephrase them. Sometimes there is more than one way to Rome. There is nothing bad about having choices. Today I'd like to show how to pick the fastest way to Rome - and how you can speed up your PowerShell code tremendously by knowing about some easy design rules.

Understanding The PowerShell Pipeline

The PowerShell pipeline is a great feature but many users aren't aware that the pipeline actually is a throttling mechanism designed to minimize memory consumption. When you read in a large file, for example, by reading this file line-by-line and processing each line over the pipeline, only one line at a time needs to be stored in memory. That's why PowerShell can easily read and process huge log files.

The backside is that speed is fueled by memory and CPU. The memory throttling done by the PowerShell pipeline takes away memory and reduces speed. And the results can be significant. Have a look:

PS> 1..100 | Get-Random
66
PS> 1..10000 | Get-Random
9746
PS> 1..1000000 | Get-Random
509967

All of these lines generate a range of numbers and then pick one random number from it. When you try this for yourself, you'll notice that the first two commands run almost instantaneously. The third line however takes forever. The overhead produced by the PowerShell pipeline accumulates with each element that needs to travel across it. So in the last command, one million numbers need to travel one by one to Get-Random, and that takes a lot of time.

In many scenarios, the great memory-saving aspect of the PowerShell pipeline is not important at all. In our current example, it definitely does not matter. So here you can speed up things easily by avoiding the pipeline and passing the data directly to the parameter that would else receive it over the pipeline:

PS> Get-Random -InputObject (1..1000000)
770786

Notice how the very same operation takes only a fraction of a second instead of many seconds.

Every Day Scenarios To Save Time

There are tons of every day scenarios that you can speed up this way. Let's start with a very common one: discarding data. When you do not need data, you have the choice of piping it to Out-Null or assigning it to the special variable $null. Guess which way is faster? Test for yourself:

PS> (Measure-Command {1..100000 | Out-Null }).TotalMilliseconds
1114,3006 #!!!!!!!!!!! Over 1 f...ing second !!!!!!!!!!)
PS> (Measure-Command {$null = 1..100000 }).TotalMilliseconds
29,3192
PS> (Measure-Command {1..100000 > $null }).TotalMilliseconds
38,1542
PS> (Measure-Command {[void](1..100000) }).TotalMilliseconds
28,2378

All of these lines do the same: they dump data. The first approach uses the pipeline and takes over a second! Assigning the data to $null just takes lightning fast 30ms. Holy Moly. Of course the performance difference depends on how much data you need to dump. Often, it is much less than 100000 elements, so the performance gain becomes less impressive. However, just imagine this statement as part of a loop that runs a number of times. Each time the loop runs, the peformance penalty sums up.

Here is yet another common scenario: a loop. Some people use the pipeline to create loops:

PS> 1..100000 | ForEach-Object { "looping for the $_. time" }

On my machine, it takes approximately 7 seconds (on yours it can be much faster, the point is just the relative comparison).

The same loop can also be written without a pipeline, like this:

PS> for ($x=1; $x -le 100000; $x++) {
"looping for the $x. time"
}

This one takes just 0,5 seconds. That's a heck of a noticable speed increase.

The Pipeline Is EveryWhere

Sometimes, it is not so apparent that PowerShell is using the Pipeline when it really does. Here is some typical code:

PS> (Measure-Command {

Get-Content $env:windir\windowsupdate.log |
Foreach-Object { "reading line $_" }

}).TotalSeconds

1.2655871

Compare this to the version that avoids the pipeline:

PS> (Measure-Command {

$all = Get-Content $env:windir\windowsupdate.log -ReadCount 0
Foreach($line in $all) { "reading line $line" }

}).TotalSeconds

0.0848887

Again, more than 14 times faster. The thing to watch out here is to use Get-Content with the parameter -ReadCount 0. Without it, Get-Content by default is optimized for the pipeline and emits each line as a single object, causing again a lot of work for the pipeline.

Conclusions

The PowerShell pipeline is a great feature! In the last example, thanks to the pipeline, just one line of a file needed to be stored in memory. The faster version without the pipeline, in contrast, needed to hold the entire text file in memory (variable $all). So this article is not about avoiding the pipeline. It is about using the PowerShell resources wisely.

You have seen that there is a tradeoff: speed or small memory footprint. Use the pipeline if you must conserve memory. Avoid the pipeline if you need speed. And take a look again at some of the initial examples: there are a lot of standard code scenarios where conserving memory really isn't an issue. So by not using the Pipeline here, you can improve performance easily.

Stay tuned...!

Tobias

Microsoft MVP PowerShell Germany

Displaying Data in Excel (and why it won't work)

2012-04-23T22:46:00Z

It's pretty easy to send PowerShell results to Excel and display them nicely as a spreadsheet. Actually, here's a simple function called Out-ExcelReport. Pipe anything to it, and it shows up as an excel sheet (provided you have indeed installed excel beforehand):

function Out-ExcelReport {
param(
$path = "$env:temp\report$(Get-Date -format yyyyMMddHHmmss).csv",
[switch]$open
)
$Input |
Export-Csv $path -NoTypeInformation -UseCulture -Encoding UTF8
if($open) { Invoke-Item $path }
}

Now creating spreadsheets is as easy as this:

Get-Process | Out-ExcelReport -open

Why Does Excel Display Garbage?

Displaying results in Excel isn't so much a problem once you figure out how to export to CSV and then import that to excel.

The problem becomes evident a bit later, when you start to have fun and then suddenly run into lines such as this one:

Get-WMIObject Win32_NetworkAdapterConfiguration -filter 'IPEnabled=True' |
Select-Object Caption, DefaultIPGateway, MACAddress, IPAddress |
Out-ExcelReport -open

This line will indeed find your active network adapters, and it will open excel and display the network properties your code selected. But... the result probably looks similar to this:

Caption DefaultIPGateway MACAddress IPAddress
[00000007] Bro... System.String[] 68:A8:6D:0B:5F:CC System.String[]

Note how some properties display correctly (such as MACAddress), while others read "System.String[]". Where did the IP address go?

Shape Up Your Results!

The problem occurs whenever the property is NOT just text. Since excel can only deal with text, anything that is not text in the first place needs to be converted to text. "System.String[]" simply is the result of such a conversion. The property you tried to display was an array of strings. That's correct BTW. A network adapter can have multiple IP addresses, so the property IPAddress is by design an array.

To create meaningful reports, you need to convert such properties to text in a more clever way. Here are two things you can do:

Join Arrays: use the -join operator to join array elements and produce descriptive text
Out-String: send the property to Out-String and take advantage of PowerShell's own built-in conversion support

So how exactly would you implement these conversion methods? Well, here's a way:

$results = Get-WMIObject Win32_NetworkAdapterConfiguration -filter 'IPEnabled=True' |
Select-Object Caption, DefaultIPGateway, MACAddress, IPAddress

$results.DefaultIPGateway = $results.DefaultIPGateway -join ','
$results.IPAddress = $results.IPAddress | Out-String

$results | Out-ExcelReport -open

Note how I just took the result produced by Select-Object and saved that in $results. That's all. Since Select-Object always creates new objects with only the properties I want, and since objects produced by Select-Object are always read/write, I now can use whatever method I prefer to beautify those properties. Then, when all is fine, I can continue and send $results on to Out-ExcelReport. The result is a report with all the multivalues displaying fine.

Phew! Done - and you now can convert your PowerShell results to excel reports - and fine-tune any property that isn't showing up the way you want it to. Stay tuned...!

Tobias

Microsoft MVP PowerShell Germany

Finding Logged On User (and Writing To Other People's Registry)

2012-04-19T12:47:00Z

How would I find out the currently logged on user on a remote machine? And how can I remotely access another person's HKEY_CURRENT_USER registry hive? Here are some tricks you may want to try...

Who's Logged On?

To find out who's logged on to another machine, you could use WMI and check for the owner of explorer.exe. Since explorer.exe is providing the GUI, by looking at its owner you find the logged on users like this:

# specify the remote computer name or IP address you want to query
# you do need remote WMI access to that machine
$ComputerName = $env:computername

Get-WMIObject Win32_Process -filter 'name="explorer.exe"' -computername $computername |
ForEach-Object {
$owner = $_.GetOwner()
'{0}\{1}' -f $owner.Domain, $owner.User
} |
Sort-Object |
Get-Unique |
ForEach-Object {
$rv = 1 | Select-Object ComputerName, User
$rv.ComputerName = $computername
$rv.User = $_
$rv
}

Writing to Other User's HKEY_CURRENT_USER Registry Hive

Another common problem is remote registry access. There are many ways how you can remotely access the registry on another machine. Today, I will be using PowerShell remoting so as a prerequisite, PowerShell remoting needs to be set up and enabled (I am not covering that part here).

All of the following examples run locally and remotely. To run them remotely, use Invoke-Command and send the code to the remote system via PowerShell Remoting.

When you connect to a remote machine via PowerShell remoting, you can easily write to HKEY_LOCAL_MACHINE, but writing to HKEY_CURRENT_USER would always write to your very own HKEY_CURRENT_USER and not to the one of the person currently logged on to that machine. This is because PowerShell remoting uses your identity (and not the one of the currently logged on remote user).

However, as long as there *is* someone logged on the remote machine, you can easily write to that person's HKEY_CURRENT_USER as well. First, you need to know who is logged on. Here's how you can find that out:

function Get-CurrentUserName {
Get-ItemProperty 'Registry::HKEY_USERS\*\Volatile*' | Select-Object -ExpandProperty UserName
}

Basically, we are looking into HKEY_USERS and search for any user that has a subkey that starts with "Volatile", then we can read the registry value "UserName". That's it.

Accessing Logged On User via HKEY_USERS

By taking the same route, we can now write to another users' private registry hive. This is how you find the hive that represents this user:

function Get-CurrentUser {
Split-Path (dir 'Registry::HKEY_USERS\*\Volatile*' | Select-Object -ExpandProperty Name)
}

To create a new key named "NewKey" in HKEY_CURRENT_USER\Software, try this:

$user = Get-CurrentUser
New-Item Registry::$user\Software\NewKey

Now, the key was written to HKCU: in the context of the currently logged on user on the remote machine.

I hope that was a useful hint. Stay tuned...!

Tobias

Microsoft MVP PowerShell Germany

Shorten Path Names (and using Low Level API functions)

2012-04-11T07:22:00Z

Path names can grow large, and sometimes there is just not enough room to display the entire path. There is no cmdlet either that would shorten the path for you. So let's take a look today at how you can access low-level Windows API functions from PowerShell to add this feature,

As an immediate benefit, after you read this article you can optimize your PowerShell prompt and make it shorter while keeping the current path meaningful.

Mission Target: Compacting a Path

What exactly is meant by "shorten a path name"? I am not talking about turning a "long" file path into the old 8.3 file path. That's a different ball game - the 8.3 path would be shorter but at the same time completely unreadable.

What I want is to shorten a path in a human readable format, keeping as much meaningful information as possible. And while there is no PowerShell cmdlet for this, there is a low level Windows API function called PathCompactPathEx() which lives in the system file shlwapi.dll.

In pre-PowerShell-times, low level API functions were only available to developers. With PowerShell, everyone can access them. And here's how.

Accessing Low Level API Functions

To access a low level API function, you need its "signature" which describes where the function lives and how it needs to be called (its parameters and data types). Typically, the signature is defined in a language like c#, but you could as well use signatures defined in any other language that is supported by the .NET framework, like VB.NET. There are plenty of web sites that list API signatures, like pinvoke.net. Here's the signature for PathCompactPathEx() in c#:

[DllImport("shlwapi.dll", CharSet = CharSet.Auto, SetLastError = true)] public static extern bool PathCompactPathEx(System.Text.StringBuilder pszOut, string pszSrc, Int32 cchMax, Int32 dwFlags);

Let's try and decipher it (you can skip this part if your hair is on fire and you just want results):

It tells you that there is a function called PathCompactPathEx() which resides in the system file shlwapi.dll and takes four input parameters and returns a Boolean value. The four parameters needed to run the function have different data types. The first one is a StringBuilder type, the second one is a String (text), and the remainder are Int32 (32bit whole numbers). The signature also states that this is a "static" function. Static functions will later become a part of a data type, so the keyword "static" is an important piece of information once you are ready to call your API function from PowerShell.
And this is how you feed a signature into PowerShell to gain access to that function:

$sig = @'
[DllImport("shlwapi.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern bool PathCompactPathEx(
System.Text.StringBuilder pszOut, string pszSrc, Int32 cchMax,
Int32 dwFlags);
'@

Add-Type -MemberDefinition $sig -Namespace Win32 -Name StringFunctions

The secret really is to feed that signature code to the cmdlet Add-Type. This cmdlet then creates a new type (which you can assign any name you want; in this example the type is called Win32.StringFunctions), and you can use the new type then to access the function.

Two things are worth mentioning here:

First, I am submitting the signature as a "Here-String", which is a multiline text delimited by @'…'@. For Here-Strings to work, it is of utmost importance that no text whatsoever (including whitespace) follows the opening tag (@'), and that no text (including whitespace) preceeds the closing tag ('@).

Second, Add-Type is actually taking the c# signature source code and is compiling a DLL in memory which then is loaded. If your signature is written in a different language (like VB.NET), you need to specify the language by adding the parameter -Language. Valid languages currently are CSharp, CSharpVersion3, CSharpVersion2, VisualBasic, and Jscript.

After running Add-Type, you actually get a new type as the name suggests. The name of this type was set by the parameters -Namespace and -Name, so in our example the name of the new type is Win32.StringFunctions. To check out which functions are available in this type, use this line:

PS> [Win32.StringFunctions] | Get-Member -Static

TypeName: Win32.StringFunctions

Name MemberType Definition
---- ---------- ----------
Equals Method static bool Equals(System.Object objA, System.Object objB)
PathCompactPathEx Method static bool PathCompactPathEx(System.Text.StringBuilder p...
ReferenceEquals Method static bool ReferenceEquals(System.Object objA, System.Ob...

And there is our new API function PathCompactPathEx()! Don't worry about the other two, Equals() and ReferenceEquals(), these are default functions provided by every type.

Turning Low-Level Functions Into Civilized Functions

Accessing our low level function is still not a piece of cake, however, because as the term "low level" suggests, these API functions work in a very basic fashion and require you to pay attention to the required data types and containers. That's why it is a best practice to turn a hard-to-use low level function into an easy-to-use PowerShell function before you actually start using it in your scripts.

Here's the result (which you can also download here: http://powershell.com/cs/media/p/15905.aspx):

function Compress-Path($Path, $Length=20) {

$sig = @'
[DllImport("shlwapi.dll", CharSet = CharSet.Auto, SetLastError = true)]
public static extern bool PathCompactPathEx(
System.Text.StringBuilder pszOut, string pszSrc, Int32 cchMax,
Int32 dwFlags);
'@

Add-Type -MemberDefinition $sig -name StringFunctions -namespace Win32

$sb = New-Object System.Text.StringBuilder(260)
if ([Win32.StringFunctions]::PathCompactPathEx($sb , $Path , $Length+1, 0)) {
$sb.ToString()
} else {
Throw "Unable to compact path"
}
}

Now, to compact a path all you need to do is call Compress-Path and tell the function how much room you have:

PS> Compress-Path -Path $env:temp -Length 30
C:\Users\Tobias\AppDat...\Temp
PS>
PS> Compress-Path -Path 'HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall' -Len 16
HKL...\Uninstall
PS> Compress-Path -Path 'HKLM:\Software\Microsoft\Windows\CurrentVersion\Uninstall' -Len 30
HKLM:\Software\Mi...\Uninstall

A better PowerShell prompt

One first use of Compress-Path could be an improved prompt function. By default, the PowerShell prompt always displays the complete path you are in, often wasting a lot of real estate in your console or editor.

Here's an improved prompt function that displays a compressed path, still giving you a good indication of where your current folder is, and displays the complete path in your window title bar.

function prompt {
'PS {0}> ' -f (Compress-Path (Get-Location) 15)
$host.UI.RawUI.WindowTitle = (Get-Location)
}

To permanently use this prompt, just make sure you add both the function Compress-Path and the function prompt into your profile script so PowerShell loads both functions automatically at startup (both functions can be downloaded here: http://powershell.com/cs/media/p/15905.aspx). The profile script is located here:

PS C:\Us...\Tobias> $profile.CurrentUserAllHosts
C:\Users\Tobias\Documents\WindowsPowerShell\profile.ps1

And here's some code that opens the profile script for you (and creates it if it does not yet exist):

if (!(Test-Path $profile.CurrentUserAllHosts))
{ $null = New-Item $profile.CurrentUserAllHosts -Force -ItemType File };
Invoke-Item $profile.CurrentUserAllHosts

I hope you had fun! You can now take this concept and wrap all kinds of low-level API functions in the same way. Stay tuned...!

Tobias

Microsoft MVP PowerShell Germany

Regular Expressions Are Your Friend (Part 3)

2011-12-02T12:27:00Z

In Part 1, you lerned how Regular Expressions can extract useful information from noise text. In Part 2, we looked at using Regular Expressions to split text, yet another powerful technique to extract the pieces of information you may need. Today, we conclude our little excurse and use Regular Expressions to replace text. Fasten your seat belts, please!

Replacing Old Text With New Stuff

Replacing some information inside a text with some other information seems to be pretty straight-forward: simply use -replace and tell the operator which text you would like to replace with other text:

PS> 'Hello World!' -replace 'World', 'PowerShell'
Hello PowerShell!

It is not obvious at first that -replace in reality takes a Regular Expression. You may notice accidentally when you try and use characters that have a special meaning, like here:

PS> 'Hello.' -replace '.', '!'
!!!!!!

Huh? Well, "." is a special RegEx placeholder and represents any character, so the result is right. If you must use special characters, they need to be escaped:

PS> 'Hello.' -replace '\.', '!'
Hello!

Remember the trick from our previous parts? If you aren't sure which characters need to be escaped, ask PowerShell!

PS> [RegEx]::Escape('Hello.')
Hello\.

Eliminating Duplicates

Being able to use Regular Expressions does add the burden of escaping, but at the same time it adds incredible power. Let's assume you want to get rid of excessive whitespace and replace multiple whitespace with just one space. Here is the solution:

PS> $text = 'Too many blank characters'
PS> $text -replace '\s{2,}', ' '
Too many blank characters

Here, -replace was looking for any whitespace (placeholder: \s) that occurs at least 2 times (quantifier: {2,}). It then replaces these instances with a single space. Done.

Using The Match

Your new replacement text does not need to be static. It can reference the original match. Have a look:

PS> $text = 'The problem was described in KB123456. Look it up.'
PS> $pattern = 'KB\d{6,8}'
PS> $text -replace $pattern, 'a KB-Article ($0 to be precise)'
The problem was described in a KB-Article (KB123456 to be precise). Look it up.

Here, we define a pattern for KB article numbers. The replacement string contains a placeholder ($0). This placeholder seems to be a PowerShell variable, but it is not. It just looks like one, which is why you must use single quotes or else PowerShell would also interpret this as a variable and resolve it.

What exactly is $0? To explain this, take a look at the result returned by -match:

PS> $text = 'The problem was described in KB123456. Look it up.'
PS> $pattern = 'KB\d{6,8}'
PS> $text -match $pattern
True
PS> $matches

Name                           Value
----                           -----
0                              KB123456

As it turns out, $matches returns a match with the name "0", so this match is what $0 represents.

If you use grouping in your pattern (place parts of it in brackets), then you have even more choices:

PS> $text = 'The problem was described in KB123456. Look it up.'
PS> $pattern = 'KB(\d{6,8})'
PS> $text -match $pattern
True
PS> $matches

Name                           Value
----                           -----
1                              123456
0                              KB123456

Now, -replace would support both the placeholders $0 and $1:

PS> $text = 'The problem was described in KB123456. Look it up.'
PS> $pattern = 'KB(\d{6,8})'
PS> $text -replace $pattern, 'a KB-Article (with number $1 to be precise)'
The problem was described in a KB-Article (with number 123456 to be precise). Look it up.

Using Backreferences

What you have just seen is often called "backreference": your new replacement text is backreferencing the original text. That technique can be extremely versatile. Let's say you want to make sure each comma in a text is followed by a space.

Here is a solution that uses backreferences:

PS> $text = 'After each comma,I want a whitespace,but only if there was no whitespace in the first place, of course.'
PS> $text -replace ',(\S)', ', $1'
After each comma, I want a whitespace, but only if there was no whitespace in the first place, of course.

The pattern looks for a comma that is immediately followed by a non-whitespace (placeholder: /S - watch out, placeholders are case-sensitive!). If that was found, it is replaced by a comma, a space and then the backreference (which is the character that previously immediately followed the comma). Mission accomplished. Thank you, backreferences!

Finding Duplicates - Backreferences In Your Pattern

You can even find and eliminate unwanted duplicates. Here is some sample code for you:

PS> $text = 'this text text contains duplicate words words following each other'
PS> $text -replace '\b(\w+)(\s+\1){1,}\b', '$1'
this text contains duplicate words following each other

In this case, the original pattern itself uses backreferences, too. It looks for a word boundary (placeholder: \b) and then for a word (\w+). Note that the word pattern is in brackets, so it is the first group with the index 1.

Then, it looks for at least one space and the same word again (placeholder: \1). So it is backreferencing the first group match. The quantifier ({1,}) tells RegEx that we are looking for one or more duplicate words.

The -replace operator then replaces all of this with the backreference to the first word, thus eliminating all the duplicate words.

So if you want to backreference matches in your replacement text, you prefix the group number with a $. If you want to backreference matches in your pattern, prefix the index with \ instead.

Capitalizing First Letter in a Word

Every now and then, even Regular Expressions can't do the job, so PowerShell scriptblocks can assist. Let's assume you want to capitalize every first letter in every word. Identifying the first letter is easy, but Regular Expressions cannot capitalize characters.

That's why you can submit a PowerShell script block that tells RegEx what to do with the match. Here is a sample:

PS> $text = 'i want to capitalize every word in this sentence.'
PS> $pattern = '\b(\w)'
PS> [RegEx]::Replace($text, $pattern, { param($letter) $letter.Value.toUpper() })
I Want To Capitalize Every Word In This Sentence.

Since -replace cannot process PowerShell script blocks, you need to use .NET directly here. The pattern describes a word boundary (placeholder: \b) and then the immediately following first character (placeholder: \w). Note that this placeholder is placed in brackets, so it is the first group.

Next, this sample uses the .NET class [RegEx] and its Replace() method which is the same dog food -replace uses internally, too. This time, though, a script block is submitted as replacement text.

This script block is often called a "delegate" because it is called for each match and determines the replacement dynamically. It receives the match through a param() block and then can use whatever logic you want to return the replacement text. In our case, it simply converts the received letter to uppercase.

What’s Next?

We are done with our tour of PowerShells' RegEx support. Next time, we check out hashtables and the clever things you can do with them. Hashtables are really everywhere in PowerShell, but they are a little shy, so you won't see them at first. Knowing about hashtables, how they work and what can be done with them will get you a lot more control over PowerShell. I am sure you'll be surprised! Stay tuned...

Tobias

Microsoft MVP PowerShell Germany

Regular Expressions Are Your Friend (Part 2)

2011-11-14T10:37:00Z

In Part 1, we used Regular Expressions to extract useful information from noise text. That was pretty cool, and we used the operator -match, the automatic PowerShell variable $matches and the powerful Get-Matches function.

Today, we'll use Regular Expressions again, but this time they are used to split text.

Chopping Text Into Pieces

Regular Expressions can be your butchers' knife to split text into pieces. Why on earth would you like do that, though?

For example, to process a comma-separated list of values. Have a look:

'comma,separated,list' -split ','

Here, the operator -split takes the text on its left side and chops the text up at each comma it finds. Yes, Regular Expressions do not need to be monsters. They can be very small. In this case, our Regular Expression simply is the comma - and represents a comma. It is - uh - a placeholder for - a comma. Easy enough.

You now could pipe the results to Foreach-Object and do stuff for all the elements in your comma-separated list:

$computers = 'pc1,pc2,server12,serverAB'
$computers -split ',' | Foreach-Object { "I am doing something with $_" }
I am doing something with pc1
I am doing something with pc2
I am doing something with server12
I am doing something with serverAB

$computers -split ',' | Foreach-Object { Get-WMIObject Win32_BIOS -computername $_ }

You can also use splitting to take apart path names. Here, the pattern is the backslash (if you want the parts of the path) or a dot (if you are after the file extension). This time, however, the pattern to use is not as straight-forward. Both the backslash and the dot are special characters in Regular Expressions. So if you really mean a backslash or a dot, you need to escape them:

PS> 'c:\windows\file.txt' -split '\.'
c:\windows\file
txt
PS> ('c:\windows\file.txt' -split '\.')[-1]
txt
PS> 'c:\windows\file.txt' -split '\\'
c:
windows
file.txt
PS> ('c:\windows\file.txt' -split '\\')[-1]
file.txt

Escaping Regular Expressions

As you have just discovered, sometimes it is necessary to escape characters in Regular Expressions so they are taken literally.

Fortunately, there is a way to automatically escape characters. If you aren't sure if something you mean literally should indeed be escaped or not, try using this trick:

PS> [Regex]::Escape('c:\folder\file.txt')
c:\\folder\\file\.txt

It takes the text you provide and returns it in escaped form so Regular Expressions wouldn't treat anything in that text as special characters.

Splitting Without Consuming

By default, when you chop up text using -split, the split character(s) are "consumed", so they are gone after the split.

PS> $text = 'This text gets splitted, and the delimiter is by default consumed, as you can see'
PS> $text -split ','
This text gets splitted
and the delimiter is by default consumed
as you can see

With look-behind expressions, you can make sure that the splitting pattern is not consumed:

PS> $text -split '(?<=, )'
This text gets splitted,
and the delimiter is by default consumed,
as you can see

Basically, "?<=" tells the RegEx engine to split right after the splitting character. Splitting without consuming can become important as you'll see in a second.

Separating Hex-Pairs

Let's assume you have a long list of hexadecimal pairs and would like to convert that list back to decimals:

'00AB1CFFAB1034'

To do that, you would somehow have to split after every second character and without consuming because you do not want to loose any information. Here is how you solve this puzzle:

PS> '00AB1CFFAB1034' -split '(?<=\G[0-9a-f]{2})(?=.)'
00
AB
1C
FF
AB
10
34

Now you can pipe the pairs to Foreach-Object and convert them to decimals or bytes or whatever you may need:

PS> '00AB1CFFAB1034' -split '(?<=\G[0-9a-f]{2})(?=.)' |
Foreach-Object { [System.Convert]::ToInt32($_, 16) }
0
171
28
255
171
16
52

PS> '00AB1CFFAB1034' -split '(?<=\G[0-9a-f]{2})(?=.)' |
Foreach-Object { [System.Convert]::ToByte($_, 16) }
0
171
28
255
171
16
52

Whew, wait a minute! This time, the Regular Expression wasn't as simple anymore. Right. Regular Expressions aren't hated without a reason. Sometimes they become tricky little monsters with mind-blowing lookaheads, lookbehinds and lookarounds. But is that a reason not to use them? No! We all use microwaves to make popcorn, but do we care about the physics? I don't.

So always remember: you do not need to care about Regular Expression mechanics once you have one that does what you want it to do. So once you have a Regular Expression that splits Hex-Pairs, you are good. Simply use it.

Instead of spending frustrating hours of crafting complex RegEx patterns, think "code reuse" and visit a search engine like google.Look for "RegEx <Pattern>", for example "RegEx IPAddress". Chances are someone else has already done the dirty job for you, and all you need to do is copy and paste the RegEx pattern.

If you do want to digest patterns and really understand what is going on, there are plenty of great RegEx tutorial sites. This one explains lookbehinds, lookaheads and lookarounds and helps you understand the previous example: http://www.regular-expressions.info/lookaround.html.

What’s Next?

We covered -match and -split, but there's a third family member: -replace. So next time we'll continue talking about RegEx and look how you can replace things, update IP lists and more. Meanwhile, if you have spare time, go visit some of the plenty sites that focus on RegEx, and try and refine your understanding of RegEx placeholders, anchors and quantifiers. Stay tuned!

Tobias

Microsoft MVP PowerShell Germany

Regular Expressions Are Your Friend (Part 1)

2011-10-27T02:04:00Z

Yes, regular expressions can be complex but hey no, they do not need to be! You can do amazing things with small and relatively simple regular expressions. So let’s take a look at it.

Finding Information: -match

Often, you need some piece of information that is buried inside of noise text. Let’s assume you’d like to find the KB article number that is in this text:

PS> $text = 'The problem was discussed in KB552356 and is solved. KB652534 was not addressed.'

Regular expressions can do this for you. All you need to do is define the pattern you are looking for. Here is a sample pattern that identifies KB article numbers:

PS> $pattern = 'KB\d{4,6}'

Before we look at the pattern and how it is constructed, let’s see results. PowerShell has an operator called –match that takes regular expressions and returns $true when it found the pattern in the text:

PS> $text -match $pattern
True

If there was a match, you even get back the expression that matched your pattern in the automatic variable $matches:

PS> $matches

Name                           Value
----                           -----
0                              KB552356

PS> if ($text -match $pattern) { $matches[0] }
KB552356

How Patterns Work

Regular expression patterns basically contain three things: anchors, placeholders, and quantifiers. In the pattern we used, we had all three elements:

“KB” was an anchor. It expected the pattern to start at the static text “KB”.
\d is a placeholder. It represents a digit.
{4,6} is a quantifier. It expects the placeholder to occur four to six times, so it matches 4 to 6 digits.

That’s not too hard, eh?

Creating Groups

Let’s assume you are just interested in the number of the KB article. To get just the number, you can create groups by using parenthesis.

Here is the refined pattern:

PS> $pattern = 'KB(\d{4,6})'
PS> $text -match $pattern
True
PS> $matches

Name                           Value
----                           -----
1                              552356
0                              KB552356

Here, $matches returns two results. Index 0 holds the complete match, and index 1 holds the match of the first group. It contains just the number of the KB article.

PS> if ($text -match $pattern) { $matches[1] }
552356

Finding Multiple Matches

Unfortunately, -match only finds the first match. That’s why it is called –match and not –matches. And this is also why it only found the first KB article number in the sample text, not the second one.

There is a built-in way in PowerShell to find multiple matches, but it is somewhat hidden. Select-String can find multiple matches:

PS> $text | Select-String -AllMatches $pattern

The problem was discussed in KB552356 and is solved. KB652534 was not addressed.

However, the result is the entire text. Only when you poke in the results will you get the individual matches:

PS> $text | Select-String -AllMatches $pattern | Select-Object -ExpandProperty Matches | Select-Object -ExpandProperty Value
KB552356
KB652534

Since that’s not really beautiful, you can create your own filter and use it to extract multiple matches:

filter Get-Matches($Pattern) {
$_ | Select-String -AllMatches $pattern |
Select-Object -ExpandProperty Matches |
Select-Object -ExpandProperty Value
}

PS> $text | Get-Matches $pattern
KB552356
KB652534

Your filter is pretty flexible. You can use it for example to parse entire log files:

PS> Get-Content $env:windir\windowsupdate.log -Encoding UTF8 |
>> Get-Matches 'successfully installed.*?update:.*'
>>
successfully installed the following update: Microsoft Silverlight (KB979202)
successfully installed the following update: Security Update for Windows 7 for x64-based
Systems (KB2532531)

(...)

Check the pattern used here:

We have an anchor (‘successfully installed’), then a placeholder (‘.’=anything) with a quantifier (‘*?’ = any number, but as short as possible), then another anchor (‘update’), and then again a placeholder (‘.’ = anything) with a quantifier (‘*’ = any number).

Essentially, this extracts all lines from the log file where updates were installed.

Finding Multiple Matches Fast

Select-String is rather slow, though. To retrieve multiple matches, using .NET directly speeds up things considerably. Here’s the Get-Matches function as a .NET implementation:

function Get-Matches($Pattern) {
begin { $regex = New-Object Regex($pattern) }
process { foreach ($match in ($regex.Matches($_))) { ([Object[]]$match.Groups)[-1].Value } }
}

PS> Get-Content $env:windir\windowsupdate.log -Encoding UTF8 | Get-Matches 'successfully installed.*?update:.*'
successfully installed the following update: Microsoft Silverlight (KB979202)
successfully installed the following update: Security Update for Windows 7 for
x64-based Systems (KB2532531)
successfully installed the following update: Update for Windows 7 for x64-based
Systems (KB2533623)

(...)

This is not just a lot faster, it also supports grouping, so if you just wanted the product name that was installed, place the last placeholder-expression into parenthesis. Get-Matches always returns the last group:

PS> Get-Content $env:windir\windowsupdate.log -Encoding UTF8 | Get-Matches 'successfully installed.*?update: (.*)'
Microsoft Silverlight (KB979202)
Security Update for Windows 7 for x64-based Systems (KB2532531)
Update for Windows 7 for x64-based Systems (KB2533623)
Security Update for Windows 7 for x64-based Systems (KB2507938)
Security Update for Windows 7 for x64-based Systems (KB2555917)

Clever RegEx Matching

To make life much easier, here’s a function that returns regular expression matches as real PowerShell objects:

function Get-Matches {
param(
    [Parameter(Mandatory=$true)]
    $Pattern,

    [Parameter(ValueFromPipeline=$true)]
    $InputObject
)

begin {

    try {
   $regex = New-Object Regex($pattern)
  }
  catch {
   Throw "Get-Matches: Pattern not correct. '$Pattern' is no valid regular expression."
  }
  $groups = @($regex.GetGroupNames() |
  Where-Object { ($_ -as [Int32]) -eq $null } |
  ForEach-Object { $_.toString() })
}

process {
  foreach ($line in $InputObject) {
   foreach ($match in ($regex.Matches($line))) {
    if ($groups.Count -eq 0) {
     ([Object[]]$match.Groups)[-1].Value
    } else {
     $rv = 1 | Select-Object -Property $groups
     $groups | ForEach-Object {
      $rv.$_ = $match.Groups[$_].Value
     }
     $rv
    }
   }
  }
}
}

Now, if you’d like to extract information, you could use it like this:

PS> "KB2786873 good KB27897983 bad " | Get-Matches 'KB\d{4,8}'
KB2786873
KB27897983

You could also create named groups which you do by adding ?<Name> to your parenthesis. Then, Get-Matches automatically turns these groups into object properties:

PS> "KB2786873 good KB27897983 bad " | Get-Matches '.*?(?<KB>KB\d{4,8}).*?(?<
word>\w+)'

KB                                     word
--                                      ----
KB2786873                       good
KB27897983                     bad

Take a look how easily you now can parse a raw text file and turn it into real PowerShell objects! First read the log file:

$text = Get-Content $env:windir\windowsupdate.log -Encoding UTF8 -ReadCount 0

Next, try a couple of RegEx patterns:

PS> $text | Get-Matches 'successfully installed.*?update: (.*)'
Microsoft Silverlight (KB979202)
Security Update for Windows 7 for x64-based Systems (KB2532531)
Update for Windows 7 for x64-based Systems (KB2533623)
Security Update for Windows 7 for x64-based Systems (KB2507938)

(...)

PS> $text | Get-Matches 'successfully installed.*?update: (?<Product>.*)'

Product
-------
Microsoft Silverlight (KB979202)
Security Update for Windows 7 for x64-based Systems (KB2532531)
Update for Windows 7 for x64-based Systems (KB2533623)
Security Update for Windows 7 for x64-based Systems (KB2507938)
Security Update for Windows 7 for x64-based Systems (KB2555917)

(...)

PS> $text | Get-Matches '(?<Date>\d{4}-\d{2}-\d{2}).*?(?<Time>\d{2}:\d{2}:\d{2}).*?successfully installed.*?update: (?<Product>.*)'

Date                       Time                       Product
----                       ----                       -------
2011-10-17                 13:05:46                   Microsoft Silverlight ...
2011-10-17                 13:59:38                   Security Update for Wi...
2011-10-17                 13:59:38                   Update for Windows 7 f...
2011-10-17                 13:59:38                   Security Update for Wi...
2011-10-17                 13:59:38                   Security Update for Wi...

(...)

PS> $text | Get-Matches '(?<Date>\d{4}-\d{2}-\d{2}).*?(?<Time>\d{2}:\d{2}:\d{2}).*?successfully installed.*?update: (?<Product>.*?) \({0,1}KB(?<KB>\d{5,8})'

Date                Time                Product             KB
----                ----                -------             --
2011-10-17          13:05:46            Microsoft Silver... 979202
2011-10-17          13:59:38            Security Update ... 2532531
2011-10-17          13:59:38            Update for Windo... 2533623
2011-10-17          13:59:38            Security Update ... 2507938
2011-10-17          13:59:38            Security Update ... 2555917

(...)

PS> Get-Content $env:windir\windowsupdate.log -Encoding UTF8 -ReadCount 0 | Get-Matches '(?<Date>\d{4}-\d{2}-\d{2}).*?(?<Time>\d{2}:\d{2}:\d{2}).*?successfullyinstalled.*?update: (?<Product>.*?) \({0,1}KB(?<KB>\d{5,8})'

(...)

Using Special Options

There are two ways for you to find matches: use the built-in operator –match, or use the .NET Framework RegEx type. Generally, you want to use the .NET approach whenever you need to find more than just the first match.

Depending on which method you choose, there are different default settings, though, that you need to know about. The –match operator always works case-insensitive whereas .NET always works case-sensitive by default.

The easiest way to make an expression case-insensitive is to use control operators. This pattern for example works case-sensitive with –match and case-insensitive with .NET:

$pattern = ‘KB\d{6,8}’

This pattern always works case-insensitive, regardless of which method you use:

$pattern = ‘(?i)KB\d{6,8}’

Another important control operator turns on single-line-mode. Let’s assume you downloaded HTML code from a web page and want to find all instances of a pattern. To find all matches, you need to use .NET. To find the pattern in the entire HTML source code, you need single-line-mode.

To make this work, your pattern should start with (?si).

What’s Next?

Next week, I’ll keep talking about RegEx and look at splitting and replacing. Meanwhile, if you have spare time, go visit some of the plenty sites that focus on RegEx, and try and refine your understanding of RegEx placeholders, anchors and quantifiers. Stay tuned!

Tobias

Microsoft MVP PowerShell Germany

Reuse Your Code - Create Modules Automagically!

2011-10-14T16:34:00Z

When you write scripts, you want to automate things, fair enough. When it comes to writing scripts, a lot of people do not automate, though. They do the same coding over and over again and waste time and consistency.

So let’s look at some simple code-reusing techniques. It’s not that hard at all. In fact, after you read this post you can create your own PowerShell modules in seconds.

Code Reuse – Join The Game!

Actually, most PowerShell script writers reuse code all the time without realizing – they’re just not reusing their own code. When you run a cmdlet, you are reusing the code someone else put into that cmdlet. You know that’s cool because it makes PowerShell scripts much shorter and easier to write.

If you want to sort stuff, use Sort-Object. That's code reuse. In the old days of VBScript you had to re-invent and code bubble-sort-algorithms all the time. No more in PowerShell. If you want to manage Active Directory, import the AD module and use Get-ADUser to search for users. Again, no need for coding tricky AD connect and search stuff. It's done for you so you can focus on your tasks. Cmdlets are a big old code reuse party.

Most PowerShell scripters happily use cmdlets but eventually find that they need additional stuff because there aren’t predefined cmdlets for everything. So they start scripting day and night and produce long and convoluted scripts that are hard to manage and adjust.

Stop that. When you need to automate something, grab a coffee and think first. What is the “vocabulary” (cmdlets, functions) you need to address the problem? Do you have all the vocabulary you need for that? What is missing? Which PowerShell module might already provide the needed vocabulary? There are great modules for VMWare, HyperV, AD, Exchange, SQLServer, you name it.

Chances are, some vocabulary you need is missing. When that occurs, design the functions that provide the missing vocabulary first. Once they are done, continue with your initial job. Does that take more time? No, it saves time. Here's how.

Adding More Vocabulary - Outputting Data to Excel (for example)

Let’s say as part of your solution you need a way to output information to Microsoft Excel.

Instead of incorporating this functionality into your overall solution (and producing monster scripts), define a function that just focuses on outputting information to Microsoft Excel. Clean and sweet. It could be as simple as this one:

function Out-ExcelReport {
param(
$path = "$env:temp\report$(Get-Date -format yyyyMMddHHmmss).csv",
[switch]$open
)

$Input |
Export-Csv $path -NoTypeInformation -UseCulture -Encoding UTF8

if($open) { Invoke-Item $path }
}

Once your function is done, test it. Run it like this:

Get-Process | Out-ExcelReport –open

Or like this:

Get-Process |
Select-Object Name, Description, Company, MainWindowTitle |
Where-Object { $_.MainWindowTitle } |
Out-ExcelReport –open

Making Your Vocabulary Reusable, Too!

Once your new function is working like a charm, you could be tempted to copy and paste the code into your overall solution. Don’t do that! Don't assimilate your great new function into some script. Keep it separate! Make it reusable!

Any script that wants to export data to Microsoft Excel should import the necessary vocabulary from PowerShell modules. Which raises the question: how would you turn your function into a module? That’s done automagically when you download, unpack and import this helper module: http://powershell.com/cs/media/p/12823.aspx. Simply import it like this:

Import-Module c:\pathtomodulefolder\ModuleHelper

Make sure you specify the path to the unzipped folder named "ModuleHelper" inside the ZIP (due to zipping, there are two nested folders, take the one inside the other), and also note that as a general prerequisite, your PowerShell execution policy should of course allow scripts to run.

Now, to save your brand new function Out-ExcelReport to a new module of your choice, use the new command Out-Module:

Out-Module -Module ReportingStuff –Function Out-ExcelReport

That’s it! You may have noticed that there was no module “ReportingStuff” before, but there is now:

Get-Module –ListAvailable

So when you import this new module, you load your function Out-ExcelReport. Essentially, you created the same great code reuse mechanism that you enjoyed from other vendors to run AD or Exchange cmdlets. Open a new PowerShell console, and try this:

Import-Module ReportingStuff
Get-Service | Out-ExcelReport -open

Your new function Out-ExcelReport just became reusable, and you can import your module in every script that needs your new reporting capabilities now. Never do you have to waste time again to create that functionality again. Cool, eh?

Adding and Managing Scripts

So far, your new module “ReportingStuff” has only one function. You can add more, though. Your new module serves as a script repository for you, and you can save all functions into that module that deal with reporting. For example, maybe you’d also like to be able to easily create text files that are wide enough to show all results without truncation. Here’s the function:

<#
.SYNOPSIS
Saves results as text to a text file
.DESCRIPTION
Saves results to text file and automatically adjusts the file width so no columns and no data is truncated.
Truncation only occurs for object properties that contain multi-line text.
.PARAMETER Path
Optional. Name of text file to create. Existing files will be overwritten. By default, a temporary file in the temp folder is created.
.PARAMETER Open
Opens the text file after creation in the application that is associated with .txt-files.
.EXAMPLE
Get-Process | Where-Object { $_.MainWIndowTitle} | Select-Object Name, Description, Company, MainWindowTitle , CPU | Out-TextReport -Open
Lists various pieces of information about running processes and writes them to a text file. The file width is automatically adjusted to provide enough room to display all information. The text file then is opened by its default application.
.EXAMPLE
Get-ACL $env:windir | Out-TextReport -open
Reads the windows folder NTFS permissions and writes the results to a text file. The text file then is opened by its default application.

#>
function Out-TextReport {
param(
$Path = "$env:temp\report.txt",
[switch]$Open
)

$Input |
Format-Table -AutoSize |
Out-File $Path -Width 10000
if($open) { Invoke-Item $Path }
}

Run it, and test it thoroughly. Once done, again save the function to your module, directly from memory:

Out-Module ReportingTools Out-TextReport

Your new module now has two pieces of vocabulary. To update it, use –force while you import it:

Import-Module ReportingTools –Force -Verbose

Adding Help and Updating Modules

Note that with Out-TextReport, I added a comment-based help block to the code. That’s a really good idea because you probably remember how you discovered and learned 3rd party cmdlets – by using Get-Help and looking at the examples.

You should provide the same helpful information for the users of your own “cmdlets”. That’s just fair (and gives them an especially professional touch, too). So take a little time and create a great help block with practical examples.

If you save a function to your module that does not yet have a help block, Out-Module adds one. So when you load and edit functions from your module (hang in, we’ll cover that in a second), you just need to fill out the placeholders and save the script.

You can even update functions in your module(s). Just assume you wanted to polish a function. Simply open the module path, and load the function into your favorite script editor. Make the adjustments, then save it back:

Explorer (Get-ModulePath ReportingTools)

Likewise, you can send a function that you changed, back to the module by using Out-Module again. It will replace the old version.

More to come...

At our first European PowerShell Deep Dive yesterday, I also did a presentation about the power of regular expressions and how beautifully PowerShell embraces them. I added all the example functions to a sample module using the technique described above. So next week, I'll share some serious RegEx magic with you. Stay tuned!

Tobias

Microsoft MVP PowerShell Germany

Applying NTFS Permissions

2011-09-21T10:21:00Z

Recently, I needed to create a folder with NTFS permissions. PowerShell can do that for you, and when you look at your weapons, you'll find that sometimes it is best to mix command types and not just stick to cmdlets. At the end, I had a handy function that would take a path and a username and do all the tricky NTFS permissions stuff for me. Here is how.

Creating Folders

Let's start with the simple part and write a function that creates a folder if that folder does not yet exist:

function Create-Folder {
param($path)

if ( (Test-Path $path) -ne $true) {
New-Item -Path $path -ItemType Directory | Out-Null
}
}

Apply Permissions

Next, I want to add NTFS permissions to that folder. A specifc user should get change permission, and all Administrators should get full permission. There are cmdlets to do the job like Get/Set-ACL, but working with them can be hard because they really are just simple wrappers for low-level .NET methods.

PowerShell is not limited to cmdlets. You can happily use established console-based applications like cacls.exe. Here is an example:

function Create-Folder {
param($path, $user)

if ( (Test-Path $path) -ne $true) {
New-Item -Path $path -ItemType Directory | Out-Null
}

CACLS $path /G "Domain Admins":F
CACLS $path /E /G $user:C
}

As it turns out, this won't work yet. The first call to CACLS removes existing permissions which is what I want. However, this triggers a confirmation, so I would have to manually type "Y" to approve the operation. The second call to CACLS won't work at all. CACLS can't understand it, so it throws back its manual to the caller.

To correct these issues, a couple of tricks are needed. To automatically send a key to a confirmation, place it on the pipeline. And to resolve the syntax issue, submit arguments to CACLS as a string so PowerShell won't get confused:

function Create-Folder {
param($path, $user)

if ( (Test-Path $path) -ne $true) {
New-Item -Path $path -ItemType Directory | Out-Null
}

'y' | CACLS """$path"" /G ""Domain Admins"":R" | Out-Null
CACLS """$path"" /E /G ""$user"":F" | Out-Null
}

Now it is easy to create new folders and apply standard NTFS security:

Create-Folder -path c:\user1 -user mydomain\username

Careful!

Unfortunately, this approach is not culture-neutral. On non-US systems, you probably will want to change two things:

Make sure you use the correct name for the Admin group. On German systems, it is called "Domänen-Admins" instead of "Domain Admins"
Make sure you send the correct confirmation key. On German systems, it is "J" rather than "Y"

I know that sucks, but for a number of scenarios, incorporating tools like CACLS can simplify life tremendously. Learning points here are: console-based applications are (almost) equal PowerShell citizens. To submit arguments to them, you may have to turn them into a string in order to avoid parsing conflicts.

See you next time!

Tobias

Microsoft MVP PowerShell Germany

Bulk-Renaming Files (and other magic)

2011-09-12T07:49:00Z

Bulk-renaming files can cause some headache: maybe you'd like to nicely rename all your pictures with a keyword and an incrementing counter, or you'd want to clean up log files and assign them a standard name based on its creation date. It's easy to do that for one file, and with our latest tip, we illustrated how you can as just as easily do this for hundreds of files, too. Heck, did we get a lot of feedback, both additional questions and great suggestions, so let's check out what other options you have to bulk-rename files.

Creating Filenames with an Incrementing Counter

In our original tip, we wanted to rename the content of a folder filled with picture files. Each picture file was supposed to get a new file name like "pictureX", where "X" was an incrementing counter. Here is the logic:

$global:i = 1

dir c:\pictures\ -Filter *.jpg |
Rename-Item -NewName {
"picture_$i.jpg"; $global:i++
}

There are two things to note here: First, we need a global variable $i which holds the incrementing counter. It needs to be global because we want to remember its setting and increment it with each new file. By default, variables are valid only within their "territory" (scriptblock) and below, so if the counter variable was not global, the scriptblock would always start with a new variable.

Second, note how Rename-Item accepts a scriptblock. Instead of assigning a static file name, a scriptblock can dynamically create the filename for you. In our example code, the new filename consists of a fixed prefix "picture_" and the counter variable. Pretty nice! But it gets better.

Taking Advantage of $_

Jacques Barathon pointed out that the scriptblock that you submit to Rename-Item of course has a $_ automatic variable. This is true for most scriptblocks that you submit to cmdlets. $_ always represents the current object you are working with.

That's cool because now you can easily use the existing file name and manipulate it. Here is what Jacques did: he wanted to replace a single word in a filename. Let's check out how he did this:

dir *.docx |
Rename-Item -NewName {$_.Name.Replace(‘chapter’, ‘chapitre’)}

Whew! That was easy! Since $_ is referring to the current file, he accessed the original filename through $_.Name. The result is the filename, and it is a string data type. So he could then use the string method Replace() to replace a word in that filename. The result was the new filename where 'chapter' was replaced by 'chapitre'. This result was then used by Rename-Item to rename the file.

The Sky Is The Limit

Of course, $_ gives you complete access to all file properties, so you can basically access and use all file properties to construct the new filename. Let's say you wanted to replace a filename by a timestamp based on the creation date. Here is some code that does it:

$code = {
$timestamp = Get-Date $_.CreationTime -format 'yyyyMMddHHmmssff'
$extension = $_.Extension
$timestamp.$extension
}

dir c:\pictures | Rename-Item -NewName $code

Note how we placed the code (scriptblock) into an extra variable $code to keep the code clean.

While you are playing with this sample, you'll definitely run into issues. When doing bulk operations, you always have to take all kinds of edge cases into consideration. What if there are two files with the very same creation date? Since no two files can have the same name, Rename-Item would fail at the second file and not continue. Bummer.

Happily, you are not bound to any limits. Your scriptblock can contain as much logic as you like. So here is a more advanced approach that takes care of duplicates. It first creates the new filename, then checks if a file with that name already exists, and if so adds a counter to the filename.

$code = {
$timestamp = Get-Date $_.CreationTime -format 'yyyyMMddHHmmssff'
$extension = $_.Extension

$counter = 0
while ($true) {
$filename = '{0}{1}{2}' -f $timestamp, $( if ($counter) { "-$counter" } else { '' }) , $extension
$filepath = Join-Path (Split-Path $_.FullName) $filename
if (Test-Path $filepath) {
$counter++
} else {
$filename
break
}
}
}

dir c:\pictures | Rename-Item -NewName $code

You can reuse this logic for all events where you cannot be sure that the filename created by your scriptblock is unique.

That's the scoop for today. Hope to see you next week,

Tobias

Microsoft MVP PowerShell Germany

Clever Splatting to Pass Optional Parameters

2011-08-03T04:15:00Z

Getting system information from WMI is a pretty straight-forward thing with Get-WMIObject. You can even specify one or more computer names or IP addresses to retrieve the information remotely. That's all great, but how would you design a function that encapsulates Get-WMIObject (or any other cmdlet that can work both locally and remotely)? That's not so trivial.

Have a look. There's a really clever solution for that!

Prerequisites: Getting Information Locally or Remotely

Get-WMIObject is a great example of a cmdlet with built-in remoting. For example, to get information about your BIOS, use this call:

PS> Get-WMIObject -Class Win32_BIOS

SMBIOSBIOSVersion : 02LV.MP00.20081121.hkk
Manufacturer : Phoenix Technologies Ltd.
Name : Phoenix SecureCore(tm) NB Version 02LV.MP00.20081121.hkk
SerialNumber : ZAMA93HS600210
Version : SECCSD - 6040000

PS> Get-WMIObject -Class Win32_BIOS -ComputerName storage1

SMBIOSBIOSVersion : P03
Manufacturer : Phoenix Technologies LTD
Name : Ver 1.00PARTTBLw
SerialNumber : 98H340ED2H9300237A30A1
Version : PTLTD - 6040000

As you can see, the cmdlet gets information both from your local system and from one (or more) remote systems - provided they are correctly configured and you have appropriate permissions. It all depends on the parameters you submitted to Get-WMIObject.

We are not digging into the security details here because that's another topic, but if you can't get Get-WMIObject to work against remote systems, try enabling the "Remote Management Exception" in your firewall when you get "RPC"-exceptions, and use the parameter -credential to log on as a different user when you get "Access Denied"-exceptions.

Creating A (Stupid) Wrapper Function

The call to Get-WMIObject is really very trivial, but let's assume you still want to encapsulate it and use it as a new function. How would you design a function that can run both against the local machine and the remote machine(s)?

In traditional thinking, you would have to check whether the user has actually submitted the -computername parameter, and then call Get-WMIObject multiple times: if the parameter was submitted, you pass it on to the cmdlet, and if not, you omit it. The result usually is pretty awkward and becomes even more convoluted when you add additional parameters. Here is an ugly example:

function Get-BIOS {
param(
$computername = $null,
$credential = $null
)

if ($credential) {
Get-WmiObject Win32_BIOS -computername $computername -credential $credential
} elseif ($computername) {
Get-WmiObject Win32_BIOS -computername $computername
} else {
Get-WmiObject Win32_BIOS
}
}

At least, it does what it was supposed to do:

PS> Get-BIOS

SMBIOSBIOSVersion : 02LV.MP00.20081121.hkk
Manufacturer : Phoenix Technologies Ltd.
Name : Phoenix SecureCore(tm) NB Version 02LV.MP00.20081121.hkk
SerialNumber : ZAMA93HS600210
Version : SECCSD - 6040000

PS> Get-BIOS -computername storage1

SMBIOSBIOSVersion : P03
Manufacturer : Phoenix Technologies LTD
Name : Ver 1.00PARTTBLw
SerialNumber : 98H340ED2H9300237A30A1
Version : PTLTD - 6040000

PS> Get-BIOS -computername storage1 -credential (Get-Credential Administrator)

SMBIOSBIOSVersion : P03
Manufacturer : Phoenix Technologies LTD
Name : Ver 1.00PARTTBLw
SerialNumber : 98H340ED2H9300237A30A1
Version : PTLTD - 6040000

Create a (Clever) Wrapper Function

When you want to pass optional parameters to cmdlets, there is a much more clever and elegant way in PowerShell: Splatting! Instead of manually handing over parameters to a cmdlet, you create a hash table with the parameters you want to pass on. Then, you submit the hash table to the cmdlet.

The advantage of this not widely known technique is obvious once you see the improved version of the wrapper function: PowerShell already makes available an object that you can use for splatting! The system variable $PSBoundParameters contains all the parameters that were submitted to your function, so whatever someone submitted can be passed on to a cmdlet without the need to check for parameters and call different code:

function Get-BIOS {
param(
$computername,
$credential
)

Get-WmiObject -Class Win32_BIOS @psboundparameters

}

Isn't that a beauty? It works exactly the same, with much less code. The function calls Get-WMIObject to retrieve WMI BIOS information and optionally passes on the computername and/or credential parameter. No extra work for you anymore.

Thanks to Aleksandar and HAL (both PS MVPs) for their valuable input!

Hope to see you next week,

Tobias

Microsoft MVP PowerShell Germany

Dealing With File Locks

2011-06-23T10:40:00Z

Sometimes, when you try and access a file, it may refuse to open because it is locked by another user or process. Likewise, you may want to lock a file yourself to make sure the file is not accessed and read/changed while you are manipulating it. Which raises the question: how do you control file locks?

Locking a file

Get-Content can read text-based files. This cmdlet is not locking the file, so while you are reading it, others can still access or even change it.

To apply a file lock, you need to use low level .NET methods. The next example illustrates how to open a file with a read/write lock. While this script is running, no one can access the file:

$file = [System.io.File]::Open('c:\files\somefile.txt', 'Open', 'Read', 'None')
Read-Host 'Press ENTER to release file'
$file.Close()

The method Open() accepts four arguments:

The path to the file you want to access
The action you want to take, for example "Open"
The access you want to use, for example "Read"
The lock. "None" grants to access to others, so the file is locked. "Read" would allow others to read the file while you are using it, and "ReadWrite" would allow others to read and/or write to the file while you are using it

Reading File Content

When you lock a file, no one else can access it, so you cannot use Get-Content to read its content anymore, either. To read the file content, you would have to refer to .NET methods. Here is an example that reads file content of a file you applied a lock on:

$file = [System.io.File]::Open('c:\files\somefile.txt', 'Open', 'Read', 'None')
$reader = New-Object System.IO.StreamReader($file)
$text = $reader.ReadToEnd()
$reader.Close()
$file.Close()

Dealing With Existing File Locks

When you try and access a file, others may have already applied locks to it. For example, windows maintains a file called $env:windir\windowsupdate.log which carries a "Write" lock: you can read its contents, but you cannot write to it.

If you tried and accessed that file with a lock of type "None" (exclusive access), your code would fail because the system already implemented a lock which is incompatible to the lock you wanted to apply. You can only lock a file for exclusive access if noone else has applied a lock before.

To deal with this, you would have to make sure your lock is not conflicting with existing locks. When you specify a "ReadWrite" lock (essentially allowing others to still read and write the file), you can safely read the file despite other locks (except if someone had applied a "None" lock):

$file = [System.io.File]::Open("$env:windir\windowsupdate.log", 'Open', 'Read', 'ReadWrite')
$reader = New-Object System.IO.StreamReader($file)
$text = $reader.ReadToEnd()
$reader.Close()
$file.Close()

This is how Get-Content works. It uses a "ReadWrite" lock. The above low level .NET access is a lot faster though because it is not reading the file line by line. Instead, it returns the entire file content as one (large) string.

Hope to see you next week,

Tobias

Microsoft MVP PowerShell Germany

Renaming Files Puzzle - And Three Golden PowerShell Rules

2011-05-23T11:57:00Z

I am moderator for one of our Ask-the-Experts forums, and every once in a while, I get questions I would like to discuss. Like this one:

"I am tring to write a quick interactive command that will rename all the jpgs in the current directory and came up with the following command:

$i=1; gci | ? { ($_.Extension -eq ".JPG") -and !($_.Name.StartsWith("cover")) } | % { Rename-Item -Path ".\$_" -NewName [string]::Format("picture_$i.

jpg"); $i++ }

There appears to be a parameter binding issue. But the 2 parameters I have specified (-Path and -NewName) I have named so I assumed that position is not of relevance. What am I missing here?"

Let's see how we can solve this puzzle and come up with an easy solution using the "3 Golden PowerShell rules"!

Think PowerShell! Use Parameters!

Whenever you work with files, Get-Childitem (aka dir or ls) is your friend. And like most Get-* cmdlets, it has a lot of parameters to help you get what you want. Of course you can always use Where-Object (short: ?) to filter the results once you got them. That is only your last resort, though. Before you do that, always make sure the original cmdlet has no way of filtering. And Get-Childitem has. So the first part would be selecting the files you want to rename like this:

Get-Childitem c:\somefolder -Filter cover*.jpg

Another thing to watch out is this: never use Get-Childitem without specifying an absolute or relative path to start with. Else, it will depend on your current path, and that path may change. So the next time you are running your code, it may no longer work because your current path is no longer pointing to the files you want to rename.

Rule #1: Before you resort to Where-Object, make sure the upstream cmdlet has no built-in parameters to do the same! It is easier and much faster!

Think PowerShell! Use Pipeline Binding!

The next thing is finding the right cmdlet for the action you want to take. Rename-Item was the perfect choice. However, it is not necessary to use Foreach-Object (short: %) to iterate through all the files. Instead, Rename-Item is able to accept the files directly via pipeline. Get-Help can tell you that:

Get-Help -Name Rename-Item -Parameter Path

It returns: Accept pipeline input? true (ByValue, ByPropertyName)

So the second part of the solution looks like this:

Get-Childitem c:\somefolder -Filter cover*.jpg | Rename-Item -NewName "somenewname.jpg"

Rule #2: Before you resport to Foreach-Object, make sure the downstream cmdlet cannot receive your results via pipeline!

Think PowerShell! Use Scriptblocks!

Of course, renaming a bunch of files to the exact same name will fail, and thus you may be intrigued to use a foreach-object loop after all so you can compose individual file names. But that is not necessary. Many cmdlets accept scriptblocks which resemble executable code, so rather than submitting a fixed name, submit code that composes the name. The final solution looks like this:

$global:i = 1; dir c:\test1\ -filter cover*.bmp | rename-item -NewName { "picture_$i.jpg"; $global:i++}

This line will rename all jpg files that start with "cover" in folder c:\test1 to pictureX.jpg where X is an incrementing index number.

Note the use of global: (you could also use script:). With this prefix, you tell PowerShell to use the same variable. If you omitted that, incrementing $i inside the scriptblock would not have an effect on the global variable $i, because inside a script block, the variable is local and would be discarded each time after the script block was done.

Rule #3: Before you design complex code to calculate results for parameters, make sure the parameters do not accept scriptblocks in the first place.

Admittedly, this third rule is not that easy to follow because there is no easy way to know whether a parameter can actually receive a script block.

Hope to see you next week,

Tobias

Microsoft MVP PowerShell Germany