Chapter 5. The PowerShell Pipeline

The PowerShell pipeline chains together a number of commands similar to a production assembly. So, one command hands over its result to the next, and at the end, you receive the result.

Topics Covered:

• Using the PowerShell Pipeline
  • Object-oriented Pipeline
  • Text Not Converted Until the End
  • Table 5.1: Typical pipeline cmdlets and functions
  • Streaming: Real-time Processing or Not?
  • "Blocking" Pipeline Commands
• Converting Objects into Text
  • Making Object Properties Visible
  • Formatting Pipeline Results
  • Displaying Particular Properties
  • Using Wildcard Characters
  • Scriptblocks and "Synthetic" Properties
  • Changing Column Headings
  • Optimizing Column Width
  • PropertySets and Views
• Sorting and Grouping Pipeline Results
  • Sort Object and Hash Tables
  • Grouping Information
  • Using Grouping Expressions
  • Using Formatting Cmdlets to Form Groups
• Filtering Pipeline Results
  • Filtering Objects Out of the Pipeline
  • Selecting Object Properties
  • Limiting Number of Objects
  • Processing All Pipeline Results Simultaneously
  • Removing Doubles
• Analyzing and Comparing Results
  • Statistical Calculations
  • Comparing Objects
  • Comparing Before-and-After Conditions
  • Detecting Changes to Objects
  • Comparing File Contents
  • Saving Snapshots for Later Use
• Exporting Pipeline Results
  • Suppressing Results
  • Changing Pipeline Formatting
  • Forcing Text Display
  • Excel: Exporting Objects
  • HTML Outputs
• The Extended Type System (Part One)
  • Rendering Text as Text and Only Text
  • Your Wish Has Priority
  • Known Objects and Formatting
  • Unknown Objects
  • Emergency Mode
  • "The Case of the Vanished Column"
  • ETS Enhancement
  • Planning Enhancement
• Summary

Using the PowerShell Pipeline

Instruction chains are really nothing new. The old console was able to forward (or "pipe") the results of a command to the next with the "pipe" operator "|". One of the more known usages was to pipe data to the tool more, which then would present the data screen page by screen page:

In contrast to the traditional concept of text piping, the PowerShell pipeline takes an object-oriented approach and implements it in real time. Have a look:

It returns an HTML report on the current directory contents sorted by file size. All of this starts with a Dir command, which then passes its result to Sort-Object. The sorted result then gets limited to only the properties you want in the report. ConvertTo-Html converts the objects to HTML which is then written to a file.

Object-oriented Pipeline

What you see here is a true object-oriented pipeline so the results from a command remain rich objects. Only at the end of the pipeline will the results be reduced to text or HTML or whatever you choose for output. Take a look at Sort-Object. It sorts the directory listing by file size. If the pipeline had simply fed plain text into Sort-Object, you would have had to tell Sort-Object just where the file size information was to be found in the raw text. You would also have had to tell Sort-Object to sort this information numerically and not alphabetically. Not so here. All you need to do is tell Sort-Object which object property you want to sort. The object nature tells Sort-Object all it needs to know: where the information you want to sort is found, and whether it is numeric or letters.

You only have to tell Sort-Object which object property to use for sorting because PowerShell sends results as rich .NET objects through the pipeline. Sort-Objectdoes all the rest automatically. Simply replace Length with another object property, such as Name or LastWriteTime, to sort according to these criteria. Unlike text, information in an object is clearly structured: this is a crucial PowerShell pipeline advantage.

Text Not Converted Until the End

The PowerShell pipeline is always used, even when you provide only a single command. PowerShell attaches to your input the cmdlet Out-Default which converts the resulting objects into text at the end of the pipeline.

Even a simple Dir command is appended internally and converted into a pipeline command:

Of course, the real pipeline benefits show only when you start adding more commands. The chaining of several commands allows you to use commands like Lego building blocks to assemble a complete solution from single commands. The following command would output only a directory's text files listing in alphabetical order:

The cmdlets in Table 5.1 have been specially developed for the pipeline and the tasks frequently performed in it. They will all be demonstrated in the following pages of this chapter.

Streaming: Real-time Processing or Not?

When you combine several commands in a pipeline, you'll want to ask when each separate command will actually be processed: consecutively or at the same time? The pipeline processes the results in real time, at least when the commands chained together in the pipeline support real-time processing. That's why there are two pipeline modes:

• Sequential (slow) mode: In sequential mode, pipeline commands are executed one at a time. So the command's results are passed on to the next one only after the command has completely performed its task. This mode is slow and hogs memory because results are returned only after all commands finish their work and the pipeline has to store the entire results of each command. The sequential mode basically corresponds to the variable concept that first saves the result of a command to a variable before forwarding it to the next command.
• Streaming Mode (quick): The streaming mode immediately processes each command result. Every single result is directly passed onto the subsequent command. It rushes through the entire pipeline and is immediately output. This quick mode saves memory because results are output while the pipeline commands are still performing their tasks. The pipeline doesn't have to store all of the command's results, but only one single result at a time.

"Blocking" Pipeline Commands

You can sort pipeline results through a blocking operation because sorting can only take place when all results are available. This also means there can be long processing times and it can even cause instability if you don't pay attention to memory requirements:

Whether a command supports streaming is up to the programmer. For Sort-Object, there are technical reasons why this command must first wait for all results. Otherwise, it wouldn't be able to sort the results. If you use commands that are not designed for PowerShell then clearly their original programmers could not have taken into account the special demands of PowerShell. For example, if you use the traditional command more.com to output information one page at a time, it will work but more.com is also a blocking command that could interrupt pipeline streaming:

But also genuine PowerShell cmdlets, functions, or scripts can block pipelines if the programmer doesn't use streaming. Surprisingly, PowerShell developers forgot to add streaming support to the integrated more function. This is why more essentially doesn't behave much differently than the ancient more.com command:

The cmdlet Out-Host means you don't have to wait. Its parameter -paging also supports page-by-page outputs. Because this cmdlet supports streaming, you won't have to sit in front of the console twiddling your thumbs:

Converting Objects into Text

At the end of a day, you want commands to return visible results, not objects. So, while results stay rich data objects while traveling the pipeline, at the end of the pipeline, they must be converted into text. This is done by (internally) adding Out-Default to your input. The following commands are identical:

Out-Default transforms the pipeline result into visible text. To do so, it first calls Format-Table (or Format-List when there are more than five properties to output) internally, followed by Out-Host. Out-Host outputs the text in the console. So, this is what happens internally:

Making Object Properties Visible

To really see all the object properties and not just the ones PowerShell "thinks" are important, use Format-Table and add a "*" to select all object properties.

You now get so much information that columns shrink to an unreadable format.

Still, the horizontal table design is unsuitable for more than just a handful of properties. This is why PowerShell uses Format-List instead of Format-Table whenever there are more than five properties to display, and you should do the same:

You will now see a list of several lines for each object's property. For a folder, it might look like this:

A file has slightly different properties:

The property names are located on the left and their content on the right. You now know how to find out which properties an object contains.

Formatting Pipeline Results

Transforming objects produced by the pipeline is carried out by formatting cmdlets. There are four choices:

These formatting cmdlets are not just useful for converting all of an object's properties into text but you can also select the properties you want to see.

Displaying Particular Properties

To do so, you should type the property that you want to see and not just an asterisk behind the cmdlet. The next instruction gets you a directory listing with only Name and Length. Because subdirectories don't have a property called Length, the Length column for the subdirectory is empty:

Using Wildcard Characters

Wildcard characters are allowed so the next command outputs all running processes that begin with "l". All properties starting with "pe" and ending in "64" are output:

If you want to use even more complex wildcards, regular expressions are permitted (more information coming in Chapter 13). For example, WMI objects that are returned by Get-WmiObject contain a number of properties that PowerShell returns and that all begin with the "__" character. To exclude these properties, you should use a wildcard like this one:

Scriptblocks and "Synthetic" Properties

Scriptblocks can be used as columns as they basically act as PowerShell instructions included in braces that work like synthetic properties to calculate their value. Within a scriptblock, the variable $_ contains the actual object. The scriptblock could convert the Length property into kilobytes if you'd like to output file sizes in kilobytes rather than bytes:

Or perhaps you'd like your directory listing to denote how many days have passed since a file or a subdirectory was last modified. While the file object doesn't furnish such information, you could calculate this by means of available properties and provide it its own new property. In the LastWriteTime property, the date of the last modification is indicated. By using the New-TimeSpan cmdlet, you can calculate how much time has elapsed up to the current date. To see how this works, look at the line below as an example that calculates the time difference between January 1, 2000, and the current date:

Use this scriptblock to output how many days have elapsed from the LastWriteTime property up to the current date and to read it out in its own column:

Dir would then return a subdirectory listing that shows how old the file is in days:

Changing Column Headings

As you use synthetic properties, you'll notice that column headings look strange because PowerShell puts code in them that computes the column contents. However, after reading the last chapter, you know that you can use a hash table to format columns more effectively and that you can also rename them:

Optimizing Column Width

Text output conforms to the width of your PowerShell console's display buffer as it tries to accommodate as much data as possible. Because the pipeline processes results in real time, Format-Table cannot know how wide of a space the column elements will occupy. As a result, the cmdlet tends to be generous in sizing columns. If you specify the -auto parameter, Format-Table will collect all results first before setting the maximum width for all elements. You can optimize output, but the results will no longer be output in real time:

PropertySets and Views

If you don't specify any particular properties behind the formatting cmdlet, PowerShell will determine which object properties to convert into text. This automatic feature comes from what is known as the Extended Type System (ETS), which you'll learn more about a bit later. For many commands, PowerShell supplies PropertySets,which are compilations of especially important object properties. They make it unnecessary to specify properties manually, yet still receive basic information.

If you output the result of Get-Process without further specifications, PowerShell will routinely convert the following Process properties objects into text:

You can set quite a different priority by specifying a PropertySet like PSResources after Format-Table:

And PowerShell will select other properties when you use the PropertySet PSConfiguration:

This raises the question of what exactly is a PropertySet and how to find out what they are about. PropertySets are defined for each cmdlet. If you want to see which PropertySets are available for the Get-Process cmdlet, use Get-Member to list all members of the PropertySet type:

The properties that make a PropertySetvisible are listed after the respective PropertySet. As you see, two PropertySets exist for the Get-Process cmdlet. No practical PropertySets are defined for most other cmdlets, but you make up for that. In the section about the ETS toward the end of this chapter, you'll learn how to define a command's properties that are most important for you as a PropertySet.

Views work in a similar way as they set not just the properties that are to be converted into text, but they can also specify column names or widths and even group information.

Views are highly specific and always apply to particular object types and particular formatting cmdlets. The Priority view applies only to Format-Table and only when you display Process objects with it. This view doesn't work for Format-List:

You'll get an error message if you try to use it to format a file listing and not processes,:

Unfortunately, there is no built-in option for finding out which views are available. In the section on the ETS, you'll learn solutions to this problem, and you'll also read about how to define your own views.

Sorting and Grouping Pipeline Results

Your first task is to process and concentrate this information since PowerShell commands often return large amounts of data. Using the cmdlets Sort-Object and Group-Object, you can sort and group other command results. In the simplest scenario, just append Sort-Object to a pipeline command and your output will already be sorted. It's really very simple:

When you do that, Sort-Object selects the property it uses for sorting. It's better to choose the sorting criterion yourself as every object property may be used as a sorting criterion. For example, you could use one to create a descending list of a subdirectory's largest files:

The parameter -property allows you to use any object property as a sorting criterion. In this case, Length is used and Sort-Object does the rest of the work itself. You need only describe where the file size is located (it is clearly available in the Length object property). You do not have to state explicitly that the file size is numeric and so has to be sorted numerically, not alphabetically. Sort-Object can sort by more than one property at the same time. For example, if you'd like to alphabetize all the files in a subdirectory by type first (Extension property) and then by name (Name property), specify both properties:

Sort Object and Hash Tables

Sort-Object not only uses properties for sorting operations. You may also use hash tables as an alternative. Let's assume that you want to sort a subdirectory listing by file size and name, while the file size must be sorted in descending and names in ascending order. How do you accomplish that? In any case, not like this:

You can solve this problem by passing Sort-Object to a hash table (see Chapter 4).

The hash table makes it possible to append additional information to a property, so you can separately specify for each property the sorting sequence you prefer.

Apropos hash tables: can you sort these, too? At first glance, it would seem so:

Yet it does work if you pass the enumerator directly to Sort-Object. This is what you'll get with GetEnumerator():

Grouping Information

Group-Object works by grouping similar objects and then reporting their number. You only need specify the property to Group-Object as your grouping criterion. The next line returns a good status overview of services:

In this case, Group-Object returns an object for every group. The number of groups depends only on how many different values could be found in the property specified in the grouping operation. The Status property always returns either the values "running" or "stopped" for services. This is why Group-Object returned exactly two objects in this example.

The results' object always contains the properties Count, Name, and Group. Services are grouped according to the desired criteria in the Group property. The following shows how you could obtain a list of all currently running services:

It works in a very similar way for other objects. In a file system, Group-Object would put file types in a subdirectory and list their frequency if you use Extension as grouping property:

Of course, you could subsequently also sort the result:

Using Grouping Expressions

Group-Object not only groups by set properties but also can use PowerShell expressions. These must be specified in braces behind Group-Object. The respective object is within the expression as is customary for a $_ variable. The expression can report back on any results. Then Group-Object groups the objects accordingly.

In the following line, the expression returns True if the file size exceeds 100 KB or False as the line returns two groups, True and False. All files larger than 100KB are in the True group:

However, the expression's return value doesn't have to be either True or False, but is arbitrary. In the next line, the expression determines the file name's first letter and returns this in capitals. The result: Group-Object groups the subdirectory contents by first letters:

If you take a closer look at the Group-Object result, you'll notice that after each group name is an array in which single group objects are summarized. So, you could output a practical, alphabetically grouped directory view from this result:

Of course, it will cost a little memory space to store the grouped objects in arrays. Use the parameter -noelement if you don't need the grouped objects.. You could then receive a quick listing of how many processes of which companies are running on your computer. However, because of the -noelement parameter, you will not be able to see any longer which processes these are in detail:

Using Formatting Cmdlets to Form Groups

Group-Object isn't the only option for grouping information. Formatting cmdlets like Format-Table or Format-List can also group information if you use the -groupBy parameter. You can specify the property that you want to use as a grouping criterion after it. For example, if you'd like to group a subdirectory's contents by file type, use the Extension property:

The result appears to be correct at first glance. However, if you look more carefully, you'll find many groups repeated as Format-Table tried not to disrupt the streaming pipeline and processed the files unleashed by Dir running through the pipeline in real time. This leads to a continual accumulation of new groups as files pass through the pipeline that no longer fit into the current group. So, if you want to form groups, you will need to interrupt pipeline streaming and sort the files first, based on the criterion you want to group them afterwards:

Filtering Pipeline Results

Pipeline filters allow only certain objects or object properties through the pipeline. That's practical, because often you will want all results that a command returns. Where-Object permits only those objects to pass through that meet certain criterion. Select-Object also allows only certain object properties to travel through the pipeline. You can use ForEach-Object to process all objects in the pipeline sequentially, enabling you to make your own filters. Finally, Get-Unique removes pipeline duplicates. Let's take a closer look at filters.

Filtering Objects Out of the Pipeline

If you're only interested in certain objects, assign Where-Object the task of closely examining all objects and allowing only those through that meet your criterion, which consists of object properties. For example, if you don't want to view all services returned by Get-Service, but only currently running services, you'll first have to know which service object property reveals whether the service is running or not. You will need more detailed knowledge about the properties supported by an object.

You already know how to ferret out these properties. If you'll recall, Format-List neatly lists all of an object's properties when you use the asterisk character as an argument. You will only need an object example that you can examine with Format-List.

To do so, use the same command that you will want to use later in your pipeline, such as Get-Service and save its result to a variable. As commands return their results in arrays and store each object in it as array elements, you can take the first element you find out of the array and pass it to Format-List:

Now, you can already see all of the object's properties and then its current values. It should be obvious right away that the information sought can be found in the Status property, so you only want to view the objects whose Status property contains the "running" value. You're now ready to use the pipeline filter:

In fact, it works just the way you want it to work so that now you can see only those services that are actually running. How does Where-Object function? The cmdlet expects you to type a PowerShell command in braces and evaluate the command for every pipeline object. The object that Where-Object was just examining can always be found in the variable $_. $_.Status returns the Status property content and needs only be compared to the value that you want to let through.

In reality, the instruction behind Where-Object works like a condition (see Chapter 7): if the expression results in $true, the object will be let through. That's why you may formulate conditions as complex as you like, but you must only make sure that your instruction results in either $true or $false.

The pipeline filter's principle may be applied to all object types and works in the same way everywhere. As an experienced administrator, you may be a little disappointed that the service objects returned by Get-Service contain relatively little information. If you want to list all services that would automatically start, but at the moment aren't running, you can leverage the built-in Windows Management Instrumentation (WMI) infrastructure as an information source to supply more data. You'll harvest much more information when you ask it about services:

The information needed for your criteria are located in the Started and StartMode properties. And because the Where-Object pipeline filter is used very often, there exists a practical abbreviation for it: "?". Here is an example of what your pipeline filter could look like:

If everything works properly, these lines shouldn't report any services at all because services in the "auto" start-up mode are automatically started and, for this reason, should be running. If you're notified of services, you should verify whether these services are (no longer) running despite auto start. One cause could be that the service has completed its task and was then ended as scheduled.

Incidentally, because WMI objects are not on the internal PowerShell list, results are always displayed as lists. For this reason, at the end our above example, we set a table format using Format-Table, which is much clearer.

Selecting Object Properties

The information contained in individual objects may be limited as well. You've just seen that some objects, depending on type, may contain many properties of which you often need only a few. By using Select-Object, you can select those properties that really interest you. All other properties will not be allowed through by Select-Object. For example, the following lines will acquire the user object for the integrated Guest account of your computer:

Most of these properties won't interest you, so Select-Object was able to remove them. In the following, just three of your specified properties were returned:

You could have had the same result if you had used a formatting cmdlet. That would even have been to your advantage since you could use the -autosize parameter to optimize column width:

The significant difference: Format-Table converts properties specified to the object into text. In contrast, Select-Object creates a completely new object containing just these specified properties:

If you type an asterisk as wildcard character after Select-Object, all properties will be marked as relevant. Formatting cmdlets will now output all object properties:

If you'd like to view nearly all of an object's properties, it's easier to display only the properties you don't want by typing the parameter -exclude to specify those properties you want to remove from the object. The next line will output all of a file's properties and directory objects, except for those beginning with "PS" (and show internal PowerShell help properties):

Limiting Number of Objects

Select-Object filters not only object properties but can also, if you prefer, reduce the number of objects allowed to traverse the pipeline. This function is considerably more interesting because it allows you to view, among others, the five largest files of a directory or the five processes that have been running the longest:

A couple of things are interesting here. For example, if you sort a list of processes by StartTime, you'll presumably get several error messages. If you aren't logged on with administrator privileges, you may not retrieve the information from some processes. However, you can avoid this difficulty by setting Sort-Object with parameter -ErrorAction (in short: -ea) to SilentlyContinue. This option is available for nearly every cmdlet and makes sure that error messages won't be displayed.

As a result of such restricted access, not all processes will have any control at all over StartTime. Wherever you can't read the start time because you don't have administrator privileges, a null value will be returned, which messes up the sorting result. You wouldn't get the right results if you wanted to use -first to view the processes that last started running,:

Sort-Object uses the value 0 for empty properties. That's why PowerShell gives you the processes for which it couldn't find any start times. This is interesting since those would be exactly the processes which you have no full access rights. However, this is a problem you can solve, and you already know how: by using the pipeline as just previously described. You should simply filter all of the pipeline's objects out that have an empty StartTime property so that you can better understand what those processes actually are, and then add the Description property in the output. That's where process objects record a brief description of the process:

Processing All Pipeline Results Simultaneously

If you prefer, you may also submit the results separately to the pipeline and then decide on a case-by-case basis what to do with them. The right tool is the ForEach-Object cmdlet that can convert objects into text:

An instruction block in braces follows ForEach-Object so you can execute as many PowerShell commands as you like as long as you separate the commands by ";". This statement block is executed for every single pipeline object: within the block the current object is available in the $_ variable. In the example, ForEach-Object output a text for every service retrieved by Get-Service and inserts into the text the three properties Name, DisplayName, and Status.

Removing Doubles

Get-Unique removes duplicate entries from a sorted list as it presumes that the list was initially sorted according to criterion to make things easier. Get-Unique goes through every element on the list and compares it with the preceding ones. If two are identical, the new object is discarded. So, if you haven't done any sorting, Get-Unique won't work:

Only after you sort the list—in this case, an array—will doubles be removed:

This method is particularly interesting when you break down text files' contents into single words. You can use the following line to do so:

Then, you could sort this list of each word of a file and then either send it to Get-Unique (the list of all words that are in a text) or to Group-Object (the number of words used in a text):

Analyzing and Comparing Results

Using the cmdlets Measure-Object and Compare-Object, you can measure and evaluate PowerShell command results. For example, Measure-Object allows you to determine how often particular object properties are distributed. Compare-Object enables you to compare before-and-after snapshots.

Statistical Calculations

Using the Measure-Object cmdlet, you can carry out statistical calculations so you can work out minimal, maximal, and average values for a particular object property. For example, if you want to know how files sizes are distributed in a directory, let Dir give you a directory listing and then examine the Length property:

Measure-Object counts by default only the specified property's frequency. You should now know that there are 50 objects that have the Length property. Use the relevant parameters if you'd also like to receive the other statistical statements:

Measure-Object can also search through other text files and ascertain the frequency of characters, words, and lines in them:

Comparing Objects

You may often want to compare "before-and-after" conditions to find out which processes have restarted since a certain point in time, or which services have changed in comparison to a particular initial state. The Compare-Object cmdlet can perform this task by making use of the fact that PowerShell commands do not retrieve text internally, but real objects.

Comparing Before-and-After Conditions

For example, you should take a snapshot first if you want to find out whether new processes have started up, or running processes, have terminated in a certain period of time::

All processes will now be stored in the variable $before. To be exact, $before is an array in which every process is represented by a process object. You can now compare the current state at any time you like with this snapshot. Just pass the snapshot list and the list of currently running processes to Compare-Object,which will subsequently establish the differences between the two lists:

The SideIndicator column (line?) reports whether a new process has started running ("=>") or has been ended in the meantime ("<="). Consequently, Compare-Object returns only those processes that are different. Use -includeEqual as an additional parameter, if you want to see the processes that have not been changed. Use the additional parameter -excludeDifferent, if you'd like to see only those processes that have not been modified.

Detecting Changes to Objects

If you use Compare-Object as described above, it will only check whether every object in one list is matched in another list. While comparing them to their initial state, may be sufficient to determine whether objects were removed or added, you can't use this approach to establish whether an object's inner status has changed.

For example, if you'd like to verify whether services have stopped or started in comparison to their defined initial state, Compare-Object won't initially help you because when a service is stopped it still exists: only its inner status has changed. You should instead instruct Compare-Object to compare one or more of the object's properties by using Format-List to easily determine which properties are available to you. You should. first acquire a service object and experiment around with it a little:

It quickly turns out that the Status property retrieves the desired information. So, you could first make another snapshot of all services, stop a service subsequently, and then instruct Compare-Object to use the Status property to ascertain differences:

If you instruct Compare-Object with the parameter -property to compare the Status and Name properties, you'll receive the information you want: the service wuauserv was executed in the list in $before, but not in the list in $after. So it was stopped.

Since the Compare-Object results consist of objects, you could make a further analysis of the result. Perhaps all that interests you are executed modifications. Use a pipeline filter, such as Where-Object, to specify to the filter that you're interested in only those objects in which the SideIndicator property corresponds to the value "=":

If you'd like to formulate the result in plain text, use a loop, such as, Where-Object, and use the information in the retrieved objects to put together the plain text:

You can use this same procedure for widely varying monitoring tasks. Think in advance about which command you could use to determine an object's status to be monitored and, which of the object's properties will describe its status. For example, if you want to find out whether files in a directory have changed, the right command would be Dir and the property could be Length (because of the changed file size) or LastWriteTime (the contents could have been changed even if its size is just as large as it was before). Here's an example:

Comparing File Contents

A special form of the "snapshot" is a file's text contents. If you read text contents using Get-Content, you'll get an array with lines of text. Compare-Object can compare this array again and determine which lines within text files have changed: Here's another example:

Saving Snapshots for Later Use

Some before-and-after comparisons may not be able to be completed in one day. Perhaps you would like to compare operating states over a longer time period, and are not sure if the computer (and your PowerShell) is running the entire time without interruptions. Or maybe you would like to use the same precisely set initial state. In this case, you can "serialize" the objects in the initial state. In other words, the objects are stored as a file in a special data format, more or less "frozen." Later, you can load the object at any time from the file and use them for comparison.

The Export-Clixml cmdlet carries out serialization. All you need to do is to specify a file name under which the objects can be saved. For example, the following line saves a list of all running processes to the file before.xml:

Because the initial state is now stored as a file, you could close PowerShell and reboot your computer. As soon as you are ready to compare the current processes with the stored initial status, you can load the file back in PowerShell:

However, if you try to compare the contents of $before with the current list of processes, Compare-Object will output an endless list of deviations:

In the simplest scenario, Compare-Object only verifies whether the objects are in both lists. But as soon as you serialize or "freeze" objects, your object type changes. If you use Import-Clixml later to input these objects, the information will be brought back to life in a different type while the objects will continue to contain all information. Why? Because the re-input objects no longer correspond to running processes but are the "unfrozen" older processes.

You already know the solution to the problem: simply instruct Compare-Object to compare particular properties because the revived objects continue to contain all the important information. As soon as you compare objects, Compare-Object doesn't care at all about the object type as long as the objects to be compared support the same properties:

You now know that a process called notepad has been added since the snapshot and a process called regedit was ended. However, you wouldn't yet know whether the processes that have the same name are in fact identical. To find out, you would have to include additional object properties in the comparison, such as the process ID, which clearly identifies processes:

Now, you can see as well that PowerShell was started up again once. The instance of PowerShell with the process ID 8004 was ended and in its place a new instance of PowerShell with the process ID 3032 was started.

Exporting Pipeline Results

You have learned that pipeline results are converted into text when they reach the pipeline's end at the latest and are output in the console because PowerShell appends the Out-Default cmdlet to the end of every entry. As a result, this cmdlet decided where pipeline results will be output. Along with Out-Default, there are a number of additional output cmdlets that you can put at your pipeline's end so the result is redirected to a file or printed out rather than output in the console. The pipeline stops its work on reaching the first output cmdlet; if you enter one, Out-Host, which PowerShell appends automatically, won't go into operation:

Suppressing Results

Send the output to Out-Null if you want to suppress command output:

Changing Pipeline Formatting

At first glance, Out-Host seems somewhat superfluous since all results will end up in the console when you don't specify any output cmdlet. So what's the use of Out-Host? On the one hand, this cmdlet supports optional parameters like -paging, which can be used to output information page by page. You already tried that at the beginning of this chapter. In addition, you can use Out-Host to control pipeline formatting, which in itself is much more important.

The reason is that all output cmdlets not only output all pipeline results to the relevant output device, but also automatically convert pipeline objects into readable text. You've already learned how this conversion works by formatting cmdlets like Format-Table. This gets interesting when you specify neither a formatting nor an output cmdlet in your pipeline. Then, PowerShell takes action automatically, though sometimes the result can be confusing.

For example, can you explain why the next instruction outputs all services in table form, but the following in list form?

In the second line, the results of two commands are mixed. That's permitted, and you just have to remember to separate individual commands by a semicolon. None of the two commands outputs its results by using an output cmdlet. That's why all results remain in the pipeline and are automatically processed at the end by Out-Host. That's exactly what causes the problem since PowerShell extends the line in the following way behind the scenes:

Out-Default determines whether you gave one of the formatting cmdlets a particular format. If not, it tries to find an appropriate format. In doing so, it takes a cue from the first object in the result, the path name of Get-Location. However, an unexpectedly colorful series of Get-Service services follows so no predefined format exists with which this muddled medley can be displayed, Out-Default falls back on the list format. You can encounter the problem described here in many places. It also affects, among others, functions or scripts:

The solution to this problem: either specify a format for the pipeline yourself or send the results of individual commands to the console:

Forcing Text Display

PowerShell delays conversion until the last possible moment and converts pipeline objects into text only until they reach the end of the pipeline since information is typically lost when objects are converted into text. However, by using Out-String, you can force PowerShell to convert objects into text any time you like. Out-String is the only output cmdlet that continues the pipeline instead of terminating it. Out-String puts the objects it receives back into the pipeline as text. You can assign the result to a variable Because it behaves like a normal pipeline command.:

The result of Out-String is always a single, complete text. That also means that Out-String blocks the pipeline stream and waits until all results arrive. If you'd prefer getting the text line by line in an array, use the -stream parameter; then Out-String will transform incoming objects into single blocks of text in real time and won't block the pipeline:

Excel: Exporting Objects

All output cmdlets convert pipeline results into text that may be displayed haphazardly. An alternative are comma-separated lists generated by Export-Csv. Comma-Separated Value (CSV) files that can then be opened in programs like Microsoft Excel allows you to continue working smoothly with the data retrieved by PowerShell. You can then turn columns of numbers into expressive graphics.

The objects returned by Dir are converted into text along with all their properties. Open the resulting CSV file and, if you have installed Microsoft Excel, the information will be displayed column-by-column as an Excel spreadsheet. You could also display the information in a text editor if you don't have Excel.

Export-Csv consequently takes care of the formatting data job by writing all object properties as arrays in comma-separated files. What happens when you mess things up by using a formatting cmdlet is shown by the next example:

The information in the CSV file is now nearly unreadable, and it becomes clear how formatting cmdlets do their work behind the scenes by embedding objects in their own formatting instructions. That's why you may never use formatting cmdlets if you want to use Export-Csv to store raw information in a file. In general, you should also use formatting cmdlets only at the end of your pipeline so that formatting instructions will not disrupt other commands.

A question remains: if you use formatting cmdlets to specify which of an object's properties you're interested in, how then can you determine which properties are written into the CSV file? The answer is to strip away the unwanted properties from the objects by using Select-Object. You can then state the property that you want to keep. All the others will be removed from the object. That's the solution, for Export-Csv always writes all (remaining) properties into the CSV file:

HTML Outputs

If you'd like, PowerShell can also pack its results into (rudimentary) HTML files. Converting objects into HTML formats is done by ConvertTo-Html:

But don't be alarmed if the procedure takes a while because PowerShell has to read out all of the objects' properties and save them as a HTML table. If you want to see only particular properties as a HTML report, as in the case of Export-Csv, you should never use formatting cmdlets. It would be better for you to use Select-Object here. You could also take this opportunity to give the HTML page a title by using the -title parameter. The title will turn up later on the title bar of the browser that is displaying your file. Unfortunately, the cmdlet doesn't have formatting options that go beyond this:

The Extended Type System (Part One)

One of the PowerShell console's most remarkable capabilities is converting any object into text. You have seen how different formatting cmdlets can turn object properties into text and output them as text either beside or below each other.

What is striking in this connection is above all that PowerShell succeeds in only converting an object's essential properties into text. PowerShell would have to fail right from the beginning if it had to convert absolutely all of an object's properties into text, for then even a simple directory listing would generate a confusing amount of information:

You don't have to make do with this raw, completely unserviceable text conversion of object properties. You can convert text in a way that makes sense in a practical way by using the Extended Type System (ETS),. Only the ETS can enable PowerShell to process internal objects, waiting until they reach the end of the pipeline before transforming them into understandable text.

The ETS consists of two parts. One part takes care of formatting objects and will be described next. The other part attends to object properties and will be explained in the next chapter.

Rendering Text as Text and Only Text

The ETS goes into action only when objects are output in the console. The ETS does nothing if the data is already available as text. So, if you wanted to use Out-String to convert a directory listing into text right from the beginning and then pass it through one of the formatting cmdlets, it would not be rendered any differently:

Your Wish Has Priority

The ETS will still remain inactive if you specify after a formatting cmdlet like Format-Table which properties should be converted into text., The conversion of objects into text is not the problem, but the selection and differentiation of important and unimportant properties is the issue. If you specify which properties should be converted, you won't let the ETS make this decision:

Known Objects and Formatting

If you use a formatting cmdlet like Format-Table without selecting properties after it, the ETS will go into action for the first time, because the way in which these objects are to be displayed and which properties are to be shown now must be selected automatically. To do this, the ETS first determines what kinds of objects are to be converted into text:

Dir returns files in a System.IO.FileInfo object and files in a System.IO.DirectoryInfo object. Then, the ETS looks in its own internal records to see how these objects must be converted into text. The records are stored in the form of XML files that have the file extension ".ps1xml":

Every object is precisely defined in these XML files. Among others, the definition includes which object properties are supposed to be converted into text and whether the object should be displayed in the form of a list or table.

Unknown Objects

If the object that the ETS is supposed to convert into text is unknown because it isn't defined in one of the ps1xml records, the ETS will flatly convert all properties of the object into text. Then, the question becomes whether the object is to be displayed as a table or a list. If there are fewer than five, the ETS uses a table view, otherwise a list view. You can verify that easily enough yourself by fabricating your own "homemade" objects:

Emergency Mode

If during output the ETS discovers a critical condition, it will automatically switch over to list view. Such a critical condition can arise, for example, when the ETS encounters unexpected objects. The following instruction will initially output the list of running processes in table view, but because file system objects turn up suddenly and unexpectedly, during the output the ETS switches over to emergency mode and lines up the remaining objects in list view.

"The Case of the Vanished Column"

When encountering unknown objects, the ETS always takes its cue from the first object that it outputs. That can cause a strange phenomenon. The ETS always shows all object properties for an unknown object, but only all object properties of the first object that the ETS outputs. If further objects follow with more properties, the present selection of properties remains and information is suppressed.

The following example shows how information can be withheld: Get-Process returns a list of running processes. They are sorted by the property StartTime and subsequently the only properties that are output are Name and StartTime:

When you execute these lines, you may possibly get a lot of error messages, but that's not your fault. Without administrator privileges, you aren't allowed to access many processes: you can't even ask what the start-up time was. As a result, you'll get a list of processes of which only a few are listed with their start times. Only the process names are output. The start times of all processes is simply suppressed. Why?

Whenever you use Select-Object to take a property away from an object, you change the object type. Get-Process retrieves Process objects, and you cannot simply cancel the properties of these objects. That's why Select-Object wraps the information of the incoming Process objects in new objects, which it creates new:

The new objects are of the PSCustomObject type. There is no entry in the ETS record for this object type, and so the ETS outputs all the properties of the first object. Because you had used Sort-Object to sort the output by ascending start times, the list begins with the objects that have no start time because of access restrictions.

As a result, the ETS recognizes only one property, Name, in the first object. It doesn't find the start time in the first object and so start times are not output for the following objects. You can solve this problem by not relying on the ETS, but instead selecting the object you want:

ETS Enhancement

If the ETS is familiar with a certain object type, it can convert it into text optimally. For unknown objects, conversion is far less elegant, possibly even useless. Fortunately, the ETS can be enhanced: all you need to do is to teach ETS how to handle new object types so that they, too, can be displayed as text optimally.

Planning Enhancement

The first step of ETS enhancement is to determine which object type you want to display better. You may frequently use Get-WmiObject to get information from the WMI service, but you're not happy with the way PowerShell displays these objects:

First, find out what type of object is returned by the command:

This shows you that you need an ETS enhancement for objects of the type System.Management.ManagementObject. Next, take a look at this object's properties and select one that you want the ETS to convert into text. For example, DeviceID, Name, and ProcessorID. Then, formulate the definition of the object in XML. In the TableHeaders area, set column headers, and in the TableRowEntries area, set object properties.

Store this XML code in a file called Win32_Processor.format.ps1xml.Thhen, use Update-FormatData to read it into the ETS:

Now, the result will be much easier to understand when you output Win32_Processorobjects again:

However, in this particular instance a mishap occurred. When you acquire other WMI objects, these will now also be displayed in the format that you just defined:

The reason has to do with special features of the WMI. It returns all WMI objects in a System.Management.ManagementObject type.

So, the ETS didn't make a mistake. Instead, the culprit is the WMI as for WMI objects (and only for these), ETS enhancements must be more specific since the type name alone is not enough. That's why WMI objects are assigned to additional object types that you can find in the PSTypeNames property:

The object name that is specific to Win32_Processor objects is called System.Management.ManagementObject#root\cimv2\Win32_Processor. So, you would have to specify this object name in your ETS enhancement so that the enhancement applies only to Win32_Processor WMI objects:

Modify your enhancement accordingly, and read it again with Update-FormatData. You can safely ignore the resulting error message. After updating, your enhancement will be valid only for Win32_Process WMI objects.

Summary

PowerShell uses a pipeline for all command entries, which feeds the results of the preceding command directly into the subsequent command. The pipeline is active even when you enter only a single command because PowerShell always automatically adds the Out-Default cmdlet at the pipeline's end so that it always results in a two-member instruction chain.

Single command results are passed as objects. The cmdlets shown in Table 5.1 can filter, sort, compare, measure, expand, and restrict pipeline elements. All cmdlets accomplish this on the basis of object properties. In the process, the pipeline distinguishes between sequential and streaming modes. In streaming mode, command results are each collected, and then passed in mass onto the next command. Which mode you use depends solely on the pipeline commands used. Output cmdlets dispose of output. If you specify none, PowerShell automatically uses Out-Host to output the results in the console. However, you could just as well send results to a file or printer.

All output cmdlets convert objects into readable text while formatting cmdlets are responsible for conversion. Normally, formatting cmdlets convert only the most important, but if requested, all objects into text. The Extended Type System (ETS) helps convert objects into text. The ETS uses internal records that specify the best way of converting a particular object type into text. If an object type isn't in an ETS internal record, the ETS resorts to a heuristic method, which is guided by, among other things, how many properties are contained in the unknown object.

In addition to traditional output cmdlets, export cmdlets store objects either as comma-separated lists that can be opened in Excel or serialized in an XML format. Serialized objects can be comfortably converted back into objects at a later time. Because when exporting, in contrast to outputting, only plain object properties, without cosmetic formatting, are stored so that no formatting cmdlets are used.