Search results for 'app:weblogs' matching tags 'Jobs', 'Georges Maheu', and 'writing'

Increase Performance by Slowing Down Your PowerShell Script

Anonymous — Fri, 03 Feb 2012 06:00:00 GMT

Summary: Microsoft PFE, Georges Maheu, further optimizes the Windows PowerShell script he presented earlier this week.

Microsoft Scripting Guy, Ed Wilson, is here. Our guest blogger today is Georges Maheu. Georges presented a script three days ago to gather Windows services information in an Excel spreadsheet. Although the script did an adequate job for a small number of servers, it did not scale well for hundreds of servers. On day one, the script took 90 minutes to document 50 computers. On day two, it took less than three minutes. Day three went down to 43 seconds. Today, Georges wants to do even better! Here are quick links to his previous blogs:

Day 1: Beat the Auditors, Be One Step Ahead with PowerShell

Day 2: Speed Up Excel Automation with PowerShell

Day 3: Speed Up Excel Automation with PowerShell Jobs

Note: All of the files from today, in addition to files for the entire week are in a zip file in the Script Repository. You can read more from Georges on the PFE Blog: OpsVault.

Now, once again, here’s Georges...

There comes a time when there is no point to optimizing a script further. Moving from 90 minutes down to less than one minute was well worth the investment in time! But going from 43 seconds to 30 is not worth the effort for this script.

Today, we tackle typical issues that are encountered when collecting large volumes of information in a distributed environment. Excel can handle 1000 computers—that’s cool, but not very practical beyond those numbers.

There are a few options for dealing with large volumes of information: one could store all the information in a database and write nice reports by using queries. Personally, I could not create a decent database to save my life! Therefore, I will go to the next option, which is to use individual files for each computer.

Instead of writing the information in an Excel tab, the information will be exported to a file in .csv format by using the Export-CSV cmdlet.

First, the following lines are replaced:

$data = ($services `

| Select-Object $properties `

| ConvertTo-Csv -Delimiter "`t" -NoTypeInformation ) -join "`r`n"

[Windows.Clipboard]::setText($data) | Out-Null

$computerSheet.range("a$mainHeaderRow").pasteSpecial(-4104) |

Out-Null #Const xlPasteAll = -4104

With these lines:

$services |

Select-Object $properties |

Export-CSV "$script:customerDataPath\$($currentJobName).csv" `

-Encoding ASCII -NoTypeInformation

Then, all the code that is related to Excel is removed from the script.

Most of the optimization techniques that I have shown so far can be reused, but there are a few additional challenges. One of them is, “What should I do if a computer does not respond?” With 50 computers, one can afford to run a 43 second script repeatedly until all the data is gathered. Not so obvious when 5000 computers or more are involved.

One could use the file system to keep track of computers already processed. If a CSV file exists, there is no need to collect the information again. If there is a need to update the information for a specific computer, simply delete that file and run the script again.

if (Test-Path "$customerDataPath\$($jobName).csv")

{

write-host "$jobName data has already been collected"

}

else

{

Start-Job -ScriptBlock `

{

param($computerName);

Get-WmiObject `

-class win32_service `

-ComputerName $computerName

} `

-Name $jobName `

-ArgumentList $computerName

"Creating file for $computerName"

} #if (Test-Path "$customerDataPath\$($jobName).csv")

Yesterday, running out of resources was avoided with a rude implementation of a producer–consumer design pattern. This same issue needs to be revisited because the script design was based on the time it took the consumer to write the data in Excel. Writing to a file is much faster, and running out of resources could occur again.

This happens if there are too many jobs running at the same time. In fact, the more jobs that are running concurrently, the longer each ones takes to complete. This can be a vicious circle.

Strangely enough, the performance of the script can be increased by slowing it down! I slow the script down a bit by using the Start-Sleep cmdlet.

if (@(get-job).count -gt 15)

{

Start-Sleep -Milliseconds 250

write-host "Slow down!" -ForegroundColor yellow

}

if (@(get-job).count -gt 5) {Get-CompletedJobs}

Slowing down the rate at which new jobs are started allows existing jobs to finish and be processed. This will help maintain the job queue size within the limits of the computer’s resources. For example, the following 500 computer test run took 8 minutes and 3 seconds after encountering a resource struggle.

After the lines were added to slow the script down, the same script completed in 3 minutes, 39 seconds.

The following screen capture shows these new lines in action after restarting a 5000 computer test run. The screen capture also shows the optimization mentioned earlier by using files as indicators that a computer has already been processed:

This is not an exact science, you will need to experiment with these numbers based on your computer resources and the time it takes to get and process the information.

This latest version of the script gets these numbers:

01 minute 19 seconds for 100 computers.

01 minutes 56 seconds for 250 computers.

03 minutes 39 seconds for 500 computers.

07 minutes 05 seconds for 1000 computers.

40 minutes 20 seconds for 5000 computers.

Not bad! However, there is one more thing to do to wrap up this project. All the services that run using nondefault service accounts need to be extracted.

Here is a second script that will process all those CSV files, extract that information, and store it in another CSV file.

clear-host

$startTime = Get-Date

$scriptPath = Split-Path -parent $myInvocation.myCommand.definition

$customerDataPath = "$scriptPath\Services Data"

$reportPath = "$scriptPath\Services Report"

$CSVfiles = Get-Childitem $customerDataPath -filter *.csv

if ($CSVfiles.count -ge 1)

{

Write-Host "There are: $($CSVfiles.count) data files"

}

else

{

Write-Host "There are no data files in $customerDataPath." -ForegroundColor red

exit

}

new-item -Path $scriptPath -name "Services Report" -force -type directory | Out-Null #Create report folder

$exceptions = @()

$CSVfiles |

ForEach-Object `

{

$services = Import-csv -Path $_.fullname

"Processing $_"

forEach ($service in $services)

{

# $service.startName

################################################

# EXCEPTION SECTION

# To be customized based on your criteria

################################################

if ( $service.startName -notmatch "LocalService" `

-and $service.startName -notmatch "Local Service" `

-and $service.startName -notmatch "NetworkService" `

-and $service.startName -notmatch "Network Service" `

-and $service.startName -notmatch "LocalSystem" `

-and $service.startName -notmatch "Local System")

{

Write-Host $service.startName -ForegroundColor yellow

$exceptions += $service

} #if ($service.startName

} #foreach ($service in $services)

} #ForEach-Object

$exceptions | Export-CSV "$reportPath\Non Standard Service Accounts Report.csv" `

-Encoding ASCII -NoTypeInformation

$endTime = get-date

"" #blank line

Write-Host "-------------------------------------------------" -ForegroundColor Green

Write-Host "Script started at: $startTime" -ForegroundColor Green

Write-Host "Script completed at: $endTime" -ForegroundColor Green

Write-Host "Script took $($endTime - $startTime)" -ForegroundColor Green

Write-Host "-------------------------------------------------" -ForegroundColor Green

"" #blank line

This script took just under 10 minutes to run. In summary, these two optimized scripts processed 5000 computers in about one hour. Not bad!

These scripts can be adapted with minor modifications to collect just about any kind of data. Today’s script focuses on large environments while yesterday’s script is appropriate for a small to mid-size environment. The next time your manager tells you that the auditors are coming, you can sit back and smile. You will be ready.

~ Georges

Thank you, Georges. This has been a great series of blogs. The zip file that you will find in the Script Repository has all the files and scripts from Georges this week.

I invite you to follow me on Twitter and Facebook. If you have any questions, send email to me at scripter@microsoft.com, or post your questions on the Official Scripting Guys Forum. See you tomorrow. Until then, peace.

Ed Wilson, Microsoft Scripting Guy

Speed Up Excel Automation with PowerShell Jobs

Anonymous — Thu, 02 Feb 2012 06:00:00 GMT

Summary: Microsoft PFE, Georges Maheu, further optimizes the Windows PowerShell script that he presented in his previous two blogs.

Microsoft Scripting Guy, Ed Wilson, is here. Our guest blogger today is Georges Maheu. Georges presented a script two days ago to gather Windows services information in an Excel spreadsheet: Beat the Auditors, Be One Step Ahead with PowerShell

Although that script did an adequate job for a small number of servers, it did not scale well for hundreds of servers. Yesterday, he optimized the script to reduce the runtime from 90 minutes to less than three minutes: Speed Up Excel Automation with PowerShell

Today, Georges wants to do even better.

Note: All of the files from today, in addition to files for the entire week are in a zip file in the Script Repository. You can read more from Georges on the PFE Blog: OpsVault.

Take it away Georges...

Today, we will see if yesterday’s script can be optimized further. As already seen, the script makes an inventory of all services running on your servers, and it identifies services that are using non-standard accounts. The inventory is presented in an Excel spreadsheet.

The first version of the script was linear and simple but somewhat slow. Here is a screen capture of the WTM while the script is running. The script took 90 minutes to perform a 50 computer inventory. CPU on the test computer remained in the 20% utilization range during this test.

Yesterday, the performance of this script was improved by focusing on a few bottlenecks. Runtime has been reduced dramatically—down to two minutes and 31 seconds for the same 50 computer inventory. Not bad!

CPU utilization varied, but it was mostly around 15%.

Now the question is, “How can we improve this further? Is it even possible?”

Using the same technique as yesterday, let’s start with the Measure-Command cmdlet:

Measure-Command {$services = Get-WmiObject win32_service `

-ComputerName $computerName}

This script provided the following information:

WMI query: < 0.25 seconds for responding local computer

WMI query: < 1.00 seconds for responding remote computer

WMI query: < 20.00 seconds for non-responding computer

Ping dead computer: < 2.5 seconds

In the test scenario, I used my desktop computer (no, I do not have 50 computers in my lab). The projection is 47*0.25 + 3*20 = 72 seconds. Knowing the script actually takes 148 seconds, the overhead is about 50%. Knowing that it is impossible to improve the time to make the WMI query, the only option is to run the queries in parallel. The total amount of data will be the same, so the gain will have to come from the latency time it takes to perform a WMI query.

Windows PowerShell 2.0 has functionality to support concurrency.

First method: Get-WMIObject –ComputerName

WMI can be used directly. The next screen capture shows the time it takes WMI to query 50 computers in a single statement:

The performance is great, but the data would need to be parsed based on the systemName field, and the script structure would have to be modified considerably.

Second method: Invoke-Command

invoke-command -ScriptBlock `

{

param($computerName);

Get-WmiObject `

-class win32_service `

-ComputerName $computerName

} `

-asJob `

-JobName "$i - $computerName" `

-ThrottleLimit 3 `

-ArgumentList $computerName `

-ComputerName LocalHost

Invoke-Command looks promising. It has a ThrottleLimit parameter, which limits the number of network connections, not the number of jobs running on a computer, as demonstrated in the following screen capture.

Invoke-Command would be nice if the queries were run on remote computers, but this is not the case. However, spreading the load across multiple computers would certainly be something to consider for the next round of optimization.

Invoke-Command needs the shell to be run as administrator. Permissions could be delegated to run Invoke-Command as a regular user, but this would increase the complexity.

Third method: Start-Job

Start-Job -ScriptBlock `

{

param($computerName);

Get-WmiObject `

-class win32_service `

-ComputerName $computerName

} `

-Name "$i - $computerName" `

-ArgumentList $computerName

Start-Job creates a background process for each WMI query. The result is great. Our script can now do 50 computers in about 42 seconds.

However, the script had to be modified—the first part now starts all the jobs, and the second part processes the results. The main differences are the following lines:

foreach ($computerName in $computers)

{

$workBook.workSheets.add() | Out-Null

$workBook.workSheets.item(1).name = "$computerName"

Start-Job -ScriptBlock `

{

param($computerName);

Get-WmiObject `

-class win32_service `

-ComputerName $computerName

} `

-Name "$computerName" `

-ArgumentList $computerName

} #forEach computer

To generate all the jobs, use the following code to retrieve the information from the job queue.

while (@(Get-Job -State Completed).count -gt 0)

{

"============"

$currentJobName = (Get-Job -State Completed |

Select-Object -First 1).name

$services = Receive-Job -name $currentJobName

Remove-Job -Name $currentJobName

#same code to paste in Excel…

When the jobs are running, they can be in one of several states: Running, Completed, or Failed. There are other states, but these are the states of interest for this script. To process the jobs, the data needs to be retrieved when the job is completed. This is done with the Receive-Job cmdlet. Note that the data returned is serialized—this means that the script is not receiving the real objects as in the previous scripts, but rather a XML representation. In this scenario, this is irrelevant because this data will be converted to an Excel spreadsheet. When the data has been retrieved, the job is removed to free up resources.

WOW, we went from 90 minutes, down to three minutes, down to 42 seconds to document 50 computers. The end result, the Excel report (for the auditors), is the same for all three versions. This is great performance but can it scale up? The test desktop computer can handle 50 computers. Can it do 100, 200, 500, or even 1000 computers?

Well, let’s try!

Here are the results for 100 computers:

The computer list includes some nonexistent computers to test nonresponding computers.

Here are the script results:

97 seconds (01 minute 37 seconds) for 100 computers.

380 seconds (06 minutes 20 seconds) for 250 computers.

1228 seconds (20 minutes 28 seconds) for 500 computers.

3450 seconds (57 minutes 30 seconds) for 1000 computers.

Scaling up requires additional care. For example, the computer may run out of resources and the script performance could seriously degrade. Worst, some data might be lost. Fortunately in this case, resolving this issue has an interesting side effect. It actually increases the performance!

To avoid running out of resources, the script will be tweaked based on a rude implementation of a producer–consumer design pattern. In other words, by limiting the number of concurrent jobs based on the speed the data is generated versus the speed it can be imported into Excel. This improved the script down to these numbers:

94 seconds (01 minute 34 seconds) for 100 computers.

215 seconds (03 minutes 35 seconds) for 250 computers.

450 seconds (07 minutes 30 seconds) for 500 computers.

904 seconds (15 minutes 04 seconds) for 1000 computers.

The last modification to the script was to move the code that retrieves the data from the job, to paste it into Excel to a function (the consumer), and to call this function from the loop that generates the jobs (the producer).

foreach ($computerName in $computers)

{

$computerName = $computerName.trim()

$workBook.workSheets.add() | Out-Null

$workBook.workSheets.item(1).name = "$computerName"

"Creating sheet for $computerName"

Start-Job -ScriptBlock `

{

param($computerName);

Get-WmiObject `

-class win32_service `

-ComputerName $computerName

} `

-Name "$computerName" `

-ArgumentList $computerName

if (@(get-job).count -gt 5) {Get-CompletedJobs}

} #forEach computer

The key line is:

if (@(get-job).count -gt 5) {Get-CompletedJobs}

This line stops the creation of new jobs while removing completed jobs from the job queue. You may want to experiment with the count greater than 5, depending on the time it takes to complete the WMI query versus the time it takes to paste it into Excel. Starting to process the data as it is being generated overlaps the overhead mentioned previously and produces the performance gain.

Optimizing is always a question of balance. How much time do you want to invest to further optimize a script? There comes a point where the gain is not worth the pain.

Having optimized the performance, the next challenge will be to scale this script from one thousand to several thousand computers. As we will see tomorrow, we will need to take a new approach to address this.

BTW, I shaved one additional minute off the 1000 computer data by changing this line:

$excel.visible = $true

To this:

$excel.visible = $false

~ Georges

Thank, Georges. The zip file that you will find in the Script Repository has all the files and scripts from Georges this week. Please join us tomorrow for the conclusion of the series.

Ed Wilson, Microsoft Scripting Guy