Simultaneous execution of multiple script blocks using Start-Job (instead of looping)
Hello to all!
I was looking for a way to make my script more efficient and I have come to the conclusion (with the help of nice people here on StackOverflow) that Start-Job is the way to go.
I have the following foreach loop that I would like to run simultaneously on all servers in $ servers. I am having trouble figuring out how I am actually collecting the information received from the Receive-Job and adding the $ serverlist to the list.
PS: I know I am a long way from getting this nailed down, but I would really appreciate some help getting started as I am very puzzled about how Start-Job and Receive-Job work.
# List 4 servers (for testing)
$servers = Get-QADComputer -sizelimit 4 -WarningAction SilentlyContinue -OSName *server*,*hyper*
# Create list
$serverlistlist = @()
# Loop servers
foreach($server in $servers) {
# Fetch IP
$ipaddress = [System.Net.Dns]::GetHostAddresses($Server.name)| select-object IPAddressToString -expandproperty IPAddressToString
# Gather OSName through WMI
$OSName = (Get-WmiObject Win32_OperatingSystem -ComputerName $server.name ).caption
# Ping the server
if (Test-Connection -ComputerName $server.name -count 1 -Quiet ) {
$reachable = "Yes"
}
# Save info about server
$serverInfo = New-Object -TypeName PSObject -Property @{
SystemName = ($server.name).ToLower()
IPAddress = $IPAddress
OSName = $OSName
}
$serverlist += $serverinfo | Select-Object SystemName,IPAddress,OSName
}
Notes
- I was outputting $ serverlist to a csv file at the end of the script
- I am listing aprox 500 servers in my complete script
source to share
Since your loop only needs to work on a string, it's easy to turn it into a parallel script.
Below is an example of creating background jobs in your loop to speed up processing.
The code will loop through the array and deploy background jobs to run the code in the script block $sb
. The variable $maxJobs
controls how many jobs are started at once, and the variable $chunkSize
controls how many servers will process each background job.
Add the rest of your processing to the script block, adding any other properties you want to return to the PsObject.
$sb = {
$serverInfos = @()
$args | % {
$IPAddress = [Net.Dns]::GetHostAddresses($_) | select -expand IPAddressToString
# More processing here...
$serverInfos += New-Object -TypeName PsObject -Property @{ IPAddress = $IPAddress }
}
return $serverInfos
}
[string[]] $servers = Get-QADComputer -sizelimit 500 -WarningAction SilentlyContinue -OSName *server*,*hyper* | Select -Expand Name
$maxJobs = 10 # Max concurrent running jobs.
$chunkSize = 5 # Number of servers to process in a job.
$jobs = @()
# Process server list.
for ($i = 0 ; $i -le $servers.Count ; $i+=($chunkSize)) {
if ($servers.Count - $i -le $chunkSize)
{ $c = $servers.Count - $i } else { $c = $chunkSize }
$c-- # Array is 0 indexed.
# Spin up job.
$jobs += Start-Job -ScriptBlock $sb -ArgumentList ( $servers[($i)..($i+$c)] )
$running = @($jobs | ? {$_.State -eq 'Running'})
# Throttle jobs.
while ($running.Count -ge $maxJobs) {
$finished = Wait-Job -Job $jobs -Any
$running = @($jobs | ? {$_.State -eq 'Running'})
}
}
# Wait for remaining.
Wait-Job -Job $jobs > $null
$jobs | Receive-Job | Select IPAddress
Here's a version that handles one server for each job:
$servers = Get-QADComputer -WarningAction SilentlyContinue -OSName *server*,*hyper*
# Create list
$serverlist = @()
$sb = {
param ([string] $ServerName)
try {
# Fetch IP
$ipaddress = [System.Net.Dns]::GetHostAddresses($ServerName)| select-object IPAddressToString -expandproperty IPAddressToString
# Gather OSName through WMI
$OSName = (Get-WmiObject Win32_OperatingSystem -ComputerName $ServerName ).caption
# Ping the server
if (Test-Connection -ComputerName $ServerName -count 1 -Quiet ) {
$reachable = "Yes"
}
# Save info about server
$serverInfo = New-Object -TypeName PSObject -Property @{
SystemName = ($ServerName).ToLower()
IPAddress = $IPAddress
OSName = $OSName
}
return $serverInfo
} catch {
throw 'Failed to process server named {0}. The error was "{1}".' -f $ServerName, $_
}
}
# Loop servers
$max = 5
$jobs = @()
foreach($server in $servers) {
$jobs += Start-Job -ScriptBlock $sb -ArgumentList $server.Name
$running = @($jobs | ? {$_.State -eq 'Running'})
# Throttle jobs.
while ($running.Count -ge $max) {
$finished = Wait-Job -Job $jobs -Any
$running = @($jobs | ? {$_.State -eq 'Running'})
}
}
# Wait for remaining.
Wait-Job -Job $jobs > $null
# Check for failed jobs.
$failed = @($jobs | ? {$_.State -eq 'Failed'})
if ($failed.Count -gt 0) {
$failed | % {
$_.ChildJobs[0].JobStateInfo.Reason.Message
}
}
# Collect job data.
$jobs | % {
$serverlist += $_ | Receive-Job | Select-Object SystemName,IPAddress,OSName
}
source to share
Something you need to understand about Start-Job is that it starts a new Powershell instance running as a separate process. Receive-job gives you a mechanism to output the results of that session to your local session so that your main script can work with it. As attractive as it might sound, running all these concurrent tasks would mean running 500 instances of Powershell on your machine, all of them running at once. This will likely lead to some unintended consequences.
Here's one way to approach dividing work if it helps:
Splits an array of computer names into $ n arrays and starts a new job using each array as an argument list in the script block:
$computers = gc c:\somedir\complist.txt
$n = 6
$complists = @{}
$count = 0
$computers |% {$complists[$count % $n] += @($_);$count++}
0..($n-1) |% {
start-job -scriptblock {gwmi win32_operatingsystem -computername $args} - argumentlist $complists[$_]
}
source to share