Run N công việc song song trong PowerShell

Question

tôi có các kịch bản PowerShell sau

$list = invoke-sqlcmd 'exec getOneMillionRows' -Server... 
$list | % { 
    GetData $_ > $_.txt 
    ZipTheFile $_.txt $_.txt.zip 
    ... 
} 

Làm thế nào để chạy các khối kịch bản ({ GetDatta $_ > $_.txt ....}) song song với số lượng tối đa hạn chế về công việc, ví dụ tối đa 8 tệp có thể được tạo cùng một lúc?

Answer 1

Lệnh ghép ngắn Start-Job cho phép bạn chạy mã trong nền. Để làm những gì bạn muốn, một cái gì đó giống như mã dưới đây sẽ hoạt động.

foreach ($server in $servers) { 
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' }) 
    if ($running.Count -le 8) { 
     Start-Job { 
      Add-PSSnapin SQL 
      $list = invoke-sqlcmd 'exec getOneMillionRows' -Server... 
      ... 
     } 
    } else { 
     $running | Wait-Job 
    } 
    Get-Job | Receive-Job 
} 

Hy vọng điều này sẽ hữu ích.

Answer 2

Cần thực sự dễ dàng với lệnh ghép ngắn Split-Pipeline của mô-đun SplitPipeline. Mã sẽ trông đơn giản như sau:

Import-Module SplitPipeline 
$list = invoke-sqlcmd 'exec getOneMillionRows' -Server... 
$list | Split-Pipeline -Count 8 {process{ 
    GetData $_ > $_.txt 
    ZipTheFile $_.txt $_.txt.zip 
    ... 
}} 

Answer 3

Công việc nền là câu trả lời. Bạn cũng có thể điều chỉnh công việc trong hàng đợi chạy bằng cách sử dụng [System.Collection.Queue]. Có một bài đăng trên blog từ nhóm PowerShell về chủ đề này: http://blogs.msdn.com/b/powershell/archive/2011/04/04/scaling-and-queuing-powershell-background-jobs.aspx

Sử dụng phương pháp xếp hàng có lẽ là câu trả lời hay nhất để điều chỉnh công việc nền.

Answer 4

Cùng ý tưởng là người dùng "Start-Tự động hoá" được đăng, mà phải sửa chữa các lỗi về việc quên để bắt đầu công việc được tổ chức trở lại khi chạm mệnh đề khác trong ví dụ của mình:

$servers = @('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n') 

foreach ($server in $servers) { 
    $running = @(Get-Job | Where-Object { $_.State -eq 'Running' }) 
    if ($running.Count -ge 4) { 
     $running | Wait-Job -Any | Out-Null 
    } 

    Write-Host "Starting job for $server" 
    Start-Job { 
     # do something with $using:server. Just sleeping for this example. 
     Start-Sleep 5 
     return "result from $using:server" 
    } | Out-Null 
} 

# Wait for all jobs to complete and results ready to be received 
Wait-Job * | Out-Null 

# Process the results 
foreach($job in Get-Job) 
{ 
    $result = Receive-Job $job 
    Write-Host $result 
} 

Remove-Job -State Completed 

Answer 5

Cũ chủ đề nhưng tôi nghĩ điều này có thể giúp:

$List = C:\List.txt 
$Jobs = 8 

Foreach ($PC in Get-Content $List) 
{ 
Do 
    { 
    $Job = (Get-Job -State Running | measure).count 
    } Until ($Job -le $Jobs) 

Start-Job -Name $PC -ScriptBlock { "Your command here $Using:PC" } 
Get-Job -State Completed | Remove-Job 
} 

Wait-Job -State Running 
Get-Job -State Completed | Remove-Job 
Get-Job 

Vòng lặp "Do" tạm dừng "foreach" khi số lượng công việc "chạy" vượt quá số lượng "$ jobs" được phép chạy. Thần chờ cho các công việc còn lại để hoàn thành và trình diễn thất bại ...

Answer 6

tôi sử dụng và improove một chức năng đa luồng, bạn có thể sử dụng nó như:

$Script = { 
    param($Computername) 
    get-process -Computername $Computername 
} 

@('Srv1','Srv2') | Run-Parallel -ScriptBlock $Script 

bao gồm mã này trong kịch bản của bạn

function Run-Parallel { 
    <# 
     .Synopsis 
      This is a quick and open-ended script multi-threader searcher 
      http://www.get-blog.com/?p=189#comment-28834 
      Improove by Alban LOPEZ 2016 

     .Description 
      This script will allow any general, external script to be multithreaded by providing a single 
      argument to that script and opening it in a seperate thread. It works as a filter in the 
      pipeline, or as a standalone script. It will read the argument either from the pipeline 
      or from a filename provided. It will send the results of the child script down the pipeline, 
      so it is best to use a script that returns some sort of object. 

     .PARAMETER ScriptBlock 
      This is where you provide the PowerShell ScriptBlock that you want to multithread. 

     .PARAMETER ItemObj 
      The ItemObj represents the arguments that are provided to the child script. This is an open ended 
      argument and can take a single object from the pipeline, an array, a collection, or a file name. The 
      multithreading script does it's best to find out which you have provided and handle it as such. 
      If you would like to provide a file, then the file is read with one object on each line and will 
      be provided as is to the script you are running as a string. If this is not desired, then use an array. 

     .PARAMETER InputParam 
      This allows you to specify the parameter for which your input objects are to be evaluated. As an example, 
      if you were to provide a computer name to the Get-Process cmdlet as just an argument, it would attempt to 
      find all processes where the name was the provided computername and fail. You need to specify that the 
      parameter that you are providing is the "ComputerName". 

     .PARAMETER AddParam 
      This allows you to specify additional parameters to the running command. For instance, if you are trying 
      to find the status of the "BITS" service on all servers in your list, you will need to specify the "Name" 
      parameter. This command takes a hash pair formatted as follows: 

      @{"key" = "Value"} 
      @{"key1" = "Value"; "key2" = 321; "key3" = 1..9} 

     .PARAMETER AddSwitch 
      This allows you to add additional switches to the command you are running. For instance, you may want 
      to include "RequiredServices" to the "Get-Service" cmdlet. This parameter will take a single string, or 
      an aray of strings as follows: 

      "RequiredServices" 
      @("RequiredServices", "DependentServices") 

     .PARAMETER MaxThreads 
      This is the maximum number of threads to run at any given time. If ressources are too congested try lowering 
      this number. The default value is 20. 

     .PARAMETER SleepTimer_ms 
      This is the time between cycles of the child process detection cycle. The default value is 200ms. If CPU 
      utilization is high then you can consider increasing this delay. If the child script takes a long time to 
      run, then you might increase this value to around 1000 (or 1 second in the detection cycle). 

     .PARAMETER TimeOutGlobal 
      this is the TimeOut in second for listen the last thread, after this timeOut All thread are closed, only each other are returned 

     .PARAMETER TimeOutThread 
      this is the TimeOut in second for each thread, the thread are aborted at this time 

     .PARAMETER PSModules 
      List of PSModule name to include for use in ScriptBlock 

     .PARAMETER PSSapins 
      List of PSSapin name to include for use in ScriptBlock 

     .EXAMPLE 
      1..20 | Run-Parallel -ScriptBlock {param($i) Start-Sleep $i; "> $i sec <"} -TimeOutGlobal 15 -TimeOutThread 5 
     .EXAMPLE 
      Both of these will execute the scriptBlock and provide each of the server names in AllServers.txt 
      while providing the results to GridView. The results will be the output of the child script. 

      gc AllServers.txt | Run-Parallel $ScriptBlock_GetTSUsers -MaxThreads $findOut_AD.ActiveDirectory.Servers.count -PSModules 'PSTerminalServices' | out-gridview 
    #> 
    Param(
     [Parameter(ValueFromPipeline=$true,ValueFromPipelineByPropertyName=$true)] 
      $ItemObj, 
     [ScriptBlock]$ScriptBlock = $null, 
     $InputParam = $Null, 
     [HashTable] $AddParam = @{}, 
     [Array] $AddSwitch = @(), 
     $MaxThreads = 20, 
     $SleepTimer_ms = 100, 
     $TimeOutGlobal = 300, 
     $TimeOutThread = 100, 
     [string[]]$PSSapins = $null, 
     [string[]]$PSModules = $null, 
     $Modedebug = $true 
    ) 
    Begin{ 
     $ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault() 
     ForEach ($Snapin in $PSSapins){ 
      [void]$ISS.ImportPSSnapIn($Snapin, [ref]$null) 
     } 
     ForEach ($Module in $PSModules){ 
      [void]$ISS.ImportPSModule($Module) 
     } 
     $RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host) 
     $RunspacePool.CleanupInterval=1000 
     $RunspacePool.Open() 

     $Jobs = @() 
    } 
    Process{ 
     #ForEach ($Object in $ItemObj){ 
      if ($ItemObj){ 
       Write-Host $ItemObj -ForegroundColor Yellow 
       $PowershellThread = [powershell]::Create().AddScript($ScriptBlock) 

       If ($InputParam -ne $Null){ 
        $PowershellThread.AddParameter($InputParam, $ItemObj.ToString()) | out-null 
       }Else{ 
        $PowershellThread.AddArgument($ItemObj.ToString()) | out-null 
       } 
       ForEach($Key in $AddParam.Keys){ 
        $PowershellThread.AddParameter($Key, $AddParam.$key) | out-null 
       } 
       ForEach($Switch in $AddSwitch){ 
        $PowershellThread.AddParameter($Switch) | out-null 
       } 
       $PowershellThread.RunspacePool = $RunspacePool 
       $Handle = $PowershellThread.BeginInvoke() 
       $Job = [pscustomobject][ordered]@{ 
        Handle = $Handle 
        Thread = $PowershellThread 
        object = $ItemObj.ToString() 
        Started = Get-Date 
       } 
       $Jobs += $Job 
      } 
     #} 
    } 
    End{ 
     $GlobalStartTime = Get-Date 
     $continue = $true 
     While (@($Jobs | Where-Object {$_.Handle -ne $Null}).count -gt 0 -and $continue) { 
      ForEach ($Job in $($Jobs | Where-Object {$_.Handle.IsCompleted -eq $True})){ 
       $out = $Job.Thread.EndInvoke($Job.Handle) 
       $out # return vers la sortie srandard 
       #Write-Host $out -ForegroundColor green 
       $Job.Thread.Dispose() | Out-Null 
       $Job.Thread = $Null 
       $Job.Handle = $Null 
      } 
      foreach ($InProgress in $($Jobs | Where-Object {$_.Handle})) { 
       if ($TimeOutGlobal -and (($(Get-Date) - $GlobalStartTime).totalseconds -gt $TimeOutGlobal)){ 
        $Continue = $false 
        #Write-Host $InProgress -ForegroundColor magenta 
       } 
       if (!$Continue -or ($TimeOutThread -and (($(Get-Date) - $InProgress.Started).totalseconds -gt $TimeOutThread))) { 
        $InProgress.thread.Stop() | Out-Null 
        $InProgress.thread.Dispose() | Out-Null 
        $InProgress.Thread = $Null 
        $InProgress.Handle = $Null 
        #Write-Host $InProgress -ForegroundColor red 
       } 
      } 
      Start-Sleep -Milliseconds $SleepTimer_ms 
     } 
     $RunspacePool.Close() | Out-Null 
     $RunspacePool.Dispose() | Out-Null 
    } 
}