PowerShell script to check monitor Status

In our environment we have a Runbook folder structure that says if you have a monitor Runbook and you want it to be running in our Production environment it must be placed in a root folder under the \Production folder. Once it is there we use a PowerShell script (initiated every 5 minutes) that ensures all of the monitors are turned on.

The script we run leverages the PowerShell commandlets from http://scorch.codeplex.com/releases/view/107858. There are actually two scripts we have that are interesting, one is for starting all monitors one level down from a folder and one is for stopping them (we target the ‘stop’ one at our standby / failover web service). The scripts are below, I will post runbooks with them integrated to the codeplex project soon. Enjoy!

Start Script

Param(
  [string]$activeEnvironmentName,
  [string]$activeEnvironmentPort,
  [string]$baseFolderPath,
  [switch]$waitForComplete
)
$modulePath = "C:\Program Files\SCOrchDev\Modules"
if(-not($Env:PSModulePath -like "*$modulePath*"))
{
	$Env:PSModulePath += ";$modulePath"
}
Import-Module scorch
$MaxConcurrency = 10
$checkRunbook = { 
Param($rb, $activeWS, $activeEnvironmentName)
	try
	{
		$modulePath = "C:\Program Files\SCOrchDev\Modules"
		if(-not($Env:PSModulePath -like "*$modulePath*"))
		{
			$Env:PSModulePath += ";$modulePath"
		}
		Import-Module scorch
		Write-Host "------------------------------------------------------------------------------------------------------------------------"
		Write-Host "Starting work on"  
		$rb.Path
		$rb = Get-SCORunbook $activeWS -RunbookGUID $rb.Id
		Write-Host ""
		
		#check for monitor running
		$newJob = $rb | Get-SCOJob $activeWS -jobStatus "Running"
		
		#job is already running
		if(($newJob | Measure-Object).count -gt 0)
		{
			Write-Host "Already running in $activeEnvironmentName"
		}
		else
		{
			Write-Host "Monitor was not running: Creating new Job"
			
			#start the job for the runbook by path
			$newJob = $rb | Start-SCORunbook $activeWS
			$rbRunningJobs = $rb | Get-SCOJob $activeWS -jobStatus "Running"
			
			#wait for the job to start
			while(($rbRunningJobs | Measure-Object).count -gt 0)
			{
				Write-Host -NoNewline .
				$rbRunningJobs = $rb | Get-SCOJob $activeWS -jobStatus "Running"
			}
		}
		Write-Host ""
		Write-Host "------------------------------------------------------------------------------------------------------------------------"
	}
	catch { throw }
}

$activeWS = New-SCOWebserverURL $activeEnvironmentName $activeEnvironmentPort
Write-Host "Active Environment:   $activeWS"

Write-Host ""
Write-Host "Loading $activeEnvironmentName Base Folders"
$activeBaseFolders = Get-SCOSubFolder $activeWS $baseFolderPath $true
$activeRBS = @()
foreach($runbook in $activeBaseFolders)
{
	$activePotentialAdd = $runbook.Runbooks | ? {$_.IsMonitor -eq $true}
	if($activePotentialAdd -ne $null)
	{
		$activeRBS += $runbook.Runbooks | ? {$_.IsMonitor -eq $true}
	}
}
$activePaths = $activeRBS | select Path;
$activePaths | ft Path

Write-Host "------------------------------------------------------------------------------------------------------------------------"
Write-Host "Checking Monitor"
foreach($rb in $activeRBS)
{
	$waited = $false
	if((Get-Job -State Running).Count -ge $MaxConcurrency) 
	{ 
		$waited = $true
		Write-Host "Max Concurrent Jobs Reached: Waiting" 
	}
	while((Get-Job -State Running).Count -ge $MaxConcurrency)
	{
		Write-Host -NoNewLine .
		sleep -Milliseconds 333
	}
	if($waited) { Write-Host "" }
	
	Write-Host "Starting" $rb.Path
	$j = Start-Job -ArgumentList @($rb, $activeWS, $activeEnvironmentName) -ScriptBlock $checkRunbook -Name $rb.path
	while($true)
	{
		$state = ($j | Get-Job).State
		if(($state -eq "Running") -or ($state -eq "Completed") -or ($state -eq "Failed"))
		{
			break
		}
	}
}
if($waitForComplete)
{
	$jArray = Get-Job
	Write-Host ""
	Write-Host "Working"
	while($true)
	{
		Write-Host -NoNewline .
		$finishedCount = ($jArray | Get-Job | ? {$_.State -eq "Completed"}).Count
		$finishedCount += ($jArray | Get-Job | ? {$_.State -eq "Failed"}).Count
		if($finishedCount -eq $jArray.Count) { break }
		sleep -Milliseconds 333
	}
	Write-Host ""
	Write-Host "------------------------------------------------------------------------------------------------------------------------"
	foreach($j in Get-Job -State Completed)
	{
		Receive-Job $j
		Remove-Job $j
	}
	Write-Host ""
	Get-Job
}
else 
{
	Write-Host ""
	Write-Host "------------------------------------------------------------------------------------------------------------------------"
}

Stop Script

Param(
  [string]$EnvironmentName,
  [string]$EnvironmentPort,
  [string]$baseFolderPath,
  [switch]$waitForComplete
)
$modulePath = "C:\Program Files\SCOrchDev\Modules"
if(-not($Env:PSModulePath -like "*$modulePath*"))
{
	$Env:PSModulePath += ";$modulePath"
}
Import-Module scorch
$MaxConcurrency = 10
$checkRunbook = { 
Param($rb, $activeWS, $EnvironmentName)
	try
	{
		$modulePath = "C:\Program Files\SCOrchDev\Modules"
		if(-not($Env:PSModulePath -like "*$modulePath*"))
		{
			$Env:PSModulePath += ";$modulePath"
		}
		Import-Module scorch
		Write-Host "------------------------------------------------------------------------------------------------------------------------"
		Write-Host "Starting work on"  
		$rb.Path
		$rb = Get-SCORunbook $activeWS -RunbookGUID $rb.Id
		Write-Host ""
		
		#check for monitor running
		$monJob = $rb | Get-SCOJob $activeWS -jobStatus "Running" -LoadJobDetails
		
		#Monitor is running
		if(($monJob | Measure-Object).count -gt 0)
		{
			Write-Host "Monitor running in $EnvironmentName"

			#If there are more than 1 jobs running wait for all but the 'monitor' to finish
			$givenMessage = $false
			while($monJob.ActiveInstances -gt 1)
			{
				if($givenMessage)
				{
					Write-Host -NoNewline .
				}
				else
				{
					Write-Host "Waiting for jobs to complete in $EnvironmentName"
					Write-Host "Instance Count: "  $monJob.ActiveInstances
					$givenMessage = $true
				}
				#sleep -Seconds 5
				$monJob = $rb | Get-SCOJob $activeWS -jobStatus "Running" -LoadJobDetails
			}
			if($givenMessage) { Write-Host "" }
			
			Write-Host "Stopping in $EnvironmentName"
			
			#Stop the job			
			$monJob | Stop-SCOJob $activeWS | Out-Null
			$monJob = $rb | Get-SCOJob $activeWS -jobStatus "Running"
			
			#wait for job to stop
			while(($monJob | Measure-Object).count -gt 0)
			{
				#update Job Status
				Write-Host -NoNewline .
				$monJob | Stop-SCOJob $activeWS
				$monJob = $rb | Get-SCOJob $activeWS -jobStatus "Running" -LoadJobDetails
			}
		}
		else
		{
			Write-Host "Monitor not running"
		}
			
		Write-Host ""
		Write-Host "------------------------------------------------------------------------------------------------------------------------"
	}
	catch { throw }
}

$activeWS = New-SCOWebserverURL $EnvironmentName $EnvironmentPort
Write-Host "Active Environment:   $activeWS"

Write-Host ""
Write-Host "Loading $EnvironmentName Base Folders"
$activeBaseFolders = Get-SCOSubFolder $activeWS $baseFolderPath $true
$activeRBS = @()
foreach($runbook in $activeBaseFolders)
{
	$activePotentialAdd = $runbook.Runbooks | ? {$_.IsMonitor -eq $true}
	if($activePotentialAdd -ne $null)
	{
		$activeRBS += $runbook.Runbooks | ? {$_.IsMonitor -eq $true}
	}
}
$activePaths = $activeRBS | select Path;
$activePaths | ft Path

Write-Host "------------------------------------------------------------------------------------------------------------------------"
Write-Host "Checking Monitor"
foreach($rb in $activeRBS)
{
	$waited = $false
	if((Get-Job -State Running).Count -ge $MaxConcurrency) 
	{ 
		$waited = $true
		Write-Host "Max Concurrent Jobs Reached: Waiting" 
	}
	while((Get-Job -State Running).Count -ge $MaxConcurrency)
	{
		Write-Host -NoNewLine .
		sleep -Milliseconds 333
	}
	if($waited) { Write-Host "" }
	
	Write-Host "Starting work on" $rb.Path
	$j = Start-Job -ArgumentList @($rb, $activeWS, $EnvironmentName) -ScriptBlock $checkRunbook -Name $rb.path
	while($true)
	{
		$state = ($j | Get-Job).State
		if(($state -eq "Running") -or ($state -eq "Completed") -or ($state -eq "Failed"))
		{
			break
		}
	}
}
if($waitForComplete)
{
	$jArray = Get-Job
	Write-Host ""
	Write-Host "Working"
	while($true)
	{
		Write-Host -NoNewline .
		$finishedCount = ($jArray | Get-Job | ? {$_.State -eq "Completed"}).Count
		$finishedCount += ($jArray | Get-Job | ? {$_.State -eq "Failed"}).Count
		if($finishedCount -eq $jArray.Count) { break }
		sleep -Milliseconds 333
	}
	Write-Host ""
	Write-Host "------------------------------------------------------------------------------------------------------------------------"
	foreach($j in Get-Job -State Completed)
	{
		Receive-Job $j
		Remove-Job $j
	}
	Write-Host ""
	Get-Job
}
else 
{
	Write-Host ""
	Write-Host "------------------------------------------------------------------------------------------------------------------------"
}
Advertisements
Posted in Uncategorized

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: