Advanced Parallel Processing of Runbook Activities

In a recent blog post Robert Hearn talked about how to enable true parallel processing in Orchestrator by using the Invoke Runbook object with the ‘wait for completion’ checkmark unchecked.  There are two challenges that this scenario presents, how do you know when all of the child workflows have been completed and how to you retrieve data from them?  For instance, what if you wanted to build 50 (or X) virtual machines?  On the one hand you know that you want to run this in parallel, waiting for that many executions to complete serially would take way too long, but you need to be able to wait for them all to be built before you can do any sort of ‘post configuration’ on them.  Furthermore you probably will need to return some sort of data about the virtual machines (their generated name, mac address, assigned IP address etc.) so how do you do this? The normal answer to this problem is to use Invoke Runbook with wait for completion check, the problem with that approach (as pointed out by Robert) is that you will still get serial execution

image

 

image

So how can you accomplish those two goals (returning data to the calling Runbook and waiting for the child Runbooks to complete before moving on) while still doing the processing in parallel (for the efficiency gains)?  One way is to initiate the child Runbooks through the web service; to facilitate this I will use the Orchestrator Web Service IP available on Codeplex. The scenario I will use is the exact same as Robert’s with two modifications, my child policy now ends with a return data object (to pass data back to the caller) and the workflow is initiated by the web service.

image

image

image

imageimage

image

So the whole workflow, initiating 50 child Runbooks in Parallel that create an event log entry, took just under 1 minute to complete.  How does this stack up against Invoke Runbook run times?  When Invoke Runbook is used serially (with wait for completion checked) the Runbook completes in just under 5 minutes.  When Invoke Runbook is used for parallel execution (wait for completion not checked) the final event log is created 10 seconds after execution

image

image

image

So, at first glance, you may be tempted to say that this method roughly 6 times less efficient than the Invoke Runbook method.  What happens though when the child policy takes longer to complete?  To test this I put a ‘delay’ of 60 seconds into my child policy and re-ran the test.  As expected the Invoke Runbook without wait for completion ran the quickest at just 1 minute and 5 seconds.  Given this time you may be tempted to expect the web service method to complete in around 6 minutes (roughly 6 times slower than Invoke Runbook) but in reality it completed in just 1 minute and 45 seconds.  Theoretically the serial Invoke Runbook test would complete is just over 50 minutes (one minute per execution).

image

image

image

So how do you configure the web service objects to work like this?

To use the ‘Start Runbook’ object you need to first setup a connection to the Orchestrator Web Service

SNAGHTMLab664bc

Then you configure the object much like you would the Invoke Runbook Object (Parameter Name / Value pairs are available under optional properties).  One important thing to note, if the Runbook you are going to initiate has any parameters defined in initialize data you must specify a value to pass to it (it can be blank) using a parameter name / value pair.

imageimage

The other important thing to note is that ‘Wait for Exit’ has been set to false.  Just like Invoke Runbook this object can be told to wait for the child Runbook to complete before moving on which would cause them to execute serially, we don’t want this so we set it to false.  We do however want to wait for the child Runbook to stop, we just want to do this action after all of the child instances have been started.  In order to do this we place a ‘Get Job Instance Details’

image

image

Once the job has completed we want to get its output parameters and for that we use the Get Job Instance Parameters Object

image

image

Advertisements
Tagged with: , , ,
Posted in Workflow
2 comments on “Advanced Parallel Processing of Runbook Activities
  1. stephen says:

    hey great post,
    I like the concept and the functionality it provides, I am however running into consistency problems with this IP when invoking runbooks using the webservice. Is there an updated version of this or prehaps another method to use to acheive the same outcome. ie powershell etc.. basically my goal is to execute a child runbook multiple times to configure a service and then return before running the next step.
    Any ideas would be helpful.

    thanks

    steve

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: