In a recent blog post Robert Hearn talked about how to enable true parallel processing in Orchestrator by using the Invoke Runbook object with the ‘wait for completion’ checkmark unchecked. There are two challenges that this scenario presents, how do you know when all of the child workflows have been completed and how to you retrieve data from them? For instance, what if you wanted to build 50 (or X) virtual machines? On the one hand you know that you want to run this in parallel, waiting for that many executions to complete serially would take way too long, but you need to be able to wait for them all to be built before you can do any sort of ‘post configuration’ on them. Furthermore you probably will need to return some sort of data about the virtual machines (their generated name, mac address, assigned IP address etc.) so how do you do this? The normal answer to this problem is to use Invoke Runbook with wait for completion check, the problem with that approach (as pointed out by Robert) is that you will still get serial execution
So how can you accomplish those two goals (returning data to the calling Runbook and waiting for the child Runbooks to complete before moving on) while still doing the processing in parallel (for the efficiency gains)? One way is to initiate the child Runbooks through the web service; to facilitate this I will use the Orchestrator Web Service IP available on Codeplex. The scenario I will use is the exact same as Robert’s with two modifications, my child policy now ends with a return data object (to pass data back to the caller) and the workflow is initiated by the web service.
So the whole workflow, initiating 50 child Runbooks in Parallel that create an event log entry, took just under 1 minute to complete. How does this stack up against Invoke Runbook run times? When Invoke Runbook is used serially (with wait for completion checked) the Runbook completes in just under 5 minutes. When Invoke Runbook is used for parallel execution (wait for completion not checked) the final event log is created 10 seconds after execution
So, at first glance, you may be tempted to say that this method roughly 6 times less efficient than the Invoke Runbook method. What happens though when the child policy takes longer to complete? To test this I put a ‘delay’ of 60 seconds into my child policy and re-ran the test. As expected the Invoke Runbook without wait for completion ran the quickest at just 1 minute and 5 seconds. Given this time you may be tempted to expect the web service method to complete in around 6 minutes (roughly 6 times slower than Invoke Runbook) but in reality it completed in just 1 minute and 45 seconds. Theoretically the serial Invoke Runbook test would complete is just over 50 minutes (one minute per execution).
So how do you configure the web service objects to work like this?
To use the ‘Start Runbook’ object you need to first setup a connection to the Orchestrator Web Service
Then you configure the object much like you would the Invoke Runbook Object (Parameter Name / Value pairs are available under optional properties). One important thing to note, if the Runbook you are going to initiate has any parameters defined in initialize data you must specify a value to pass to it (it can be blank) using a parameter name / value pair.
The other important thing to note is that ‘Wait for Exit’ has been set to false. Just like Invoke Runbook this object can be told to wait for the child Runbook to complete before moving on which would cause them to execute serially, we don’t want this so we set it to false. We do however want to wait for the child Runbook to stop, we just want to do this action after all of the child instances have been started. In order to do this we place a ‘Get Job Instance Details’
Once the job has completed we want to get its output parameters and for that we use the Get Job Instance Parameters Object
