Many times when designing a pipeline for a use case one may need to make a call to an external service (REST/SOAP). In SnapLogic REST and SOAP snaps operate serially, meaning they can only process one document at a time. Some service calls may take an extended period of time to come back, so this will essentially block oneโs pipeline execution until a response is received. This may be an acceptable for a pipeline with little volume, but if one is processing thousands of documents or more and each call takes a few seconds to complete, it can rack up quite the runtime.
To get around this, one can distribute the incoming documents over a number of outputs and add more snaps to make service calls.
To evenly distribute incoming documents across multiple outputs, one can use the router snap in โautoroutingโ mode. What this will do is round-robin incoming documents over the number of given outputs. To setup, drag a router snap onto oneโs pipeline, add the desired number of output views, and leave the expressions correlating to those output views blank.
After that is setup, copy oneโs snap making the service call and add one to each of the router outputs.
Union the REST outputs to recombine the streams, and then reconnect the remaining pipeline logic.
Thats it!
If your service calls block even a few seconds and one is making a large number of calls, this should decrease the runtime a bit. The exact amount of parallelism to use will be a factor of the use case and the service endpoint. Iโve found that two calls has worked best most of the time, only on extreme volume was an advantage gained by going to 3 or more.
