Designing snaplogic pipelines as a microservice

I like the idea! I’ve only just begun to research triggered tasks, but I think that it would be interesting to see how a pipeline performs under load. If your micro service pipeline performs a small set of focused tasks, and is stateless, that is a plus.

If the pipeline were, for example, to write to a file, it could be the case that multiple calls might overwrite the data in the file. But thinking along the lines of perhaps a Workday onboarding. If the pipeline were to insert a new employee X into some subsystem Y, that sounds like something that could be exposed as a REST service.

This leads me to a couple concerns. I might need input from Platform folks on this. The pipeline, under load, may require more memory to be allocated to your JCC. Also, what are SnapLogic’s internal mechanisms to handle multiple calls coming into the triggered pipeline? Are the requests queued up, or are we able to essentially run multiple instances of the pipeline? That would be cool, like an auto elastic compute feature. But The mechanics behind how this is handled, I would like to hear input from Platform folks.

Disregarding the typically loose use of REST in this discussion, we have found the triggered tasks quite useful, especially when used as HTTP callback handlers (aka WebHooks 😑).

I’ve experimented somewhat with creating a RESTful interface using pipelines and triggered tasks as the underlying implementation and using Amazon’s AWS API Gateway to provide the bookmark-able entry point URLs.

Another benefit of the API Gateway is that you can use multiple implementation technologies where appropriate.

TK

That’s a great topic for discussion. I’d be curious to see who is actively using triggered tasks or if someone is avoiding them for any particular reason.

Depending on your configuration, you might be able to have one primary triggered task that fire off the others. We have a case where I have one snaplogic REST API that takes in parameters to invoke other pipelines ( > 20). So I don’t have 20 different REST URL’s to provide - just one and then let the parameters determine what should be executed next. If you’re having 20/sec, the snaplex you set up should have multiple nodes to handle the traffic. Each snaplex can handle 10 concurrent processes last I heard. So you would need to have multiple nodes for that snaplex to distribute the work.

Consider reading through the Ultra Task/Ultra Pipeline documentation. I believe this feature has been used to build REST end points. One novel thing I saw demoed was using the Ultra Pipeline to return HTML. Then the user pointed his browser at the endpoint and the pipeline’s output was rendered as a web page.

For more information regarding the scalability, maintenance, traceability, take a look at this blog post on the topic by Nidhi Maniyar.