Designing snaplogic pipelines as a microservice

Community, I Need you suggestions :

Is it a good idea to expose multiple pipelines as triggered tasks i.e exposing them as a rest services ?
These triggered tasks would be called by other application as typical REST end points.
Basis idea is to have collections of pipelines, (under a project) to act as a microservice.

What do community members think about such implementation. Good or bad ? Provided that these REST services may be called multiple times a second (20 times a second).

Want your views from scalability ,maintenance,traceability perspective.

That’s a great topic for discussion. I’d be curious to see who is actively using triggered tasks or if someone is avoiding them for any particular reason.

I like the idea! I’ve only just begun to research triggered tasks, but I think that it would be interesting to see how a pipeline performs under load. If your micro service pipeline performs a small set of focused tasks, and is stateless, that is a plus.

If the pipeline were, for example, to write to a file, it could be the case that multiple calls might overwrite the data in the file. But thinking along the lines of perhaps a Workday onboarding. If the pipeline were to insert a new employee X into some subsystem Y, that sounds like something that could be exposed as a REST service.

This leads me to a couple concerns. I might need input from Platform folks on this. The pipeline, under load, may require more memory to be allocated to your JCC. Also, what are SnapLogic’s internal mechanisms to handle multiple calls coming into the triggered pipeline? Are the requests queued up, or are we able to essentially run multiple instances of the pipeline? That would be cool, like an auto elastic compute feature. But The mechanics behind how this is handled, I would like to hear input from Platform folks.

1 Like

Depending on your configuration, you might be able to have one primary triggered task that fire off the others. We have a case where I have one snaplogic REST API that takes in parameters to invoke other pipelines ( > 20). So I don’t have 20 different REST URL’s to provide - just one and then let the parameters determine what should be executed next. If you’re having 20/sec, the snaplex you set up should have multiple nodes to handle the traffic. Each snaplex can handle 10 concurrent processes last I heard. So you would need to have multiple nodes for that snaplex to distribute the work.

Very interesting! Also, if you have a link to the documentation specifying that a Snaplex can handle 10 concurrent processes, could you share that data? I would like to learn more about that.

@chenry, we were told that as best practice from the PS team I believe. I don’t have any written documentation on that part.

Hi Everyone,

This is an interesting topic and it would be great if we can get some kind of documentation on the limitation of using trigger tasks.

In my project we are also trying to do something similar. Our use case is we are getting data from Kinesis stream and since currently Snaplogic doesn’t have any snap for it, we are thinking to use Snaplogic trigger task URL as a post request from lambda function. So, whenever our stream gets data the lambda will execute and it will send that data to Snaplogic pipeline using the post request. We did a test with sample stream and it worked fine. However, we are not sure how it is going to perform with actual stream data with lots of data coming every minute. It would be great if get any kind of documentation on trigger task limitation so that we can take care of it in our implementation.

Thanks

Consider reading through the Ultra Task/Ultra Pipeline documentation. I believe this feature has been used to build REST end points. One novel thing I saw demoed was using the Ultra Pipeline to return HTML. Then the user pointed his browser at the endpoint and the pipeline’s output was rendered as a web page.

For more information regarding the scalability, maintenance, traceability, take a look at this blog post on the topic by Nidhi Maniyar.

Disregarding the typically loose use of REST in this discussion, we have found the triggered tasks quite useful, especially when used as HTTP callback handlers (aka WebHooks :expressionless:).

I’ve experimented somewhat with creating a RESTful interface using pipelines and triggered tasks as the underlying implementation and using Amazon’s AWS API Gateway to provide the bookmark-able entry point URLs.

Another benefit of the API Gateway is that you can use multiple implementation technologies where appropriate.

TK

1 Like