Designing snaplogic pipelines as a microservice

[satish2230]

Community, I Need you suggestions :

Is it a good idea to expose multiple pipelines as triggered tasks i.e exposing them as a rest services ?
These triggered tasks would be called by other application as typical REST end points.
Basis idea is to have collections of pipelines, (under a project) to act as a microservice.

What do community members think about such implementation. Good or bad ? Provided that these REST services may be called multiple times a second (20 times a second).

Want your views from scalability ,maintenance,traceability perspective.

[dmiller]

That’s a great topic for discussion. I’d be curious to see who is actively using triggered tasks or if someone is avoiding them for any particular reason.

[chenry]

I like the idea! I’ve only just begun to research triggered tasks, but I think that it would be interesting to see how a pipeline performs under load. If your micro service pipeline performs a small set of focused tasks, and is stateless, that is a plus.

If the pipeline were, for example, to write to a file, it could be the case that multiple calls might overwrite the data in the file. But thinking along the lines of perhaps a Workday onboarding. If the pipeline were to insert a new employee X into some subsystem Y, that sounds like something that could be exposed as a REST service.

This leads me to a couple concerns. I might need input from Platform folks on this. The pipeline, under load, may require more memory to be allocated to your JCC. Also, what are SnapLogic’s internal mechanisms to handle multiple calls coming into the triggered pipeline? Are the requests queued up, or are we able to essentially run multiple instances of the pipeline? That would be cool, like an auto elastic compute feature. But The mechanics behind how this is handled, I would like to hear input from Platform folks.

[slaytanic70]

Depending on your configuration, you might be able to have one primary triggered task that fire off the others. We have a case where I have one snaplogic REST API that takes in parameters to invoke other pipelines ( > 20). So I don’t have 20 different REST URL’s to provide - just one and then let the parameters determine what should be executed next. If you’re having 20/sec, the snaplex you set up should have multiple nodes to handle the traffic. Each snaplex can handle 10 concurrent processes last I heard. So you would need to have multiple nodes for that snaplex to distribute the work.

[chenry]

Very interesting! Also, if you have a link to the documentation specifying that a Snaplex can handle 10 concurrent processes, could you share that data? I would like to learn more about that.