Designing snaplogic pipelines as a microservice

[satish2230]

Community, I Need you suggestions :

Is it a good idea to expose multiple pipelines as triggered tasks i.e exposing them as a rest services ?
These triggered tasks would be called by other application as typical REST end points.
Basis idea is to have collections of pipelines, (under a project) to act as a microservice.

What do community members think about such implementation. Good or bad ? Provided that these REST services may be called multiple times a second (20 times a second).

Want your views from scalability ,maintenance,traceability perspective.

[slaytanic70]

@chenry, we were told that as best practice from the PS team I believe. I don’t have any written documentation on that part.

[aditya_sharma]

Hi Everyone,

This is an interesting topic and it would be great if we can get some kind of documentation on the limitation of using trigger tasks.

In my project we are also trying to do something similar. Our use case is we are getting data from Kinesis stream and since currently Snaplogic doesn’t have any snap for it, we are thinking to use Snaplogic trigger task URL as a post request from lambda function. So, whenever our stream gets data the lambda will execute and it will send that data to Snaplogic pipeline using the post request. We did a test with sample stream and it worked fine. However, we are not sure how it is going to perform with actual stream data with lots of data coming every minute. It would be great if get any kind of documentation on trigger task limitation so that we can take care of it in our implementation.

Thanks

[tlikarish]

Consider reading through the Ultra Task/Ultra Pipeline documentation. I believe this feature has been used to build REST end points. One novel thing I saw demoed was using the Ultra Pipeline to return HTML. Then the user pointed his browser at the endpoint and the pipeline’s output was rendered as a web page.

For more information regarding the scalability, maintenance, traceability, take a look at this blog post on the topic by Nidhi Maniyar.

[tk42]

Disregarding the typically loose use of REST in this discussion, we have found the triggered tasks quite useful, especially when used as HTTP callback handlers (aka WebHooks 😑).

I’ve experimented somewhat with creating a RESTful interface using pipelines and triggered tasks as the underlying implementation and using Amazon’s AWS API Gateway to provide the bookmark-able entry point URLs.

Another benefit of the API Gateway is that you can use multiple implementation technologies where appropriate.

TK