How to integrate 3rd party job scheduler (e.g. Cisco Tidal) to kickoff and monitor SnapLogic pipeline execution

[pnarayan]

This document demonstrates how to integrate 3rd party job scheduler (e.g. Cisco Tidal) to kickoff and monitor snaplogic pipeline execution.

Invoke:
Deploy the SnapLogic integration pipeline as a trigger task
Invoke the SnapLogic pipeline execution from Tidal through the URL (ground or cloud)
Design the long running pipeline to be asynchronous to avoid timeout (firewall, Snaplogic control plane)

Monitor:
Use the SnapLogic Pipeline Monitoring API to check for execution status
You can find the Snaplogic pipeline monitoring API here

Case Study:
A master pipeline to accept all the Tidal request and distribute accordingly (Triggered)
Call different snaplogic trigger task based on the request.
e.g. https://elastic.snaplogic.com:443/api/1/rest/slsched/feed/organization_name/projects/project_name/pipeline_name
Log the Tidal Job ID & SnapLogic pipeline runtime ID into Database for status check
Database table to include the key fields: Tidal Job ID, SnapLogic Pipeline Runtime ID, Status
Response to Tidal request to close the connection to avoid time-out
A monitoring pipeline to update Tidal regarding the execution status (Scheduled)
Query the Database for running pipeline
Obtain the pipeline execution status using the SnapLogic pipeline runtime ID (Rest Call) e.g. https://elastic.snaplogic.com/api/1/rest/public/runtime/organization_name/ruuid?
Update the Tidal regarding the execution status using Tidal Job ID
Update the Database with the new status

[nsmith]

I’m getting a “504 Gateway Time-out” html response from triggered tasks right at the 15 minute mark. The pipeline is running a >15 minute MySQL stored procedure, so there aren’t any documents coming out during that time. Any ideas? Is there a way to increase the timeout?

[nsmith]

Okay we’re going to try doing this instead.

Have a triggered task that performs a pipeline execute on different pipelines. It will return the runtime id of the pipeline (ruuid) and then we’ll check the snaplogic “Pipeline Monitoring API” every 5 minutes for the ruuid status. Hopefully this method works, unless somehow we can increase the gateway timeout threshold.