cancel
Showing results for 
Search instead for 
Did you mean: 

List started pipelines regardless of start date

kmiesse
Contributor

I need to determine which pipelines are currently running regardless of when they started. It appears the Pipeline Monitoring API only returns pipelines that have started within the last_hours parameter (although I can’t seem to determine proper values for the start and end parameters - they are defined as “timestamp in milliseconds” - milliseconds since when?).
We have continuously running pipelines that may have been started more than 90 days ago, which I believe is beyond the start limit of the Pipeline Monitoring API. I need this information to determine if all pipelines are executing as they should. How can I determine which pipelines are running regardless of their start date? Thanks in advance!

1 ACCEPTED SOLUTION

ken
Former Employee

As an interim solution, have you tried manipulating the data returned by the results of the Snaplex Monitoring API? There is quite a bit of information returned, but at a Snaplex node level, you will have visibility of all active runtime IDs in the active_pipelines array. The runtime IDs could then be resolved using the Pipeline Monitoring API to mine additional, granular details.

View solution in original post

18 REPLIES 18

nganapathiraju
Former Employee

IF you look at the documentation,

https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438155/Pipeline+Monitoring+API

last_hours - integer - fetch data for last n hours. Max limit is 2160 hours (90 days).
default = 1
pipe_name - string - fetch data for a particular pipeline
org_wide - Boolean - fetch data for whole org including all phases
state - string - fetch data for particular pipeline states.
State: [NoUpdate, Prepared, Started, Queued, Stopped, Stopping, Completed, Failed]. (case sensitive)
    Queued - Pipeline is queued for processing, all pipelines start in this state
    NoUpdate - Intermediate state at startup
    Prepared - Pipeline has been prepared
    Started - Pipeline has started execution
    Completed - Pipeline completed successfully
    Failed - Pipeline failed
    Stopped - Pipeline was stopped, by user or by the system, indicated by the 'stopper' key
    Stopping - Pipeline is stopping
    Example: state=Completed,Failed


In your case, you would want to put a max time for last_hours and fetch your interested statuses

Hope that works out of you.

But that won’t work if a pipeline was started more than 90 days ago, which is the max, correct? It appears to only return pipelines started within the last_hours param. I need to list those that are currently running but may have started more than 90 days ago.

We dont keep anything beyond that time period which is why its available
through PUBLIC APIs.

If you wanted, you can store all that data through SPLUNK or some other
kind of monitoring tools.

Thanks & Regards,
Naveen

We archive these data in our warehouse in a daily basis to allow analysis like this.