List started pipelines regardless of start date

I need to determine which pipelines are currently running regardless of when they started. It appears the Pipeline Monitoring API only returns pipelines that have started within the last_hours parameter (although I can’t seem to determine proper values for the start and end parameters - they are defined as “timestamp in milliseconds” - milliseconds since when?).
We have continuously running pipelines that may have been started more than 90 days ago, which I believe is beyond the start limit of the Pipeline Monitoring API. I need this information to determine if all pipelines are executing as they should. How can I determine which pipelines are running regardless of their start date? Thanks in advance!

IF you look at the documentation,

https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438155/Pipeline+Monitoring+API

last_hours - integer - fetch data for last n hours. Max limit is 2160 hours (90 days).
default = 1
pipe_name - string - fetch data for a particular pipeline
org_wide - Boolean - fetch data for whole org including all phases
state - string - fetch data for particular pipeline states.
State: [NoUpdate, Prepared, Started, Queued, Stopped, Stopping, Completed, Failed]. (case sensitive)
    Queued - Pipeline is queued for processing, all pipelines start in this state
    NoUpdate - Intermediate state at startup
    Prepared - Pipeline has been prepared
    Started - Pipeline has started execution
    Completed - Pipeline completed successfully
    Failed - Pipeline failed
    Stopped - Pipeline was stopped, by user or by the system, indicated by the 'stopper' key
    Stopping - Pipeline is stopping
    Example: state=Completed,Failed


In your case, you would want to put a max time for last_hours and fetch your interested statuses

Hope that works out of you.

But that won’t work if a pipeline was started more than 90 days ago, which is the max, correct? It appears to only return pipelines started within the last_hours param. I need to list those that are currently running but may have started more than 90 days ago.

We dont keep anything beyond that time period which is why its available
through PUBLIC APIs.

If you wanted, you can store all that data through SPLUNK or some other
kind of monitoring tools.

Thanks & Regards,
Naveen

We archive these data in our warehouse in a daily basis to allow analysis like this.

How does Dashboard report them as “Started” if they were started more than 90 days ago? I just want a simple list of what pipelines are running currently without having to run some OS or JCC-level commands on the node.

The only possibility for a task to be in a “started” mode is an Ultra.

I have not seen an scheduled task run that long.

Triggered task is out of question.

We have non-ultra pipelines running continuously due to the JMS Consumer snap configured to continuous mode as follows:
image

What makes the difference in whether a pipeline was started with a scheduled task vs. a triggered task for how long the pipeline runs?

Also, since SnapLogic quarterly updates require the nodes to be restarted nearly every 90 days no pipeline would run longer than the time between updates. If the API would allow us to go back as far as the last platform update that would resolve my issue. This should only be a few more days or weeks beyond the 90 day limit now, correct?

Or if there was a way to simply list the pipelines running on a snaplex, that would resolve the issue also… Is there a way I can do that within a pipeline?

did you try these options?

plex_label - string - fetch runtime for a particular Snaplex label
cc_label - string - fetch runtime for a particular cc label (jcc’s hostname)
user_id - string - fetch runtime for particular users
Example: user_id=user_a@example.com,user_b@example.com
project_path - string - fetch runtime for particular project
Example: project_path=/Snaplogic/projects/test_project
Format: /<org_name>/projects/<project_name>
Format: /<org_name>/shared

I added the plex_label param and removed the last_hours param, assuming it still wouldn’t return executions starting prior to the 1 hour default of last_hours and that are still running. And they were not returned, as expected, but other executions that were started in the last hour were returned. I don’t think this API will simply list running pipelines regardless of start time. Is there another option?

I added Enhancement Request: List Started Pipelines. I also submitted a Support ticket for it.

As an interim solution, have you tried manipulating the data returned by the results of the Snaplex Monitoring API? There is quite a bit of information returned, but at a Snaplex node level, you will have visibility of all active runtime IDs in the active_pipelines array. The runtime IDs could then be resolved using the Pipeline Monitoring API to mine additional, granular details.

1 Like

Wow! Thank you! That just might do the trick.

@ken, do you know if the active_pipelines contains IDs regardless of their start time so I should see IDs from executions starting before the previous 90 days that the Pipeline Monitoring API is restricted to?

That is correct, Kim. The active pipeline runtimes returned by the API should all be active pipelines that the Control Plane sees as running regardless of when they the time at which they were started.

Wonderful. Thanks again.

@ken, do you know if the active_pipelines would be all in the Started state or could they be any one of the various states on startup like Prepared or Queued also?

The runtime IDs returned can be in different states of activity. You would need to cross-reference them individually using the Pipeline Monitoring API to get that level of detail.

1 Like