Forum Discussion

kmiesse's avatar
kmiesse
Contributor
8 years ago
Solved

List started pipelines regardless of start date

I need to determine which pipelines are currently running regardless of when they started. It appears the Pipeline Monitoring API only returns pipelines that have started within the last_hours parameter (although I can’t seem to determine proper values for the start and end parameters - they are defined as “timestamp in milliseconds” - milliseconds since when?).
We have continuously running pipelines that may have been started more than 90 days ago, which I believe is beyond the start limit of the Pipeline Monitoring API. I need this information to determine if all pipelines are executing as they should. How can I determine which pipelines are running regardless of their start date? Thanks in advance!

  • As an interim solution, have you tried manipulating the data returned by the results of the Snaplex Monitoring API? There is quite a bit of information returned, but at a Snaplex node level, you will have visibility of all active runtime IDs in the active_pipelines array. The runtime IDs could then be resolved using the Pipeline Monitoring API to mine additional, granular details.

18 Replies

  • ken's avatar
    ken
    Former Employee

    As an interim solution, have you tried manipulating the data returned by the results of the Snaplex Monitoring API? There is quite a bit of information returned, but at a Snaplex node level, you will have visibility of all active runtime IDs in the active_pipelines array. The runtime IDs could then be resolved using the Pipeline Monitoring API to mine additional, granular details.

    • kmiesse's avatar
      kmiesse
      Contributor

      Wow! Thank you! That just might do the trick.

      • kmiesse's avatar
        kmiesse
        Contributor

        @ken, do you know if the active_pipelines contains IDs regardless of their start time so I should see IDs from executions starting before the previous 90 days that the Pipeline Monitoring API is restricted to?

  • ken's avatar
    ken
    Former Employee

    The runtime IDs returned can be in different states of activity. You would need to cross-reference them individually using the Pipeline Monitoring API to get that level of detail.

  • IF you look at the documentation,

    https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438155/Pipeline+Monitoring+API

    last_hours - integer - fetch data for last n hours. Max limit is 2160 hours (90 days).
    default = 1
    pipe_name - string - fetch data for a particular pipeline
    org_wide - Boolean - fetch data for whole org including all phases
    state - string - fetch data for particular pipeline states.
    State: [NoUpdate, Prepared, Started, Queued, Stopped, Stopping, Completed, Failed]. (case sensitive)
        Queued - Pipeline is queued for processing, all pipelines start in this state
        NoUpdate - Intermediate state at startup
        Prepared - Pipeline has been prepared
        Started - Pipeline has started execution
        Completed - Pipeline completed successfully
        Failed - Pipeline failed
        Stopped - Pipeline was stopped, by user or by the system, indicated by the 'stopper' key
        Stopping - Pipeline is stopping
        Example: state=Completed,Failed
    
    
    

    In your case, you would want to put a max time for last_hours and fetch your interested statuses

    Hope that works out of you.

    • kmiesse's avatar
      kmiesse
      Contributor

      But that won’t work if a pipeline was started more than 90 days ago, which is the max, correct? It appears to only return pipelines started within the last_hours param. I need to list those that are currently running but may have started more than 90 days ago.

      • nganapathiraju's avatar
        nganapathiraju
        Former Employee

        We dont keep anything beyond that time period which is why its available
        through PUBLIC APIs.

        If you wanted, you can store all that data through SPLUNK or some other
        kind of monitoring tools.

        Thanks & Regards,
        Naveen

    • aleung's avatar
      aleung
      Contributor III

      We archive these data in our warehouse in a daily basis to allow analysis like this.

      • kmiesse's avatar
        kmiesse
        Contributor

        How does Dashboard report them as “Started” if they were started more than 90 days ago? I just want a simple list of what pipelines are running currently without having to run some OS or JCC-level commands on the node.

  • ken's avatar
    ken
    Former Employee

    That is correct, Kim. The active pipeline runtimes returned by the API should all be active pipelines that the Control Plane sees as running regardless of when they the time at which they were started.

      • kmiesse's avatar
        kmiesse
        Contributor

        @ken, do you know if the active_pipelines would be all in the Started state or could they be any one of the various states on startup like Prepared or Queued also?