Forum Discussion

ManojD's avatar
ManojD
New Contributor
3 years ago

Require API to build a pipeline to find the high CPU and memory utilization of an individual pipeline during its runtime

We are trying to design a pipeline in Snaplogic to identify the high CPU and memory utilization of an individual pipelines that ran in a month using public API. We tried using runtime API but we are not able to get the expected output.

API used:
“https://elastic.snaplogic.com/api/1/rest/public/runtime/ ”
“https://elastic.snaplogic.com/api/1/rest/public/runtime/Environment/ ”+$Runtime_ID

Is there an API for it? which can give the expected output.

  • Hello,

    Are you referring to triggered tasks? This is a great suggestion, but I’m curious if you have any specific ideas about how you’d want to configure the task so that the label includes the information you care about.

    Thanks,
    Patrick

  • I imagine it like something that is akin to how the pipeline execute task works. With the pipeline execution, there is an expression that can be used for the pipeline’s label. The label value is what gets presented on the dashboard. The same type of thing could be set up between the task and the pipeline that it is invoking. The pipeline’s edit configuration could contain a place where an expression could be entered and that expression could accept values that are passed to it from the task that is invoking it.

    I hope that makes sense. Let me know. I could try to draw a some illustrations.

    • ptaylor's avatar
      ptaylor
      Employee

      Yes, it does, thanks. That’s basically what I thought you meant. But this is a bit of a different problem than for Pipe Exec, which evaluates the expression to set the child pipeline’s label for each input document in the context of the parent pipeline, before the child pipeline starts. With a triggered task, it raises some questions. A triggered task only executes a single pipeline per request, and if the request contains an array, the first snap will receive each element as a separate document. So the question is how to evaluate the expression used to set the pipeline’s label before the pipeline is running, and what exactly should it be evaluated against? Just pipeline parameters (which includes the request metadata)? Or also documents? Just the first document if there’s more than one? Or the array before it’s split into multiple documents?

      • Something like this is what I was imagining, where a execution label expression could be added to the pipeline’s parameters. This could either go here on on the task. Might be better on the task.