Pipeline Patterns: What would you like to see?

[dmiller]

We’ve been discussing the idea of making sample pipelines available in the Community. While the specifics of how this would happen still needs to be worked out, we’d liked to know what kind of use cases would you like to see?

[murphy-brian]

I would love to see a whole category on performance monitoring and tuning tips. Environment, Pipelines, Snaps and expressions.

[dmiller]

I would love to see a whole category on performance monitoring and tuning tips. Environment, Pipelines, Snaps and expressions.

Do you mean a category within the Community for Performance? If so, that would be Monitoring Pipelines and Plexes.

[Ksivagurunathan]

Is there a way to figure out which snap in a pipeline is causing the performance bottleneck ? Not sure how accurate the statistics information of snap in pipeline execution statistics .

[tstack]

Ksivagurunathan:

Is there a way to figure out which snap in a pipeline is causing the performance bottleneck ?

Have you gone through the documentation for the execution statistics dialog? In particular, pay attention to the Duration bars for the Snaps. Snaps with long blue bars are spending their time executing, while Snaps with long green and purple bars are waiting for upstream or downstream snaps to finish working. So, snaps with long blue bars are typically the bottleneck.

Ksivagurunathan:

Not sure how accurate the statistics information of snap in pipeline execution statistics .

Do you have reason to believe they are not accurate? If you can provide some examples, we can take a look.

[Ksivagurunathan]

Not only more time consumed also other factors like memory & CPU usage . Also I meant to do this in a pipeline and send an alert about that pipeline, snaps and other required details. When multiple team using the tool and all of them sharing same nodes for execution, this would be really helpful.