Forum Discussion

MattBurden's avatar
MattBurden
New Contributor II
5 years ago

How to rerun any Pipeline based on the Ruuid of a previous execution

I’m seeking a way to rerun any pipeline with the exact details used in a previous execution. Ideally I’m looking for a generic solution that works for any Ruuid that requires little or no additional development to the pipelines themselves - my use case is only for pipelines that accept parameters (not payloads) i.e GET not POST.

My intention is to create a pipeline that is passed a single parameter (the ruuid of a previous pipeline execution). I can then use the public API’s to determine the execution details (pipeline, parameters etc…)

However…

  • I can’t see a generic way to use the ‘Pipeline Execute’ snap as the types and numbers of pipeline parameters will vary.

  • Alternatively I could create Triggered Tasks for the pipelines I want to run (and then determine the appropriate Task) before executing it via a REST GET Snap , but doing so would require maintenance of those Tasks going forward and generate a lot of Tasks that have no other purpose/use.

Has anyone solved a similar problem in a elegant way? 🤞

2 Replies

  • tlikarish's avatar
    tlikarish
    Employee
    5 years ago

    I don’t think there is an elegant solution to this at the moment. You rightly point out the big issue

    I can’t see a generic way to use the ‘Pipeline Execute’ snap as the types and numbers of pipeline parameters will vary.

    Since there isn’t a way to dynamically pass the pipeline parameters I don’t think there is much you can do.

    A less elegant approach could be to devise some kind of pipeline parameter convention. Like all pipelines would accept n parameters and you’d pass n even when the pipeline uses only 3. There could be other conventions you use, but you’d have to consider the tradeoffs with what you’re trying to achieve.

    Personally I think this could be an interesting enhancement to the product, so I’ll share with product owners to see what they think.

    Could you talk a bit more about what is motivating the need to re-run a pipeline?

    Have you taken a look at resumable pipelines before? Maybe that is something that could help your use case?

    https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/721944618/Resumable+Pipelines

  • MattBurden's avatar
    MattBurden
    New Contributor II
    5 years ago

    Thanks tlikarish, my use case is a end user facing Job Management interface embeded within our ERP solution. I want to allow users to retry whole or in most cases just parts of pipelines in the event of errors and I wanted to do so without needing to manually capture additional info for every pipeline - hence wanting to base the soution around the Ruuid and the associated info available to me from the SnapLogic Public API’s.

    I have briefly looked at resumable pipelines thankyou, but in addition to the need for a subscription, unavailability of an endpoint is just one potential use case. More commonly its related to data quality issues, we want to allow users to resolve and then recover any related failed integration messages (or more cruicially just failed parts without rerunning the whole).

    I’ve gone down the route of checking for and if not found creating what is a throwaway triggered Task in a dedicated project that I can then call, pass parameters (and as a bonus if needed a payload) to via a REST GET/POST. I think this (along with the info I am capturing in my own logs) will give me the technical flexibility to retry any failed pipelines (parent or child) without having to specifically configure/build them to allow me to do so.

    Its borderline ellegant… or at least I hope has the potential to be so.