Feature Request: Flow Control Snaps

When designing Pipelines we’re often faced with wanting to control the sequencing of Snap execution in different branches – e.g. ‘don’t start executing Snap Y until Snap X has completed’. Often this is solved by using a Union Snap and then a Sort Snap (which collates all documents in the stream). For example:

This is required in order to prevent “Update Task in Env” from starting execution until “Update Pipeline in Env” is complete. Often I also end up doing a Copy of the document stream, Union and then a Filter Snap to exclude the extra copies of the documents.

We’ve requested the feature to have Snaps which allows for flow control / sequencing / dependencies between different branches of a Pipeline, and would allow for simpler and more readable Pipeline designs (Service Request #18184).

I’d be keen to hear any thoughts or feedback from the community - is this a feature you would like to see too?

3 Likes

@cj.ruggles Thanks for your raising this enhancement request. We are investigating the possibility of adding some more specific Flow Control Snaps & Process Flow in future releases.

Your point is a good one - it is not obvious how to make the pipeline wait until the prior steps are completed. I used to “aggregate”, but now I prefer to use “tail” to force the pipeline to wait until the next snap executes. I think it would be quicker and use less memory than “sort” or “aggregate”. A “wait” snap would be nice, but not a high priority for the experienced pipeline developer.

2 Likes

Did you give the Number of documents* has some random maximum value ?

Use 1 as the number of documents so you wait for the very last one.

But how I will get all the documents back as tail will filter send only the last one document .

You are correct. You would have to park the data in a file or something in order to get the data set AND use a tail to wait for processing to end, or do some other copy/join mechanism. It’s ugly. It would be nice if they had a snap to wait until completion of a step before another one starts for your use case.

1 Like

Does Snaplogic team have update on this request ? We have had many conversations with snaplogic team on this. As mentioned by others, we often(almost every usecase) have do workaround for this. If snaplogic could develop control feature , it would really really benefit us.

The product management may be able to answer this question better as to when such snap is going to be ready. Some other alternative methods are to use a join operation which is a very nice way to achieve synchronization. Another good option would be is to use the file poller snap that polls the given directory or files for every few seconds until the specified time out is reached. If the presence of the files or directory is determined, it will produce an output that can be used to trigger the downstream pipelines. Polling can be done for an indefinite period as well by setting the polling timeout value to be 1.

We are using the copy and Join but am sorry its really not nice way, if you have do that more than one place in the same pipeline , it didn’t look good. This use case is not just one scenario , if faced this situation in many times when I’m developing the pipelines. I wish Snaplogic team take some quick action and provides us snap which meet this requirement.

Can you please send a screen shot of the pipeline where you are having this scenario?

when I say scenario, I meant the downstream snap need to execute only once for any no of input but it needs all input information. If you use Head or Tail, we lose all information except 1 record. I would prefer to share my screen in a meeting. Please feel free to setup one.

I see the problem. It is the synchronization issue that we are discussing in this thread. We need a kind of a Gate Snap that will allow the output to be written out when all the inputs are processed. But one aspect that is unclear to me is if the purpose is that of synchronization, then why do we need the data coming from the upstream snaps? Can we simply use Head or Tail Snaps?

Even if you use Join Snaps, the result of the Join Snap is more dependent on the join condition, the number of input tables to the join.

As discussed in the thread above even UNION , followed SORT snaps also help in synchronization which will produce a different output compared to JOIN

What is most important we need to understand what kind of outputs are we expecting from this snap?

Would you be willing to get on a call with us? We are working on conceptualizing workflow enhancements so this is timely. Diby Malakar (VP Product Management, dmalakar@snaplogic.com)

Sure. am available this afternoon if you would like to setup one. I sent an email to you. Thanks!!

For the time being, you can use the GroupByN snap to collect all of the input documents into an array in a single document and then process that array in the subsequent snaps.

This feature would be very useful for us as well.

1 Like