How can we pass a dynamically assigned variable from one end of a pipeline to the other?

[swright]

Our organization has decided that they only want a single pipeline for each integration we develop (no execute pipeline to pass parameters to another pipeline). They say that this is due to the way their error handling works and that they only want one pipeline referenced when they have an error.

I explain the above because people will probably suggest assigning the variable in a parent pipeline and passing it as a parameter to a child.

I need to pass variables assigned earlier in a pipeline to snaps that occur later in the same pipeline. For instance, I assign a filename variable like:

“test_”+Date.now().toLocaleDateTimeString({“format”:“yyyyMMdd”})+“.csv”

I use it in a File reader snap at the beginning of the pipeline and need to use it again multiple times near the end of the pipeline. At the end of the pipeline I move the file to an archive location and need the file name to do that.

I can’t set parameters dynamically in “Edit Pipeline” or within the pipeline, and it is difficult to pass a variable from one end of the pipeline to the other. In searching this forum I find that others have this same issue but I didn’t see any solutions other than using parent/child pipelines. Is there an easy way to set a variable early in a pipeline and use it in other snaps that come later in the same pipeline? Other workarounds…?

Thanks!

[ptaylor]

Depending on what you’re trying to do later in your pipeline, the approaches that don’t involve Pipeline Execute are either awkward or not possible.

With Pipeline Execute, this becomes straightforward. Urge your organization to reconsider their decision to avoid one of our product’s most powerful and useful features.

[swright]

Thanks. I think that this is good advice.

[Supratim]

@swright Yes, this is one of the challenging part in snaplogic which I have faced during SOA to Snaplogic migration. There are work around but it’s make your pipeline lengthy.

1. Use copy snap where you have exact name of the file. You might need multiple output view as per the requirement.
2. Then use gate snap just before the snap where you want to reuse the name of file reader.
   3.After gate either you can use mapper or can directly use filename on your write snap.

[swright]

Another good method! Thanks!

[del]

Hey Scott,
As mentioned already, there are several complex options that can be used as a workaround. Those mentioned already are as good as any, so I refrain from mentioning more. Also, I agree with Patrick, that it may behoove your organization to reconsider their position on the Pipeline Execute snap.

However, based solely on your example of needing a dynamically generated date stamp constant throughout your pipeline, this may be a less clunky option for you:

Try using pipe.startTime (or a formatted derivative) as part of your filename expression.
"test_" + pipe.startTime.toLocaleDateTimeString({"format":"yyyyMMdd"}) + ".csv"
I believe pipe.startTime should remain constant throughout the pipeline run and this might provide what you can use in your specific use case example.

This won’t solve other pass through scenarios though; which is basically the meaning of the title of your post. However, another option for those scenarios might be to place your dynamic expression in a pipeline parameter and use the eval() function to evaluate the parameter. I have used this technique before. As long as it is not an expression result that will change with run time, it should work.