How to pass a variable along the pipeline?

[graham-onpoint]

I have a pipeline that looks like this:

The Snaps are:

1. List all files in a given Box directory
2. Filter the files according to a mask
3. Read the matching Excel file into the pipeline
4. Convert the first Worksheet to CSV
5. Write a CSV file to an archive folder in Box
6. Delete the original Excel file

Everything is working up to step 5, where I need to access the output variables of step 2 again, the FileName is needed to create the matching file name in the archive folder (except this time with a .csv extension) and then the FileId is needed in step 6 in order to delete the Excel file.

The business reason behind doing this is that the file coming in to the Box folder will have a date prefix at the front of it. There is no set frequency for this file, it is practically arriving at random intervals. Therefore, I have no way to hard code the actual file name into the Pipeline and must dynamically check for a new file every day.

How can I store the output fields from step 2 somewhere in memory so that the final two Snaps in the pipeline can access them?

[tstack]

graham-onpoint:

I can see how that would happen, but could that not be controlled with a ForEach Snap that serializes the processing?

Hmm, I’m not quite sure what you mean here. The PipeExec snap basically obsoletes the ForEach snap.

graham-onpoint:

however it is not a requirement for the pipeline I am developing as (a) there will only be one file, and (b) it is acceptable to stop processing and throw an error if there is more than one file.

That’s not going to be true in the general case. So, we can’t add something so error prone and hope that people only use it in the right situations.

[graham-onpoint]

I believe that this would be very helpful if added to the “Parameters and Fields” documentation.

[Pavol]

Thanks @tstack for the suggestion. To be honest I find this quite a common requirement that is very complicated to implement. Parent/Child pipelines are much harder to debug, and it is strangely inconsistent. The Box Read snap outputs the “content location” i.e. the filename, but the CSV/Excel Parser snaps drops everything passed to it, except the file contents. A viable solution could be an option for the CSV/Excel Parser to include the originating filename in its output, much like Alteryx does

[graham-onpoint]

This is another use case that is effectively asking for this feature. The parallel issue comes up again, of course, but that doesn’t negate the functional requirement.

Retrieve Filename into Mapper from Box Read Designing Pipelines

How can I catch the Filename “Box Read” Snap opened into the “Mapper” snap? [Box%20Read%20File%20Name]