How to get the file name(s) from a Multi File Reader

@sghneim

Strange.

As a workaround, you can achieve the same by using Directory Browser Snap and Parent-Child pipeline configuration.

1. Parent Pipeline

   • Use Directory Browser Snap to list the files from s3 location
   • Use Pipeline Execute Snap to call the child pipeline and send the file name as a pipeline parameter

2. Child Pipeline

   • Create pipeline parameter to receive the file names from the Parent pipeline.
   • Read the files using File Reader Snap
   • Map the content and file name from parameter using Mapper Snap

This is actually what was proposed above by tstack.

t’s currently not possible to pass the binary header that contains the file name through the CSVParser. Instead, you can use a DirectoryBrowser snap to get the file names and then kick off child pipelines to read the files and do the SnowflakeUpserts. In that case, you’ll be passing the filename as a pipeline parameter to the child pipeline, so you can use a Mapper to add the parameter into the documents that are going into the Upsert. A side-benefit of this is that you can process multiple files in parallel.

Regards,
Spiro Taleski