How to get the file name(s) from a Multi File Reader

[mxpeskir]

I have a Multi File Reader reading a series of S3 files using a wildcard, and writing the data to Snowflake. There is a Mapper in between. Functionally, everything is working as expected.
I’d like to get the name of the file in which the data was read, and write it to a Meta_FileName column. How do I retrieve the file name from the Multi File Reader? I’m sort of assuming it’s an expression to be added in the Mapper but not sure. TIA!

[tstack]

mxpeskir:

I have a Multi File Reader reading a series of S3 files using a wildcard, and writing the data to Snowflake. There is a Mapper in between.

Is there a parser snap that is converting the binary documents?

[mxpeskir]

My apologies for lack of detail, yes there is indeed a CSV parser following the multi file reader.

[tstack]

It’s currently not possible to pass the binary header that contains the file name through the CSVParser. Instead, you can use a DirectoryBrowser snap to get the file names and then kick off child pipelines to read the files and do the SnowflakeUpserts. In that case, you’ll be passing the filename as a pipeline parameter to the child pipeline, so you can use a Mapper to add the parameter into the documents that are going into the Upsert. A side-benefit of this is that you can process multiple files in parallel.

[KishorePradeep]

hi tstack,

Is there an inbuilt snaplogic functionality for the same ?