I have a Multi File Reader reading a series of S3 files using a wildcard, and writing the data to Snowflake. There is a Mapper in between. Functionally, everything is working as expected.
I’d like to get the name of the file in which the data was read, and write it to a Meta_FileName column. How do I retrieve the file name from the Multi File Reader? I’m sort of assuming it’s an expression to be added in the Mapper but not sure. TIA!
Is there a parser snap that is converting the binary documents?
My apologies for lack of detail, yes there is indeed a CSV parser following the multi file reader.
It’s currently not possible to pass the binary header that contains the file name through the CSVParser. Instead, you can use a DirectoryBrowser snap to get the file names and then kick off child pipelines to read the files and do the SnowflakeUpserts. In that case, you’ll be passing the filename as a pipeline parameter to the child pipeline, so you can use a Mapper to add the parameter into the documents that are going into the Upsert. A side-benefit of this is that you can process multiple files in parallel.
hi tstack,
Is there an inbuilt snaplogic functionality for the same ?
Hi Snaplogic Team,
Could you please advise on the solution for the same?
Was this ever resolved I am facing the same issue currently pulling files from s3 then using json parser and mapper to write data into a file. Is there a way to get the file name in a column ?
I am using the multi file reader.
Please find the attached sample pipeline:
Sample_pipe_2022_01_15.slp (7.9 KB)
Hope this helps.
Regards,
Spiro Taleski
Hi Spiro,
i am getting an error on the join shape .
.
not sure what i am doing wrong. my multifile reader is set with a wildcard and the subfolders option is checked. any idea what might be the cause of this error ?
