Forum Discussion

mxpeskir's avatar
mxpeskir
New Contributor
7 years ago

How to get the file name(s) from a Multi File Reader

I have a Multi File Reader reading a series of S3 files using a wildcard, and writing the data to Snowflake. There is a Mapper in between. Functionally, everything is working as expected. I’d like...
expression_language
sghneim's avatar
sghneim
New Contributor
4 years ago

Hi Spiro,

i am getting an error on the join shape .
.

not sure what i am doing wrong. my multifile reader is set with a wildcard and the subfolders option is checked. any idea what might be the cause of this error ?

  • SpiroTaleski's avatar
    SpiroTaleski
    Valued Contributor
    4 years ago

    @sghneim

    Hmm, the same code it is working on my side. So, I am wondering if the file data can have some impact.

    Could you try to re-drag the Join Snap(first delete existing Join, and drag new one from Snap Pallet again), and add the same configuration.

    Inner Join with “1” for Left and Right Join Paths.

    Regards,
    Spiro Taleski

    • sghneim's avatar
      sghneim
      New Contributor
      4 years ago

      i did delete the join and dragged a new one to the canvas , the set up is below but still no luck with this .

      • SpiroTaleski's avatar
        SpiroTaleski
        Valued Contributor
        4 years ago

        @sghneim

        Strange.

        As a workaround, you can achieve the same by using Directory Browser Snap and Parent-Child pipeline configuration.

        1. Parent Pipeline

          • Use Directory Browser Snap to list the files from s3 location
          • Use Pipeline Execute Snap to call the child pipeline and send the file name as a pipeline parameter

        2. Child Pipeline

          • Create pipeline parameter to receive the file names from the Parent pipeline.
          • Read the files using File Reader Snap
          • Map the content and file name from parameter using Mapper Snap

        This is actually what was proposed above by tstack.

        t’s currently not possible to pass the binary header that contains the file name through the CSVParser. Instead, you can use a DirectoryBrowser snap to get the file names and then kick off child pipelines to read the files and do the SnowflakeUpserts. In that case, you’ll be passing the filename as a pipeline parameter to the child pipeline, so you can use a Mapper to add the parameter into the documents that are going into the Upsert. A side-benefit of this is that you can process multiple files in parallel.

        Regards,
        Spiro Taleski