Forum Discussion

Surendar's avatar
Surendar
Contributor
8 years ago

Generate multiple files based on single column

I need to generate multiple files based on category, say for example, The table having list of columns and category column have multiple values.

Category:

  • milk_products
  • water_products
  • oil_products
  • rice_products

So from above category, I need to generate 4 files with its records in separate files dynamically in Snap Logic.

Can any one have any idea/suggestion how to achieve this?

pipeline
sample_pipeline
transform

6 Replies

  • tstack's avatar
    tstack
    Former Employee
    8 years ago

    I’m attaching an example pipeline that generates a separate JSON file for each document in a particular category:

    FilePerCategory_2017_11_17.slp (7.0 KB)

    Here’s some detail on what’s going on in the pipeline:

    • The Group By Fields Snap can be used to do the category grouping (make sure the input documents are sorted by category, otherwise I think you’ll get multiple outputs for the same category).
    • Once the categories are grouped together in a single document, you can write them out to a file using the JSON Formatter. This JSON Formatter has the ‘Format each document’ option checked, which means that a separate binary output document will be generated for each input document. We want this since every input document will be the collection of items in the category and that’s what we want in each file. In addition, we set the ‘Content’ property to ‘$group’ since that contains the array of items in the category. The last thing for the formatter is to compute the file name based on the category using the ‘Binary Header Properties’ to set the ‘content-location’ property in the binary output document header.
    • Finally, there’s a File Writer that writes out a file using the name as found in the ‘content-location’ property.
    • Surendar's avatar
      Surendar
      Contributor
      8 years ago

      Thank you for the input and solution for json file. Is there any approach for CSV file format?

      because there is no specific way to write our contents($group) to in CSV parser/formatter. But we have it in JSON for-matter “Content” option.

      • tstack's avatar
        tstack
        Former Employee
        8 years ago

        As you point out, the CSV Formatter does not have the Binary Header or Content properties like the JSON Formatter. As an alternative, you can pass the category groups into a child pipeline and do the processing there. Using a child pipeline allows you to specify the file name through a pipeline parameter that is referenced in the File Writer. The child pipeline should consist of the following snaps:

        The splitter will break apart the category groups into individual documents again, which is what the CSV Formatter is expecting. The File Writer is then configured to write to a file using a file name from a pipeline parameter.

        In the parent pipeline, replace the JSON Formatter/File Writer with a Pipeline Execute snap that calls the child pipeline and passes the file name in a pipeline parameter.

  • tstack's avatar
    tstack
    Former Employee
    8 years ago

    What is the format of these files (e.g. CSV, JSON, XML, …)?

    • Surendar's avatar
      Surendar
      Contributor
      8 years ago

      Any files either CSV or JSON files. by default is CSV

      • nganapathiraju's avatar
        nganapathiraju
        Former Employee
        8 years ago

        can you post the sample structure? There can be many ways of doing it!