Group By N

[pjanapati]

Hi,

I am building a pipeline where I am getting the 100 records. And I want to group 5 records and send the file to FTP. Here, for grouping 5 records I have used group by N. After this, I am unable to format the CSV data.

In csv formatter, I am getting the below error.

Can you please anyone help me on this. After group by N, how to format that data to CSV data.

[Aleksandar_A]

Hello @pjanapati.

You can try by using a child pipeline after you group the documents, and in that pipeline you just split the group and write it as a csv file.

Sample pipelines:
SL-comm-csv-write_2023_03_06.slp (5.3 KB)
SL-comm-csv-batches_2023_03_06.slp (5.8 KB)

Let me know if this helps you.

BR.
Aleksandar.

[Aleksandar_A]

Or you can omit the Group By N Snap, and specify a Batch Size in the Pipeline Execute Snap.

And in the child you just format and write the documents.

[pjanapati]

Hi @AleksandarAngelevski ,

Thanks for the solution.
In JSON, we have 12 records. Coming to CSV files I am getting only 2 files with 5 records each. Here we are losing 2 records. Is there any solution for this.

Thank you.

[Aleksandar_A]

That should not be the case, the Group By N will produce 3 arrays(groups) if you have 12 records and the group size is set to 5.
Maybe you should check the naming of the files, maybe they are being overwritten?