โ02-27-2020 10:30 PM
I have 3 binaries as three documents. Is it possible to merge the three binaries into a single document and write it to a file?
โ03-02-2020 01:27 PM
Would a Join Snap set to Merge work?
โ03-02-2020 01:35 PM
Whether I use a Union โ Document to Binary โ File Writer or Join (Merge) โ Pivot โ Document to Binary โ File Writer, I run into the same issue. I have three documents and only the final document is written to the file. How do I write all three documents to a single file?
โ03-02-2020 01:38 PM
In the FileWriter, are you using the โAppendโ setting for the โFile Actionโ? What protocol are you using? Are you sure that protocol supports doing an append?
โ03-02-2020 01:39 PM
Append in the File Writer was the first thing I tried last week but Append isnโt supported in S3. I would like to avoid having to use SFTP to accomplish this. That complicates things in other ways.
โ03-02-2020 02:03 PM
Note that using a Union snap will not work like you want since it will immediately pass whatever input comes in to the output. Since Snaps all run in parallel, there is no guarantee you will get your desired order of documents on the output (i.e. header, detail, trailer). It could theoretically come out as: trailer, header, detail.
How much data is in the Detail branch of the screenshot above? Hundreds of rows, millions? If itโs more on the order of tens/hundreds, the new Gate snap might work for this use case. You can replace CSVFormatters, Bin2Doc, and Union with the Gate snap followed by a Mapper, JSON Splitter, and a CSV Formatter. Iโm attaching an example pipeline that does just that.
CommunityMergeBinary_2020_03_02.slp (10.1 KB)