02-27-2020 10:30 PM
I have 3 binaries as three documents. Is it possible to merge the three binaries into a single document and write it to a file?
03-02-2020 01:27 PM
Would a Join Snap set to Merge work?
03-02-2020 01:35 PM
Whether I use a Union → Document to Binary → File Writer or Join (Merge) → Pivot → Document to Binary → File Writer, I run into the same issue. I have three documents and only the final document is written to the file. How do I write all three documents to a single file?
03-02-2020 01:38 PM
In the FileWriter, are you using the “Append” setting for the “File Action”? What protocol are you using? Are you sure that protocol supports doing an append?
03-02-2020 01:39 PM
Append in the File Writer was the first thing I tried last week but Append isn’t supported in S3. I would like to avoid having to use SFTP to accomplish this. That complicates things in other ways.
03-02-2020 02:03 PM
Note that using a Union snap will not work like you want since it will immediately pass whatever input comes in to the output. Since Snaps all run in parallel, there is no guarantee you will get your desired order of documents on the output (i.e. header, detail, trailer). It could theoretically come out as: trailer, header, detail.
How much data is in the Detail branch of the screenshot above? Hundreds of rows, millions? If it’s more on the order of tens/hundreds, the new Gate snap might work for this use case. You can replace CSVFormatters, Bin2Doc, and Union with the Gate snap followed by a Mapper, JSON Splitter, and a CSV Formatter. I’m attaching an example pipeline that does just that.
CommunityMergeBinary_2020_03_02.slp (10.1 KB)