Merge binaries into single document

[dwhansen-cbg]

I have 3 binaries as three documents. Is it possible to merge the three binaries into a single document and write it to a file?

[dmiller]

Would a Join Snap set to Merge work?

[dwhansen-cbg]

Whether I use a Union → Document to Binary → File Writer or Join (Merge) → Pivot → Document to Binary → File Writer, I run into the same issue. I have three documents and only the final document is written to the file. How do I write all three documents to a single file?

[tstack]

dwhansen-cbg:

I have three documents and only the final document is written to the file.

In the FileWriter, are you using the “Append” setting for the “File Action”? What protocol are you using? Are you sure that protocol supports doing an append?

[dwhansen-cbg]

Append in the File Writer was the first thing I tried last week but Append isn’t supported in S3. I would like to avoid having to use SFTP to accomplish this. That complicates things in other ways.

[tstack]

Note that using a Union snap will not work like you want since it will immediately pass whatever input comes in to the output. Since Snaps all run in parallel, there is no guarantee you will get your desired order of documents on the output (i.e. header, detail, trailer). It could theoretically come out as: trailer, header, detail.

How much data is in the Detail branch of the screenshot above? Hundreds of rows, millions? If it’s more on the order of tens/hundreds, the new Gate snap might work for this use case. You can replace CSVFormatters, Bin2Doc, and Union with the Gate snap followed by a Mapper, JSON Splitter, and a CSV Formatter. I’m attaching an example pipeline that does just that.

CommunityMergeBinary_2020_03_02.slp (10.1 KB)