UNION: Preserve source data sequence

#1

I need to preserve the sequence of source data to produce line by line output. It seems UNION snap cannot do that as I just find “December 20, 2014: Preserve order was removed.” comment in SnapLogic docs. So then, what option I have???
Fox example, if in the source data file, there are 3 records, with X, Y, Z in each line sequentially, then in the target file, after conversion, it should generate same sequence, say X1, Y1, Z1 in each line.

#2

Hi @Abhijit, I think you are talking about the following scenario - you have two streams of documents that are being combined through the use of the Union Snap, but the order of the documents that was originally present in both streams has been lost, like so:

community-union-question.png2138×1258 221 KB

If this is what you are describing, this is the nature of SnapLogic’s streaming architecture - documents are streamed into downstream Snaps as soon as they become available. If you wish to impose a natural order again, I would recommend using the Sort Snap immediately after the Union Snap:

image.png896×372 46.4 KB

Is this what you were experiencing? If not, please let me know and perhaps share a screenshot of what is happening.

#3

In addition to Robin’s comment on this. If you have a source that doesn’t give you a ‘natural’ ordering, you can pass it through the Sequence snap to add a sequential value field. Then, later, you can use the Sort snap to put everything back in order.

1 Like
#4

I also have a similar use case and want to confirm the functionality of the Union snap. I have a single input file which contains a mixed set of records for dogs and cats. Within my pipeline I use a Router snap to send dog records down one path and cat records down another. Both of these paths are then merged through the use of the Union snap.

I don’t need to preserve the order of the input data, but I do need to preserve the order for each of the paths, cats and dogs.

If the input data contains cat1, cat2, cat3, dog1, dog2, dog3 I am OK if the Union output is cat1, dog1, cat2, dog2, cat3, dog3 as the Union output continues to preserve the order of the records in their respective sets.

Is it guaranteed that the Union output will preserve the order of the records through each of the paths/streams or is it possible that cat1, dog2, cat3, dog1, cat2, dog3 could be the output of Union?

#5

Yes, the order of the streams are preserved.