cancel
Showing results for 
Search instead for 
Did you mean: 

UNION: Preserve source data sequence

Abhijit
New Contributor

I need to preserve the sequence of source data to produce line by line output. It seems UNION snap cannot do that as I just find “December 20, 2014: Preserve order was removed.” comment in SnapLogic docs. So then, what option I have???
Fox example, if in the source data file, there are 3 records, with X, Y, Z in each line sequentially, then in the target file, after conversion, it should generate same sequence, say X1, Y1, Z1 in each line.

4 REPLIES 4

robin
Former Employee

Hi @Abhijit, I think you are talking about the following scenario - you have two streams of documents that are being combined through the use of the Union Snap, but the order of the documents that was originally present in both streams has been lost, like so:

9c03edb3871083748b36ce6001b192d06450c890.png

If this is what you are describing, this is the nature of SnapLogic’s streaming architecture - documents are streamed into downstream Snaps as soon as they become available. If you wish to impose a natural order again, I would recommend using the Sort Snap immediately after the Union Snap:

3505d802b9788d63d346fc8c23447fcbe9664b03.png

Is this what you were experiencing? If not, please let me know and perhaps share a screenshot of what is happening.

ncrouch
Former Employee

In addition to Robin’s comment on this. If you have a source that doesn’t give you a ‘natural’ ordering, you can pass it through the Sequence snap to add a sequential value field. Then, later, you can use the Sort snap to put everything back in order.

ScottWiemann
New Contributor

I also have a similar use case and want to confirm the functionality of the Union snap. I have a single input file which contains a mixed set of records for dogs and cats. Within my pipeline I use a Router snap to send dog records down one path and cat records down another. Both of these paths are then merged through the use of the Union snap.

I don’t need to preserve the order of the input data, but I do need to preserve the order for each of the paths, cats and dogs.

If the input data contains cat1, cat2, cat3, dog1, dog2, dog3 I am OK if the Union output is cat1, dog1, cat2, dog2, cat3, dog3 as the Union output continues to preserve the order of the records in their respective sets.

Is it guaranteed that the Union output will preserve the order of the records through each of the paths/streams or is it possible that cat1, dog2, cat3, dog1, cat2, dog3 could be the output of Union?

Yes, the order of the streams are preserved.