Forum Discussion
marenas - Here is an updated version of the pipeline to preserve the order of the files. Basically, I've simply added a Mapper to each path and placed a file-number and record-number to the documents of both paths before the Union, then sorted the data to ensure proper record ordering, and finally removed the sorting fields from the document.
So a couple new concepts here:
- snap.in.totalCount is a built-in value that gives us the number of records seen by the input view of the snap.
- I enabled Pass-Through in the Mapper settings for the "Add fileN sortKey" snaps to allow the rest of the data to flow through the snap without having to specify it. In the "Remove sortKey" Mapper, I use Pass-Through again to allow all the fields to pass through except the fields that I have in the Expression without specifying the Target-Path - this effectively deletes just those 2 fields.
Hope this helps!
- agarwal_rohit2 years agoNew Contributor
Szymon - Did you get any solution for this? I am also looking for the same
Related Content
- 11 months ago
- 6 months ago
- 3 years ago
- 4 years ago