Forum Discussion
marenas - I would not recommend using Gate to combine the data files - it can cause excessive memory consumption for very large files since the data has to be stored completely in memory. I recommend the attached approach.
The trick here is in the Mapper on the bottom path and the second input view added to the CSV Parser. If you review the documentation, you will see that the second input view allows you to specify a header and also datatypes, if you choose. I simply added the header in the Mapper.
Then in the Union, it combines the data in the way you are looking for.
One thing to note is that Union will take documents as they come from each input view - meaning in this case that if both CSV Parsers are sending a large volume of records, you will see them intermixed - it does not wait for all of the documents on the first path before consuming the documents from the second path. There are easy fixes for this, but thought I would mention it in case it is a requirement that the data ordering be preserved between the input files.
Hope this helps!
- marjan_karafilo3 years agoContributor
Hi Igor,
Then maybe Group by fields snap is the one you are looking for?
You can choose by which object to group by.
Let me know if this helps you.
BR,
Marjan