Parsing large files
Hey all,
I am trying to tread a File (260+Mb) and parse it and and then insert the data to a Table. And when i m using SQL Bulk load The csv parser always seems to be stuckt at output 4100 docs
...
Forum Discussion
Thanks,seems like it was problem with, the reason I used join with tail snap is because i do not want the pipeline to continue executing untill the insert operation is fully completed, So used tail snap to act as a waiting snap
this is the updated pipeline
This might, and probably will result in proceeding with the pipeline even before the data is loaded into the table. The Join is one of the solutions to achieve what you need, but you need to be careful while configuring. Combine both, the new setup + Join (1on1)
That will result in having one object on output, it will wait both inputs to complete and will move the execution forward.
