Wait in the parent pipeline till all the documents are processed in the child pipelines

[aprabhav]

I need to load tables in postgres db. I receive the data as csv files (one each for each table). Some of the tables are master data tables and the other are transaction data tables. The transaction data tables have foreign key relationship with the master data tables. Hence while uploading the data to DB, I need to make sure that the master data tables are updated first before updating the transaction data tables. However, I would like to avail the parallel processing (using child pipelines) while updating the master data tables (there are multiple master data tables that don’t have dependencies amongst themselves). Once all the master data tables are updated then I would like to start another parallel processing flow for updating multiple transaction data tables.

Please let me know the ways to achieve this.

[pmancevski]

Hi @aprabhav,

There are couple of ways to achieve this and also I’m assuming that in the child pipeline you have some separate logic for transaction data and you can know which one has dependency. Fastest solution i can think of is

With the router you will separate the data between master data and transaction data, “Gate” snap will not complete executing until the execution of all upstream Snaps is done. After that in the mapper you can get only transaction data ($input1[0]), which you separate before in the router.

Thanks,
Pero M.

[aprabhav · Accepted solution]

Thanks a lot, Pero. This helped and it worked. Thanks again!!!