I need to load tables in postgres db. I receive the data as csv files (one each for each table). Some of the tables are master data tables and the other are transaction data tables. The transaction data tables have foreign key relationship with the master data tables. Hence while uploading the data to DB, I need to make sure that the master data tables are updated first before updating the transaction data tables. However, I would like to avail the parallel processing (using child pipelines) while updating the master data tables (there are multiple master data tables that don’t have dependencies amongst themselves). Once all the master data tables are updated then I would like to start another parallel processing flow for updating multiple transaction data tables.
Please let me know the ways to achieve this.