I am trying to create source target table row count validate pipeline for which I have used following design in Snaplogic:
- Read all the source tables name from one of the table (table names are same in both source and target).
- Create query: select count(*) from table name (got from above snap).
- Pass the above query to Child Pipeline.
- Select all the records from audit table where source table row count is not matching to target table row count.
- Write all the records from above snap to a file and send an email.
1)Execute the query received from parent pipeline on the source tables and on the target tables.
2) Store the row count of source and target in one of the audit table in target.
My problem is In parent pipeline if the number of child pipeline execute is 100(in step 3) , the next snap(step 4) is executing 100 times due to which if my audit table has 10 non matching records, it is throwing 1000 records output which is wrong.