How to stop the data flow after the pipeline execute snap

[aditya_sharma]

Hi,

I am trying to create source target table row count validate pipeline for which I have used following design in Snaplogic:

Parent pipeine:

1. Read all the source tables name from one of the table (table names are same in both source and target).
2. Create query: select count(*) from table name (got from above snap).
3. Pass the above query to Child Pipeline.
4. Select all the records from audit table where source table row count is not matching to target table row count.
5. Write all the records from above snap to a file and send an email.

Child Pipeline:

1)Execute the query received from parent pipeline on the source tables and on the target tables.
2) Store the row count of source and target in one of the audit table in target.

My problem is In parent pipeline if the number of child pipeline execute is 100(in step 3) , the next snap(step 4) is executing 100 times due to which if my audit table has 10 non matching records, it is throwing 1000 records output which is wrong.

[nsmith]

Hi Aditya,

You can use the “tail” snap to limit document passthrough. It will only output 1 document, no matter how many input documents it receives. You might need this in multiple places. I think this will fix your problem.

If you need the 100 documents combined into 1 single document, you might want to check out the “aggregate” snap.

[aditya_sharma]

Thanks, using tail snap it worked fine. I didn’t tried the aggregate or group by n, but i am sure using these snap will also work.

[cstewart]

Take a look at using the “Group By N” Snap at the end of your child pipeline, with a “Group Size” set to 0 (zero) it will combine all output documents into a single output document.
Hope this helps.