Forum Discussion

aditya_sharma's avatar
aditya_sharma
New Contributor III
8 years ago

How to stop the data flow after the pipeline execute snap

Hi,

I am trying to create source target table row count validate pipeline for which I have used following design in Snaplogic:

Parent pipeine:

  1. Read all the source tables name from one of the table (table names are same in both source and target).
  2. Create query: select count(*) from table name (got from above snap).
  3. Pass the above query to Child Pipeline.
  4. Select all the records from audit table where source table row count is not matching to target table row count.
  5. Write all the records from above snap to a file and send an email.

Child Pipeline:

1)Execute the query received from parent pipeline on the source tables and on the target tables.
2) Store the row count of source and target in one of the audit table in target.

My problem is In parent pipeline if the number of child pipeline execute is 100(in step 3) , the next snap(step 4) is executing 100 times due to which if my audit table has 10 non matching records, it is throwing 1000 records output which is wrong.

3 Replies

  • nsmith's avatar
    nsmith
    New Contributor III
    8 years ago

    Hi Aditya,

    You can use the “tail” snap to limit document passthrough. It will only output 1 document, no matter how many input documents it receives. You might need this in multiple places. I think this will fix your problem.

    If you need the 100 documents combined into 1 single document, you might want to check out the “aggregate” snap.

    • aditya_sharma's avatar
      aditya_sharma
      New Contributor III
      8 years ago

      Thanks, using tail snap it worked fine. I didn’t tried the aggregate or group by n, but i am sure using these snap will also work.

  • cstewart's avatar
    cstewart
    Former Employee
    8 years ago

    Take a look at using the “Group By N” Snap at the end of your child pipeline, with a “Group Size” set to 0 (zero) it will combine all output documents into a single output document.
    Hope this helps.