Child and Parent pipeline having count issue

When I am executing the parent pipeline the child pipeline has a different count and in the parent, the pipeline has a different count.

child pipeline execution :

parent pipeline execution (pipeline execute snap)

The child and parent pipeline don’t have any filter or router.
Child pipeline

Parent :slight_smile:

Hi @Rahul,

Can you check how much documents were processed in the child pipeline in the “CSV Parser” snap ? I can see that there is 1 input document that goes into the child pipeline which then the child generates 16,797 documents on the output but 16,838 document were processed. Did you check if there were any errors ? I can see that you have an error view on your pipeline execute snap, maybe some of the documents that were generated failed during the csv parser process or during the REST Get API ?

1 Like

@j.angelevski In child pipeline CSV parser process 16838 records.

@ljupcho_machkovski @j.angelevski @dmiller There is the snaplogic bug in pipeline execute snap .?

@Rahul

As an addition to @j.angelevski 's post. You probably have some errors in the CSV data, so your pipe execute snap forwards those to the error view, and sends the rest to output. You can validate that just by hovering with your mouse on the Generate Report snap after execution, or click the Pipeline Statistics icon in the toolbar and check if something went to the error view on those particular snaps.

@bojanvelevski There are no records that went in error view.

Hi @Rahul ,

Did you check for duplicate rows (data) while processing in child pipeline?
Also, are you ignoring empty data in child pipeline in CSV Parser?

Maybe in Parent pipeline, the CSV Formatter is set to ignore empty stream, so you see different count values.

BR,
Marjan

Can you please provide a link to the parent execution?

@marjan.karafiloski in CSV parser I am using ignore empty data and child pipeline and pipeline execute have a different count.

child :

pipeline execute in parent pipeline :

@ptaylor Can you please explain me “Can you please provide a link to the parent execution?”

Highlighted pipeline execute calling same child pipeline.

Open the completed pipeline execution. Then copy the URL from the browser tab and paste it here.

pipeline runtime id: SnapLogic Designer

Thanks. I don’t have a good explanation for the discrepancy between the number of output documents reported by the child (16838) vs the Pipe Exec that executed that child (16797). I think there might be a bug as you suggested. I suggest opening a ticket with SnapLogic Support about this. Please include this info:

  • Parent: pipe_runtime_id=58b7a2d080b28239d980b31f_e742d951-e350-4805-91e6-a37ca471ab18
  • Child: pipe_runtime_id=58b7a2d080b28239d980b31f_1f5879d3-d325-4949-ac78-bef6d0d40361
  • Slack (internal discussion thread)

Also, please note that this child execution was actually the 4th attempt. There were 3 failed attempts of that child pipeline in the few minutes before the one that succeeded where the REST Get failed with a 404. Here’s one of them:
https://elastic.snaplogic.com/sl/designer.html#pipe_runtime_id=58b7a2d080b28239d980b31f_c6780f37-34ce-4191-a801-5cf202449d7e

I don’t know whether these failures are related to the document count discrepancy for the child that succeeded.