Capture runtime document in ultra pipeline for debugging purpose

[eguo]

Sometimes you want to capture input or output document of snaps at runtime in ultra pipeline for debugging purpose. Here’s a quick way:

1. Don’t use File Writer Snap as it break the lineage in ultra pipeline
2. Use a Pipeline Execute Snap
3. Send the logging content as one or multiple parameter to the child pipeline
4. Make sure the child pipeline do nothing heavy weight
5. Make sure set the child pipeline capture the parameter in question
6. View the captured parameter in Dashboard
7. You can copy the captured parameter and paste it to a JSON pretty print tool for better viewing

[vyasyash]

Hello Eguo,

Thanks for suggesting the way to capture the input/output document of snaps.

I’m new to SnapLogic world, and come from traditional ESB background. Wanted to understand if SnapLogic provides any out of the box feature to capture the input & output document for auditing and logging purpose (both in case of synchronous and async scenario). In production scenarios input/out payload will come handy to conduct RCA of the failed interface pipeline.

Thanks in advance.

Best,
Yashu Vyas

[eguo]

@vyasyash, welcome to the post-ESB paradigm!

No there is no out of box feature to persist messages like in traditional ESB, hence created the approach in this post to enable troubleshooting activities.

Benefits of this approach:

• It avoids introducing new I/O endpoints into the pipeline. As a result, the pipeline has less state to manage in the runtime.
• You could use Pipeline Monitoring API to consume the captured information to satisfy certain use cases.

Limitations:

• Only the last rolling hundred sub-pipeline’s runtime logs will be kept by SL platform therefore visible in the dashboard and accessible via the Pipeline Monitoring API.
• A runtime log will be kept in SL platform for 90 days.
• Can not enable the reuse in the pipeExec snap.

Eddy

[vyasyash]

Thanks Eddy for your response.

If for some specific business requirement we’ve to build a reusable pipeline for payload auditing purpose, do you suggest to use the Copy snap to capture the payload or SnapLogic follows a proprietary message structure for internal processing from where document data can be directly accessed ?

Best,
Yashu Vyas

[krupalibshah]

@eguo

Do you know like we have it in the typical ESB system, for error handling. If the error occurred at say 5th step then it would roll back and end up at the 1st step where we generally handle them.

However it does not seem to be same in SnapLogic. Hence we need to have error handling at each step?

/Krupali