How to audit payload in pipeline?

Hi,

You’re right, the Public API just provides access to the log data captured in Dashboard, which doesn’t include any document data from the pipelines.

One of our Snaplogic users solved this by creating a re-usable ‘Logger’ Pipeline which is invoked via a Pipeline Execute at the relevant points within their Pipelines. The Logger is very lightweight and captures some basic document data and pushes it to our centralised logging, in our case LogStash.

However we need to be careful how much we use it, particularly in production, as it adds the overhead of an additional pipeline execution every time. At the very least it should be designed so that you can ‘Reuse Executions’. We also don’t consider it good practice to capture significant amounts of document data in our logs as:

a) it will fill up log storage unnecessarily, when for most successful transactions we won’t ever look at it
b) it presents a security / privacy risk if any sensitive business data is included (e.g. employee date of birth) because logs are usually not as tightly secured

Generally we’ll just log the unique identifiers (e.g. employee ID) or if you’re using one, correlation ID.

Cheers,
C.J.

Thanks a lot CJ for your quick response.

I understood, as SnapLogic is more of a data streaming platform, the underlying philosophy doesn’t support capturing document data for auditing purpose.

In our project we’re prepping for a large implementation with around 700 + integration scenarios, which includes most of the business critical processes for the enterprise.

Depending on our typical requirements, if we still have to capture the incoming/outgoing document data, can you please suggest if for internal document processing SnapLogic follows any specific message(document) format from where document data can be accessed or using Copy snap is the only possible solution.

Regards,
Yashu Vyas

Thanks Naveen for sharing your perspective. I completely agree it’s not advisable to log each and every pipeline step as it’ll be an overhead.

My query is more to do with the auditing part of it where we want to capture the document payload at entry & exit point. Do we’ve some OOTB mechanism in SnapLogic to capture that or we’ll have to leverage Copy snap and submit it to audit pipeline for subsequent processing.

Hope the requirement is more clear now.

Regards,
Yashu Vyas

i have the same requirement … did u find any solution ?

I had a similar requirement. Request you to let me know, if there is an out of the box solution for it?

Thanks,
Srivatsa