cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Keep data "through" snaps which don't provide original input under $original in their output

JoeDyndale
New Contributor

SnapLogic isnโ€™t consistent in how data is passed through various snaps. Itโ€™s often necessary to keep some data obtained early in the flow (e.g. from the source) for use later (e.g. when writing to target). However, some snaps, like the XML Parser, requires that only the data to be parsed is passed as input to it while also not supporting binary headers or such mechanisms to forward data from one side of the snap to the other - effectively removing everything except the data it cares about from the stream.

Thereโ€™s an enhancement request for fixing this posted here somewhere, and weโ€™ve written about this to our SnapLogic contacts, so hopefully the following work-around wonโ€™t be necessary for very long, but here it is:

Move the โ€œproblemโ€ snap to a child pipeline and call it via the Pipeline Execute snap, making sure โ€œReuse executions to process documentsโ€ is not checked (wonโ€™t work if it is). If needed, at the start of the child pipeline, remove any data not to be used with the โ€œproblemโ€ snap. The Pipeline Execute snap will output the original input data under $original (as the โ€œproblem snapโ€ should have done).

1 REPLY 1

joel_bourgault
New Contributor II

Hello,

Very useful, thanks!

An addition though: `original` is appended if both following conditions are met:

  • "Reuse executions to process documents" is *not* checked
  • "Batch size" is 1

However, this comes at the prize of a significantly slower execution. Itโ€™s still possible to set Pool Size though.

Too bad thereโ€™s no "pass-through" option for Pipeline Execute snap.