Passing parameters/session variables between nested pipelines/snaps

[mikeandrews]

Early warning - I’m a newbie to the platform…

I want to wrap up various bits of logic/lookups/data retrieval in reusable pipelines that mask the underlying complexity (eg finding and reading a particular data source without knowing the storage protocol, access keys etc), but cant get to grips with passing metadata in and out between pipelines and snaps. Should I be passing a [control] document stream in (with 1 doc, eg a json containing key/value metadata pairs) so that i can pass onwards to the next step (and another doc stream with some actual data documents) rather than passing parameters (which I cant see a way of updating/passing out again)?

If it should be via parameters I’d need to use the Script snap to read/write these, but cant find any examples/refs for reading pipeline parameters or writing/updating parent parameters? Just input/output/error/log objects available via Script snap?

If its via a control/metadata doc is there a way to define the document schema so i can use mapper/filter etc or will I just need to handwrite some python/javascript to read/parse/do-work/create-updated-metadata-doc/self.output.write(newmetadata) ?

Thanks,
Mike

[mikeandrews]

Sorry my point was that File Reader doesn’t accept a $variable parameter, so I am at a dead end as I can’t write a value to the pipeline _parameter either which it does accept.

[Bhavin]

File reader does accetpt $ and _ variables, did you toggle the = button ?

toggle-me

[del]

@Bhavin, I think what Mike is saying is that the File Reader has a binary input which requires a binary formatter of some sort which does not pass through the document variables that would otherwise be available to the File Reader if it had a JSON input.

@mikeandrews, I’ve run into the same issue and the only way I’ve been able to get this to work in the past is with two pipelines. The first ending with a Pipeline Execute or For Each snap which would assign the filename variable value to a parameter provided by the second pipeline which contained the File Reader snap.

For the same or similar use case, a couple of years ago, I had suggested to a SnapLogic representatives the need to make pipeline parameters variable rather than static. I even suggested a new variable array could be added as a property to the pipe object. However, this was an in-person, casual conversation and may not have made it to the request queue.

[Added: I have made corrections to this post in my following response]

[tstack]

del:

@Bhavin, I think what Mike is saying is that the File Reader has a binary input which requires a binary formatter of some sort which does not pass through the document variables that would otherwise be available to the File Reader if it had a JSON input.

Binary documents have a “header” with properties that can be referenced using dollar variables. When looking at the preview data for a binary view in Designer, you should be able to see what properties are available, if any. Unfortunately, most Formatter snaps don’t have a way to fill in the header document with properties, which limits the possibilties. It’s a gap that needs to be addressed.

del:

For the same or similar use case, a couple of years ago, I had suggested to a SnapLogic representatives the need to make pipeline parameters variable rather than static. I even suggested a new variable array could be added as a property to the pipe object. However, this was an in-person, casual conversation and may not have made it to the request queue.

They are effectively variable when used through PipeExec, as you described, and that is the correct approach at this time. The reason parameters cannot be set is because all of the snaps are running in parallel. So, it’s impossible to say what the value of a parameter is at any point is if they can be changed. For example, if you had a Mapper setting a pipeline parameter followed by a JSON-Formatter and a File-Writer, the Mapper could have processed one to hundreds of documents before the Writer read the value of the pipeline parameter. Now, imagine if the value computed for the parameter changed for every document that it processed, you would never get consistent results.

[del]

I should apologize and make some corrections after re-reading the above… My last comment probably didn’t make much sense because my use case was around the File Writer instead of the File Reader, so it didn’t fit this post. Nevertheless, now that I’ve accidentally hijacked the post and the conversation has continued… 🙂

@mikeandrews, I agree @Bhavin’s previous post should resolve your issue. What has not been included (if it’s not obvious) is that you can configure the File Reader snap with an input view so that the variables can be passed through.

Now, back to my unintentional hijack… @tstack,

tstack:

The reason parameters cannot be set is because all of the snaps are running in parallel. So, it’s impossible to say what the value of a parameter is at any point is if they can be changed. For example, if you had a Mapper setting a pipeline parameter followed by a JSON-Formatter and a File-Writer, the Mapper could have processed one to hundreds of documents before the Writer read the value of the pipeline parameter. Now, imagine if the value computed for the parameter changed for every document that it processed, you would never get consistent results.

Thank you very much for that explanation; that makes complete sense!

I’ve personally not run into any other use case other than the above mentioned, so if the Formatter snap can be extended in some way to pass along variable header info, then the issue is solved (and I can reduce a small number of secondary pipelines).

Thanks
- Del