Passing parameters/session variables between nested pipelines/snaps

[mikeandrews]

Early warning - I’m a newbie to the platform…

I want to wrap up various bits of logic/lookups/data retrieval in reusable pipelines that mask the underlying complexity (eg finding and reading a particular data source without knowing the storage protocol, access keys etc), but cant get to grips with passing metadata in and out between pipelines and snaps. Should I be passing a [control] document stream in (with 1 doc, eg a json containing key/value metadata pairs) so that i can pass onwards to the next step (and another doc stream with some actual data documents) rather than passing parameters (which I cant see a way of updating/passing out again)?

If it should be via parameters I’d need to use the Script snap to read/write these, but cant find any examples/refs for reading pipeline parameters or writing/updating parent parameters? Just input/output/error/log objects available via Script snap?

If its via a control/metadata doc is there a way to define the document schema so i can use mapper/filter etc or will I just need to handwrite some python/javascript to read/parse/do-work/create-updated-metadata-doc/self.output.write(newmetadata) ?

Thanks,
Mike

[mikeandrews]

I can successfully pass in/out/thru parameters using the Execute Pipeline snap - I couldnt get the Child Pipeline approach to work ($content, _param, $var, $_param never came out the other end) so will just Execute Pipeline with flow control items/choices - I’m feeling pretty confident now so thanks 🙂 And will be using the suggested python syntax when in Script snaps.
Mike

[mikeandrews]

Sorry to come back on this thread but…

I can’t update/set a pipeline parameter via Mapper - when I type “_mypipelineparam” as the Target Path in a Mapper, it gets corrected to “$_mypipelineparam”. I finally worked out the param wasnt being set by dumping it out via a Mapper to copy param to $content then write to File Writer.

When I use a variable eg “$myvariable” in the Mapper, the value is set fine. Confirmed with the same dump to file flow.

But I am needing to use this dynamically calculated value (variable or parameter) as the URI/location for a File Reader snap which does accept a parameter (which I cannot set in the Mapper) but doesnt accept a variable (which I can set).

Is there a way to pass a dynamic value to File Reader? This is quite frustrating and seems inconsistent you can use $xx for some Snaps and not others, and that you can’t set _parameters in the Mapper. Again apologies for any newbie mistakes…

[nisars]

Thank you for your information on the syntax of pipeline parameters in the script! Couldn’t find this in the documentation.

[dmiller]

I’ll have development look at the Snaps vs Scripts table in the developer docs.

[Bhavin]

_Param’s cannot be set using a mapper or any other means except pipeline param’s, think of it as a global param which you set it at pipeline level, you can use it across pipelines but can be set at a global/pipeline level. Now when it comes to mapper or any other snap that lets you set variables via $varName notation, this are like local vars which can have values set via mapper and are available to the immediate following snap.

when I type “_mypipelineparam” as the Target Path in a Mapper, it gets corrected to “$_mypipelineparam

and this is the expected behavior, now lets say you are getting some value from an external call, ex: you invoke a triggered task and when you do that you pass on a “fileName” param to this task which is then used in the pipeline as a fileName + timeStamp + .extn in your file writer snap. In that case you can use something like this in your File Writer snap settings

‘/some/project/path’ + _fileName + $timeStamp + ‘.json’

where timeStamp is set via a mapper like this

mapper

you could have also created a $fileToWriteTo variable in a mapper which will have this expression

‘/some/project/path’ + __fileName + ‘_’ + Date.now().toLocaleDateTimeString() + ‘.json’

Hope it makes sense!