Passing parameters/session variables between nested pipelines/snaps

[mikeandrews]

Early warning - I’m a newbie to the platform…

I want to wrap up various bits of logic/lookups/data retrieval in reusable pipelines that mask the underlying complexity (eg finding and reading a particular data source without knowing the storage protocol, access keys etc), but cant get to grips with passing metadata in and out between pipelines and snaps. Should I be passing a [control] document stream in (with 1 doc, eg a json containing key/value metadata pairs) so that i can pass onwards to the next step (and another doc stream with some actual data documents) rather than passing parameters (which I cant see a way of updating/passing out again)?

If it should be via parameters I’d need to use the Script snap to read/write these, but cant find any examples/refs for reading pipeline parameters or writing/updating parent parameters? Just input/output/error/log objects available via Script snap?

If its via a control/metadata doc is there a way to define the document schema so i can use mapper/filter etc or will I just need to handwrite some python/javascript to read/parse/do-work/create-updated-metadata-doc/self.output.write(newmetadata) ?

Thanks,
Mike

[shodge]

If you have sample pipelines, seeing those would be helpful, if possible.

While the exact answer is going to depend on application, I’d generally say to use documents if you are going to be using Pipeline Execute (https://doc.snaplogic.com/wiki/display/SD/Pipeline+Execute), since with reuse turned on (e.g., you spin up the child pipeline once, instead of spinning it up and tearing it down per document) the pipeline parameters can only be set once.

You should be able to access pipeline parameters in the Mapper by using an underscore instead of a dollarsign - i.e. $foo if it’s a variable, _foo if it’s a parameter. If you need the parameters returned, you can add a Mapper at the end, check “Pass Through”, and then add in the parameters you want to return using this syntax.

Is that along the lines of what you’re looking for?

Shayne

[mikeandrews]

thanks a lot Shayne, I will go with parameters and try using the underscore syntax to pass them out and along the workflow. will post again later with an update on how i got on 🙂

[mikeandrews]

hmm, snaps vs scripts table could be a bit more prominent in the help/docs… ie Script snap:

Does not support pipeline _parameters
Supports only Document input/output
Does not support Accounts
Often awkward to debug
Cannot be unit tested

I agree the Script snap is awkward to debug… :). And am surprised there’s no feature to pass in/out parameters. Some stuff cant be achieved with the standard Transform snaps, but i have to fake a document as a ScriptHook input to pass in a parameter and then try and work out the syntax with the Mapper to take the subsequent output and write that to a pipeline parameter to be used later in the flow by other snaps. I thought it would be simple/quick to do some basic js/python in a Script but looks like i may have to author a custom Snap with the SDK. Will persevere a few more hours with the Script snap…

[Bhavin]

“Does not support pipeline _parameters” - on the contrary Script snap does support pipeline params

It can be accessed in a script snap by $_pipeline_param_name

Example:

Python

        data["pipelineparam"] = $_pipeline_param

JavaScript

         new_data.pipelineparam = $_pipeline_param;

also documented over here - https://doc.snaplogic.com/wiki/display/SD/Parameters+and+Fields