Forum Discussion

mikeandrews's avatar
mikeandrews
New Contributor III
9 years ago

Passing parameters/session variables between nested pipelines/snaps

Early warning - I’m a newbie to the platform…

I want to wrap up various bits of logic/lookups/data retrieval in reusable pipelines that mask the underlying complexity (eg finding and reading a particular data source without knowing the storage protocol, access keys etc), but cant get to grips with passing metadata in and out between pipelines and snaps. Should I be passing a [control] document stream in (with 1 doc, eg a json containing key/value metadata pairs) so that i can pass onwards to the next step (and another doc stream with some actual data documents) rather than passing parameters (which I cant see a way of updating/passing out again)?

If it should be via parameters I’d need to use the Script snap to read/write these, but cant find any examples/refs for reading pipeline parameters or writing/updating parent parameters? Just input/output/error/log objects available via Script snap?

If its via a control/metadata doc is there a way to define the document schema so i can use mapper/filter etc or will I just need to handwrite some python/javascript to read/parse/do-work/create-updated-metadata-doc/self.output.write(newmetadata) ?

Thanks,
Mike

15 Replies

  • mikeandrews's avatar
    mikeandrews
    New Contributor III
    9 years ago

    thanks a lot Shayne, I will go with parameters and try using the underscore syntax to pass them out and along the workflow. will post again later with an update on how i got on 🙂

    • mikeandrews's avatar
      mikeandrews
      New Contributor III
      9 years ago

      hmm, snaps vs scripts table could be a bit more prominent in the help/docs… ie Script snap:

      Does not support pipeline _parameters
      Supports only Document input/output
      Does not support Accounts
      Often awkward to debug
      Cannot be unit tested

      I agree the Script snap is awkward to debug… :). And am surprised there’s no feature to pass in/out parameters. Some stuff cant be achieved with the standard Transform snaps, but i have to fake a document as a ScriptHook input to pass in a parameter and then try and work out the syntax with the Mapper to take the subsequent output and write that to a pipeline parameter to be used later in the flow by other snaps. I thought it would be simple/quick to do some basic js/python in a Script but looks like i may have to author a custom Snap with the SDK. Will persevere a few more hours with the Script snap…