Early warning - I’m a newbie to the platform…
I want to wrap up various bits of logic/lookups/data retrieval in reusable pipelines that mask the underlying complexity (eg finding and reading a particular data source without knowing the storage protocol, access keys etc), but cant get to grips with passing metadata in and out between pipelines and snaps. Should I be passing a [control] document stream in (with 1 doc, eg a json containing key/value metadata pairs) so that i can pass onwards to the next step (and another doc stream with some actual data documents) rather than passing parameters (which I cant see a way of updating/passing out again)?
If it should be via parameters I’d need to use the Script snap to read/write these, but cant find any examples/refs for reading pipeline parameters or writing/updating parent parameters? Just input/output/error/log objects available via Script snap?