โ07-20-2017 05:12 AM
Early warning - Iโm a newbie to the platformโฆ
I want to wrap up various bits of logic/lookups/data retrieval in reusable pipelines that mask the underlying complexity (eg finding and reading a particular data source without knowing the storage protocol, access keys etc), but cant get to grips with passing metadata in and out between pipelines and snaps. Should I be passing a [control] document stream in (with 1 doc, eg a json containing key/value metadata pairs) so that i can pass onwards to the next step (and another doc stream with some actual data documents) rather than passing parameters (which I cant see a way of updating/passing out again)?
If it should be via parameters Iโd need to use the Script snap to read/write these, but cant find any examples/refs for reading pipeline parameters or writing/updating parent parameters? Just input/output/error/log objects available via Script snap?
If its via a control/metadata doc is there a way to define the document schema so i can use mapper/filter etc or will I just need to handwrite some python/javascript to read/parse/do-work/create-updated-metadata-doc/self.output.write(newmetadata) ?
Thanks,
Mike
โ07-26-2017 08:29 AM
I can successfully pass in/out/thru parameters using the Execute Pipeline snap - I couldnt get the Child Pipeline approach to work ($content, _param, $var, $_param never came out the other end) so will just Execute Pipeline with flow control items/choices - Iโm feeling pretty confident now so thanks ๐ And will be using the suggested python syntax when in Script snaps.
Mike
โ07-27-2017 06:10 AM
Sorry to come back on this thread butโฆ
I canโt update/set a pipeline parameter via Mapper - when I type โ_mypipelineparamโ as the Target Path in a Mapper, it gets corrected to โ$_mypipelineparamโ. I finally worked out the param wasnt being set by dumping it out via a Mapper to copy param to $content then write to File Writer.
When I use a variable eg โ$myvariableโ in the Mapper, the value is set fine. Confirmed with the same dump to file flow.
But I am needing to use this dynamically calculated value (variable or parameter) as the URI/location for a File Reader snap which does accept a parameter (which I cannot set in the Mapper) but doesnt accept a variable (which I can set).
Is there a way to pass a dynamic value to File Reader? This is quite frustrating and seems inconsistent you can use $xx for some Snaps and not others, and that you canโt set _parameters in the Mapper. Again apologies for any newbie mistakesโฆ
โ02-07-2018 03:51 AM
Thank you for your information on the syntax of pipeline parameters in the script! Couldnโt find this in the documentation.
โ07-25-2017 03:56 PM
Iโll have development look at the Snaps vs Scripts table in the developer docs.
โ07-27-2017 08:10 AM
_Paramโs cannot be set using a mapper or any other means except pipeline paramโs, think of it as a global param which you set it at pipeline level, you can use it across pipelines but can be set at a global/pipeline level. Now when it comes to mapper or any other snap that lets you set variables via $varName notation, this are like local vars which can have values set via mapper and are available to the immediate following snap.
when I type โ_mypipelineparamโ as the Target Path in a Mapper, it gets corrected to โ$_mypipelineparam
and this is the expected behavior, now lets say you are getting some value from an external call, ex: you invoke a triggered task and when you do that you pass on a โfileNameโ param to this task which is then used in the pipeline as a fileName + timeStamp + .extn in your file writer snap. In that case you can use something like this in your File Writer snap settings
โ/some/project/pathโ + _fileName + $timeStamp + โ.jsonโ
where timeStamp is set via a mapper like this
mapper
you could have also created a $fileToWriteTo variable in a mapper which will have this expression
โ/some/project/pathโ + __fileName + โ_โ + Date.now().toLocaleDateTimeString() + โ.jsonโ
Hope it makes sense!