Referencing an earlier Snap
Hello.
I'm trying to pass a JSON as a pipeline parameter, and I've prefixed my pipeline with a Mapper Snap to "parse" the JSON and the output of the Mapper is a JSON object I can reference later in my pipeline. A sample of this JSON is such:
{
"extract": {
"source": {
"schema": "public",
"table": "table",
"account": "../../shared/source-dev",
"pipeline_name": "projects/shared/query_extract",
"load_type": {
"full-load": {}
},
"target": {
"s3_account": ",../../shared/s3",
"target_bucket": "dev-bucket",
"target_path": "snaplogic_test.csv"
}
}
}
}
And here is the Mapper (the output is "input_params"):
I've reduced my pipeline to the minimum pipeline where I still have this problem.
I have my Mapper mentioned above, a Pipeline Execute to query against the source, a Router for doing different actions based on if there are results, and if so a Pipeline Execute to write that data to S3. I think I'm missing a few things like parsing documents into a JSON or whatever for s3, but I'll cross that bridge when I get there.
The problem I am running into is that in the router and pipeline execution, I can reference "$input_params.etc" in the expression logic, but once I am past that I can no longer reference it.
So my questions are:
- How can I reference that mapper output "input_params" for my s3 pipeline?
- Is there a better way to take a single JSON parameter and make its parts available throughout the entire pipeline?
Ah, I don't know how I missed this. So I was able to do something like "JSON.parse(_input_params).foo.bar.bin". It's the same step as was in the mapper.
I'm not sure if this is more efficient than having a "set variable" function, but since that's not possible I think this will suffice.