We are looking at creating a reusable pipeline to load data from table1 from source DB to table1 in target DB. Plan is to reuse the same pipeline for n tables.
Option 1 -
Created master pipeline and child pipeline. Master pipeline gets table name list from source DB.
Then uses mapper and foreach snap to execute the child pipeline.
Child pipeline loads table from source DB to target DB.
Issue- Unable to use the parameter/variable value from master pipeline-foreach snap to child pipeline -snowflake select/execute snap.
Anyone faced similar error or has recommendation for the approach on reusable pipeline.
Why not have the output on Master pipeline be a document list where each document is a command. Then the child pipe runs 1 command, they will all get called and the output will results in ach documents which can be processed separately or merged into 1 doc
Thanks for your input @jcornelius.
Do you have any sample pipeline with it? I am new to Snaplogic so appreciate if you could share any reference.
Tried passing dynanic values from parent to child pipeline and it is working fine. But the child pipeline during validation shows error but runs fine when dynamic values are passed from master pipeline
Is this expected?
Please see the reference screenshots.
You need to set _name if you are going to use “Pipeline Parameter”
You can use “Document Parameters” and they are referenced with $. Lokk at the output of Snowflake - Execute by clicking the Red doc icon, this will show into the next snap.
ie is output is “tableName”: “table1” then change select to "select * from " $tableName
Thanks @AleksandarAngelevski , @jcornelius , @bojanvelevski
Not sure on how to use document parameters. Tried with pipeline parameters and assigned default values for pipeline parameter to get rid of validation error in child pipeline. During execution, child pipeline parameter default values got over written by values passed by parent pipeline.