I’ve two pipelines, let’s call them Parent Pipeline and Child Pipeline:
In the Child pipeline, I am using Snowflake Upsert snap like below:
The key column i.e.
_KEY_COLUMN is passed in Parent Pipeline via Execute Pipeline snap parameters.
Now, I want to add multiple key columns i.e. consider below image:
but the trick is I want to be able to decide which one gets invoked i.e. is there a way I can say from the parent that _KEY_COLUMN_1 is not required?
Now, there can ofcourse be n no. of key columns, but want to start with a finite (say 4) at the moment.
More context about the nature of the pipelines:
- the child pipeline is basically a generic pipeline which accepts a dataset and tablename which it upserts into and also creating the audit entries
- the parent pipeline can be any other pipeline which is responsible for a specific dataset and when it comes to upserting that in SF, it calls the child pipeline.
But some datasets may have 1 key column, some can have 2, some can have more. How can I control this generically?
Any suggestions is appreciated.