Dynamically taking S3 path from metadata
Hi All,
I need to give a s3 path dynamically from redshift metadata table in S3 File writer.
Because it has csv formatter before it, not able to give redshift execute and mapper to pass the value...
Forum Discussion
Hi Shrabanti,
You can create a child pipeline that will contain at least the CSV Formater and the S3 File writer snaps. From there you can send the path value in Pipeline Parameters and you can read the path from the parameter dynamically in the created child pipeline.
Please let me know if you have any questions.
Regards,
Nikola Minovski
-
I have passed the value from parent.I can see the value in json
-
I have declared pipeline parameter in child pipeline
3.But in s3 file reader I am not getting the value I am passing.
Is it something wrong I am doing here?
Thanks & Regards,
SHrabantiHi Shrabanti, I would use the dropdown on the right and click the button on the upper right in the dropdown to visually see if the parameter(s) isn’t showing values.
If you still can’t see the value, checking the parent execute snap and child parameter to verify that there aren’t any typos.@Shrabanti As per your 1 step, only directory you have mention, should have filename as well.(because in your file write you didn’t add any filename)
In your step 2, don’t use $ac_feed_path (even you didn’t change technically shouldn’t issue)
Before step 3, use Pipeline execute snap and pass ac_feed_path value from pipeline input param.
Then it should work.
You can pass ac_feed_path to execution label so can track which file has been processed in Dashboard.
-
