Forum Discussion

Shrabanti's avatar
Shrabanti
New Contributor II
5 years ago

Dynamically taking S3 path from metadata

Hi All, I need to give a s3 path dynamically from redshift metadata table in S3 File writer. Because it has csv formatter before it, not able to give redshift execute and mapper to pass the value...
sample_pipeline
Shrabanti's avatar
Shrabanti
New Contributor II
5 years ago

  1. I have passed the value from parent.I can see the value in json

  2. I have declared pipeline parameter in child pipeline

    3.But in s3 file reader I am not getting the value I am passing.

Is it something wrong I am doing here?

Thanks & Regards,
SHrabanti

Supratim's avatar
Supratim
Contributor III
5 years ago

@Shrabanti As per your 1 step, only directory you have mention, should have filename as well.(because in your file write you didn’t add any filename)
In your step 2, don’t use $ac_feed_path (even you didn’t change technically shouldn’t issue)
Before step 3, use Pipeline execute snap and pass ac_feed_path value from pipeline input param.
Then it should work.
You can pass ac_feed_path to execution label so can track which file has been processed in Dashboard.