Reading multiple files from ftp and loading it in snowflake table

[pritam605]

I am trying to create and load a snowflake table based on multiple csv files that are stored in a ftp location.

As of now, my approach looks like -

Step 1: parent pipeline 

File poller -》 Pipeline Execute

(Created an entry for pipeline parameter with name as "filepath" and parameter value as "$path")

Step 2 -》 child pipeline 

Zip file read -》 csv parser - 》 mapper - 》 snowflake bulk load

(Created an entry for pipeline parameter with key as "filepath" and parameter value as "$_pipeline_parameters.filepath")

In the zipfile read snap, i meed to fill value for the "File*" section. As of now i am filling it with the value "_filepath" but it throws error aayinf "unable to read from $pipeline_parameters.filepath. Reason: File not found on ............$pipeline_parameters.filepath. check for url syntax and file access permission. 

IIin the parent pipeline, file poller is working fine and i am getting a list of 10 diff zip files as $path. Not sure, what is going wrong after that. 

Please assist. Not sure waht is going wrong here.

[Ranjith]

Hi @pritam605 ,

It appears that the _filepath, based on the provided context and the attached image, is not carrying the correct SFTP path for the CSV file. The Zip FileReader requires the SFTP path for each CSV file specified in the file settings at the snap level (e.g., sftp://sample.sftp.com/folder/filename). Currently, based on the snap configuration, it is looking for the file at the SLDB level instead of the correct SFTP path.