โ08-24-2020 08:30 AM
Hey All,
So Iโve been trying to develop a pipeline that reads a bunch of files from an s3 bucket, checks which ones are no more than a week old, and then sends those to be parsed and eventually uploaded into snowflake.
Iโve already been able to read a single csv file from my s3 bucket, parse it, and insert it into snowflake. Having trouble now with setting up a file filter within the directory browser snap to only send me the files that are a week old or younger.
Any ideas on what filter expression I can use?
โ08-24-2020 08:55 AM
Hi @amubeen,
One possible solution would be if you list all the files from s3 bucket using the Directory browser snap. As an output you will get the creation date of the files. Then by using the filter snap you can filter the files that are week old or younger(ex: $[โCreation dateโ] >= Date.now().minusDays(7)).
Regards,
Spiro Taleski
โ08-24-2020 10:55 AM
If you use the Directory Browser to get the list of files, you can then use a Filter snap downstream to filter based on your date selection criteria. I donโt know that there is any way to push the selection down to S3 in the directory browser.