Forum Discussion

amubeen's avatar
amubeen
New Contributor II
5 years ago

Getting list of csv files through directory browser and parsing by date

Hey All,

So I’ve been trying to develop a pipeline that reads a bunch of files from an s3 bucket, checks which ones are no more than a week old, and then sends those to be parsed and eventually uploaded into snowflake.

I’ve already been able to read a single csv file from my s3 bucket, parse it, and insert it into snowflake. Having trouble now with setting up a file filter within the directory browser snap to only send me the files that are a week old or younger.

Any ideas on what filter expression I can use?

2 Replies

  • SpiroTaleski's avatar
    SpiroTaleski
    Valued Contributor
    5 years ago

    Hi @amubeen,

    One possible solution would be if you list all the files from s3 bucket using the Directory browser snap. As an output you will get the creation date of the files. Then by using the filter snap you can filter the files that are week old or younger(ex: $[‘Creation date’] >= Date.now().minusDays(7)).

    Regards,
    Spiro Taleski

  • cstewart's avatar
    cstewart
    Former Employee
    5 years ago

    If you use the Directory Browser to get the list of files, you can then use a Filter snap downstream to filter based on your date selection criteria. I don’t know that there is any way to push the selection down to S3 in the directory browser.