How to load Today Date Files from s3 Bucket to SLDB

Hi @sathishmadavali,

One way to filter only today’s files is to browse the S3 Bucket using the S3 Browser and after that to use a Filter Snap with the following expression:

Date.parse($lastModified) >= Date.parse(Date.now().toLocaleDateString())

If your file pool in S3 is growing at increasing rate you might consider another approach.

Let me know if this helps you.

BR,
Aleksandar.

@AleksandarAngelevski ,THANK YOU

For your reference, Date.now().minusDays(1).toLocaleDateString()
Try to incorporate this in your pipeline and test it out.

Regards,
Darsh

@sathishmadavali : You can use 2 snaps over here:

1. Multi-File Reader and provide the path ensuring that the wildcard only takes into consideration the format you’re looking for (e.g. csv). You will have to specify multiple paths if you’re wanting to read multiple files.

2. Using router → if you go into views, you can add additional views for which you can write different functions like $objectKey == “testing.csv”, second function would be $objectKey == “testing.json” so on and so forth. This works if you’re having $objectKey in your upstream.

Moreover, I would also urge to open a new topic as we have branched out from reading today’s file to separating files. Thankyou.

CC: @dmiller @AleksandarAngelevski

@sathishmadavali - I have created a new topic: Routing files from S3

@sathishmadavali - Please mark the solution provided by @AleksandarAngelevski as “solution” so that this thread can be closed.

You had to tweak a little bit, are you having $lastModified in your upsteam?