Fetch data whose date is less than 10hrs from the existing date!

[darshthakkar]

Hi Team,

I’m trying to achieve a filter condition but haven’t found any luck so far. My data has a field as Last_Updated which has date stored in the format 2022-06-23 03:54:45

I want to consider only those records whose date is less than 10 hours than the existing date. How can I achieve this? Shall I use a mapper or a filter snap and I would really appreciate if the logic behind this can be shared.

If the format of the date that is stored in the Last_Updated needs to be transformed as well, please let me know.
Thanks!

Regards,
Darsh

[bojanvelevski]

In order to exclude duplicates, you’ll have to use a Diff mechanism. Practically, when get records that are updated in the last 48 hours, you need to compare them with Salesforce records added in the last 48 hours by using a Diff snap.

[darshthakkar]

Thank you @bojanvelevski, cheked Diff snap, wasn’t aware of this up until now. It suggests that the input streams should be sorted, I will have to use a sorter as my entire pipeline was unsorted.

[bojanvelevski]

Yes @darshthakkar, you’ll have to sort the streaming data on the same field you’ll differentiate on.

[darshthakkar]

@bojanvelevski: From the documentation on Diff snap, it can be understood that the input documents should be flat files. For my use case, I will have to compare 2 different systems, one is snowflake and another is Salesforce plus after the comparison I need to update Salesforce.

Does Diff snap allow us to update records to Salesforce if we put a salesforce upsert/salesfroce update after Modified path of Diff?

Thanks!

[koryknick]

@darshthakkar - I would recommend carefully considering if the Diff snap is really what you want to use. The Diff snap compares the incoming JSON documents as a whole for equivalence - if anything in the objects are different, it considers it as modified. This includes everything from a difference in the value of a given element, the order of the elements within the object, missing / added elements, element name case sensitivity, or even differences in datatypes.

I think what you probably want to consider instead is the use of a Join snap - probably using Left Outer join type - to check the existence of the target record and apply insert/update methodology based on that condition.

Also, rather than opening a new discussion on a closed topic, please open a new thread so the information shared is under a logical heading for other members to find more easily.