Data reconciliation solutions?
One of my company's use cases for SnapLogic today is replication of data from Salesforce into internal Kafka topics for use throughout the enterprise. There have been various instances of internal c...
Forum Discussion
What would be the issue if you use SystemModStamp as your state.
e.g.
Assuming Data Warehouse is one of the consumer
Data Warehouse : select Max(SystemModStamp) > SFDC > REST SOQL where SystemModStamp > $SystemModStamp via SL > Kafka Producer > Data Warehouse
I'm assuming the above response assumes some sort of custom data reconciliation happening within the data warehouse?
If so, a custom reconciliation is one option being considered, and would potentially be similar to what you've described.
However, the intention of this thread was to seek recommendations for any existing offering (commercial or open source) that can perform this type of reconciliation.
To the point of SystemModstamp specifically, it would not work for all objects.
1.) Not every object being replicated / needing reconciliation has SystemModstamp
2.) For objects that have SystemModstamp, it's almost never used as part of the query for the existing replication architecture
A reason for this is that updates in Salesforce having nothing to do with a change in data can result in SystemModstamp being updated on records.
During validation of the current replication architecture when SystemModstamp was used for replication, I observed a situation where retrieval was being attempted for tens of millions of records for an object when there was no update in the actual data that would have been replicated.
