Data reconciliation solutions?
One of my company's use cases for SnapLogic today is replication of data from Salesforce into internal Kafka topics for use throughout the enterprise. There have been various instances of internal c...
Forum Discussion
Hi Scott,
Thanks for the response.
My organization is not using the CDC API today. We are using a REST and Bulk APIs to replicate data. Salesforce support and solution architects have recommended CDC instead of our current architecture.
Do you have more information about the usability, reliability, and scalability limitations of CDC?
Also, can you advise on what the new API is that Salesforce has introduced?
Regarding handling the missed events in Kafka, my organization uses Kafka to allow multiple internal applications to consume updates from Salesforce for objects of interest.
My team is responsible for the replication (today using SnapLogic) as well as the Kafka platform. We are interested in validating that all of the expected records from Salesforce have successfully reached Kafka.
If a record has not reached Kafka, we are interested in triggering an automated process to retrieve the missing record(s) and ensure that all of the latest, expected records have reached Kafka.
So, I am seeking input from the community of any tools that might support this use case. If the tool requires data to be extracted from Kafka and into a traditional database for the reconciliation / audit with Salesforce, this would be fine.
What would be the issue if you use SystemModStamp as your state.
e.g.
Assuming Data Warehouse is one of the consumer
Data Warehouse : select Max(SystemModStamp) > SFDC > REST SOQL where SystemModStamp > $SystemModStamp via SL > Kafka Producer > Data Warehouse
I'm assuming the above response assumes some sort of custom data reconciliation happening within the data warehouse?
If so, a custom reconciliation is one option being considered, and would potentially be similar to what you've described.However, the intention of this thread was to seek recommendations for any existing offering (commercial or open source) that can perform this type of reconciliation.
To the point of SystemModstamp specifically, it would not work for all objects.
1.) Not every object being replicated / needing reconciliation has SystemModstamp
2.) For objects that have SystemModstamp, it's almost never used as part of the query for the existing replication architecture
A reason for this is that updates in Salesforce having nothing to do with a change in data can result in SystemModstamp being updated on records.
During validation of the current replication architecture when SystemModstamp was used for replication, I observed a situation where retrieval was being attempted for tens of millions of records for an object when there was no update in the actual data that would have been replicated.
