Data reconciliation solutions?
One of my company's use cases for SnapLogic today is replication of data from Salesforce into internal Kafka topics for use throughout the enterprise. There have been various instances of internal c...
Forum Discussion
Hi feenst,
Different data replication strategies can be implemented based on business requirements, whether the need is for real-time, near-real-time, or scheduled synchronization.
For scheduled data replication, SnapLogic provides a straightforward solution using the Salesforce Snappack. Data can be retrieved from any Salesforce object by retrieving data based on LastModifiedDate (or SystemModstamp) field. At the end of each sync cycle, the most recent timestamp can be stored and used in the next cycle to fetch only those records that have been modified since the last successful run.
For real-time or near-real-time replication, in addition to the Change Data Capture (CDC) feature, there can be used Outbound Messages in Salesforce. Outbound Messages can be configured to trigger SnapLogic tasks. In this case, Ultra Tasks are ideal for real-time processing due to their low-latency response time and built-in retry capabilities. Similarly, HTTP Callouts from Salesforce can also be used to invoke SnapLogic tasks, typically implemented via @future methods or queueable Apex classes to ensure proper bulkification and asynchronous execution.
We’ve successfully implemented these approaches multiple times and found them to be reliable and scalable.
If this aligns with your needs, I’d be happy to set up a call to walk through the different architecture options for replicating Salesforce data using SnapLogic.
Best regards, Lazo.
Hi lazo_ilijoski,
Thanks for the reply. My company today uses the following approach for replication:
"For scheduled data replication, SnapLogic provides a straightforward solution using the Salesforce Snappack. Data can be retrieved from any Salesforce object by retrieving data based on LastModifiedDate (or SystemModstamp) field. At the end of each sync cycle, the most recent timestamp can be stored and used in the next cycle to fetch only those records that have been modified since the last successful run."
I am also working with our Salesforce IT team on a PoC using Change Data Capture.
However, my original question isn't around data reconciliation / auditing solutions.
Do you have any experience with data reconciliation / auditing solutions that support ingesting data from Salesforce and Kafka?
