Data reconciliation solutions?

[feenst]

One of my company's use cases for SnapLogic today is replication of data from Salesforce into internal Kafka topics for use throughout the enterprise.

There have been various instances of internal consumers of the Kafka data reporting missing records.

Investigations have found multiple causes for these data drops. Some of the causes are related to behavior that Salesforce describes as "Working As Designed".

Salesforce has recommended other replication architectures, but there are various concerns with my company about using them (license cost, platform load) ... and we might still end up with missing data.

So, we're looking into data reconciliation / auditing solutions. Are there any recommendations on a tool that can:

* Identify record(s) where the record in Salesforce does not have a matching record (e.g. same timestamp) existing in Kafka

* Generate a message containing relevant metadata (e.g. record Id, Salesforce object, Kafka topic) to be sent to a REST endpoint / message queue for reprocessing

[lazo_ilijoski]

Hi @feenst,

Different data replication strategies can be implemented based on business requirements, whether the need is for real-time, near-real-time, or scheduled synchronization.

For scheduled data replication, SnapLogic provides a straightforward solution using the Salesforce Snappack. Data can be retrieved from any Salesforce object by retrieving data based on LastModifiedDate (or SystemModstamp) field. At the end of each sync cycle, the most recent timestamp can be stored and used in the next cycle to fetch only those records that have been modified since the last successful run.

For real-time or near-real-time replication, in addition to the Change Data Capture (CDC) feature, there can be used Outbound Messages in Salesforce. Outbound Messages can be configured to trigger SnapLogic tasks. In this case, Ultra Tasks are ideal for real-time processing due to their low-latency response time and built-in retry capabilities. Similarly, HTTP Callouts from Salesforce can also be used to invoke SnapLogic tasks, typically implemented via @future methods or queueable Apex classes to ensure proper bulkification and asynchronous execution.

We’ve successfully implemented these approaches multiple times and found them to be reliable and scalable.

If this aligns with your needs, I’d be happy to set up a call to walk through the different architecture options for replicating Salesforce data using SnapLogic.

Best regards, Lazo.

[feenst]

Hi @lazo_ilijoski,

Thanks for the reply. My company today uses the following approach for replication:

"For scheduled data replication, SnapLogic provides a straightforward solution using the Salesforce Snappack. Data can be retrieved from any Salesforce object by retrieving data based on LastModifiedDate (or SystemModstamp) field. At the end of each sync cycle, the most recent timestamp can be stored and used in the next cycle to fetch only those records that have been modified since the last successful run."

I am also working with our Salesforce IT team on a PoC using Change Data Capture.

However, my original question isn't around data reconciliation / auditing solutions.

Do you have any experience with data reconciliation / auditing solutions that support ingesting data from Salesforce and Kafka?