One of my company's use cases for SnapLogic today is replication of data from Salesforce into internal Kafka topics for use throughout the enterprise.
There have been various instances of internal consumers of the Kafka data reporting missing records.
Investigations have found multiple causes for these data drops. Some of the causes are related to behavior that Salesforce describes as "Working As Designed".
Salesforce has recommended other replication architectures, but there are various concerns with my company about using them (license cost, platform load) ... and we might still end up with missing data.
So, we're looking into data reconciliation / auditing solutions. Are there any recommendations on a tool that can:
* Identify record(s) where the record in Salesforce does not have a matching record (e.g. same timestamp) existing in Kafka
* Generate a message containing relevant metadata (e.g. record Id, Salesforce object, Kafka topic) to be sent to a REST endpoint / message queue for reprocessing
