Help with DIFF SNAP

[PayalS]

Hi,

I am looking for some help regarding DIFF Snap.
Below is the scenario, I am working on:

I have two Source DBs - A & B. DIFF snap is being used to identify the eligible data for Insert, Update and Delete, and accordingly operations performed in Source B as Target DB.
e.g.
Lets say 10 records are there in A
20 records are there in B
then DIFF will identify 10 records for deletion from B.

Now, the problem I am facing is,
if due to any data issue or connection failure, if no records are coming from Source A, then DIFF is identifying all records from B to flow in the deletion link, which eventually deleting all the data from B.

I dont want this to happen. I want my pipeline to stop/fail, if any such error happend in the source, so that target cannot be empty in any case.
I am using Error Pipeline, which is tracking the error fine but then not stopping the pipeline.

Is there a way, I can track the error and stop the pipeline in such scenarios of connection failure or Data Issue ?

Quick Help will be really appreciated.

Thanks in Advance,
Payal Srivastava

[SpiroTaleski]

Hi @PayalS

You can try by splitting the pipeline in two pipelines(parent/child).

In the parent pipeline you can add a logic where you will check if there are records coming from Source A or not(ex:count if number of records is bigger than 0).

In the child pipeline you can add the DIFF snap together with Source B.

If there are no records from Source A then do not call the child pipeline(ex: by using Filter Snap that will not proceed pipeline execution and call child pipeline if there are no records on input).

This is just an idea that you can try and check if it works for you.

Regards,
Spiro Taleski

[PayalS]

Thanks Spiro for the quick reply. Much Appreciated. 🙏

Yeah, I too thought the same approach, but then in our project we do have a total count of 70 pipelines (which are already created and we identified this bug later). Now, with this change, we will have to modify all 70 pipelines and along with that we will have to create another set of 70 pipelines.

Therefore, I was thinking if we could have any alternate approach to fix the same. May be by just modifying the error pipeline or may a small fix which can be done only to already created 70 pipelines.

[SpiroTaleski]

Thanks @PayalS

The other approach that I was thinking is replacing the DIFF snap and implementing custom logic for identifying which data should be inserted, updated or deleted.

But, I assume that this approach will require big changes on the existing pipelines as well.

Regards,
Spiro Taleski

[PayalS]

Thanks @Spiro_Taleski
True, that will again require big changes in all 70 pipelines ☹️