Issues using Diff Snap

[dmcnichol]

Hi,

I’m new to snap logic, I have two data sources (Oracle, SQL) the new is Oracle and existing is SQL.

The pipeline’s first run, runs as expected, but the second, where I would expect all data to be in the “unmodified” output data preview is not working as expected. I’m trying to find where my settings are incorrect.

Entire Pipeline:

Diff:

I believe my issue is similar to this

Any help is appreciated thank you,

[ptaylor]

Are you executing the pipeline using the Validate button? If so, try holding the Shift key when you click the Validate button. That will bypass caching.

[bojanvelevski]

@dmcnichol
My guess is that you have large number of records in both of the sources, and when you hit execute, the pipeline works fine, detects modifications and unmodified rows accordingly. But when you try to validate the data, you are validating the pipeline, and by that, you end up with 2 completely different sets of data from both sources. Meaning, the IDs from the “original” set are not in the “new” set.
So as a result of that, all of the IDs from one source will go to Deletions, and from the other source, will go to Insertions.

Find a way to validate the data differently. One suggestion is to take specific group of IDs, and when you validate, you’ll make sure to use those IDs in both sources.

[dmcnichol]

Thank you both for your quick responses. 😀

I’m dealing with a small set of data 498 records.

I’ve been using ‘Shift click Validate’ as a fall back when snaps are not behaving as I thought they should or I change the underlying file or database structure.

in order to simplify my issue (pretty sure this is new user growing pains) I’ve filtered both New and old data sources by id=271967386.

First run performs as expected:

Second run not so much (expected 1 record in the ‘Unmodified’ output)

I suspect it is due to data type? could anyone possibly confirm?

Oracle (new) data preview before Diff Snap:

SQL (old/existing) data preview before Diff Snap

I’m going to try using the mapper snap or another transform to solve.

Thank you all again for your time.

[dmcnichol]

Update:

I cut down the data to only the ID field and recreated the destination SQL table, so 1 record 1 field and the issue was still present.

Adding a mapping snap which changed the ID(Oracle Source) from number to string seems to resolve it, but when I try to add other fields (still 1 record) the record is routed to the “Modified” output.

I’ve used notepad++ compare plugin on the two Diff snap’s inputs and they are a match, this has really confused me

Mapper Snap Details:

Pipeline with mapper:

any advice appreciated