How to use Deduplicate snap?

Designing Pipelines Solutions
#1

I plan to use Deduplicate snap to remove duplicate records from a CSV file. In my CSV file, let’s say I have 3 records with the same ID. I just want to keep 1 record.

So I am using Deduplicate snap and set it up like this

image
image1028×441 19.2 KB

It works but it also remove some of the not duplicated records. Why?

#2

The thresholds in the snap default to values suited for approximate match. For exact match, you can set a higher threshold, like 0.9 and set a low criteria of 0.01 and high of 0.99