Forum Discussion

mtran21's avatar
mtran21
New Contributor III
5 years ago

How do you keep only one record of duplicated records?

My pipeline is a simple pipeline that execute an Oracle query then parse the data to a CSV file. The result of the query gives some of the duplicate records (Same ID), that because the database has ...
vineesha's avatar
vineesha
Employee
5 years ago

Hi, unique snap eliminates duplicate documents in a document stream, such as duplicate rows.
use unique snap before mapper and let me know if you still have any issues.

mtran21's avatar
mtran21
New Contributor III
5 years ago

My goal is to keep 1 record from each duplicate pair like this

So using Unique snap, will it remove the entire duplicate pair?