cancel
Showing results for 
Search instead for 
Did you mean: 

How to ignore duplicate rows when inserting record into table

maahutch
New Contributor

I'm developing a pipeline to request data from an API and load it into a MS SQL Server table. The first run of the pipeline loaded 50 records to the table, each with a unique 'user_name' value. The table constraints require the 'user_name' field be unique. A new user has been added to the source platform so I want to run the pipeline again and ignore the duplicate records and update the destination table with the additional, 51st, record. Right now, I just get an error from the first record telling me I have a duplicate key error. 

I've tried using the 'Update' snap with a condition like "user_name != $user_name" but with no success. How can I run this pipeline so it will skip duplicate key values and add any new records? 

 

2 REPLIES 2

ssapa
Employee
Employee

@maahutch Try using SQL server merge snap if that works.

alchemiz
Contributor III

Hi @maahutch ,

Good day, you can also use the sql execute or sql stored procedure snaps and script the condition there if you're not expecting to update existing records otherwise @ssapa suggestion is the solution as it will do upsert ( insert new record/ update existing record )
example:

alchemiz_1-1691268031217.png

Thanks,
EmEm