Forum Discussion

maahutch's avatar
maahutch
New Contributor
2 years ago

How to ignore duplicate rows when inserting record into table

I'm developing a pipeline to request data from an API and load it into a MS SQL Server table. The first run of the pipeline loaded 50 records to the table, each with a unique 'user_name' value. The table constraints require the 'user_name' field be unique. A new user has been added to the source platform so I want to run the pipeline again and ignore the duplicate records and update the destination table with the additional, 51st, record. Right now, I just get an error from the first record telling me I have a duplicate key error. 

I've tried using the 'Update' snap with a condition like "user_name != $user_name" but with no success. How can I run this pipeline so it will skip duplicate key values and add any new records? 

 

2 Replies

  • alchemiz's avatar
    alchemiz
    Contributor III
    2 years ago

    Hi maahutch ,

    Good day, you can also use the sql execute or sql stored procedure snaps and script the condition there if you're not expecting to update existing records otherwise ssapa suggestion is the solution as it will do upsert ( insert new record/ update existing record )
    example:

    Thanks,
    EmEm