Hi,
I have a scenario to read records from Redshift table based on this condition below which I have it in all my Redshift snaps for the pipeline I am using.
weekof < Post_prod_support7 and weekof != Post_prod_support7
At the end it checks for any records with the above condition as true to self call the same pipeline with filter and pipeline execute snap.
Records properly inserted with correct count with this approach till above condition is true. But after inserting records for the above condition, it keep inserting the last set of records with same values again and again even the condition is false.
Ideally I have this check in all the snaps and even in Bulk upsert snap I have all the fields to look for existing records to avoid this issue. Still it inserts the last set of records again and again even the condition is false.
PFB attachments of pipeline, snap and table screenshots.
Does anyone faced such issues while inserting? Any suggestion to avoid this last set duplicate insertion would be helpful.
Regards,
Amar.
