@darshthakkar - I would recommend carefully considering if the Diff snap is really what you want to use. The Diff snap compares the incoming JSON documents as a whole for equivalence - if anything in the objects are different, it considers it as modified. This includes everything from a difference in the value of a given element, the order of the elements within the object, missing / added elements, element name case sensitivity, or even differences in datatypes.
I think what you probably want to consider instead is the use of a Join snap - probably using Left Outer join type - to check the existence of the target record and apply insert/update methodology based on that condition.
Also, rather than opening a new discussion on a closed topic, please open a new thread so the information shared is under a logical heading for other members to find more easily.
I agree to your comments on opening a new thread. I will do the needful.
Regarding Diff snap, I haven’t used it so far, or to put it correctly, I wasn’t even aware of that until today.
So, I have used Joins, Inner join to be precise and then routing the error view to get the actual output.
The concerning part is duplicates so how would I eliminate those will require some serious brainstorming from my side.
Thank you for your suggestion on new thread.
Regards,
Darsh
Let me see if I understand:
You are reading from a csv file, then calling an API with a parameter that is a value from that file?
You then want to store data in a file --the same CSV or a different one?
And you getting an error -- what is the error?
Can you include a sample pipeline or screenshots?
You can route your error to error view and save it in another csv file. If the returned error is a custom error and route the response in another downstream. Also in csv formatter, check option "ignore empty stream" to avoid creation of file if no documents exists in the upstream.
