I have a pipeline with duplicate check logic in place which queries a endpoint and if no matching record is found then stores the file at the endpoint.
My doubt is ,for a bulk upload call to this pipeline, will that duplicate check work or not. I understand that each snap process the incoming requests separately and lets say two parellel processes Process1 and Process2 are at the duplicate check, but as none of them have created the document , both will pass and will create duplicate document in case of a bulk request.
please correct me if my understanding of pipeline processing document is not correct or if you have a solution in mind to tackle this issue. I donโt want to add the delay after duplicate check as that will increase my pipeline execution time. Any help will be greatly appreciated.
Thank you
