I’m working on a project in which i receive a post call with data everytime someone register on one website. I need to post this data to another site but i dont want to post them individually and as frequent as they come. so im thinking about building a first pipeline that appends posted data into a JSON. Another pipeline will run once a day and is responsible for reading this JSON file, posting all its data to final destination, cleaning it, saving logs, sending reports etc.
I didn’t load tested it, so i wanna know if theres a better way of doing it. Im afraid it wont work fine with many concurrent post calls, mainly when both pipelines are running simultaneously. i dont know what happens once one pipeline is trying to append something to a json, and another one is reading it to post, at the same time. what if it tries to append info to same json at the same time, with many concurrent rest post to it.
Whats the best way of gathering data for this purpose using snaplogic?
