Showing results for 
Search instead for 
Did you mean: 

Creating a Buffer for Data

New Contributor III

I’m working on a project in which i receive a post call with data everytime someone register on one website. I need to post this data to another site but i dont want to post them individually and as frequent as they come. so im thinking about building a first pipeline that appends posted data into a JSON. Another pipeline will run once a day and is responsible for reading this JSON file, posting all its data to final destination, cleaning it, saving logs, sending reports etc.
I didn’t load tested it, so i wanna know if theres a better way of doing it. Im afraid it wont work fine with many concurrent post calls, mainly when both pipelines are running simultaneously. i dont know what happens once one pipeline is trying to append something to a json, and another one is reading it to post, at the same time. what if it tries to append info to same json at the same time, with many concurrent rest post to it.

Whats the best way of gathering data for this purpose using snaplogic?


Contributor III

Hi @rmedeiros,

Good day, another alternative is to use Amazon SQS snap pack (if you do have AWS available) where your 1st pipeline will push this messages directly in AWS SQS (using SQS producer)… then the 2nd pipeline will just listen from the AWS SQS (using SQS consumer set message count to 0)