Yes if the location is made sure to exist before this is written.
I am pretty sure you wont have to insert 100s of locations every load. It is a one time thing.
Glad you figured it!!
If its the same data and you’re doing an update, it seems like it’d be a non issue.
I’d like to ask why understand your use case more, You have a child pipeline running to do individual updates to a database?That’d eat a lot of resources spinning up and down those parallel executions, even if you reuse. From what you’ve described here it sounds like you’re injecting parallelism just for the sake of it.
It’d most likely be more efficient and performant to use a single db snap to do your updates and edit the batch settings on your db account.
Also you should not think of pipelines as threads, but a process that’s a collection of threads. In reality, each snap in a pipeline has its own thread. Those snaps combine to perform a semi-serial execution, and pipeline executions are generally not aware of each.
Best,
unfortunately to insert the data in the db, its not via SQL, we have to call SOAP API to insert/update single location into that systems’DB.
Thanks, that approach makes more sense now.
As far as locking goes there isn’t anything in the product that would function like you were thinking at this point.
What Naveen mentioned is right. Is there any way to possibly preprocess the data so that you’d only have one instance of a place across your incoming documents (could be dangerous, if the one with the data fails)?
I’m not sure how your SOAP endpoint functions, are you just publishing flat employee records in one soap call or are you making multiple soap calls per incoming document to write pieces of the employee record to different places?
Is there any danger from your side having the duplicate data overwrite for your application?
Thanks, in our case, we have to make multiple soap calls per incoming document. There’s no other choice to do that.
But anyway. I somehow organize the data a little bit, removed the duplicate location from the data.
Now I am able to leverage the pool size of Pipeline Execute without worrying about the race condition.
Thanks again
Yes if the location is made sure to exist before this is written.
I am pretty sure you wont have to insert 100s of locations every load. It is a one time thing.
Glad you figured it!!
