Parallel processing by REST Post snap
Hi Team,
How can i improve performance of RestPost snap.
The jms snap process data at high throughput i want to send this to an ultra child pipe so calling ultra by RestPost snap. It process docs...
Forum Discussion
Yes i am using Confluent Kafka snap and sending its docs to the ultra which i am calling from restpost. And i am running 10 instance of this ultra still rest post processed 25-30 dc/sec, if making 5 parallel restpost each restpost started processing 5 doc/sec.
In child ultra pipe i am having only one mapper snap just to test it but not able to get the throughput.
I don’t think the RESTPost is your problem, it’s more likely something wrong with the ultra task or the network. Take a look at the detailed statistics for the REST snaps (see the Execution Statistics documentation), you’ll be able to see how much time they are spending waiting to receive data over the network.
If you hover over the memory/network bars over the rows, the following dropdown should appear.
You can then hover over the bars for the Network Usage to get even more detail on how much data was transferred and how long it took.
I’m going to guess that the REST snaps are spending most of their time waiting on the network, in which case it’s not an issue with the REST snaps.
@tstack
Hello Tim,
I am following with network team here for this issue.
We updated/enabled the Accelerated networking for all azure cloud servers/node of the plex as well resized to Standard D16s v3(16-CPU & 64 –RAM).
But still not able to receive the throughput.Any suggestion what change it needs to get the throughput while calling ultra pipe from restpost snap?
Reproduction step-
Create 10k doc by seq snap and send them to restpost, restpost is calling one ultra pipe url(ultra pipe has single mapper with passthrough)Thanks!
