Parallel processing by REST Post snap

[KumarR]

Hi Team,
How can i improve performance of RestPost snap.
The jms snap process data at high throughput i want to send this to an ultra child pipe so calling ultra by RestPost snap. It process docs around 25-30 doc/ sec.
Tried doing it by parallel processing by adding Router with 5 views and used 5 rest post at router to get speed 5 time increased(to achieve 30*5~150doc/sec).
But above routing trick did not work and dashboard shows Rest post snap has reduced its speed by 5 times i.e with last test single rest post process speed was 25doc/sec, with 5 restpost each started processing at 5doc/sec so getting same speed.
Any thoughts to improve performance to call ultra child by RestPost?

[KumarR]

tstack:

Can I ask why you’re doing it this way instead of using PipeExec in the JMS pipeline?

@tstack
@skidambi
@cstewart
Any workaround for this, As i don’t find a way to get the kafka snap message in error pipe in case of main ultra pipe fails(no i/p no o/p unlinked) , so using another ultra with 1 i/p 1 o/p and sending kafka message to this pipe so i will be able to get ultra payload in error pipe.
But while calling this ultra pipe from restpost parallel processing is not working.

[KumarR]

@tstack- Do you have any thoughts how to achieve parallel process by rest post here.

[KumarR]

Yes i am using Confluent Kafka snap and sending its docs to the ultra which i am calling from restpost. And i am running 10 instance of this ultra still rest post processed 25-30 dc/sec, if making 5 parallel restpost each restpost started processing 5 doc/sec.

In child ultra pipe i am having only one mapper snap just to test it but not able to get the throughput.

[tstack]

KumarR:

And i am running 10 instance of this ultra still rest post processed 25-30 dc/sec, if making 5 parallel restpost each restpost started processing 5 doc/sec.

I don’t think the RESTPost is your problem, it’s more likely something wrong with the ultra task or the network. Take a look at the detailed statistics for the REST snaps (see the Execution Statistics documentation), you’ll be able to see how much time they are spending waiting to receive data over the network.

If you hover over the memory/network bars over the rows, the following dropdown should appear.

You can then hover over the bars for the Network Usage to get even more detail on how much data was transferred and how long it took.

I’m going to guess that the REST snaps are spending most of their time waiting on the network, in which case it’s not an issue with the REST snaps.

[KumarR]

tstack:

I’m going to guess that the REST snaps are spending most of their time waiting on the network, in which case it’s not an issue with the REST snaps.

@tstack
Hello Tim,
I am following with network team here for this issue.
We updated/enabled the Accelerated networking for all azure cloud servers/node of the plex as well resized to Standard D16s v3(16-CPU & 64 –RAM).
But still not able to receive the throughput.

Any suggestion what change it needs to get the throughput while calling ultra pipe from restpost snap?

Reproduction step-
Create 10k doc by seq snap and send them to restpost, restpost is calling one ultra pipe url(ultra pipe has single mapper with passthrough)

Thanks!