Parallel processing by REST Post snap

[KumarR]

Hi Team,
How can i improve performance of RestPost snap.
The jms snap process data at high throughput i want to send this to an ultra child pipe so calling ultra by RestPost snap. It process docs around 25-30 doc/ sec.
Tried doing it by parallel processing by adding Router with 5 views and used 5 rest post at router to get speed 5 time increased(to achieve 30*5~150doc/sec).
But above routing trick did not work and dashboard shows Rest post snap has reduced its speed by 5 times i.e with last test single rest post process speed was 25doc/sec, with 5 restpost each started processing at 5doc/sec so getting same speed.
Any thoughts to improve performance to call ultra child by RestPost?

[tstack]

KumarR:

The jms snap process data at high throughput i want to send this to an ultra child pipe so calling ultra by RestPost snap.

Can I ask why you’re doing it this way instead of using PipeExec in the JMS pipeline?

[KumarR]

Hi Tim,
As of now i have ultra pipe say X without open views with kafka snaps as source and destination. When any error happens in pipe, i want to capture original kafka message in error pipeline(snap.oroginal.load()) to reprocess message, But as snap.oroginal.load() method will only work in one i/p one o/p ultra.
To overcome with this i want to keep source kafka snap in a ultra pipe X and at next step will call another ultra pipe Y(with open i/p o/p) by Restpost snap, by this i will be able to get original message of Y pipe in error pipe in case of error in Y pipe.
But high throughput is key in kafka and RetsPost is not processing doc with that latency even with parallel processing.

[tstack]

KumarR:

As of now i have ultra pipe say X without open views with kafka snaps as source and destination.

Are you using the Confluent Kafka snaps?

KumarR:

But high throughput is key in kafka and RetsPost is not processing doc with that latency even with parallel processing.

Are you sure it’s the RESTPost snap and not the ultra pipeline that is the bottleneck here? How many instances of the ultra are running?

[KumarR]

tstack:

Are you sure it’s the RESTPost snap and not the ultra pipeline that is the bottleneck here?

I tried sending payload to the ultra pipe from jmeter, jmeter was able to send ~6k docs per minute to ultra pipe.