Feedmaster queue pileup and the response time for ultra pipeline

[KumarR]

Hi team,
I am getting huge variation in the response time while processing request in ultra pipeline(sample pipe not much logic or processing).
I have one ultra pipe running with multiple instances, and hitting 11000+ records to pipe from jmeter.
Most of the request 90% are getting response in less than 2 sec and 99% are at around 4 sec.
Avg time process is 1.2 sec but 1% request are taking more than 10 sec.

My usecase can not accept latency beyond 4 sec but i am facing issue here as some req. are taking over 10 sec. What is the reason and what should i do for this to make it consistent, please suggest ?
Avg time is ~1.2 sec so variation of 1 or 2 sec is still ok but i am experiencing too much variation of over 8 sec. Please check status below.
Thanks,

[KumarR]

Hi Aditya,
As per my understanding it was happening because of network hops.
The client/jmeter was running on one network and ultra pipes were on different so due to hop it was taking time.
I was running 4 instances of 1 ultra pipe so first req on each instance was taking some time(~10sec) due to some security constraints/checks on our network, 1st req onward it was processing smoothly.
Thanks!