Forum Discussion

KumarR's avatar
KumarR
New Contributor
7 years ago

Parallel processing by REST Post snap

Hi Team,
How can i improve performance of RestPost snap.
The jms snap process data at high throughput i want to send this to an ultra child pipe so calling ultra by RestPost snap. It process docs around 25-30 doc/ sec.
Tried doing it by parallel processing by adding Router with 5 views and used 5 rest post at router to get speed 5 time increased(to achieve 30*5~150doc/sec).
But above routing trick did not work and dashboard shows Rest post snap has reduced its speed by 5 times i.e with last test single rest post process speed was 25doc/sec, with 5 restpost each started processing at 5doc/sec so getting same speed.
Any thoughts to improve performance to call ultra child by RestPost?

10 Replies

  • tstack's avatar
    tstack
    Former Employee
    7 years ago

    Can I ask why you’re doing it this way instead of using PipeExec in the JMS pipeline?

    • KumarR's avatar
      KumarR
      New Contributor
      7 years ago

      Hi Tim,
      As of now i have ultra pipe say X without open views with kafka snaps as source and destination. When any error happens in pipe, i want to capture original kafka message in error pipeline(snap.oroginal.load()) to reprocess message, But as snap.oroginal.load() method will only work in one i/p one o/p ultra.
      To overcome with this i want to keep source kafka snap in a ultra pipe X and at next step will call another ultra pipe Y(with open i/p o/p) by Restpost snap, by this i will be able to get original message of Y pipe in error pipe in case of error in Y pipe.
      But high throughput is key in kafka and RetsPost is not processing doc with that latency even with parallel processing.

      • tstack's avatar
        tstack
        Former Employee
        7 years ago

        Are you using the Confluent Kafka snaps?

        Are you sure it’s the RESTPost snap and not the ultra pipeline that is the bottleneck here? How many instances of the ultra are running?

    • KumarR's avatar
      KumarR
      New Contributor
      7 years ago

      @tstack
      @skidambi
      @cstewart
      Any workaround for this, As i don’t find a way to get the kafka snap message in error pipe in case of main ultra pipe fails(no i/p no o/p unlinked) , so using another ultra with 1 i/p 1 o/p and sending kafka message to this pipe so i will be able to get ultra payload in error pipe.
      But while calling this ultra pipe from restpost parallel processing is not working.

  • KumarR's avatar
    KumarR
    New Contributor
    7 years ago

    @tstack- Do you have any thoughts how to achieve parallel process by rest post here.

  • KumarR's avatar
    KumarR
    New Contributor
    7 years ago

    Yes i am using Confluent Kafka snap and sending its docs to the ultra which i am calling from restpost. And i am running 10 instance of this ultra still rest post processed 25-30 dc/sec, if making 5 parallel restpost each restpost started processing 5 doc/sec.

    In child ultra pipe i am having only one mapper snap just to test it but not able to get the throughput.

    • tstack's avatar
      tstack
      Former Employee
      7 years ago

      I don’t think the RESTPost is your problem, it’s more likely something wrong with the ultra task or the network. Take a look at the detailed statistics for the REST snaps (see the Execution Statistics documentation), you’ll be able to see how much time they are spending waiting to receive data over the network.

      If you hover over the memory/network bars over the rows, the following dropdown should appear.

      You can then hover over the bars for the Network Usage to get even more detail on how much data was transferred and how long it took.

      I’m going to guess that the REST snaps are spending most of their time waiting on the network, in which case it’s not an issue with the REST snaps.

      • KumarR's avatar
        KumarR
        New Contributor
        7 years ago

        @tstack
        Hello Tim,
        I am following with network team here for this issue.
        We updated/enabled the Accelerated networking for all azure cloud servers/node of the plex as well resized to Standard D16s v3(16-CPU & 64 –RAM).
        But still not able to receive the throughput.

        Any suggestion what change it needs to get the throughput while calling ultra pipe from restpost snap?

        Reproduction step-
        Create 10k doc by seq snap and send them to restpost, restpost is calling one ultra pipe url(ultra pipe has single mapper with passthrough)

        Thanks!

  • KumarR's avatar
    KumarR
    New Contributor
    7 years ago

    Thanks Tim, i will go through the link shared for more analysis. Below is the screen shot for 1 restpost exec status i got from dashboard.

    Is this time waiting happening due to any configuration while creating ultra task or do i need to talk to network people for this. Please suggest as i can see wait time here but not sure the root cause.