How are pipeline executions distributed across a Snaplex?

When a request to run a pipeline is submitted against a Snaplex,the Control Plane must decide which member node to send the request to.

In making that decision,the Control Plane assumes that each node in a Snaplex is homogenous in terms of both hardware specifications (CPU, RAM, disk space) and installed software (operating system, SnapLogic Snaplex binaries, Java Runtime Environment) to derive a baseline utilization score.

It then factors in the number of active pipelines running on each member node to identify the nodes that are the least utilized.

It is out of this subset of least utilized nodes that some randomization is added to determine the Snaplex node that will receive the pipeline execution request.

If the hardware specifications member nodes are dissimilar or if there are additional applications or processing running on them locally,the pipeline load distribution calculations will be skewed which can manifest in the following behavior:

  • Irregular distribution of pipeline executions within the Snaplex
  • Pipelines rarely or never execute on specific Snaplex member nodes

To correct this behavior, ensure that all nodes in a Snaplex are at parity with one another in terms of both hardware specifications and installed software.

1 Like

Thank you Robin, this makes a lot of sense in regards to evening spreading the load.

As we try to make our servers equal in hardware and software specifications, there will inevitably come a time when that’s not practically possible (i.e. Server minimum CPU availability raises across the board) and we will be in a situation where newer servers will inherently be more powerful. At that point, we would be looking at either upgrading some servers to make them comparable or replacing servers.

Is there any plans of supporting different configurations of hardware in some sort of load balancing configuration setting? ServerA - 1.0x, ServerB - 1.0x, ServerC = 1.5x capicity, etc.

Thanks!

@brettdorsey don’t believe it’s on the immediate roadmap but something that we can discuss with you going forward. I’ll make our PM team (including @cstewart, VP of Product) aware.

Brett, we are aware of this requirement. It is our practice to specify that nodes in a single Snaplex be of similar configuration, and with the majority of customers using some form of server virtualisation this has not been, and continues to reduce being, over time as an issue.
That said, it is certainly on the “would like to enhance” list, but not at this point scheduled.
Thanks for raising this, the feedback from our customers is always welcome as it helps us gauge the importance of issues amongst customers.
Craig