Architecture: Pipeline Execution Mechanism - Some Questions

How does the SnapLogic Control Plane identify a specific node in a Snaplex for executing a pipeline? How does it identify if a node is busy? Is it only checking one time for the node’s availability?

This is a continuous process:

  1. When a node is fired up (started), it establishes a connection with the SnapLogic Control Plane over a non-persistent outbound connection via port 443.
  2. This is in essence, a registration process based on the configuration file (Basically the node tells the Control Plane, “I am a node, that belongs to a particular plex which is an asset within a particular org”).
  3. Once this is done, the node and Control Plane start communicating with each other via web socket connections. The node keeps sending information such as heart beat, process details to the Control Plane continually.
  4. Based on this info, the Control Plane is kept aware of the number of active threads in each node continually (On the Dashboard, you can get to the Snaplex-> Node-> Additional Information). So this is not a one-time activity but an ongoing activity.
  5. Among the nodes within a Snaplex, the Control Plane makes a choice of that node which has the least number of active threads (In other words, that is the “least busy” node), to send the pipeline executions .
2 Likes