Architecture: Pipeline Execution Mechanism - Some Questions

[skidambi]

How does the SnapLogic Control Plane identify a specific node in a Snaplex for executing a pipeline? How does it identify if a node is busy? Is it only checking one time for the node’s availability?

This is a continuous process:

1. When a node is fired up (started), it establishes a connection with the SnapLogic Control Plane over a non-persistent outbound connection via port 443.
2. This is in essence, a registration process based on the configuration file (Basically the node tells the Control Plane, “I am a node, that belongs to a particular plex which is an asset within a particular org”).
3. Once this is done, the node and Control Plane start communicating with each other via web socket connections. The node keeps sending information such as heart beat, process details to the Control Plane continually.
4. Based on this info, the Control Plane is kept aware of the number of active threads in each node continually (On the Dashboard, you can get to the Snaplex-> Node-> Additional Information). So this is not a one-time activity but an ongoing activity.
5. Among the nodes within a Snaplex, the Control Plane makes a choice of that node which has the least number of active threads (In other words, that is the “least busy” node), to send the pipeline executions .

[anubhav_nautiya]

Hi I have a query on this, if we have two Snaplexes in our org say Snaplex A and Snaplex B. I have a pipeline which calls a child pipeline, so if a parent pipline in running in one of the nodes of Snaplex A, could child pipeline run on a node of Plex B? or will it always run on a node of Plex A?

Please note we are using pipe.plexpath in Snaplex property of PE snap

Regards
Anubhav

[skidambi]

Hello - Based on your question, with Snaplex A and Snaplex B in place; if the parent pipeline is getting executed in one of the nodes of Snaplex A, and I am assuming you are calling the child pipeline using the Pipeline Execute snap. The Pipeline Execute snap has the following three options for the developer to choose from (on the field labeled Execute On😞

1. SNAPLEX_WITH_PATH

• When you choose this option, another field Snaplex Path would appear.
• NOT PROVIDING any info on this Snaplex Path field, would result in the (chosen) child pipeline getting executed in the SAME NODE as the parent. For example, if parent pipeline is executing on a node in Snaplex A, the child pipeline specified in the Pipeline Execute snap would execute in the same node where the parent is being executed.
• Providing a valid Snaplex path, would result in the child running in any of the available nodes in the Snaplex provided.

2. LOCAL_NODE

• Child pipeline gets executed in the same node (of the same Snaplex) where the parent is being executed.

3. LOCAL_SNAPLEX

• Child pipeline gets executed in the same Snaplex where is parent is being executed. Note that this only means the same Snaplex but the child and parent may get executed in different nodes depending upon how busy each of the nodes in that Snaplex are.

[anubhav_nautiya]

Thanks for the detailed Explanation. 🙂