Groundplex without internet

[dksrivatsa]

Does a Groundplex continue to execute pipelines and schedules in the event of losing internet connectivity? And does it push the execution information back to control plane after the internet connectivity is restored?

[akidave]

For Ultra pipeline instances that are already running (and do not have any dependency on the control plane or other network resources), the pipelines will continue processing documents for up to two days if the heartbeat with the control plane is not happening.

For other functionality, like scheduled pipelines and Snaplex triggers, short network disruptions are tolerated. Longer disruptions will result in failures. New pipeline execution information is cached in memory and pushed to the control plane when the network is restored. How much data can be cached will depend on the pipeline load on the node and the memory available.

Any node which cannot talk to the control plane for two days will restart itself, to try and re-establish communication through the restart.

[Matthias · Accepted solution]

an additional Note on this topic:
pipelines that are already running continue to run (for example a pipeline running a FilePoller snap), even without the requirement of ultra pipelines.

Obviously, the node restart after 2 days will stop the pipeline, but we’ve successfully done a test with this over several hours - after the Node was back online, all metrics showed up in the control plane after a short amount of time.
This obviously assumes that the pipeline is built in a way to not require connectivity (no network connections to the internet, no SLDB references) - but for “pure offline” tasks (like in our case, file movements) is perfectly fine.

Ultrapipelines themselfs are not “as useful” in a scheduled pipeline scenario - as you’d have to maintain an external Scheduler, to “trigger” the already running ultra-pipeline, which makes management of this very cumbersome.

Snaplogic would have a quite powerfull feature at hand for exactly this usecase since the move of the Scheduler to be triggered “on the groundplex” 2 releases ago - however the scheduler itself will be disabled after 11 missed heartbeats (according to logs in our groundplex) - which is roughly 15 minutes - and cannot currently be configured as it’s hardcoded in the snaplogic code.

[dksrivatsa]

Thanks for the detailed response Ajay and Matthias, this helps a lot.