Pipeline Design and Performance Optimization Guide
Introduction This document serves as a comprehensive best practice guide for developing efficient and robust Pipelines within the SnapLogic Platform. It offers guidelines that aim to optimize performance, enhance maintainability, reusability, and provide a basis for understanding common integration scenarios and how best to approach them. The best practices encompass various aspects of Pipeline design, including Pipeline behavior, performance optimization and governance guidelines. By adhering to these best practices, SnapLogic developers can create high-quality Pipelines that yield optimal results while promoting maintainability and reuse. The content within this document is intended for the SnapLogic Developer Community or an Architect, in addition to any individuals who may have an influence on the design, development or deployment of Pipelines. Authors: SnapLogic Enterprise Architecture team Why good Pipeline Design is important The SnapLogic Pipeline serves as the foundation for orchestrating data across business systems, both within and outside of an organization. One of its key benefits is its flexibility and the broad range of "Snaps" that aim to reduce the complexity involved in performing specific technical operations. The “SnapLogic Designer”, a graphical low-code environment for building an integration use case with Snaps, provides a canvas enabling users with little technical knowledge to construct integration Pipelines. As with any user-driven environment, users must exercise careful attention to ensure they not only achieve their desired business goals but also adhere to the right approach that aligns with industry and platform best practices. When dealing with a SnapLogic Pipeline, these best practices may encompass various considerations: Is my Pipeline optimized to perform efficiently? Will the Pipeline scale effectively when there's an increase in data demand or volume? If another developer were to review the Pipeline, would they easily comprehend its functionality and intended outcome? Does my Pipeline conform to my company's internal conventions and best practices? Not considering these factors may cause undesirable consequences for the business and users concerned. Relative to the considerations stated above, these consequences could be as follows: If data is not delivered to the target system, there may be financial consequences for the business. The business may experience data loss or inconsistency when unexpected demand occurs. Development and project teams are impacted if they are unable to deliver projects in a timely fashion. Lack of internal standardization limits a company's ability to govern usage across the whole business, thus making them less agile. Therefore, it is essential that users of the Platform consider best practice recommendations and also contemplate how they can adopt and govern the process to ensure successful business outcomes. Understanding Pipeline Behavior To better understand how Pipelines can be built effectively within SnapLogic, it is essential to have an understanding of the Pipeline’s internal characteristics and behaviors. This section aims to provide foundational knowledge about the internal behavior of Pipelines, enabling you to develop a solid understanding of how they operate and help influence better design decisions. Pipeline Execution States The execution of a SnapLogic Pipeline can be initiated either via a Triggered, Ultra or Scheduled task. In each case, the Pipeline transitions through a number of different ‘states’ with each state reflecting a distinct processing the lifecycle of the Pipeline, from invocation, preparation, execution to completion. The following section of the document will look to highlight this process in more detail and explain some of the internal behaviors. The typical Pipeline execution flow is as follows: Initialize Pipeline. Send Metadata to Snaplex. Prepare Pipeline, fetch & decrypt account credentials. Connect to endpoint security. Send execution metrics. Pipeline completes, and resources are released. The following section describes the different Pipeline state transitions & respective behavior in sequential order. State Purpose NoUpdate A pre-preparing state. This indicates a request to invoke a Pipeline has been received but the leader node or control plane is trying to establish which Snaplex node it should run on. (This state is only relevant if the Pipeline is executed on the leader node). Preparing Indicates the retrieval of relevant asset metadata including dependencies from the control plane relating to the invoked Pipeline. This process also carries out pre-validation of snap configuration alerting the user of any missing mandatory snap attributes. Prepared Pipeline is prepared and is ready to be executed Executing Pipeline executes and processes data, connecting to any Snap Endpoints using the specified protocols. Completed Pipeline execution is complete and the teardown resulting in the releasing of compute resources within the Snaplex Node. Final Pipeline execution metrics and sent to the Control Plane. Table 1.0 Pipeline state transitions Pipeline execution flow Pipeline Design Decision Flow The following decision tree can be used to establish the best Pipeline Design approach for a given use case. Snap Execution Model Snaps can be generally categorized into these types: Fully Streaming Most Snaps follow a fully streaming model. i.e. Read one document from the Input view (or from the source endpoint for Read Snaps), and write one document to the Output view or to the target endpoint. Streaming with batching Some Snaps are streaming with batching behavior. For example, the DB Insert Snap reads N documents and then makes one call to the database (where N is the batch size set in the database account). Aggregating Aggregating type Snaps (e.g. Aggregate, Group By, Join, Sort, Unique etc.) read all input documents before any output is written to the Output view. Aggregating Snaps can change the Pipeline execution characteristics significantly as these Snaps must receive all upstream documents before processing and sending the documents to the downstream Snaps. Pipeline Data Buffering Connected Snaps with a Pipeline communicate with one another using Input and Output views. An Input view accepts data being passed from an upstream snap, it operates on the data and then passes the data to its Output view. Each view implements a separate in-memory ring buffer at runtime. Given the following example, the Pipeline will have three separate ring buffers. These are represented by the circular connections between each snap (diamond shaped connections for binary Snaps). The size of each ring buffer can be configured by setting the below feature flags on the org. The default values are 1024 and 128 for DOCUMENT and BINARY data formats respectively. com.snaplogic.cc.jstream.view.publisher.AbstractPublisher.DOC_RING_BUFFER_SIZE=1024 com.snaplogic.cc.jstream.view.publisher.AbstractPublisher.BINARY_RING_BUFFER_SIZE=128 The values must be set as powers of two. The source Snap reads data from the endpoint and writes to the Output view. If the buffer is full (i.e. if the Consumer Snap is slow), then the Producer Snap will block on the write operation for the 1025th document. Pipeline branches execute independently. However in some cases, the data flow of a branch in a Pipeline can get blocked until another branch completes streaming the document. Example: A Join Snap might hang if its upstream Snaps (e.g. Copy, Router, Aggregator, or similar) has a blocked branch. This can be alleviated by setting Sorted streams to Unsorted in the Join Snap to buffer all documents in input views internally. The actual threads that a Pipeline consumes can be higher than the number of Snaps in a Pipeline. Some Snaps such as Pipeline Execute, Bulk loaders, and Snaps performing input/output, can use a higher number of threads compared to other Snaps. Sample Pipeline illustration for threads and buffers The following example Pipeline demonstrates the practical example of how the usage and composition of Snaps within a Pipeline change the characteristics of how the Pipeline will operate once it is executed. Segment 1 Segment 2 Six threads are initialized at Pipeline startup. There are a total of seven ring buffers. The Copy Snap has two buffers, all other Snaps have one output buffer each. There are two segments that run in parallel and are isolated (other than the fact that they run on the same node, sharing CPU/memory/IO bandwidth). The first segment has two branches. Performance of one branch can impact the other. For example, if the SOAP branch is slow, then the Copy Snap’s buffer for the SOAP branch will get full. At this point, the Copy Snap will stop processing documents until there is space available in the SOAP branch’s buffer. Placing an aggregating Snap like the Sort Snap in the slow branch changes the performance characteristics significantly as the Snap must receive all upstream documents before processing and sending the documents to the downstream Snaps. Memory Configuration thresholds Property / Threshold Where configured Default value Comments Maximum memory % Node properties tab of the Snaplex 85 (%) Threshold at which no more Pipelines will be assigned to a node Pipeline termination threshold Internal (Can be configured by setting the feature flag at the org level com.snaplogic.cc.snap.common. SnapThreadStatsPoller. MEMORY_HIGH_WATERMARK_PERCENT) 95 (%) Threshold at which the active Pipeline management feature kicks in and terminates Pipelines when the node memory consumption exceeds the threshold. Ideal range: 75-99 Pipeline restart delay interval Internal (Can be configured by setting the feature flag at the org level com.snaplogic.cc.snap.common. SnapThreadStatsPoller. PIPELINE_RESTART_DELAY_SECS) 30 (seconds) One Pipeline is terminated every 30 seconds until the node memory goes below the threshold (i.e. goes below 95%) Table 2.0 Snaplex node memory configurations The above thresholds can be optimized to minimize Pipeline terminations due Out-of-Memory exceptions. Note that the memory thresholds are based on the Physical memory on the node, and not the Virtual / Swap memory. Additional Reference: Optimizations for Swap Memory Hypothetical scenario Add 16 GB swap memory to a Snaplex node with 8 GB physical memory. Property Comments Swap Space on the server Add 16 GB of swap / virtual memory to the node. Total Memory Total Memory is now = 24 GB (8 GB Physical plus 16 GB Virtual) Maximum Heap Size Set to 90% (of 24 GB) = 22 GB Maximum Memory Set to 31% rounded (of 22 GB) = 7 GB The intent of the above calculation is to ensure that the JCC utilizes 7GB of the available 8GB memory for normal workloads. Beyond that, the load balancer can queue up additional Pipelines or send them to other nodes for processing. If Pipelines that are running collectively start using over 7GB of memory, then the JCC can utilize up to 22GB of the total heap memory by using the OS swap space per the above configuration. Table 3.0 Snaplex node memory configurations By updating the memory configurations as in the above example, the JCC utilizes 7 GB of the available 8 GB memory. Beyond that value, the load balancer would queue up additional Pipelines or distribute them across other nodes. Use the default configurations for normal workloads, and use Swap-enabled configuration for dynamic workloads. When your workload exceeds the available physical memory and the swap is utilized, the JCC can become slower due to additional IO overhead caused by swapping. Hence, configure a higher timeout for jcc.status_timeout_seconds and jcc.jcc_poll_timeout_seconds for the JCC health checks. We recommend that you limit to 16 GB the maximum swap to be used by the JCC. Using a larger swap configuration causes performance degradation during the JRE garbage collection operations Modularization Modularization can be implemented in SnapLogic Pipelines by making use of the Pipeline Execute Snap. This approach enables you to: Structure complex Pipelines into smaller segments through child Pipelines. Initiate parallel data processing using the Pooling option. Reuse child Pipelines. Orchestrate data processing across nodes, within the Snaplex or across Snaplexes. Distribute global values through Pipeline parameters across a set of child Pipeline Snaps. Modularization best practices: Modularize by business or technical functions. Modularize based on functionality and avoid deep nesting or nesting without a purpose. Modularize to simplify overly-complex Pipelines and reduce in-page references. Use the Pipeline Execute Snap over other Snaps such as Task Execute, ForEach, Auto-router (i.e. Router Snap with no routes defined with expressions), or Nested Pipelines. Pipeline Reuse with Pipeline Execute Detailed documentation with examples can be found in the SnapLogic documentation for Pipeline Execute. Use Pipeline Execute when: The child Pipeline is CPU/memory heavy and parallel processing can help increase throughput. Avoid when: The child Pipeline is lightweight where the distribution overhead can be higher than the benefit. Additional recommendations and best practices for the Pipeline Execute Snap: Use Reuse mode to reduce child runtime creation overhead. Reuse mode allows each child Pipeline instance to process multiple input documents. Note that the child Pipeline must be a streaming Pipeline for reuse mode. Use the batching (Batch size) option to batch data (avoid grouping records in parent). Use the Pool size (parallelism) option to add concurrency. If the document count is low then use the Pipeline Execute Snap for structuring Pipelines else embed the child segment within the Parent Pipeline instead of using Pipeline Execute. Set the Pool Size to > 1 to enable concurrent executions up to the specified pool size. Set Batch Size = N (where N > 1). This sends N number of documents to the child Pipeline input view. Use Execute On to specify the target Snaplex for the child Pipeline. Execute On can be set to one of the below values: LOCAL_NODE. Runs the child Pipeline on the same node as the parent Pipeline. This is recommended when the child Pipeline is being used for Pipeline structuring and reuse rather than Pipeline workload distribution. This option is used for most child Pipeline executions. LOCAL_SNAPLEX. Runs the child Pipeline on one of the available nodes in the same Snaplex as the parent Pipeline. The least utilized node principle is applied to determine the node where the child Pipeline will run.This has dependency on the network, and must be used when workload distribution within the Snaplex is required. SNAPLEX_WITH_PATH. Runs the child Pipeline on a user-specified Snaplex. This allows high workload distribution, and must be used when the child Pipeline has to run on a different Snaplex for endpoint connectivity restrictions or for effective workload distribution. This option also allows you to use Pipeline parameters to define relative paths for the Snaplex name. Additional Pipeline design recommendations This section lists some recommendations to improve Pipeline efficiency SLDB Note: SLDB should not be used as a file source or as a destination in any SnapLogic orgs (Prod / Non-Prod). You can use your own Cloud storage provider for this purpose. You may encounter issues such as file corruption, pipeline failures, inconsistent behavior, SLA violations, and platform latency if using SLDB instead of a separate Cloud storage for the file store. This applies to all File Reader / Writer Snaps and the SnapLogic API. File Read from an SLDB File source. File Write operations to SLDB as a destination. Use your own Cloud storage instead of SLDB for the following (or any other) File Read / Write use-cases: Store last run timestamps or other tracking information for processed documents. Store log files. Store other sensitive information. Read files from SLDB store. Avoid using Record Replay Snap in Production environments as the recorded documents are stored in an SLDB path making them visible to users with Read access. Snaps Enable Pagination for Snaps where supported (e.g. REST Snaps, HTTP Client, GraphQL, Marketo, etc.). There should also always be a Pagination interval to ensure that too many requests are not made in a short time. Use the Group By N Snap where there is a requirement to limit request sizes. E.g. Marketo API request. The Group By Fields Snap creates a new group every time a record with a different Group Field value is received. Place a Sort Snap before Group By Fields to avoid multiple sets of documents with the same group value. XML Parser Snap with a Splitter expression reduces memory overhead when reading large XML files. Use an Email Sender Snap with a Group By Snap to minimize the number of emails that get sent out. Pipelines Batch Size (only available if the Reuse executions option is not enabled) is used to control the amount of records that are passed into a child Pipeline. Setting this value to 1 will pass a single record for each instance of the child Pipeline. Avoid using this approach when processing large volumes of documents. Do not schedule a chain reaction. When possible, separate a large Pipeline into smaller pieces and schedule the individual Pipelines independently. Distribute the execution of resources across the timeline and avoid a chain reaction. Integration API limits must not exceed across all integrations running at the same time. Group By Snaps or Pipeline Execute can be used to achieve this. Optimization recommendations for common scenarios Scenario Recommendation Feature(s) Multiple Pipelines with similar structure Use parameterization with Pipeline Execute to reuse Pipelines Pipeline Execute Pipeline parameters Bulk Loading to target datasource Use Bulk Load Snaps where available (e.g. Azure SQL - Bulk Load, Snowflake - Bulk Load) Bulk Loading Mapper snap contains a large amount of mappings where the source & target field names are consistent Enable “Pass through” setting on the Mapper. Mapper - Pass Through Processing large data loads Perform target load operation within a Child Pipeline using the “Pipeline Execute” snap with “Execute On” set to “LOCAL_SNAPLEX”. Pipeline Execute Performing complex transformations and/or JOIN/SORT operations across multiple tables Perform transformations & operations within SQL query SQL Query Snaps High Throughput Message Queue to Database ingestion Batch polling and ingestion of messages by: Specifying matching values for Max Poll Record (Consumer Snap) with Batch Size (Database Account Setting). Performing database ingestion within a child Pipeline with Reuse Enabled on the Pipeline Execute Snap. Consumer Snaps Database Load Snaps Table 4.0 Optimization recommendations Configuring Triggered and Ultra Tasks for Optimal Performance Ultra Tasks Definition and Characteristics An Ultra Task is a type of task which can be used to execute Ultra Pipelines. Ultra Tasks are well-suited for scenarios where there is a need to process large volumes of data with low latency, high throughput, and persistent execution. While the performance of an Ultra Pipeline largely depends on the response times of the external applications to which the Pipeline connects to, there are a number of best practice recommendations that can be followed to ensure optimal performance and availability. General Ultra Best Practices Before building an Ultra Pipeline, consult the “Snap Support for Ultra Pipelines” documentation to understand if the desired Snaps are supported. For optimal Ultra performance, deploy a dedicated Snaplex to support Ultra workloads. There are two modes of Ultra Tasks - Headless Ultra and Low Latency Ultra API with each mode being characterized by the design of the Pipeline which is invoked by the Ultra Task. The modes are described in more detail below. Headless Ultra A Headless Ultra Pipeline is an Ultra Pipeline which does not require a Feedmaster, and where the data source is a Listener or Consumer type construct, for example Kafka Consumer, File Poller, SAP IDOC Listener (For a detailed list of supported Snaps, please click here). The Headless Ultra Pipeline executes continuously and polls the data source according to the frequency configured within the Snap passing documents from the source to downstream Snaps. Use Cases Processing real-time data streams such as message queues. High volume message or file processing patterns with concurrency. Publish/Subscribe messaging patterns. Best Practices Deploy multiple instances of the Ultra Task for High Availability. Decompose complex Pipelines into independent Pipeline using a Publish-Subscribe pattern. Lower the dependency on the Control Plane by avoiding the use of expressions to declare queue names, account paths etc. Set the ‘Maximum Failures’ Ultra Task configuration threshold according to the desired tolerance for failure. For long running Ultra Pipelines, set a higher ‘Max In-Flight’ option to a higher value within the Ultra Task configuration. When slow performing endpoints are observed within the Pipeline, use the Pipeline Execute Snap with Reuse mode enabled and the Pool Size field set to > 1 to create concurrency across multiple requests to the endpoint. Additional reference: Ultra Tasks Low Latency API Ultra Low Latency API Ultra is a high-performance API execution mode designed for real-time, low-latency data integration and processing. The Pipeline invoked by the Ultra Task is characterized by having an open input view for the first Snap used in the Pipeline (typically a HTTP Router or Mapper Snap). Requests made to the API are brokered through a ‘FeedMaster Node’, guaranteeing at least once message delivery. Use Cases High frequency & high throughput request-response use cases. Sub-second response times requirement. Best Practices Deploy multiple Feedmasters for High Availability. Deploy multiple instances of the Ultra Task for High Availability running within the same Snaplex. Leverage the ‘Alias’ setting within the Ultra Task configuration to support multi Snaplex High Availablity. To support unpredictable high volume API workloads, leverage the ‘Autoscale based on Feedmaster queue’ instance setting in the Ultra task configuration. When slow performing endpoints are observed within the Pipeline, use the Pipeline Execute Snap with the Reuse mode enabled and the Pool Size field set to > 1 to create concurrency across multiple requests to the endpoint. Use the HTTP Router Snap to handle supported & unsupported HTTP methods implemented by the Pipeline. Handle errors that may occur during the execution of the Pipeline and return the appropriate HTTP status code within the API response. This can be done either by using the Mapper, JSON Formatter or the XML Formatter Snap. Reference request query parameters using the $query object. Set the ‘Maximum Failures’ Ultra Task configuration setting according to the desired tolerance for failure. For long running Ultra Pipelines, set a higher ‘Max In-Flight’ setting within the Ultra Task configuration. Triggered Tasks Definition and Characteristics Triggered Tasks offer the method of invoking a Pipeline using an API endpoint when the consumption pattern of the API is infrequent and/or does not require low latency response times. Use Cases When a batch operation is required within the Pipeline, e.g. Join, Group By, Sort etc. Integrations that need to be initiated on-demand. Non-real time data ingestion. File ingestion and processing. Bulk data export APIs. Best Practices Avoid deep nesting of large child Pipelines. Use Snaplex URL to execute Triggered Tasks for reduced latency response times. Handle errors that may occur during the execution of the Pipeline and return the appropriate HTTP status code within the API response. This can be done either by using the Mapper, JSON Formatter or the XML Formatter Snap. Use the HTTP Router snap to handle supported & unsupported HTTP methods implemented by the Pipeline. Parallelize large data loads using the “Pipeline Execute” Snap with Pool Size > 1Snaplex Capacity Tuning Guide
Introduction This document serves as a comprehensive best practice guide for developing efficient and robust Pipelines within the SnapLogic Platform. It offers guidelines that aim to optimize performance, enhance maintainability, reusability, and provide a basis for understanding common integration scenarios and how best to approach them. The best practices encompass various aspects of Pipeline design, including Pipeline behavior, performance optimization and governance guidelines. By adhering to these best practices, SnapLogic developers can create high-quality Pipelines that yield optimal results while promoting maintainability and reuse. The content within this document is intended for the SnapLogic Developer Community or an Architect, in addition to any individuals who may have an influence on the design, development or deployment of Pipelines within the SnapLogic platform. Authors: SnapLogic Enterprise Architecture team Snaplex Planning Snaplexes are a grouping of co-located nodes which are treated as a single logical entity for the purpose of Pipeline execution. The SnapLogic Control plane automatically performs load balancing of Pipeline workload within a Snaplex. Nodes in Snaplexes should be homogeneous, with the same CPU/memory/disk sizing and network configurations per node type (i.e. JCC / FeedMaster). The JCC and Feedmaster nodes in a Snaplex can be of different sizes. Examples of recommended configurations: Snaplex configurations JCC node count - 4 JCC node size for each node - Large Feedmaster node count - 2 Feedmaster node size for each node - Medium JCC node count - 4 JCC node size for each node - X-Large Feedmaster node count - 2 Feedmaster node size for each node - Large Object Definition Node A Node is a JVM (Java Virtual Machine) process which is installed on a server such as Windows or Linux. JCC Node The JCC node is responsible for: Preparation, validation, and execution of Pipelines. Send heartbeat to the Snaplogic Control plane indicating the health of the node. FeedMaster Node The FeedMaster node acts as an interface between the JCC nodes and the client. The main functions of a FeedMaster node are: Manage message queues. Send heartbeat to the SnapLogic Control plane indicating the health of the node. When setting up Snaplexes, it is recommended to plan out the number of Snaplexes to configure along with the usage criteria to achieve isolation across workloads. Snaplexes can be organized in various ways such as: Pipeline Workload - Organize Snaplexes by workload type: Batch, Low latency, and On-demand. Business Unit - Organize Snaplexes by business units. Geographical location - Organize Snaplexes by data center or geographic location. The recommendation is to use a combination of the above to optimize resource usage and achieve workload isolation. Snaplex Network Requirements Snaplexes should have the below network characteristics: Within a Snaplex: Less than 10 ms round trip latency between Snaplex nodes. Greater than 40 MB/sec throughput between Snaplex nodes. Snaplex to Control Plane: Less than 50 ms round trip latency to the SnapLogic Control plane. Greater than 20 MB/sec throughput to the SnapLogic Control plane. Pipeline Execute Pipeline execution using the Pipeline Execute Snap, nodes communicate with each other using HTTPS on port 8081. There is some resiliency to network failures and HTTPS requests are retried in the case of failures. Even though requests are retried, high network latency and dropped connections can result in Pipeline execution failures. Regular Pipeline executions run within a node, requiring no communication with other nodes in the Snaplex. When a Pipeline Execute Snap is used to run child Pipelines, there are three options: Option Comments LOCAL_NODE This option is recommended when the child Pipeline is being used for Pipeline structuring and reuse rather than Pipeline workload distribution. Use this option for most regular child Pipeline executions. LOCAL_SNAPLEX The network communication is optimized for streaming data processing since the child Pipeline is on the local Snaplex. Use this option only when workload distribution within the Snaplex is required. SNAPLEX_WITH_PATH This has high dependency on the network. The network communication is optimized for batch data processing since the child Pipeline is on a remote Snaplex. Use this option only when the child Pipeline has to run on a different Snaplex, either because of endpoint connectivity restrictions or for workload distribution. Ultra Pipelines The JCC nodes communicate with the FeedMaster nodes over TCP with SSL on port 8084 when executing Ultra Pipelines. The communication between nodes is based on a message queue. This communication is not resilient to network failure, so a reliable network is required between the Snaplex nodes for Ultra Pipeline processing. In case of any network failures, the currently processing Ultra requests will be retried or in some instances fail with errors. If there is a communication failure between the JCC and Feedmaster nodes, then the request will be retried for up to five times. This is controlled by the ultra.max_redelivery_count Snaplex configuration. There is an overall 15-minute timeout for an Ultra request to the Feedmaster that is configurable at the request level using the X-SL-RequestTimeout HTTP request header or at the Snaplex level by using the llfeed.request_timeout config setting. Note that both ultra.max_redelivery_count and llfeed.request_timeout are configured under Node Properties -> Global Properties for GroundPlexes. You can submit a support request to configure these properties for your Cloudplexes. Pipeline Load Balancing The Control plane performs load balancing for Pipeline execution requests on a Snaplex. The following table lists the configurations that are involved: Property / Threshold Where configured Default value Comments Maximum Slots Node properties tab of the Snaplex 4000 One slot = One Snap = One active thread on the node A percentage of slots (configurable with the Reserved slot % property) are reserved for interactive Pipeline executions and validations thru the Designer tool. Pipelines will be queued if the threshold is reached. Some Snaps such as Pipeline Execute, Bulk loaders, and Snaps performing input/output, can use a higher number of threads compared to other Snaps. Maximum memory % Node properties tab of the Snaplex 85 (%) Threshold at which no more Pipelines will be assigned to a node Snaplex node resources (CPU, FDs, Memory) Node server configurations Configurable If the Control plane detects that there are not enough resources available on the Snaplex, then the Pipeline execution requests will be queued up on the control plane, and resume when resources are available. The Control plane dispatches the Pipeline to the node which has the most available capacity in terms of CPU/memory and file descriptors. For child Pipeline executions using the Pipeline Execute Snap, there is a preference given for running the child on the local node to avoid the network transfer penalty. Table 1.0 Configurations for Pipeline load balancing Snaplex Resource Management Capacity Planning This section provides some guidelines for Snaplex capacity planning and tuning. Configuration / Use-case Comments Workload isolation Isolate workloads across Snaplexes based on workload type, geographic location, and business unit. Node sizing Size the node (CPU, RAM, disk space) in a Snaplex based on Pipeline workload type. Batch data processing needs larger nodes while Streaming/API processing can use smaller nodes. Maximum Slots One slot = One Snap = One active thread on the node A percentage of slots (configurable with the Reserved slot % property) are reserved for interactive Pipeline executions and validations thru the Designer tool. Pipelines will be queued if the threshold is reached. Some Snaps such as Pipeline Execute, Bulk loaders, and Snaps performing input/output, can use a higher number of threads compared to other Snaps. The general recommendation is to configure this property based on the node memory configuration. Example: 8 GB - 2000 Slots 16 GB - 4000 Slots API Workloads For API workloads, the rule of thumb is to have 100 active ultra API calls per 8 GB of RAM, or 20 active triggered API calls per 8 GB of RAM. So a 16 GB node can have 200 active ultra API calls or 40 active triggered API calls. Node sizing The number of nodes in a Snaplex can be estimated based on the count of batch and streaming Pipelines. The number of FeedMaster nodes can be half of the JCC node count, with a minimum of two recommended for high availability. For active Pipeline count estimates, error Pipelines can be excluded from the count since they do not consume resources under the normal workload. Table 1.1 Configurations for Snaplex capacity planning Capacity Tuning Below are some best practices for Snaplex capacity tuning: Configuration / Use-case Comments Slot counts The Maximum slot count can be tuned based on the alerts and dashboard events. It is not required to restart the nodes for this configuration to take effect. Queued Pipelines - Increase slot count by 25% Busy nodes - Reduce slot count by 25% The slot count should not be set to more than 50% above the recommended value for the node configuration. e.g. The recommended slot count on a node with 16 GB RAM is 4000. Setting it to higher than 6000 is not advisable.If you observe high CPU / memory consumption on the node despite lowering the slot count by 25%, then consider allocating additional resources to the Snaplex nodes. Workloads Batch Workloads: Expand the node memory up to 64 GB, and deploy additional nodes for increased capacity. API Workloads: Deploy additional nodes instead of expanding the memory on the current node. Active Pipelines As a general rule, it's suggested to maintain fewer than 500 active Pipeline instances on a single node. Exceeding this threshold can lead to communication bottlenecks with the Control plane. If the number of active Pipeline instances exceeds 500, then the advisable course of action is to consider the addition of more nodes. CPU CPU consumption can be optimized by setting the Pool size and Batch size options on Pipeline Execute Snaps. Memory See Table 3.0 below Additional Reference: Optimizations for Swap Memory Table 2.0 Configurations for Snaplex capacity tuning Memory Configuration thresholds Property / Threshold Where configured Default value Comments Maximum memory % Node properties tab of the Snaplex 85 (%) Threshold at which no more Pipelines will be assigned to a node Pipeline termination threshold Internal (Can be configured by setting the feature flag at the org level com.snaplogic.cc.snap.common.SnapThreadStatsPoller. MEMORY_HIGH_WATERMARK_PERCENT) 95 (%) Threshold at which the active Pipeline management feature kicks in and terminates pipelines when the node memory consumption exceeds the threshold. Ideal range: 75-99 Pipeline restart delay interval Internal (Can be configured by setting the feature flag at the org level com.snaplogic.cc.snap.common.SnapThreadStatsPoller. PIPELINE_RESTART_DELAY_SECS) 30 (seconds) One Pipeline is terminated every 30 seconds until the node memory goes below the threshold (i.e. goes below 95%) Table 3.0 Snaplex node memory configurations The above thresholds can be optimized to minimize Pipeline terminations due Out-of-Memory exceptions. Note that the memory thresholds are based on the Physical memory on the node, and not the Virtual / Swap memory. Snaplex Alerts SnapLogic supports alerts and notifications through email and Slack channels. These can be configured in the Manager interface under Settings. The recommended alerts are listed in the table below. Alert type Comments Snaplex status alerts Status alerts can be created at the org level or the Snaplex level (in the Snaplex properties). These allow notifications to be sent when the Snaplex node is unable to communicate with the SnapLogic control plane or there are other issues detected with the Snaplex. Snaplex Resource usage alerts Set up alerts for these event types: Snaplex congestion Snaplex load average Snaplex node memory usage Snaplex node disk usage Table 4.0 Recommended Snaplex Alerts Reference: Alerts Slack notificationsSigma Framework for Operational Excellence
Sigma Framework for Operational Excellence Operationalizing innovation to maximize business value Modern Digital Platforms such as SnapLogic empower enterprises to unlock the value held in existing systems and data, allowing new services and strategies to be achieved. To support these best practices, governance and a culture of innovation are needed. Enterprises are adopting or evolving the concept of a Center of Excellence to foster and support Digital Transformation. This concept is far from new, but trends such as Agile, DevOps, API strategy, AI, and the rise of the Citizen Developer have also forced these Operating Models to change, or risk being considered outdated and relegated as not fit for purpose to support Digital Transformation. The Sigma Framework for Operational Excellence is a set of best practices distilled from the experience of SnapLogic architects — customers, partners and employees. SnapLogic is dedicated to continuously increasing the value our technology delivers to customers over time. Meanwhile, SnapLogic customer architects require a formalized operational framework to guide them from an initial tactical project to true strategic value realization through adoption across multiple roles and departments. The Sigma Framework delivers a standardized, integrated, holistic, and opinionated set of cross-disciplinary best practices, ready for adoption by customers to support sophisticated enterprise-scale deployments. Different assets in this section are designed for the various phases of a customer journey, addressing different personas and stages of maturity. By adopting the Sigma Framework for Operational Excellence, users will be able to extract maximum value from their investment in SnapLogic — both the technology, and the time commitment to develop required skills and competence. Scope of the Framework Sigma Framework for Operational Excellence is structured to reflect the way your enterprise works. We coordinate, collaborate, and consolidate expertise between lines of business, development teams, and your enterprise architecture group. leadership and advice. Let us help you operationalize and scale data-driven, 360- degree innovation throughout your organization. The Sigma Framework for Operational Excellence encompasses several domains, including strategy, technology, process, and the culture of the users of the SnapLogic Platform. The aim of the Sigma Framework is to facilitate the adoption of SnapLogic at scale across the customer organization to achieve the required positive business outcomes. This framework will be shared with the SnapLogic community and maintained and developed as new technical capabilities and implementation best practices emerge over time. Notes 12/11/2023: Diagram updated for style. PDF now in brand formatting. No content changes.Project Structures and Team Rights Guide
Overview This document outlines the recommended organizational structure for Project Spaces, Projects and team rights within your SnapLogic org. Authors: SnapLogic Enterprise Architecture team Integration Storage Hierarchy In the SnapLogic platform, integrations (pipelines) are managed within the following hierarchy: Organization - Multiple customer organizations are configured based on their development environment e.g. DEV, QA, STAGING, UAT, and PROD etc. Project Space - Team in a Business Unit or Business Group with an access to a workspace to collaborate and implement integration at one central location Project - Group pipelines with an integration, aggregation, and reporting required for the type of function Pipeline - A specific implementation of an integration For example, the path “/MyDevOrg/SLEntArch/SamplePipelines/WorkdayToSnowflake” is broken down into the following components: Organization - MyDevOrg Project Space - SLEntArch Project - SamplePipelines Pipeline - WorkdayToSnowflake Recommended Hierarchy Naming Conventions Clarity is one of the most important factors when naming projects spaces, projects, and pipelines. Naming each level of the hierarchy must make sense to all developers and administrators so that integrations can be found quickly and easily. If using Triggered Tasks, it is important to ensure that no characters are used that will violate HTTP URI naming conventions. Further, you may wish to use characters that don’t require an URL encoding, such as spaces, commas, etc. Below is an example naming convention for your project spaces, projects, and pipelines: Project spaces are named based on business unit or project team: Sales_and_Marketing EnterpriseBI ProfessionalServices Projects are named based on business target endpoint Sales_and_Marketing/MarketingForecast EnterpriseBI/SalesDashboardReporting ProfessionalServices/CommunityExamples Integrations (pipelines) are named based on business function Sales_and_Marketing/MarketingForecast/Fall_Marketing_New_Customers_To_SalesForce EnterpriseBI/SalesDashboardReporting/Supplier_Invoices_To_Workday ProfessionalServices/CommunityExamples/Community_27986_Example_JSON_MultiArray_Join Keep in mind the shallow hierarchy (project space/project) when considering your naming scheme for project spaces and projects. In most orgs, it is acceptable to assign a project space to each business unit and allow that business unit to create projects within their project space based on integration function or target. However, If you expect a very large number of pipelines to be created by a single business unit, you might want to consider an allowance for multiple project spaces for a given business unit. Shared Folders Root Shared A special project named “shared” is added to each SnapLogic Organization (org). Using the org name in the above example, this would be /MyDevOrg/shared. This is commonly referred to as the “root shared” folder. This folder will always exist and is automatically assigned with Full Access (read, write, execute) to all members of the “admins” group of the org and Read-Execute access to all other users. As a best practice, the root shared folder should only contain objects (accounts, files, pipelines, and tasks) that all SnapLogic users in your org should have access to use. Some examples may include: SMTP account used for the Email Sender snap Readonly database account used to access common, public tables Shared expression libraries that contain global static variables or user defined functions for common string/date manipulation Shared pipelines such as error handlers or globally re-usable code Project Space Shared Another special project named “shared” is added to each project space in the org. Using the example path above, this would be /MyDevOrg/SLEntArch/shared. This folder will always exist under each project space and inherits the permissions assigned to the project space. As a best practice, the project space shared folder should only contain objects (accounts, files, pipelines, and tasks) that all SnapLogic users with access to the Project Space should have access to use. Some examples may include: Database accounts used to access common tables within your business unit Shared expression libraries that contain static variables and user defined functions common to your business unit Shared/reusable pipelines common to your business unit User Groups We recommend that you create the following Groups in all of your SnapLogic orgs: Operators - this group contains the users that may need to manually execute a pipeline but do not require Full Access to the projects Migrators - this group contains the users that will perform object migrations in your orgs but do not need to Execute pipelines You should also create Developer groups specific to each Project Space and/or Project within your org. Using the example project spaces and projects listed in the Naming Conventions section of this document, you may want to add the following Groups: Groups specific to Project Space Sales_and_Marketing_Developers EnterpriseBI_Developers ProfessionalServices_Developers Groups specific to Project MarketingForecast_Developers SalesDashboardReporting_Developers CommunityExamples_Developers You may choose to enable access for developers to only see objects within the project they are working in, or you could allow read-only access to all projects within their project space to allow for some cross-project design examples. Typically, the Developer groups will have Full Access in your development org for the Projects that they are working in, with Read-Execute access to the Project Space “shared” folder and Read-Execute access to the root “shared” folder. Developer groups will also have Read-Only access in all non-development orgs for the same Project Space “shared” and Projects that they can access in your development org. If you have a larger SnapLogic development community in your organization, you may wish to distribute the administration of Projects and create Admin groups for each Project Space who will be assigned ownership of the Project Space, which allows them to create new Projects and maintain all permissions within the Project Space. Default Users We recommend that the following service accounts be added to your org(s): snaplogic_admin@<yourdomain> - this user should own the root SnapLogic shared folder, and all/most SnapLogic Project Spaces in your org(s); add this user to the “admins” group snaplogic_service@<yourdomain> - this user should own all of your SnapLogic tasks and have permissions related to executing tasks for all Projects. Note that Read-Execute Access is required as a minimum; Full Access is required if any files are written back to the SLDB of the Project during processing. Add this user to the “Operators” group Note that during migration of tasks to your non-development org(s), you should either use the snaplogic_service@<yourdomain> user to perform the migration, or use the Update Asset Owner API to change the owner of the task after migration. Tasks are owned by the user that creates it; so if a user in the Migrators group performs the migration, they will be assigned as the owner and may not have permissions to successfully execute the task in the target org(s). Hierarchy Permissions Recommended access to the root “shared” project: admin@snaplogic.com - Owner “admins” group - Full Access “members” group - Read/Execute Access “Operators” group - Read/Execute Access “Migrators” group - Full Access “Support” group - Read/Execute Access You may wish to limit Execute Access to only certain teams. If so, change the “members” group to Read Only Access and grant Read/Execute Access to your desired team groups. If you perform migrations only within specific day/time windows, you can add/remove users from the Migrators group using a scheduled task that calls the Groups API to replace all members of the Migrators group and either remove all users from the group (close the migration window) or restore users to the group (open the migration window). Recommended access to the Project Space “shared” project: admin@snaplogic.com - Owner “admins” group - Full Access “members” group - Read-Only Access (optional) “Operators” group - Read/Execute Access “Migrators” group - Full Access “<Project>_Admins” group(s) - Full access in development “<Project>_Developers” group(s) - Read/Execute Access in development You may choose to grant Read-Only access to your <Project>_Admins and <Project>_Developers groups in non-development environments depending on your support team structure Recommended access to the Projects: admin@snaplogic.com - Owner “admins” group - Full Access “members” group - Read-Only Access (optional) “Operators” group - Read/Execute Access “Migrators” group - Full Access “<Project>_Admins” group(s) - Full Access (only in development) “<Project>_Developers” group(s) - Full Access (only in development) You may choose to grant Read-Only access to your <Project>_Admins and <Project>_Developers groups in non-development environments depending on your support team structure.Collaborative Development Best Practices
Collaborative Development Best Practices Sharing knowledge to ensure repeatable project success Executive Summary This guide is designed for organizations wishing to take full advantage of the potential value of the SnapLogic integration platform through adoption of field-tested best practices for collaborative development. These policies and advice will help users develop smoothly from the technical success of an initial project, to repeatable success of follow-on projects, enabling true digital transformation. The SnapLogic pla:orm offers a number of capabilities that support the goal of achieving greater organizational maturity and increased collaboration. This document aims to highlight some of these and place them in the correct context for users to be able to determine the precise steps that are most relevant in their own environment and organization. By implementing these best practices, SnapLogic users in both IT and line-of-business teams will be able to optimize their efforts and ensure the delivery of maximum value from the original investment in SnapLogic. This document is not an exhaustive how-to guide; the SnapLogic online product documentation remains the authoritative source for specific configurations. In addition, SnapLogic offers a number of consultative programs for customers who are looking to accelerate their digital transformation. Please consult your account manager to find out more about which of these resources may be right for your situation. Introduction The graphical low-code/no-code nature of the SnapLogic development environment makes it easy to get started and build useful automation that solves a user’s pressing business problem, lets them answer an urgent question about their data, or enables them to integrate previously disconnected systems via API. The introduction of generative integration through SnapGTP further lowers the barrier to entry, capturing users’ intent in natural language and generating the corresponding pipeline or query automatically. This rapid prototyping and development is key to enabling users to achieve their goals quickly, which in turn ensures rapid return on investment (RoI) for the adoption of SnapLogic technology. SnapLogic’s integration platform as a service (iPaaS) is a full-fledged development environment, but it can sometimes be under-estimated, precisely because of how easy it is to use. IT teams may not think to put in place the sorts of best practices that they would for other, more traditional development tools. In turn, users outside of IT may not have the specific experience to expect such best practices to be implemented. The risk is that multiple projects may start in parallel and operate in isolation, resulting in duplication of effort and un-optimized use of infrastructure. SnapLogic architects — employees, partners, and customers — have developed best practices that should be followed to ensure the success of collaborative development projects. These general principles are distilled from hundreds of engagements where users were able to transition from tactical, piecemeal automation to true enterprise integration, enjoying the benefits of that deep architecture work. These recommendations are ready to be adopted and adapted to specific environments and existing policies and standards. Shared Center of Excellence The first important recommendation is that a central point of contact be defined for all activities relating to SnapLogic. This group should not be seen as the “owners” of the platform in the sense that they might be with more traditional development environments, developing components in response to requests from users in line-of-business teams. Instead, they should act as expert consultants, combining a close understanding of the wider business priorities with a specific understanding of the local SnapLogic deployment. The Center of Excellence (CoE) should administer the SnapLogic platform, deploying, scaling, securing, and retiring components as necessary. Beyond these inward-facing tasks, the CoE should also ensure the integration of the SnapLogic platform into existing operational management systems, making logs and events from the SnapLogic infrastructure and pipelines visible and accessible to central SRE teams. Another useful point of integration is version control and backup, ensuring that business-critical integrations developed inside SnapLogic are managed with the same level of care and attention as those built with more traditional development tools. While much of the routine maintenance work can be carried out entirely within an IT group, certain actions may require consultation with users. Examples of technical actions to support end-user needs include: Scaling the SnapLogic infrastructure to match planned business growth Deploying Snaplexes in a distributed fashion to satisfy business requirements (regional access, performance, regulatory or policy compliance) Managing credentials and access required for integration with external systems, ensuring those integrations are available to users while also enforcing security mandates Reviewing new or proposed developments against best practices Developing shared components to avoid duplication of effort and facilitate standardization The CoE should be staffed primarily by employees of the customer company or long-term consultants who have a deep understanding of the business context within which the SnapLogic pla:orm has been adopted and is expected to operate. Strong relationships with users in line-of-business (LoB) teams will facilitate an ongoing consultative engagement between the CoE and end-users. Customers who desire specific support may choose to augment their own CoE with a Resident Architect (RA), an expert SnapLogic employee who is assigned to support a specific customer organization. The RA will share best practices and guide the CoE personnel in the most effective deployment and use of SnapLogic technology, ensuring accelerated return on the investment in SnapLogic technology. Shared Component Library — Design for Reuse In traditional, text-oriented development, it has been best practice almost since the beginning of programming to establish shared libraries of functionality. With increasing levels of abstraction in the design of programming languages themselves, these shared libraries moved from straigh:orward re-use of code through copying, to becoming external resources that can be called upon as easily as internal components of a program. In the same way, mature organizations will ensure that pipelines are created with the expectation of reuse built into their design. These reusable pipelines should be made available as part of a shared component library. SnapLogic offers a number of Pattern Pipelines for this purpose as part of the Cloud Pattern Catalog. Meanwhile, Pattern Pipelines created within the customer organization will be listed under Projects. In both cases, the Pattern Pipelines are available for immediate use, but will generally require users to supply credentials and potentially some minor environment customization. This organization-level shared library should be owned by the CoE in order to ensure that expected standards are followed and all required documentation is provided. The CoE may choose to develop components directly, harvest components that have been created for specific purposes and develop them into reusable components, or some combination of the two. SnapLogic allows for the same mechanisms to be implemented as in any other development suite. The main principles that apply are the following: Modularity: break complex workflows up into smaller pipelines, reusing as appropriate Loose coupling: avoid dependencies between pipelines, or on specific environmental concerns High cohesion: grouping related functionality in a pipeline helps ensure robustness, reliability, reusability, and understandability Information hiding (encapsulation): design pipelines so that areas expected to change regularly are grouped together to avoid regular wide-ranging changes Separation of concerns: concentrate interactions with particular third-party systems, in order to facilitate any future changes which might be required (e.g. breaking upgrades or migrations) Pipeline Nesting The simplest form of component reuse is nesting pipelines, that is, invoking one pipeline from another. This action relies on the Pipeline Execute Snap, which allows for one pipeline to call another, ensuring that complex or commonly-required functionality can be built once and made available wherever it is needed. The called pipeline can also execute on a different Snaplex than the parent pipeline, which enables further architectural flexibility. For instance, a pipeline running on less-secure infrastructure such as a DMZ could access functionality that it would not otherwise have direct connectivity with, by invoking a pipeline that executes on secured infrastructure. The benefit here is twofold: the sensitive functionality is only accessed by a single pipeline that can be exhaustively secured and debugged, and network segmentation can be used to limit access to that single point. Configurable Pipelines Users can also create an expression library file that contains the details of the environment and import that configuration into pipelines as required. For example, individual teams within the wider organization might have their own specific configuration library with the appropriate settings for the environment. Pipelines can then be moved around from one environment to another without needing to be updated, further facilitating sharing. The same mechanism should be used to facilitate promotion between development, test, and production environments, without needing to reconfigure the pipelines themselves. These two goals can support one another, further stimulating adoption of best practices. Error Handling A particular type of pipeline that may well be overlooked by users without specific development experience is the Error Pipeline. The baseline requirement of this type of pipeline is to handle errors that may occur during execution of the business logic in the main pipeline. However, in a larger and more articulated organization, especially where there is extensive code reuse, error pipelines will also need to be standardized. Standardization of error pipelines satisfies two requirements: Handling of common error conditions – It is expected that many pipelines in an organization will interact with common systems and therefore will encounter the same or similar error conditions. These situations should be handled in a uniform fashion, both to avoid duplicate work to handle the same error, and to ensure that the error condition is indeed handled correctly every time it occurs, regardless of which pipeline has triggered it. Easier troubleshooting – when users are presented with an error message, rapid resolution requires that it be easy to understand and that it be obvious which actions can be taken (and by whom) to resolve the issue. One of the best ways to deliver on this goal of reducing mean time to resolution (MTTR) is to ensure that the same situation will always trigger the same clear and comprehensible error message. An Error Pipeline is simply a pipeline that is created specifically for handling errors; it processes error documents produced by other Pipelines in the environment. The Error Pipeline runs even if errors are not encountered in the Snaps from the main Pipeline. Error Pipelines also capture the data payload that triggered the error, which can be vitally important to error resolution. Since the SnapLogic platform will not store customer data, the error pipeline is the most standardized way to capture the actual data which was being processed by the pipeline when the error occurred. Architecture Best Practices for Collaboration Environment Separation It is recommended to create separate environments, at a minimum for development and production, and ideally with an explicit testing area as well. This practice will help to avoid production outages, since errors can be found and corrected in the pre-production environments, before the pipeline is promoted to production where errors would have greater impact. Different permissions and access controls should be set on the different environments. The development environments should be as open as possible, to enable the maximum number of users to take advantage of the ease of use of the SnapLogic platform to deliver business value. Promotion to testing and production environments on the other hand should be tightly restricted and controlled. SnapLogic offers a powerful hierarchical model of assets and containers, with individual permissions to enable the desired security level to be set for each item. The Configurable Pipelines feature helps to facilitate promotion of pipelines between environments without the need for manual reconfiguration, which itself would risk introducing errors. By configuring the pipelines in this way, the promotion process can be made entirely automatic. This automation in turn makes it possible to include automated testing and rollback as a routine part of the promotion process. Operational Best Practices Versioning and Backup As the SnapLogic platform is used and new automations are developed, significant business value will be captured within the pipelines that are created. The impact of the loss of any of these developments would be correspondingly large, whether due to human error or technical failure. The built-in versioning capabilities allows users to replace an existing pipeline with a newer one or to rollback to a previous version in the event that something went wrong. Each version is tagged automatically with information about its creator, as well as the time and date of creation, but it is recommended that users also document the state and purpose of the pipeline in the notes. It is recommended that users take advantage of these features to safeguard the valuable automations and integrations that users develop over time, and to avoid the negative consequences for the organization that would occur if those were to be lost. The CoE should work to make sure that users without a background in development be made aware of the importance of versioning their work correctly. In addition to these built-in capabilities, the SnapLogic iPaas offers out-of-the-box integration with industry-standard Git version-control, and specifically the following implementations (as of Fall 2023): GitHub GitHub Enterprise Server (GHES) GitLab.com GitLab (self-managed) Azure Repos Through Git integration, users can track, update, and manage versions of SnapLogic project files, pipelines, tasks, and accounts, using either the SnapLogic graphical interface or the SnapLogic Public APIs. These operations include checkout, add, remove, pull, and commit, as well as creating branches and tags. Git integration supports both Gitflow and trunk-based development; however, SnapLogic recommends trunk-based development. Simultaneous Edits It is important to point out that SnapLogic does not currently support simultaneous collaboration in the same pipeline. If one user is editing a pipeline and a different user makes a change to the same pipeline, the original user will be prompted that a new version of the pipeline exists. This is another reason to adopt explicit version management strategies, in order to avoid confusing users who may be used to simultaneous collaborative editing in other tools such as Google Docs. Secrets Management Secrets Management is an established best practice in traditional development environments, enabling organizations to use a third-party secrets manager to store endpoint credentials. Instead of entering credentials directly in SnapLogic accounts and relying on SnapLogic to encrypt them, the accounts contain only the information necessary to retrieve the secrets. During validation and execution, pipelines obtain the credentials directly from the secrets manager. With Secrets Management, the configured secrets are never stored by the SnapLogic control plane or by Snaplex nodes, facilitating security hardening and simplifying compliance with internal or external policies. Currently, SnapLogic support integration with the following secrets managers (as of Fall 2023): Avoid relying on the tool to monitor itself Reduce Total Cost of Ownership (TCO) by managing monitoring, alerts, and notification system in one place Manage the historical retention of data for audit needs Correlate data from different tools in order to understand correctly the cause and impact of an issue, avoiding duplication of troubleshooting efforts The SnapLogic platform uses OpenTelemetry (As of Fall 2023, this feature is available as a beta release, enabling monitoring of Pipeline execution runtime logs in Datadog.) to support telemetry data integration 3 with third-party Observability tools. Conclusion The SnapLogic iPaaS is a full-fledged development environment, and should be considered as such when it comes to both management of assets developed within the platform, and industry-standard integration with third-party tools. The ease of use of SnapLogic naturally lends itself to wide adoption by users outside of IT, and this trend is only expected to accelerate further with the introduction of generative integration via SnapGPT. The speed with which these users can achieve their goals and deliver business value may however be overlooked by centralized IT teams, leading them to underestimate the potential business value that can be achieved through implementation of industry-standard best practices for development environments. Fortunately, most of these well-understood best practices — collaborative development, modular architectures, staged infrastructure, and integration with operational management tools — are available out of the box with SnapLogic. All that is required is active engagement between core IT, the CoE team responsible for the SnapLogic, and the end-users themselves, who are after all the ones closest to the business value that is being delivered by the platform.Disaster Recovery
How SnapLogic will recover the platform and data in the event of a disaster; and a forward look at SnapLogic’s disaster recovery strategy. Introduction The purpose of this whitepaper is to present SnapLogic’s method and means to recover from a disruptive event to ensure workloads run as expected and remain durable. Background Businesses create and manage large volumes of data. As companies extend their use of artificial intelligence into daily operations, data becomes mission critical. The impact of data loss or corruption can be significant. Preparing plans to continue business in response to an event is not only good management, but essential in today’s business climate. At SnapLogic, we are revising our view of resiliency and taking a broader approach. Beyond the standard disaster recovery and cybersecurity requirements, real resiliency should include being able to handle a full range of responses a company needs to keep its business going, whatever happens. SnapLogic’s Disaster Recovery Model SnapLogic’s infrastructure is deployed on Amazon Web Services. When users access the application, their request is filtered through a load balancer which is a cloud networking device to help distribute large volumes of internet traffic. Requests are passed from the load balancer to a cluster of web servers which present the application(s) to the user. The application servers, sometimes referred to as the Control Plane, facilitate and manage pipeline creation, updates, and monitoring. The data execution servers, sometimes referred to as Cloudplexes, Groundplexes, or the Data Plane, are the heart of pipeline execution. Database servers store pipeline metadata and log data each time a pipeline is run. The last component are Snaps, sometimes referred to as connectors or APIs, they are small computational units that process data. These assets are monitored for performance 24x7x365. If performance drops below a certain threshold, an automated process notifies the appropriate team(s) to investigate and if necessary, recover or replace an infrastructure asset. Assets are managed using a combination of automated and manual processes. The presentation, data execution, and database layers would be restored manually. The application and Snaps layers leverage fully automated processes. In the event of a disaster, SnapLogic’s end to end recovery time for the entire infrastructure is within 48 hours of being hard down (inoperable). The amount of data loss would be 2 hours or less. The data execution servers, sometimes referred to as Cloudplexes, Groundplexes, or the Data Plane, have a shared responsibility for resiliency. Groundplex resiliency is solely the responsibility of the customer. Cloudplex resiliency is shared. SnapLogic is responsible for the resiliency of the infrastructure, once recovered, the customer is responsible for recovering / restarting their pipeline(s). This outlines SnapLogic’s disaster recovery model. If you have any questions about this white-paper or would like to discuss resiliency in more detail, please contact your SnapLogic representative who can schedule a deeper discussion.
