Basics of SnapLogic
Introduction SnapLogic is a cloud-based integration Platform-as-a-Service (iPaaS) that provides tools for connecting various applications, data sources, and APIs. It enables businesses to automate and streamline their data integration processes by offering pre-built connectors and a visual interface for designing integration workflows. The SnapLogic platform uses a SnapLogic pipeline, a series of connected "Snaps" (pre-built components) that define the flow and transformation of data between various systems and applications. In a SnapLogic pipeline, data flows from one Snap to another, with each Snap performing a specific function, such as data extraction, transformation, or loading (ETL). SnapLogic Designer The SnapLogic Designer is the user interface that enables you to develop pipelines. You can see the example page below. But in SnapLogic with a feature called “Asset Palette,” you may see the different styles of Side Panel view. But the features are the same as those of the side panel view. The designer page consists of three main parts: Canvas - The field for visualizing and editing the pipeline Side Panel / Asset Palette - The panel contains the menu list. (The left picture is the Side Panel view. The right picture is Asset Palette enabled) Snaps Catalog - lists all available Snaps. https://docs-snaplogic.atlassian.net/wiki/x/ePIV Pipelines Catalog - list all pipelines that you can access. https://docs-snaplogic.atlassian.net/wiki/x/w-IV Patterns Catalog - list all the patterns that you can access. https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/3022160260/Patterns+Catalog Toolbar - The list of tools for the pipeline Icons Description Execute Pipeline Execute the pipeline. Validate Pipeline Validate the pipeline. Any unsaved changes will be saved before validation. Clicking the button while a validation is in process cancels that validation. Shift-clicking the button will clear the cache before validating. Edit Pipeline Properties You specify properties when creating a pipeline. Click this button to modify the properties. Check Pipeline Statistics As a pipeline executes, the statistics are updated periodically so that you can monitor its progress. Create Task Create a Task for the current pipeline. Save Pipeline Save the current pipeline. Export Pipeline Export the current pipeline. Copy Pipeline Copy the pipeline from one project to another. Move Pipeline Move the pipeline from one project to another. Delete Pipeline Delete the current pipeline Pipeline Versions Create versions of the pipeline. Compare Pipeline Compare the current pipeline with the target pipeline. Notes Add a note or delete an existing note. Notes are saved with the pipeline. Print Print the pipeline. Snaps Snaps are the building blocks of a pipeline. Each Snap performs a single function, such as reading, parsing, transforming, or writing data. You can view the Snaps available to you (or your account) in the Snaps Catalog on the left-hand side of the SnapLogic Designer. You can drag a Snap from the Snap Catalog onto the Canvas to use it in a pipeline. Snaps Type SnapLogic includes the following basic types of Snaps with distinct icons. Icon Snap Type Description Read Specifies data sources in the pipeline. Examples: File Reader, CSV Generator, Birst Query Parse Takes the input of unstructured data and generates an output of structured data. Examples: XML Parser, Sequence Parser, JSON Parser Transform Modifies data significantly. Examples: Mapper, Aggregate, Join Flow Changes the output or direction of data in a pipeline. Examples: Router, Gate, Union Format Changes the data format. Examples: CSV Formatter, JSON Formatter, Excel Formatter Write Specifies data destinations in a pipeline. Examples: File Writer, REST Post, Email Delete Connecting Snaps The key to creating a Pipeline in SnapLogic is connecting Snaps. There are a few things to consider when placing Snaps in a Pipeline. Connection Shapes Like puzzle pieces, only Snaps with matching connection pairs (circles or diamonds) can be connected between the input and output of two snaps. When you drag a snap and place it next to or in front of another snap, the snap will automatically connect both snaps, and the connection will change color, which means it connects successfully. If the color doesn’t change, you need to recheck that both connection shapes are the same and re-connect it again. Disconnect Linked Snaps Unlinked Snaps can be moved apart or placed next to each other. Make sure the circle or diamond connector is colored Blue, which indicates that the Snaps are linked. To disconnect linked Snaps, click on the Blue connector. This clears the color and allows you to rearrange the Snaps. Remote-Connect Link Snaps You can connect to Snaps, but not next to each other, using a remote-connect link. For example, click and hold on the Mapper Snap connector until it turns Yellow, then drag it to the Copy Snap connector. When both connections turn Blue, release the mouse button. A number is placed in both connectors to let you know they are connected. Note: The number is only temporary until the Pipeline is saved. At this point, a new, permanent number may be assigned. You can also click and hold on one connection, and both Snaps connected by this link will darken. This feature is helpful for large pipelines where it may take much work to visualize the connections quickly. Data model SnapLogic will pass the data between Snaps with two models: Document data The document data models will be represented by a circle shape. This data type uses the JSON format as a container of the data. The support data type in this model is similar to the JSON standard in that it includes string, boolean, number, array, object, and null. Binary data The document data models will be represented by a diamond shape. This data type will wrap the binary data in SnapLogic’s model. Mostly, this will be inputted to the file writer and parser and outputted from the file reader and formatter. Configuration Snaps You have two options to open the configuration dialog. First, left-click on the Snap that you want to configure. The dialog will show up immediately. The second way is right-clicking at the Snap, and the menu displays options available in all Snaps through a dropdown list will be shown. Then click “Edit” in the menu. Each Snap will have different configurations. You can learn more about the configuration of each snap by clicking the question mark icon on the top right of the dialog. Expression The SnapLogic expression language is a utility that is available to Snaps. You can use expressions (JavaScript syntax) to access functions and properties to set field values dynamically. You can also use the expression language to manipulate data. Example $text == "NFL" ? "foo" : "bar" $counter > 1 ? ($counter < 3 ? 50 : 100) : -1 Expressions are available across multiple Snaps. If the Snap exposes the functionality of the expression for a property, then the icon appears in front of the property's text box. You can toggle on or off by clicking on the icon. When the toggle is on, the down arrow within the field will appear. You can click to see the list of functions and properties available. Operations List of supported and unsupported operations available on (document https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438042/Understand+Expressions+in+the+SnapLogic+Platform) Accessing Pipeline Parameters Parameters allow a pipeline to be reused in multiple situations. For example, a File Writer Snap can be configured to write to a file path specified by a parameter, which allows the same pipeline to write to different files. The parameters for a pipeline can be defined by using the Edit Pipeline properties dialog. The name of each parameter must only contain alpha-numeric characters, and the value will be converted to a string. The value for a parameter defined in the pipeline properties dialog is treated as the default when running the pipeline in Designer. Parameters can also be passed to the Pipeline Execute Snap. Any parameters not passed down from the Task or Snap will use the defaults specified in the properties dialog. To access a pipeline parameter from the expression language, you must prefix the parameter name with an underscore. For example, given the following parameters: Key Value firstName Bob numValue 12 path $.age The "firstName" parameter can then be accessed using _firstName, as in: "Hello, " + _firstName // result: Hello, Bob Since the value of a parameter is always a string, you'll need to convert any string to numeric values before operating on them. For example, simply adding two to the "numValue" parameter will append the character "2" to "12" and yield "122": _numValue + 2 // result: "122" Instead, you need to use the parseInt/parseFloat functions to parse the string into a value and then add two to it: parseInt(_numValue) + 2 // result: 14 You need to parameterize your pipeline with an expression. You can use the eval() function to evaluate an expression stored in a string. For example, to read the document field specified by the "path" parameter, you can use: eval(_path) // result: <the value of the "age" field in the current document> Accessing Input View Variables as Part of Expressions An input view schema attribute can be used as part of the expression using the dollar sign ($) prefix. Example The REST Put Snap provides a URL. The URL can be toggled into an expression, and the expressions could be created by dynamically substituting the variables from an input view, such as: 'http://someplace:someport/somepart/' + $inputvar + '/somemoreparts' Accessing secret value from the secrets manager Any expression-enabled authentication field in a Snap or Account can be used with Secrets Management. You can enter an expression that retrieves a secret stored in your secrets manager, such as an access token, a username, or a password. To use the values from the secrets manager, you must first create secrets myaccesskey and mysecretkey in the Secrets Manager vault. Then, create or modify the Account and enter an expression in the required fields. Learn more: Configure Accounts to use secrets. Account An account represents an object that encompasses details to connect to an endpoint. Accounts play a crucial role in integrating applications. Any Snap that communicates with an external endpoint needs an authenticated account to access the resources on the endpoint. For example, a MySQL Snap requires authenticated access to a MySQL database. In SnapLogic, you create an Account to store credentials and any other information necessary to connect, such as a URL, hostname, and port number. You can create an account from Designer or Manager. In Designer, when working on pipelines, every Snap needing an account prompts you to create a new account or use an existing one. To use an existing account, you can click the dropdown icon to show all the available accounts for the snaps. To create a new account, click the “Add Account” button below the property field and follow the steps. The account will be created in your selected location on the first step. You can manage the created account on the Manager page in that location. Note: You can learn more about account type and each property by clicking the icon question mark in the top right corner. Validation & Execute Pipeline Sometimes, we want to test the pipeline by dry-running it without running the write snaps. You can use the validate function on the toolbar menu. The difference between validate and execute is before each snap runs. It will check the property called “Snap execution.” There are three opinions on how to trigger the snaps. Validate & Execute - this option makes the snaps run on both the validation and execution steps. Execute only - this option makes the snaps run on only the execution step. The snap writer type uses this as a default value. Disabled - this option prevents the snaps from running. Note: By default, the validation will be triggered every time we change the configuration of the snaps in the pipeline. Preview Data After executing or validating the pipeline, they will have a preview icon in the connection joint. The preview dialog will appear when you click on it, showing the snaps' output data. For example, when we click the preview icon, the pipeline above will show output data from JSON Generator snaps. The preview dialog has three types: JSON, Table, and Raw. You can select the dropdown on Preview Type and choose the type you like. JSON Table Raw Create First Pipeline This section will show how you start creating the pipeline, from the requirement to checking the result and running the final pipeline. For the example scenario, we want to calculate the employees list to check who needs to be assigned marketing training. The list of our employees looks like the data below. [ { "Name": "Albert Maro", "Location": "Field", "Extension": 4357, "Email": "amaro@company.com", "Title": "Director, Eastern US", "Department": "Sales", "Dept ID": 1100 }, { "Name": "Anthony Dunn", "Location": "HQ", "Extension": 4387, "Email": "adunn@company.com", "Title": "Social Media Director", "Department": "Marketing", "Dept ID": 1200 }, { "Name": "Rich Harris", "Location": "CO", "Extension": 4368, "Email": "rharris@company.com", "Title": "Principal Developer", "Department": "Engineering", "Dept ID": 1300 } // more data ] The constraint of needing training is an employee in the marketing department working at “HQ.” We want the list of employees with Firstname, Lastname, Email, Title, and Training fields. The result should look like below. [ { "Firstname": "Albert", "Lastname": "Maro", "Email": "amaro@company.com", "Title": "Director, Eastern US", "Training": false }, { "Firstname": "Anthony", "Lastname": "Dunn", "Email": "adunn@company.com", "Title": "Social Media Director", "Training": true }, { "Firstname": "Rich", "Lastname": "Harris", "Email": "rharris@company.com", "Title": "Principal Developer", "Training": false } // more data ] Steps 1. Open the Designer page. 2. Click to create a new pipeline. 3. Change the label to “Employees training” and click save. 4. At this step, we already have a new empty pipeline. Then, find the “JSON Generator” snap from the side panel and drag it to the canvas screen. This snap generates a JSON document for the next snap in the pipeline. We will set it as an input source. 5. Click at the JSON Generator snap to open the configuration dialog and click “Edit JSON.” Then, replace all JSON with the value below. [ { "Name": "Albert Maro", "Location": "Field", "Extension": 4357, "Email": "amaro@company.com", "Title": "Director, Eastern US", "Department": "Sales", "Dept ID": 1100 }, { "Name": "Anthony Dunn", "Location": "HQ", "Extension": 4387, "Email": "adunn@company.com", "Title": "Social Media Director", "Department": "Marketing", "Dept ID": 1200 }, { "Name": "Rich Harris", "Location": "CO", "Extension": 4368, "Email": "rharris@company.com", "Title": "Principal Developer", "Department": "Engineering", "Dept ID": 1300 } // more data ] Click “Ok” and save button ( ) before close the dialog. 6. Wait for the validation to finish. If it doesn’t run validation, click the validation button to manually validate the pipeline. 7. Find the “Mapper” snap and drag it to after the JSON generator. The Mapper snap transforms incoming data with the specific mappings and produces new output data. 8. Click on the Mapper snap to open the configuration dialog. We focus on the five blocks at the bottom of the dialog. Input Schema - shows the schema of input data Mapping table - is the configuration to map from input data to new output data Target Schema -shows the schema of output data. But this snap hasn’t been validated yet, so it shows nothing. Input Preview - shows the current input data Output Preview - shows the current output data Next, set the mapping table with the information below. To add multiple mapping, click in the top right corner. Expression Target path $Name.split(' ')[0] $Firstname $Name.split(' ')[1] $Lastname $Email $Email $Title $Title $Location == "HQ" && $Department == "Marketing" $Trainging The finish configuration will look like this. Click save and close the dialog. 9. Click the preview button after the Mapper snap. The output should be like this. SnapGPT SnapGPT is an interactive tool inside SnapLogic Designer. It uses the power of LLMs to democratize integration by helping users create and manage integrations using natural language prompts. The SnapGPT can do six main functions in SnapLogic. Generate pipelines Describe pipelines Analyze pipelines Ask anything about the SnapLogic Intelligent Integration Platform (IIP) Generate SnapLogic expressions Create SQL queries Usage SnapGPT You can open the SnapGPT panel by clicking on the SnapGPT logo in the header bar. Then, the panel will be displayed with a welcome message. Next, we will show how to use each feature of SnapGPT on the SnapLogic platform. Generate pipelines Prompt direct to the SnapGPT Example prompts: Extract opportunity object records from Salesforce and add them to Snowflake Create a Pipeline using Salesforce Read to fetch my Opportunities, Filter out any opportunities outside of the last fiscal quarter, then write them to Snowflake. Extract opportunity object records from Salesforce closed before “2022-10-01” and add them to Snowflake. Create a pipeline that fetches my SnapLogic Activity Logs from the SnapLogic API. Describe pipelines Open the pipeline you want to describe, then go to the SnapGPT panel and ask, “Describe the pipeline.” Example prompts: Describe the pipeline Analyze pipelines Open the pipeline you want to analyze, then go to the SnapGPT panel and ask, “Analyze the pipeline.” Example prompts: Analyze the pipeline Result: SnapGPT identifies issues with the pipeline and makes suggestions for improvement, and Snaps in the pipeline offers suggestions. Ask anything about the SnapLogic Intelligent Integration Platform (IIP) Example prompts: How do I build a pipeline? When and how should I use the Salesforce SOQL snap? How can one pipeline call another pipeline? Can pipelines use recursion? How is an Ultra pipeline different from a regular pipeline? Generate SnapLogic expressions To begin, simply open a snap and select the icon. This action activates the expression generation feature, enabling me to assist you in creating expressions. I can start creating expressions for you either in our chat or inside the expression-enabled field itself by typing the prompt and then clicking on the SnapGPT icon . Example prompts: Generate an expression to filter my closed lost opportunities. Generate an expression to grab the current date and time. Create SQL queries Open a Snap that supports SQL or SOQL queries and open SnapGPT. For example, if you open the Salesforce SOQL Snap, the suggestion Create SQL query appears above the SnapGPT prompt. SnapGPT generates the query and displays it in the SQL Preview panel. You can review the generated SQL before applying to the snap. Example prompt: Generate a SQL query to get the total amount of opportunities closed within the last quarter grouped by the account's country and deal status.
Postman Collection for API Management Functionality
Organizations typically have DevOps procedures and tooling for automation around their legacy microservices or services. SnapLogic has a robust Public API available for those teams to automate publication, versioning, deprecation, migration, and even retirement of both APIs hosted in SnapLogic APIM, and API proxies managed in SnapLogic. We've created a Postman Collection and set of environments that those DevOps teams might leverage to help them manage their APIs in SnapLogic even faster.Using Mustache Templating with the Prompt Generator Snap in SnapLogic
In the world of AI-driven data integration, the ability to dynamically generate prompts is crucial for creating adaptable and responsive workflows. The Prompt Generator Snap in SnapLogic leverages Mustache templating to allow users to craft dynamic text outputs based on input data. This whitepaper aims to educate users on the fundamentals of Mustache templating and how to effectively utilize it within the Prompt Generator Snap.
Multimodal Processing in LLM
Multimodal processing in Generative AI represents a transformative leap in how AI systems extract and synthesize information from multiple data types—such as text, images, audio, and video—simultaneously. Unlike traditional single-modality AI models, which focus on one type of data, Multimodal systems integrate and process diverse data streams in parallel, creating a holistic understanding of complex scenarios. This integrated approach is critical for applications that require not just isolated insights from one modality, but a coherent synthesis across different data sources, leading to outputs that are contextually richer and more accurate. Generative AI, with multimodal processing, is redefining text extraction, surpassing traditional OCR by interpreting text within its visual and contextual environment. Unlike OCR, which only converts images to text, generative AI analyzes the surrounding image context, layout, and meaning, enhancing accuracy and depth. For instance, in complex documents, it can differentiate between headings, body text, and annotations, structuring information more intelligently. Additionally, it excels in low-quality or multilingual texts, making it invaluable in industries requiring precision and nuanced interpretation. In video analysis, a generative AI equipped with Multimodal processing can simultaneously interpret the visual elements of a scene, the audio (such as dialogue or background sounds), and any associated text (like subtitles or metadata). This allows the AI to produce a description or summary of the scene that is far more nuanced than what could be achieved by analyzing the video or audio alone. The interplay between these modalities ensures that the generated description reflects not only the visual and auditory content but also the deeper context and meaning derived from their combination. In tasks such as image captioning, Multimodal AI systems go beyond simply recognizing objects in a photo. They can interpret the semantic relationship between the image and accompanying text, enhancing the relevance and specificity of the generated captions. This capability is particularly useful in fields where the context provided by one modality significantly influences the interpretation of another, such as in journalism, where images and written reports must align meaningfully, or in education, where visual aids are integrated with instructional text. Multimodal processing enables AI to synthesize medical images (such as X-rays or MRIs) with patient history, clinical notes, and even live doctor-patient interactions in highly specialized applications like medical diagnostics. This comprehensive analysis allows the AI to provide more accurate diagnoses and treatment recommendations, addressing the complex interplay of symptoms, historical data, and visual diagnostics. Similarly, in customer service, Multimodal AI systems can improve communication quality by analyzing the textual content of a customer's inquiry and the tone and sentiment of their voice, leading to more empathetic and effective responses. Beyond individual use cases, Multimodal processing plays a crucial role in improving the learning and generalization capabilities of AI models. By training on a broader spectrum of data types, AI systems develop more robust, flexible models that can adapt to a wider variety of tasks and scenarios. This is especially important in real-world environments where data is often heterogeneous and requires cross-modal understanding to interpret fully. As Multimodal processing technologies continue to advance, they promise to unlock new capabilities across diverse sectors. In entertainment, Multimodal AI could enhance interactive media experiences by seamlessly integrating voice, visuals, and narrative elements. In education, it could revolutionize personalized learning by adapting content delivery to different sensory inputs. In healthcare, the fusion of Multimodal data could lead to breakthroughs in precision medicine. Ultimately, the ability to understand and generate contextually rich, Multimodal content positions Generative AI as a cornerstone technology in the next wave of AI-driven innovation. Multimodal Content Generator Snap The Multimodal Content Generator Snap encodes file or document inputs into the Snap's multimodal content format, preparing it for seamless integration. The output from this Snap must be connected to the Prompt Generator Snap to complete and format the message payload for further processing. This streamlined setup enables efficient multimodal content handling within the Snap ecosystem. The Snap Properties Type - Select the type of multimodal content. Content Type - Define the specific content type for data transmitted to the LLM. Content - Specify the content path to the multimodal content data for processing. Document Name - Name the document for reference and identification purposes. Aggregate Input - Enable this option to combine all inputs into a single content. Encode Base64 - Enable this option to convert the text input into Base64 encoding. Note: The Content property appears only if the input view is of the document type. The value assigned to Content must be in Base64 format for document inputs, while Snap will automatically use binary as content for binary input types. The Document Name can be set specifically for multimodal document types. The Encode Base64 property encodes text input into Base64 by default. If unchecked, the content will be passed through without encoding. Designing a Multimodal Prompt Workflow In this process, we will integrate multiple Snaps to create a seamless workflow for multimodal content generation and prompt delivery. By connecting the Multimodal Content Generator Snap to the Prompt Generator Snap, we configure it to handle multimodal content. The finalized message payload will then be sent to Claude by Anthropic Claude on AWS Messages. Steps: 1. Add the File Reader Snap: Drag and drop the File Reader Snap onto the designer canvas. Configure the File Reader Snap by accessing its settings panel, then select a file containing images (e.g., a PDF file). Download the sample image files at the bottom of this post if you have not already. Sample image file (Japan_flowers.jpg) 2. Add the Multimodal Content Generator Snap: Drag and drop the Multimodal Content Generator Snap onto the designer and connect it to the File Reader Snap. Open its settings panel, select the file type, and specify the appropriate content type. Here's a refined description of the output attributes from the Multimodal Content Generator: sl_content: Contains the actual content encoded in Base64 format. sl_contentType: Indicates the content type of the data. This is either selected from the configuration or, if the input is a binary, it extracts the contentType from the binary header. sl_type: Specifies the content type as defined in the Snap settings; in this case, it will display "image." 3. Add the Prompt Generator Snap: Add the Prompt Generator Snap to the designer and link it to the Multimodal Content Generator Snap. In the settings panel, enable the Advanced Prompt Output checkbox and configure the Content property to use the input from the Multimodal Content Generator Snap. Click “Edit Prompt” and input your instructions 4. Add and Configure the LLM Snap: Add the Anthropic Claude on AWS Message API Snap as the LLM. Connect this Snap to the Prompt Generator Snap. In the settings, select a model that supports multimodal content. Enable the Use Message Payload checkbox and input the message payload in the Message Payload field. 5. Verify the Result: Review the output from the LLM Snap to ensure the multimodal content has been processed correctly. Validate that the generated response aligns with the expected content and format requirements. If adjustments are needed, revisit the settings in previous Snaps to refine the configuration. Multimodal Models for Advanced Data Extraction Multimodal models are redefining data extraction by advancing beyond traditional OCR capabilities. Unlike OCR, which primarily converts images to text, these models directly analyze and interpret content within PDFs and images, capturing complex contextual information such as layout, formatting, and semantic relationships that OCR alone cannot achieve. By understanding both textual and visual structures, multimodal AI can manage intricate documents, including tables, forms, and embedded graphics, without requiring separate OCR processes. This approach not only enhances accuracy but also optimizes workflows by reducing dependency on traditional OCR tools. In today’s data-rich environment, information is often presented in varied formats, making the ability to analyze and derive insights from diverse data sources essential. Imagine managing a collection of invoices saved as PDFs or photos from scanners and smartphones, where a streamlined approach is needed to interpret their contents. Multimodal large language models (LLMs) excel in these scenarios, enabling seamless extraction of information across file types. These models support tasks such as automatically identifying key details, generating comprehensive summaries, and analyzing trends within invoices whether from scanned documents or images. Here’s a step-by-step guide to implementing this functionality within SnapLogic. Sample invoice files (download the files at the bottom of this post if you have not already) Invoice1.pdf Invoice2.pdf Invoice3.jpeg (Sometimes, the invoice image might be tilted) Upload the invoice files Open Manager page and go to your project that will be used to store the pipelines and related files Click the + (plus) sign and select File The Upload File dialog pops up. Click “Choose Files” to select all the invoice files both PDF and image formats (download the sample invoice files at the bottom of this post if you have not already) Click Upload button and the uploaded files will be shown. Building the pipeline Add the JSON Generator Snap: Drag and drop the JSON Generator onto the designer canvas. Click on the Snap to open settings, then click the "Edit JSON" button Highlight all the text from the template and delete it. Paste all invoice filenames in the format below. The editor should look like this. Click "OK" in the lower-right corner to save the prompt Save the settings and close the Snap Add the File Reader Snap: Drag and drop the File Reader Snap onto the designer canvas Click the Snap to open the configuration panel. Connect the Snap to the JSON Generator Snap by following these steps: Select Views tab Click plus(+) button on the Input pane to add the input view(input0) Save the configuration The Snap on the canvas will have the input view. Connecting it to the JSON Generator Snap In the configuration panel, select the Settings tab Set the File field by enabling expression by clicking the equal sign in front of the text input and set it to $filename to read all the files we specified in the JSON Generator Snap Validate the pipeline to see the File Reader output. Fields that will be used in the Multimodal Content Generator Snap Content-type shows file content type Content-location shows the file path and it will be used in the document name Add the Multimodal Content Generator Snap: Drag and drop the Multimodal Content Generator Snap onto the designer canvas and connect to the File Reader Snap Click the Snap to open the settings panel and configure the following fields: Type: enable the expression set the value to $['content-location'].endsWith('.pdf') ? 'document' : 'image' Document name enable the expression set the value to $['content-location'].snakeCase() Use the snake-case version of the file path as the document name to identify each file and make it compatible with the Amazon Bedrock Converse API. In snake case, words are lowercase and separated by underscores(_). Aggregate input check the checkbox Use this option to combine all input files into a single document. The settings should now look like the following Validate the pipeline to see the Multimodal Content Generator Snap output. The preview output should look like the below image. The sl_type will be document for the pdf file and image for the image file and the name will be the simplified file path. Add the Prompt Generator Snap: Drag and drop the Prompt Generator Snap onto the designer canvas and connect to the Multimodal Content Generator Snap Click the Snap to open the settings panel and configure the following fields: Enable the Advanced Prompt Output checkbox Set the Content to $content to use the content input from the Multimodal Content Generator Snap Click “Edit Prompt” and input your instructions. For example, Based on the total quantity across all invoices, which product has the highest and lowest purchase quantities, and in which invoices are these details found? Add and Configure the LLM Snap: Add the Amazon Bedrock Converse API Snap as the LLM Connect this Snap to the Prompt Generator Snap Click the Snap to open the configuration panel Select the Account tab and select your account Select the Settings tab Select a model that supports multimodal content. Enable the Use Message Payload checkbox Set the Message Payload to $messages to use the message from the Prompt Generator Snap Verify the result: Validate the pipeline and open the preview of the Amazon Bedrock Converse API Snap. The result should look like the following: In this example, the LLM successfully processes invoices in both PDF and image formats, demonstrating its ability to handle diverse inputs in a single workflow. By extracting and analyzing data across these formats, the LLM provides accurate responses and insights, showcasing the efficiency and flexibility of multimodal processing. You can adjust the queries in the Prompt Generator Snap to explore different results.SnapLogic deployment on Kubernetes - A reference guide
Overview SnapLogic supports the deployment of Groundplexes on Kubernetes platforms, thus enabling the application to leverage the various capabilities of Kubernetes. This document explains a few best practice recommendations for the deployment of SnapLogic on Kubernetes along with a sample deployment example using GKE. The examples in this document are specific to the GKE platform however the concepts can be applied to other Kubernetes platforms such as AWS and Azure. Author: Ram Bysani SnapLogic Enterprise Architecture team Helm Chart A Helm chart is used to define the various deployment configurations for an application on Kubernetes. Additional information about Helm charts can be found here. The Helm chart package for a SnapLogic deployment can be downloaded from the Downloads section. It contains the following files: Artifact Comments values.yaml This file defines the default configuration for the SnapLogic Snaplex deployment. It includes variables like the number of JCC nodes, container image details, resource limits, and settings for Horizontal Pod Autoscaling (HPA). Reference: values.yaml Chart.yaml This file defines the metadata and version information for the Helm chart. templates folder This directory contains the Kubernetes manifest templates which define the resources to be deployed into the cluster. These templates are YAML files that specify Kubernetes resources with templating capabilities that allow for parameterization, flexibility, and reuse. templates/deployment.yaml This file defines a Kubernetes Deployment resource for managing the deployment of JCC instances in a cluster. The deployment is created only if the value of jccCount is greater than 0, as specified in the Helm chart's values.yaml file. templates/deployment-feed.yaml This file defines a Kubernetes Deployment resource for managing the deployment of Feedmaster instances. The deployment is conditionally created if the feedmasterCount value in the Helm chart's file values.yaml file is greater than 0. templates/hpa.yaml The hpa.yaml file defines a Horizontal Pod Autoscaler (HPA) resource for a Kubernetes application. The HPA automatically scales the number of pod replicas in a deployment or replica set based on observed metrics such as CPU utilization or custom metrics. templates/service.yaml The service.yaml file describes a Kubernetes service that exposes the JCC component of your Snaplex. It creates a LoadBalancer type service, which allows external access to the JCC components through a public IP address. The service targets only pods labeled as 'jcc' within the specified Snaplex and Helm release, ensuring proper communication and management. templates/service-feed.yaml The service-feed.yaml file describes a Kubernetes service that exposes the Feedmaster components. The service is only created if the value of feedmasterCount in the Helm chart’s values.xml file is > 0. It creates a LoadBalancer type service, which allows external access to the Feedmaster components through a public IP address. templates/service-headless.yaml The service-headless.yaml file describes a Kubernetes service for IPv6 communication. The service is only created if the value of enableIPv6 in the Helm chart’s values.xml file is set to true. Table 1.0 Helm Chart configurations Desired State vs Current State The configurations in the various yaml files (e.g. Deployment, HPA, values, etc.) represent the “Desired” state of a Kubernetes deployment. The Kubernetes controllers constantly monitor the Current state of the deployment to bring it in alignment with the Desired state. Horizontal Pod Autoscaling (HPA) Horizontal Pod Autoscaling (HPA) is a feature in Kubernetes that automatically adjusts the number of replicas (pods) for your deployments based on resource metrics like CPU utilization and memory usage. SnapLogic supports HPA for deployments in a Kubernetes environment. The add-on Metrics server must be installed. Reference: Metrics-Server. Metrics collection is enabled by default in GKE as part of Cloud Monitoring. Note that Custom Metrics and External Metrics, and Vertical Pod Autoscaling (VPA) are not supported for SnapLogic deployments on Kubernetes. Groundplex deployment in a GKE environment - Example In this section, we will go over the various steps for a SnapLogic Groundplex deployment in a GKE environment. Groundplex creation Create a new Groundplex from the Admin Manager interface. Reference: Snaplex_creation. The nodes for this Snaplex will be updated when the application is deployed to the GKE environment. New Snaplex creation GKE Cluster creation Next, we create the GKE cluster on the Google Cloud console. We have created our cluster in Autopilot mode. In this mode, GKE manages the cluster and node configurations including scaling, load balancing, monitoring, metrics, and workload optimization. Reference: GKE Cluster GKE cluster Configure the SnapLogic platform Allowlist Add the SnapLogic platform IP addresses to the Allowlist. See Platform Allowlist. In GKE, this is usually done by configuring an Egress Firewall rule on the GKE cluster. Please refer to the GKE documentation for additional details. Firewall rule - Egress Helm configurations values.yaml The below table explains the configurations for some of the sections from the values.yaml file which we have used in our set up. The modified files are attached to this article for reference. Reference: Helm chart configuration Section Comments # Regular nodes count jccCount: 3 # Feedmaster nodes count feedmasterCount: 0 This defines the number of JCC pods. We have enabled HPA for our test scenario, so the jccCount will be picked from the HPA section. (i.e. minReplicas and maxReplicas). The pod count is the number of pods across all nodes of the cluster. No Feedmaster pods are configured in this example. Feedmaster count can be half of the JCC pod count. Feedmaster is used to distribute Ultra task requests to the JCC pods. HPA configuration is only applicable to the JCC pods and not to the Feedmaster pods. # Docker image of SnapLogic snaplex image: repository: snaplogic/snaplex tag: latest This specifies the latest and most recent release version of the repository image. You can specify a different tag if you need to update the version to a previous release for testing, etc. # SnapLogic configuration link snaplogic_config_link: https://uat.elastic.snaplogic.com/api/1/rest/plex/config/ org/proj_space/shared/project Retrieve the configuration link for the Snaplex by executing the Public API. The config link string is the portion before ?expires in the output value of the API. Example: snaplogic_config_link: https://uat.elastic.snaplogic.com/api/1/rest/plex/config/ QA/RB_Temp_Space/shared/RBGKE_node1 # SnapLogic Org admin credential snaplogic_secret: secret/mysecret Execute the kubectl command: kubectl apply -f snapSecret.yaml Please see the section To create the SnapLogic secret in this document: Org configurations. # CPU and memory limits/requests for the nodes limits: memory: 8Gi cpu: 2000m requests: memory: 8Gi cpu: 2000m Set requests and limits to the same values to ensure resource availability for the container processes. Avoid running other processes in the same container as the JCC so that the JCC can have the maximum amount of memory. # Default file ulimit and process ulimit sl_file_ulimit: 8192 sl_process_ulimit: 4096 The value should be more than the # of slots configured for the node. (Maximum Slots under Node properties of the Snaplex). If not set, then the node defaults will be used. (/etc/security/limits.conf). The JCC process is initialized with these values. # JCC HPA autoscaling: enabled: true minReplicas: 1 maxReplicas: 3 minReplicas defines the minimum number of Pods that must be running. maxReplicas defines the maximum number of Pods that can be scheduled on the node(s). The general guideline is to start with 1:2 or 1:3 Pods per node. The replica Pods are across all nodes of a deployment and not per node. targetAvgCPUUtilization: 60 targetAvgMemoryUtilization: 60 To enable these metrics, the Kubernetes Metrics Server installation is required. Metrics collection is enabled by default in GKE as part of Cloud Monitoring. targetAvgCPUUtilization: Average CPU utilization percentage (i.e. 60 = 60%) This is the average CPU utilization across all Pods. HPA will scale up or scale down Pods to maintain this average. targetAvgMemoryUtilization: Average memory utilization percentage. This parameter is used to specify the average memory utilization (as a percentage of the requested memory) that the HPA should maintain across all the replicas of a particular deployment or stateful set. scaleDownStabilizationWindowSeconds: 600 terminationGracePeriodSeconds: 900 # Enable IPv6 service for DNS routing to pods enableIPv6: false scaleDownStabilizationWindowSeconds is a parameter used in Kubernetes Horizontal Pod Autoscaler (HPA) It controls the amount of time the HPA waits (like a cool-down period) before scaling down the number of pods after a decrease in resource utilization. terminationGracePeriodSeconds defines the amount of time Kubernetes gives a pod to terminate before killing it. If the containers have not exited after terminationGracePeriodSeconds, then Kubernetes sends a SIGKILL signal to forcibly terminate the containers, and remove the pod from the cluster. Table 2.0 - values.yaml Load balancer configuration The service.yaml file contains a section for the Load balancer configuration. Autopilot mode in GKE supports the creation of a Load balancer service. Section Comments type: LoadBalancer ports: - port: 8081 protocol: TCP name: jcc selector: A Load balancer service will be created by GKE to route traffic to the application’s pods. The external IP address and port details must be configured on the Settings tab of the Snaplex. An example is included in the next section of this document. Table 3.0 service.yaml Deployment using Helm Upload the helm zip file package to the Cloud Shell instance by selecting the Upload option. The default Helm package for SnapLogic can be downloaded from here. It is recommended to download the latest package from the SnapLogic documentation link. The values.yaml file with additional custom configurations (as described in Tables 2.0 / 3.0 above) is attached to this article. Execute the command on the terminal to install and deploy the Snaplex release with a unique name such as snaplogic-snaplex using the configurations from the values.yaml file. The release name is a unique identifier, and can be different for multiple deployments such as Dev / Prod, etc. helm install snaplogic-snaplex . -f values.yaml <<Output>> NAME: snaplogic-snaplex NAMESPACE: default STATUS: deployed REVISION: 5 TEST SUITE: None NOTES: You can run this command to update an existing deployment with any new or updated Helm configurations. helm upgrade snaplogic-snaplex . -f values.yaml View the deployed application under the Workloads tab on the Google Cloud Console. Workloads This command returns the HPA details. $ kubectl describe hpa Name: snaplogic-snaplex-hpa Namespace: default Labels: app.kubernetes.io/instance=snaplogic-snaplex app.kubernetes.io/managed-by=Helm app.kubernetes.io/name=snaplogic-snaplex app.kubernetes.io/version=1.0 helm.sh/chart=snaplogic-snaplex-0.2.0 Annotations: meta.helm.sh/release-name: snaplogic-snaplex meta.helm.sh/release-namespace: default Deployment/snaplogic-snaplex-jcc Metrics: ( current / target ) resource cpu on pods (as a percentage of request): 8% (153m) / 60% resource memory on pods (as a percentage of request): 28% (1243540138666m) / 60% Min replicas: 1 Max replicas: 3 Run the kubectl command to list the services. You can see the external IP addresses for the Load balancer service. kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 34.118.224.1 <none> 443/TCP 16d kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kubernetes ClusterIP 34.118.224.1 <none> 443/TCP 16d snaplogic-snaplex-regular LoadBalancer 34.118.227.164 34.45.230.213 8081:32526/TCP 25m Update Load balancer url on the Snaplex Note the external IP address for the LoadBalancer service, and update the host and port on the Load balancer field of the Snaplex. Example: http://1.3.4.5:8081 Load balancer Listing pods in GKE The following commands can be executed to view the pod statuses. The pod creation and maintenance is fully managed by GKE. $ kubectl top pods $ kubectl get pods kubectl get pods --field-selector=status.phase=Running NAME READY STATUS RESTARTS AGE snaplogic-snaplex-jcc-687d87994-crzw9 0/1 Running 0 2m snaplogic-snaplex-jcc-687d87994-kks7l 1/1 Running 0 2m38s snaplogic-snaplex-jcc-687d87994-pcfvp 1/1 Running 0 2m24s View node details in the SnapLogic Monitor application Each pod represents a JCC node. The maxReplica value is set to 3 so you would see a maximum of 3 nodes (pods) deployed. (Analyze -> Infrastructure tab). Snaplex nodes The below command uninstalls and deletes the deployment from the cluster. All deployed services, metadata, and associated resources are also removed. helm uninstall <deployment_name> Pod registration with the SnapLogic Control Plane Scenario Comments How are the Pod neighbors resolved and maintained by the SnapLogic Control Plane? When a JCC/FeedMaster node (Pod) starts, it registers with the SnapLogic Control Plane, and the Control Plane maintains the list of Pod neighbors. When a JCC/FeedMaster node (Pod) registers, it also publishes its IP address to the Control Plane. An internal list of Pod IP addresses is updated dynamically for neighbor to neighbor communication. DNS resolution is not used. How are the container repository versions updated? The latest Snaplex release build is updated in the docker repository version tagged ‘latest’. The pods will be deployed with this version on startup by referencing the tags from the values.yaml file. If the Snaplex version is updated on the Control Plane to a different version (e.g. main-2872), then the JCC nodes (pods) will be updated to match that version (i.e. main-2872). Reference Groundplex Deployment on Kubernetes https://kubernetes.io/ GKE HPAUnlocking the Power of LLMs with OpenAPI Tool Integration
Large Language Models (LLMs) are revolutionizing the way we interact with digital systems, from conversational agents to intelligent automation. But to truly harness their capabilities, especially in enterprise and developer ecosystems, it’s essential to bridge the gap between LLMs and external systems through tools—specifically APIs. This is where OpenAPI plays a pivotal role. What is OpenAPI? OpenAPI (formerly Swagger) is an open-source specification that defines a standard, machine-readable format for describing RESTful APIs. It enables developers and automated systems to understand an API’s structure—including endpoints, request parameters, authentication methods, and response types—without relying on traditional documentation or access to source code. Its adoption spans industries such as technology, finance, and healthcare, thanks to its interoperability with a wide array of tools and frameworks. Why OpenAPI Matters for LLMs Integrating OpenAPI with LLMs enhances their ability to interact with real-world systems. Here's how: Universal Interface: OpenAPI acts as a universal bridge to RESTful APIs, making it possible for LLMs to interact with services ranging from cloud infrastructure to productivity apps. Standardized Format: The standardized schema helps LLMs accurately interpret API functionality—including expected inputs and outputs—without ambiguity. Accelerated Tool Creation: Developers can efficiently build LLM-compatible tools by parsing OpenAPI definitions directly. Seamless Integration: With broad support from API tooling ecosystems, OpenAPI enables quick embedding of LLM agents into existing workflows. Supports Tool Calling: Tool calling allows LLMs to autonomously select and invoke relevant APIs based on user prompts—a key feature unlocked by structured OpenAPI descriptions. Enabling LLM Tool Calling with SnapLogic To connect LLMs with OpenAPI-defined tools, the OpenAPI Function Generator Snap plays a crucial role. This component converts any OpenAPI spec into a tool object that LLMs can use through the Tool Calling pipeline in SnapLogic. Input Options for the Generator Snap The generator supports multiple input methods: URL: Directly fetch the OpenAPI spec from a provided URL. Text Editor: Paste the raw spec into a built-in editor. Input Document: Pass the OpenAPI string as part of an input document via expression. File Upload: Select a spec file stored in the SLDB. Output Structure The generated tool output includes: sl_tool_metadata: Metadata such as security parameters, headers, and base URLs. json_schema: A schema of the input parameters. These tools can be passed into the Tool Calling Snap, which then resolves runtime variables like headers and endpoint URLs dynamically. Developers can chain this with an HTTP Client Snap to perform real API calls based on LLM outputs. Passing Through the Tool Calling Snap When the tool is passed through the Tool Calling Snap, it dynamically processes and resolves several key components using the metadata and user input: Resolved URL: The base URL and path parameters from the OpenAPI spec are combined with user-supplied values to generate the final API endpoint. Headers: Custom headers, or content-type headers are filled in based on the OpenAPI security definitions or context provided by the LLM. This resolved output makes it simple for downstream snaps (like HTTP Client) to directly execute the API call. Action tools with HTTP Client Snap Once the Tool Calling Snap generates the resolved tool data, this output can be piped directly into an HTTP Client Snap for execution: This setup effectively turns a static OpenAPI definition into a fully dynamic and executable workflow, allowing LLMs to autonomously interact with real services. Real-World Use Cases With the right configuration, LLMs can interact with virtually any OpenAPI-compliant service. This opens up a wide range of practical applications across productivity tools, developer APIs, data services, and more. Example Use Case: Load Products from FakeStore API and Save as CSV in GitHub Gist This example shows how an LLM can orchestrate a two-step integration using OpenAPI specs and tool calling via SnapLogic: Fetch Data: Retrieve product data from FakeStore API. Transform & Upload: Format the data as CSV and post it as a public GitHub Gist using GitHub’s Gist API. Main Pipeline (download) Loop Pipeline (download, github openapi file, fake store openapi file) Prompt to LLM: “Load all products from FakeStore API and upload them as a CSV file to GitHub Gist.” Pipeline Flow Breakdown Step 1: FakeStore API Tool Call OpenAPI Tool: FakeStore API spec (loaded via URL or file). LLM Task: Recognize the available /products endpoint and trigger a GET request to retrieve the full list of products. Tool Calling Snap Output: Resolved URL to https://fakestoreapi.com/products, method GET, no authentication needed. Step 2: GitHub Gist API Tool Call OpenAPI Tool: GitHub Gist API spec, with token-based authentication defined in sl_tool_metadata. LLM Task: Use the POST /gists endpoint, and construct the request body with: description: e.g., "FakeStore Products Export" public: true files: A JSON object with one file (e.g., "products.csv": { content: "<csv data>" }) Step 3: Summarize the Result LLM Task: Extract and present key details from the final Gist API response, such as: Total number of products exported Link to the created Gist (e.g., html_url) Confirmation message for the user Final Result:
SnapLogic MCP Support
Introduction Since the inception of the Model Context Protocol (MCP), we've been envisioning and designing how it can be integrated into the SnapLogic platform. We've recently received a significant number of inquiries about MCP, and we're excited to share our progress, the features we'll be supporting, our release timeline, and how you can get started creating MCP servers and clients within SnapLogic. If you're interested, we encourage you to reach out! Understanding the MCP Protocol The MCP protocol allows tools, data resources, and prompts to be published by an MCP server in a way that Large Language Models (LLMs) can understand. This empowers LLMs to autonomously interact with these resources via an MCP client, expanding their capabilities to perform actions, retrieve information, and execute complex workflows. MCP Protocol primarily supports: Tools: Functions an LLM can invoke (e.g., data lookups, operational tasks). Resources: File-like data an LLM can read (e.g., API responses, file contents). Prompts: Pre-written templates to guide LLM interaction with the server. Sampling (not widely used): Allows client-hosted LLMs to be used by remote MCP servers. An MCP client can, therefore, request to list available tools, call specific tools, list resources, or read resource content from a server. Transport and Authentication MCP protocol offers flexible transport options, including STDIO or HTTP (SSE or Streamable-HTTP) for local deployments, and HTTP (SSE or Streamable-HTTP) for remote deployments. While the protocol proposes OAuth 2.1 for authentication, an MCP server can also use custom headers for security. Release Timeline We're excited to bring MCP support to SnapLogic with two key releases: August Release: MCP Client Support We'll be releasing two new snaps: the MCP Function Generator Snap and the MCP Invoke Snap. These will be available in the AgentCreator Experimental (Beta) Snap Pack. With these Snaps, your SnapLogic agent can access the services and resources available on the public MCP server. Late Q3 Release: MCP Server Support Our initial MCP server support will focus on tool operations, including the ability to list tools and call tools. For authentication, it will support custom header-based authentication. Users will be able to leverage the MCP Server functionality by subscribing to this feature. If you're eager to be among the first to test these new capabilities and provide feedback, please reach out to the Project Manager Team, at pm-team@snaplogic.com. We're looking forward to seeing what you build with SnapLogic MCP. SnapLogic MCP Client MCP Clients in SnapLogic enable users to connect to MCP servers as part of their Agent. An example can be connecting to the Firecrawl MCP server for a data scraping Agent, or other use cases that can leverage the created MCP servers. The MCP Client support in SnapLogic consists of two Snaps, the MCP Function Generator Snap and the MCP Invoke Snap. From a high-level perspective, the MCP Function generator Snap allows users to list available tools from an MCP server, and the MCP Invoke Snap allows users to perform operations such as call tools, list resources, and read resources from an MCP server. Let’s dive into the individual pieces. MCP SSE Account To connect to an MCP Server, we will need an account to specify the URI of the server to connect to. Properties URI The URI of the server to connect to. Don’t need to include the /sse path Additional headers Additional HTTP headers to be sent to the server Timeout The timeout value in seconds, if the result is not returned within the timeout, the Snap will return an error. MCP Function Generator Snap The MCP Function Generator Snap enables users to retrieve the list of tools as SnapLogic function definitions to be used in a Tool Calling Snap. Properties Account An MCP SSE account is required to connect to an MCP Server. Expose Tools List all available tools from an MCP server as SnapLogic function definitions Expose Resources Add list_resources, read_resource as SnapLogic function definitions to allow LLMs to use resources/read and resources/list (MCP Resources). Definitions for list resource and read resource [ { "sl_type": "function", "name": "list_resources", "description": "This function lists all available resources on the MCP server. Return a list of resources with their URIs.", "strict": false, "sl_tool_metadata": { "operation": "resources/list" } }, { "sl_type": "function", "name": "read_resource", "description": "This function returns the content of the resource from the MCP server given the URI of the resource.", "strict": false, "sl_tool_metadata": { "operation": "resources/read" }, "parameters": [ { "name": "uri", "type": "STRING", "description": "Unique identifier for the resource", "required": true } ] } ] MCP Invoke Snap The MCP Invoke Snap enables users to perform operations such as tools/call, resources/list, and resources/read to an MCP server. Properties Account An account is required to use the MCP Invoke Snap Operation The operation to perform on the MCP server. The operation must be one of tools/call, resources/list, or resources/read Tool Name The name of the tool to call. Only enabled and required when the operation is tools/call Parameters The parameters to be added to the operation. Only enabled for resources/read and tools/call. Required for resources/read, and optional for tools/call, based on the tool. MCP Agents in pipeline action MCP Agent Driver pipeline An MCP Agent Driver pipeline is like any other MCP Agent pipeline; we’ll need to provide the system prompt, user prompt, and run it with the PipeLoop Snap. MCP Agent Worker pipeline Here’s an example of an MCP Agent with a single MCP Server connection. The MCP Agent Worker is connected to one MCP Server. MCP Client Snaps can be used together with AgentCreator Snaps, such as the Multi-Pipeline Function Generator and Pipeline Execute Snap, as SnapLogic Functions, tools. This allows users to use tools provided by MCP servers and internal tools, without sacrificing safety and freedom when building an Agent. Agent Worker with MCP Client Snaps SnapLogic MCP Server In SnapLogic, an MCP Server allows you to expose SnapLogic pipelines as dynamic tools that can be discovered and invoked by language models or external systems. By registering an MCP Server, you effectively provide a API that language models and other clients can use to perform operations such as data retrieval, transformation, enrichment, or automation, all orchestrated through SnapLogic pipelines. For the initial phase, we'll support connections to the server via HTTP + SSE. Core Capabilities The MCP Server provides two core capabilities. The first is listing tools, which returns structured metadata that describes the available pipelines. This metadata includes the tool name, a description, the input schema in JSON Schema format, and any additional relevant information. This allows clients to dynamically discover which operations are available for invocation. The second capability is calling tools, where a specific pipeline is executed as a tool using structured input parameters, and the output is returned. Both of these operations—tool listing and tool calling—are exposed through standard JSON-RPC methods, specifically tools/list and tools/call, accessible over HTTP. Prerequisite You'll need to prepare your tool pipelines in advance. During the server creation process, these can be added and exposed as tools for external LLMs to use. MCP Server Pipeline Components A typical MCP server pipeline consists of four Snaps, each with a dedicated role: 1. Router What it does: Routes incoming JSON requests—which differ from direct JSON-RPC requests sent by an MCP client—to either the list tools branch or the call tool branch. How: Examines the request payload (typically the method field) to determine which action to perform. 2. Multi-Pipeline Function Generator (Listing Tools) What it does: Converts a list of pipeline references into tool metadata. This is where you define the pipelines you want the server to expose as tools. Output: For each pipeline, generates: Tool name Description Parameters (as JSON Schema) Other metadata Purpose: Allows clients (e.g., an LLM) to query what tools are available without prior knowledge. 3. Pipeline Execute (Calling Tools) What it does: Dynamically invokes the selected SnapLogic pipeline and returns structured outputs. How: Accepts parameters encoded in the request body, maps them to the pipeline’s expected inputs, and executes the pipeline. Purpose: Provides flexible runtime execution of tools based on user or model requests. 4. Union What it does: Merges the result streams from both branches (list and call) into a single output stream for consistent response formatting. Request Flows Below are example flows showing how requests are processed: 🟢 tools/list Client sends a JSON-RPC request with method = "tools/list". Router directs the request to the Multi-Pipeline Function Generator. Tool metadata is generated and returned in the response. Union Snap merges and outputs the content. ✅ Result: The client receives a JSON list describing all available tools. �� tools/call Client sends a JSON-RPC request with method = "tools/call" and the tool name + parameters. Router sends this to the Pipeline Execute Snap. The selected pipeline is invoked with the given parameters. Output is collected and merged via Union. ✅ Result: The client gets the execution result of the selected tool. Registering an MCP Server Once your MCP server pipeline is created: Create a Trigger Task and Register as an MCP Server Navigate to the Designer > Create Trigger Task Choose a Groundplex. (Note: This capability currently requires a Groundplex, not a Cloudplex.) Select your MCP pipeline. Click Register as MCP server Configure node and authentication. Find your MCP Server URL Navigate to the Manager > Tasks The Task Details page exposes a unique HTTP endpoint. This endpoint is treated as your MCP Server URL. After registration, clients such as AI models or orchestration engines can interact with the MCP Server by calling the /tools/list endpoint to discover the available tools, and the /tools/call endpoint to invoke a specific tool using a structured JSON payload. Connect to a SnapLogic MCP Server from a Client After the MCP server is successfully published, using the SnapLogic MCP server is no different from using other MCP servers running in SSE mode. It can be connected to by any MCP client that supports SSE mode; all you need is the MCP Server URL (and the Bearer Token if authentication is enabled during server registration). Configuration First, you need to add your MCP server in the settings of the MCP client. Taking Claude Desktop as an example, you'll need to modify your Claude Desktop configuration file. The configuration file is typically located at: macOS: ~/Library/Application Support/Claude/claude_desktop_config.json Windows: %APPDATA%\Claude\claude_desktop_config.json Add your remote MCP server configuration to the mcpServers section: { "mcpServers": { "SL_MCP_server": { "command": "npx", "args": [ "mcp-remote", "http://devhost9000.example.com:9000/mcp/6873ff343a91cab6b00014a5/sse", "--header", "Authorization: Bearer your_token_here" ] } } } Key Components Server Name: SL_MCP_server - A unique identifier for your MCP server Command: npx - Uses the Node.js package runner to execute the mcp-remote package URL: The SSE endpoint URL of your remote MCP server (note the /sse suffix) Authentication: Use the --header flag to include authorization tokens if the server enabled authentication Requirements Ensure you have Node.js installed on your system, as the configuration uses npx to run the mcp-remote package. Replace the example URL and authorization token with your actual server details before saving the configuration. After updating the configuration file, restart Claude Desktop for the changes to take effect. To conclude, the MCP Server in SnapLogic is a framework that allows you to expose pipelines as dynamic tools accessible through a single HTTP endpoint. This capability is designed for integration with language models and external systems that need to discover and invoke SnapLogic workflows at runtime. MCP Servers make it possible to build flexible, composable APIs that return structured results, supporting use cases such as conversational AI, automated data orchestration, and intelligent application workflows. Conclusion SnapLogic's integration of the MCP protocol marks a significant leap forward in empowering LLMs to dynamically discover and invoke SnapLogic pipelines as sophisticated tools, transforming how you build conversational AI, automate complex data orchestrations, and create truly intelligent applications. We're excited to see the innovative solutions you'll develop with these powerful new capabilities.
Revolutionizing Software Testing: How LLMs are Powering Automated Test Case and Data Generation
Tired of writing endless test cases and crafting complex test data manually? Discover how Large Language Models (LLMs) are transforming the QA landscape by automating test case and test data generation with remarkable accuracy and speed. In this article, we explore how LLMs—when paired with tools like SnapLogic Agent Creator—can accelerate testing cycles, boost coverage, and reduce QA efforts by up to 90%. Step into the future of intelligent, AI-driven software testing
SnapLogic Test Automation with Robot Framework: A Complete Testing Solution
Introduction In today's fast-paced integration landscape, ensuring the reliability and performance of your SnapLogic pipelines is crucial. We're excited to introduce a comprehensive test automation framework that combines the power of Robot Framework with SnapLogic's APIs to deliver a robust, scalable, and easy-to-use testing solution. This approach leverages the snaplogic-common-robot [PyPI-published library] to provide prebuilt Robot Framework keywords for interacting with SnapLogic Public APIs, integrated within a Docker-based environment.. This lets teams spin up full SnapLogic environments on demand—including Groundplex, databases, and messaging services—so tests run the same way everywhere This blog post explores two key components of our testing ecosystem: snaplogic-common-robot: A PyPI-published library https://pypi.org/project/snaplogic-common-robot/ providing reusable Robot Framework keywords for SnapLogic automation snaplogic-robotframework-examples: A public repository providing a complete testing framework with baseline test suites and Docker-based infrastructure for comprehensive end-to-end pipeline validation Key Features and Benefits 1. Template-Based Testing The framework supports template-driven test cases, allowing you to: Define reusable test patterns Parameterize test execution Maintain consistency across similar test scenarios 2. Intelligent Environment Management The framework automatically: Loads environment variables from multiple .env files Auto-detects JSON values and converts them to appropriate Robot Framework variables Validates required environment variables before test execution Why Robot Framework for SnapLogic Testing? Robot Framework offers several advantages for SnapLogic test automation: Human-readable syntax: Tests are written in plain English, making them accessible to both technical and non-technical team members Keyword-driven approach: Promotes reusability and reduces code duplication Extensive ecosystem: Integrates seamlessly with databases, APIs, and various testing tools Comprehensive reporting: Built-in HTML reports with detailed execution logs CI/CD friendly: Easy integration with Jenkins, GitLab CI, and other automation platforms The Power of Docker-Based Testing Infrastructure One of the most powerful features of our framework is its Docker-based architecture. Isolated Test Environments: Each test run operates in its own containerized environment Groundplex Control: Automatically spin up and tear down Groundplex instances for testing Database Services: Pre-configured containers for Oracle, PostgreSQL, MySQL, SQL Server, DB2, and more Message Queue Systems: Integrated support for Kafka, ActiveMQ, and other messaging platforms Storage Services: MinIO for S3-compatible object storage testing This architecture allows below capabilities: Test in production-like environments without affecting actual production systems Quickly provision and tear down complete testing stacks Run parallel tests with isolated resources Ensure consistency across different testing environments snaplogic-common-robot Library Installation The snaplogic-common-robot library is published on PyPI, making installation straightforward https://pypi.org/project/snaplogic-common-robot/ pip install snaplogic-common-robot Core Components The library provides the below components SnapLogic APIs: Low-level keywords for direct API interactions SnapLogic Keywords: High-level business-oriented keywords for common operations Common Utilities: Database connections, file operations, and utility functions. Dependency Libraries: Install all necessary dependency libraries to run Robot Framework tests for SnapLogic. These libraries support API testing, database operations, Docker container testing, JMS messaging, and AWS integration tools. The following libraries are automatically installed as dependencies when you install snaplogic-common-robot, providing comprehensive API support. This library ecosystem continues to expand as we add support for additional features and capabilities. Snaplogic RobotFramework-examples Repository The snaplogic-robotframework-examples repository demonstrates how to build a complete testing framework using the snaplogic common library.https://github.com/SnapLogic/snaplogic-robotframework-examples Framework Overview Note: This project structure is continuously evolving! We're actively working to make the framework easier and more efficient to use,The structure is subject to change as we iterate on improvements to enhance developer experience and framework efficiency. The framework follows a modular architecture with clear separation of concerns: Configuration Layer .env and .env.example manage environment variables for sensitive credentials and URLs env_files/ folder have all details required for creating accounts Makefile provides a central command interface for all build and test operations docker-compose.yml orchestrates the entire multi-container environment with a single command Build Automation makefiles/ directory contains modular scripts organized by service type (databases, messaging, mocks) Each service category has dedicated makefiles for independent lifecycle management Infrastructure docker/ holds Docker configurations for all services (Groundplex, Oracle, PostgreSQL, Kafka) env_files/ stores service-specific environment variables to isolate configuration Containerized approach ensures reproducible test environments across all systems Test Organization test/suite/ contains Robot Framework test suites organized by pipeline functionality test/test_data/ provides input files and expected outputs for validation Tests are grouped logically (Oracle, PostgreSQL+S3, Kafka) for easy maintenance Pipeline Assets src/pipelines/ stores the actual SnapLogic SLP files being tested src/tools/ includes helper utilities and requirements.txt with Python dependencies The snaplogic-common-robot library is installed via requirements.txt, providing reusable keywords Test Reporting Robot Framework automatically generates comprehensive test reports after each execution report.html provides a high-level summary with pass/fail statistics and execution timeline log.html offers detailed step-by-step execution logs with keyword-level information output.xml contains structured test results in XML format for CI/CD integration Reports include screenshots, error messages, and detailed traceability for debugging All reports are timestamped and can be archived for historical analysis Supporting Components travis_scripts/ enables CI/CD automation for continuous testing README/ holds project documentation and setup guides Key Architecture Benefits Modular design allows independent service management Docker isolation ensures consistent test environments Makefile automation simplifies complex operations Clear directory structure improves maintainability CI/CD integration enables automated testing workflows Integration with CI/CD Pipelines One of the most powerful aspects of our Robot Framework testing solution is its seamless integration with CI/CD pipelines. This enables continuous testing, ensuring that every code change is automatically validated against your SnapLogic integrations. Why CI/CD Integration Matters In modern DevOps practices, manual testing becomes a bottleneck. By integrating our Robot Framework tests into your CI/CD pipeline, you achieve: Automatic Test Execution: Tests run automatically on every commit, pull request, or scheduled interval Early Bug Detection: Issues are caught immediately, not days or weeks later in production Consistent Testing: The same tests run every time, eliminating human error and variability Fast Feedback Loop: Developers know within minutes if their changes broke existing integrations Quality Gates: Prevent deployments if tests fail, ensuring only validated code reaches production Jenkins is one of the most popular CI/CD tools, and integrating our Robot Framework tests is straightforward. How It Works? Stage 1: Prepare Environment Install SnapLogic common Robot library and required dependencies Stage 2: Start Docker Services Launches Groundplex, Oracle DB, Kafka, and MinIO containers Stage 3: Run Robot Framework Tests Execute test suites in parallel across 4 threads using pabot Stage 4: Publish Test Results Generate HTML reports, XML results, and test artifacts and can upload to S3. Stage 5: Send Notifications Distributes test results via Slack. Post: Cleanup Tears down containers, removes temp files, archives logs Slack Notifications include the below details Our CI/CD pipeline automatically sends detailed test execution reports to Slack, providing immediate visibility into test results for the entire team. HTML Reports have the below details Robot Framework automatically generates comprehensive HTML reports after each test execution, providing detailed insights into test results, performance, and execution patterns. Real-World Benefits Here's what this means for your team: For Developers Push code with confidence - Tests run automatically Get feedback in minutes - No waiting for QA cycles Fix issues immediately - While the code is still fresh in your mind For QA Teams Focus on exploratory testing - Let automation handle regression Better test coverage - Tests run on every single change Clear reports - See exactly what's tested and what's not Future Enhancements We're continuously improving the framework with planned features include: Enhanced support for more end points Integration with cloud storage services Advanced performance testing capabilities Enhanced security testing features Conclusion The combination of snaplogic-common-robot library and snaplogic-robotframework-examples framework provides a powerful, scalable solution for SnapLogic test automation. By leveraging Docker's containerization capabilities, Robot Framework's simplicity, and SnapLogic's robust APIs, teams can: Reduce testing time from hours to minutes Increase test coverage with automated end-to-end scenarios Improve reliability through consistent, repeatable tests Enable continuous testing in CI/CD pipelines Whether you're testing simple pipeline transformations or complex multi-system integrations, this framework provides the tools and patterns needed for comprehensive SnapLogic testing. Getting Involved We welcome contributions from the SnapLogic community! Here's how you can get involved: Try the Framework: Install snaplogic-common-robot and run the example tests Report Issues: Help us improve by reporting bugs or suggesting enhancements Contribute Code: Submit pull requests with new keywords or test patterns Share Your Experience: Let us know how you're using the framework in your organization Resources snaplogic-common-robot on PyPI: pip install snaplogic-common-robot https://pypi.org/project/snaplogic-common-robot/ snaplogic-robotframework-example repo: https://github.com/SnapLogic/snaplogic-robotframework-examples Documentation: Comprehensive HTML documentation available after installation via README folder Community Support: Join the discussion in SnapLogic Community forums Start automating your SnapLogic tests today and experience the power of comprehensive, containerized test automation! Questions? We're Here to Help! We hope this comprehensive guide helps you get started with automated testing for your SnapLogic integrations. The combination of snaplogic-common-robot and Docker-based infrastructure provides a powerful foundation for building reliable, scalable test automation. Have questions or need assistance implementing this framework? The SLIM (SnapLogic Intelligent Modernization) team is here to support you! We'd love to hear about your use cases, help you overcome any challenges, or discuss how this framework can be customized for your specific needs. Contact the SLIM Team: Reach out to us directly through the SnapLogic Community forums Tag us in your questions with @slim-team Email us at: slim-team@snaplogic.com We're committed to helping you achieve testing excellence and look forward to seeing how you leverage this framework to enhance your SnapLogic automation journey! Happy Testing! The SLIM TeamIndustrial IoT – Turbine Lubrication Oil Level Monitoring & Alert Mechanism via OPC UA and SnapLogic
In the energy sector, turbine lubrication oil is mission-critical. A drop in oil level or pressure can silently escalate into major failures, unplanned shutdowns, and expensive maintenance windows. In this blog, we showcase a real-world implementation using SnapLogic and OPC UA, designed to: 🔧 Continuously monitor turbine lubrication oil levels 📥 Ingest real-time sensor data from industrial systems 📊 Store telemetry in data lakes for analytics and compliance 📣 Real-time Slack alerts to engineers — before failures strike This IIoT-driven solution empowers energy providers to adopt predictive maintenance practices and reduce operational risk
