Scalable Analytics Platform: A Data Engineering Journey
Scalable Analytics Platform: A Data Engineering Journey - Explore SnapLogic's innovative Medallion Architecture approach for handling massive data, improving analytics with S3, Trino, and Amazon Neptune. Learn about cost reduction, scalability, data governance, and enhanced insights.
Unlock the Full Potential of Your Data with SnapLogic AutoSync
SnapLogic AutoSync: Your Agile Chopper for Data Integration In the world of enterprise data, long-haul flights are essential—but sometimes you need to lift off quickly, land precisely, and get the job done without waiting for a runway. Think of SnapLogic’s Intelligent Integration Platform (IIP) as your data jumbo jet: powerful, scalable, and built for complex, high-volume integrations across global systems. Now imagine you need something faster, more nimble—something that doesn’t require a flight crew to get airborne. Enter SnapLogic AutoSync—the agile chopper in your integration fleet. Whether you're syncing Salesforce data after an acquisition, uploading spreadsheets for instant analysis, or automating recurring flows between systems like Marketo and Redshift, AutoSync lifts your data with just a few clicks. It empowers business users to move quickly and experiment safely, without compromising on governance or control. With AutoSync, you’re not just reducing engineering cycles—you’re accelerating the entire journey from raw data to actionable insight.Data At Scale For AI At Scale: How To Think About Data Readiness
This week's episode of the Enterprise Alchemists is another live recording from Integreat 2024 in London! This week we have Maks Shah of Syngenta; we had a fascinating conversation during the after-event cocktail party — which is why this episode is a bit shorter than normal. Maks's key takeaway was that "there was a common theme throughout all the presentations this afternoon, and that was that your data has to be fit for it". There is no hashtag#AI success without the data to feed it with. Establishing a solid data foundation is step zero on your journey to hashtag#GenAI.Using SAP S/4HANA Snaps to extract Operational Data Provider (ODP) data
SAP's Operational Data Provisioning (ODP) provides a technical infrastructure that you can use to support two different application scenarios. The first of these is Operational Analytics for decision making in operative business processes. The other is data extraction and replication. It has been around for quite some time and has replaced all other methods when extracting doing full data as well as delta extraction into BW/4HANA. SAP has down-ported the framework to older versions of their ERP Suite like SAP ECC, CRM, SRM, SCM to make it widely available and that lead many 3rd party vendors to tap into the Remote Function Calls build for ODP to leverage the framework and it's capabilities. In February of 2024 SAP has released a Note 3255746 stating that going forward, SAP will not permit the use of RFCs for ODP data extraction. SAP clearly states that the RFC modules are only intended for SAP-internal applications and can be modified by SAP at any time, without notice. It also reserves the right to put in place technical measures that will restrict and audit the unpermitted use of RFC modules of the ODP Data Replication API. It further goes on to state any issues experienced or caused by third-party applications or customer applications by using the RFC modules, are entirely at the risk of the customer, and SAP is not responsible for resolving such issues, nor will it provide any support. As a solution SAP states that the Operational Data Provisioning (ODP) framework for data extraction and replication is exposed by an official and externally-available OData API and that all customer and third-party applications should be built using this API as it provides a stable interface (link to documentation). This blog aims to show you how you can accomplish the extraction of data from the ODP framework via OData and our SnapLogic S/4HANA Snap Pack. What is ODP and what can I do with it I mentioned already that SAP made the ODP the standard extraction method for BW/4 HANA. The Operational Data Provisioning (ODP) framework for data distribution provides a consolidated technology for data provisioning and consumption. ODP supports data extraction and replication for multiple targets and supports capturing of changes at source. This means that you are able to extract a wide variety of sources through the framework. Below is a drawing often used by SAP to depict the capabilities and visualize the various options. All of these ODP scenarios (often referred to as contexts) can be consumed by SnapLogic. ODP - SAP Extractors (ODP_SAP): This context exposes BW DataSources as Operational Data Providers. A BW DataSource defines an extraction structure that is populated by extraction code, which implements the logic to retrieve relevant data from the ABAP system. The ODP framework, without additional configuration, supports DataSources released by the application owner. Most of these DataSources are part of the SAP Business Suite. ODP - ABAP CDS Views (ODP_CDS): ABAP Core Data Services (CDS) enhance ABAP SQL to enable the definition and consumption of semantically rich data models natively. This enhancement boosts productivity, consumability, performance, and interoperability. The key concept for expressing the semantics of ABAP CDS models is annotations, which are attached to elements of the model. By applying annotations with analytical semantics, ABAP CDS views can be annotated so that the resulting metadata allows the generation of transient Operational Data Providers. ODP - BW (ODP_BW): SAP BW/4HANA exposes its data as Operational Data Providers for extraction purposes. All relevant SAP BW/4HANA InfoProviders are supported as sources. ODP - SLT Queue (ODP_SLT): Utilizing the ODP infrastructure and the trigger-based replication of the SAP Landscape Transformation Replication Server (SLT), data can be transferred in real-time from multiple source systems to one or more target systems. The SLT Replication Server serves as a provider for the ODP infrastructure, making tables from SAP sources available as delta queues. How do I expose an ODP Data Source via OData To capture deltas, the source data is written to an Operational Delta Queue through an update process. The process supports package-based full extraction for very large volumes using the operational delta queue. After the initial full extraction, it can also capture and extract deltas for changes using the operational delta queue. However, the ODP data source must support deltas for this functionality. SnapLogic can then retrieve data from this delta queue, which also aids in monitoring the data extraction and replication process. Exposure of the data is achieved using the SAP Gateway Foundation and by generating an SAP OData service with the SAP Gateway Service Builder for the ODP Data Source. This service (URL) can be accessed from an external OData client application via OData/HTTP. Generating the OData Service in the Gateway Service Builder The process is entirely non code, everything that is required in SAP will be generated including the OData service in the Gateway Service Builder. Simply navigate to transaction SEGW in SAP. Start by creating a new Project using the first icon in the menu bar marked in red in the below screenshot. In the popup give your project a name and description, choose Service with SAP Annotations and select the package you want the generated code to be stored under. If you do not need to transport the generated code to other SAP systems you can choose $TMP as the package Once the project is created select Data Model and choose Redefine -> ODP Extraction from the context menu. A wizard will now guide you through the steps in which you can select from all possible ODP data sources. Go through the wizard by choosing the Next button until the wizard finishes by generating the Model. When the Model is generated you continue to generate the OData Services by choosing the Generate Runtime Objects button. Choose the Defaults for all settings and take note of the Technical Service Name. This is the name that your OData service will have once it is generated. Activate the OData Service To activate the newly created service head to transaction /IWFND/MAINT_SERVICE in SAP. Choose the Add Service button. Enter the Technical Service Name and hit Enter. Then select the Technical Service Name from the List at the bottom and click the Add Selected Services button. Once activated you can find the service in the Service Catalog by filtering by Technical Service Name How do I consume the OData Service in SnapLogic On the SnapLogic side you use the SAP S/4HANA Read Snap to read data from the exposed OData Service. The OData Service uses an account that points to the OData Version 2 catalog containing all services as show in the screenshot blow. On the Settings screen of the SAP S/4HANA Read Snap, search for the OData Service in the API Object or select the service from the dropdown list. The service creation from the extractor gives you two entities to use: One with a name starting with EntityOf<objectName>, FactsOf<objectName> or AttrOf<objectName> depending on the type of extractor or view that represents the data source model. One one that starts with DeltaLinksOf<objectName> that exposes information about current and past delta tokens. By default, if you send a request to the first service, you will retrieve a full dataset, just like you’d work with any other OData service. However, if you add the Prefer: odata.track-changes special request header as show in the screenshot below the OData Service will subscribe to the Operational Delta Queue (ODQ) and return __delta links that allow you to request the delta that accumulated. Be sure that you removed the default entry in the Output Entry Limit field that represents the top=X URL attribute in OData in the Snap. If I kept the default entry during my tests, our S/4HANA system would not register the request in the ODQ at all. Lastly make sure that you check the Raw response checkbox for the Snap to add the __delta link to the output for downstream snaps. Retrieving ODP data in packages Should you want the ODP data to be retrieved in packages, you can specify an additional value in the Prefer header called odata.maxpagesize=<N> where <N> is the number of packages that should be prepared for retrieval, for example, odata.maxpagesize=50. This starts a background job in SAP for paging and the data is cached in ODQ. Using the odata.maxpagesize value you can specify the quantity of packages to be retrieved from the delta queue. The end of the response contains a relative link in the attribute __next, which can be used to retrieve the next package. This link can be fed into the Has Next and Next URL fields to make the S/4HANA Read Snap page through the packages stored in ODQ by the background job. How Can I make use of the Deltatoken in a SnapLogic Pipeline If subscribed successfully to ODQ the output of the S/4HANA Read snap will contain the __delta link. This link contains the query parameter !deltatoken. This URL can be used in subsequent calls to retrieve the delta update from the point in time when the token was issued to your subscription. You can either store the URL or only the token and use it in subsequent runs of the pipeline or alternatively use the DeltaLinksOf<objectName> described above to retrieve all tokens that have been issued for your subscription and then use the last token in the list to read the delta. Understanding the information received with a delta update With every row that you receive from a delta you also get two additional fields named ODQ_ENTITIYCNTR and ODQ_CHANGEMODE. The two fields will need to be considered in the target to understand if they are changed or new records. Tracking and Terminating a Subscription. Every OData Service generated for ODP also contains two additional Entities, SubscribedToFactsOf <objectName> to determine whether a delta subscription exists for the user and TerminateDeltasForFactsOf<objectName> to terminate delta extraction for the user. In our S/4HANA system the entities where not exposed through the ODP catalog but could be called without issues. The attached pipeline makes use of the SubscribedToFacts service to check if we have subscribed in a previous run to ODQ.
From Fragile to Agile: How AI Is Changing Integration
Generative AI is changing the blueprint for integration. It’s changing the way we think about the economics of team velocity, project structure, and delivery models. This technology, when applied to the software interactions we use every day, has the potential to play a transformative role in all types of organizations. SnapLogic recently released SnapGPT, the world’s first generative integration solution. SnapGPT takes this a step further by automating the creation and ongoing curation of data and process flows, allowing businesses to quickly introduce changes around new opportunities.SQL Insert from JSON change type and only type
Hi All, I have a Pipeline that takes data from a Rest API and converts it to a SQL database. This is all working and we are able to create the table without having to define the Schema in a mapper (we have hundreds of tables with thousands of fields and it would be very difficult to map them all individually). My current issue is that when converting from the JSON to SQL Server it is automatically converting the type from string to varchar(8000). We have Japanese characters etc that are being lost because of this conversion. I need to have it be nvarchar(4000) or if the string is >4000 be nvarchar max. The question is how do I make that conversion without having to map/define each and every field. And at the same time making sure I don’t truncate anything. Thanks for your help.
Purple to Redshift-WiFi Analytics
Created by @asharifian , SnapLogic This pattern integrates Purple with Redshift. Purple is a WiFi analytics application that builds detailed visitor profiles and tracks visitors in your venues, such as conferences, workshops, gatherings, etc. At Purple, all of the data collected is stored within a centralized, enterprise-class reporting suite, available for analysis. This pattern retrieves the analytics data from Purple and bulk upserts it into Redshift for reporting and additional data analytics purposes. Configuration A child pipeline is used to generate the HMAC SHA256 key. This is a requirement for the Purple API’s. Sources: Purple Venues, Visitors Targets: Redshift database tables to store venue and visitor data. Snaps used: Mapper, Pipeline Execute, REST Get, JSON Splitter, Copy, Redshift Bulk Upsert, Script, Join Downloads Purple to Redshift-WiFi Analytics.slp (21.4 KB) HMAC_SHA256.slp (18.5 KB)
