SAP's Operational Data Provisioning (ODP) provides a technical infrastructure that you can use to support two different application scenarios. The first of these is Operational Analytics for decision making in operative business processes. The other is data extraction and replication. It has been around for quite some time and has replaced all other methods when extracting doing full data as well as delta extraction into BW/4HANA. SAP has down-ported the framework to older versions of their ERP Suite like SAP ECC, CRM, SRM, SCM to make it widely available and that lead many 3rd party vendors to tap into the Remote Function Calls build for ODP to leverage the framework and it's capabilities.
In February of 2024 SAP has released a Note 3255746 stating that going forward, SAP will not permit the use of RFCs for ODP data extraction. SAP clearly states that the RFC modules are only intended for SAP-internal applications and can be modified by SAP at any time, without notice. It also reserves the right to put in place technical measures that will restrict and audit the unpermitted use of RFC modules of the ODP Data Replication API. It further goes on to state any issues experienced or caused by third-party applications or customer applications by using the RFC modules, are entirely at the risk of the customer, and SAP is not responsible for resolving such issues, nor will it provide any support.
As a solution SAP states that the Operational Data Provisioning (ODP) framework for data extraction and replication is exposed by an official and externally-available OData API and that all customer and third-party applications should be built using this API as it provides a stable interface (link to documentation).
This blog aims to show you how you can accomplish the extraction of data from the ODP framework via OData and our SnapLogic S/4HANA Snap Pack.
What is ODP and what can I do with it
I mentioned already that SAP made the ODP the standard extraction method for BW/4 HANA. The Operational Data Provisioning (ODP) framework for data distribution provides a consolidated technology for data provisioning and consumption. ODP supports data extraction and replication for multiple targets and supports capturing of changes at source. This means that you are able to extract a wide variety of sources through the framework. Below is a drawing often used by SAP to depict the capabilities and visualize the various options. All of these ODP scenarios (often referred to as contexts) can be consumed by SnapLogic.
ODP - SAP Extractors (ODP_SAP):
This context exposes BW DataSources as Operational Data Providers. A BW DataSource defines an extraction structure that is populated by extraction code, which implements the logic to retrieve relevant data from the ABAP system. The ODP framework, without additional configuration, supports DataSources released by the application owner. Most of these DataSources are part of the SAP Business Suite.
ODP - ABAP CDS Views (ODP_CDS):
ABAP Core Data Services (CDS) enhance ABAP SQL to enable the definition and consumption of semantically rich data models natively. This enhancement boosts productivity, consumability, performance, and interoperability. The key concept for expressing the semantics of ABAP CDS models is annotations, which are attached to elements of the model. By applying annotations with analytical semantics, ABAP CDS views can be annotated so that the resulting metadata allows the generation of transient Operational Data Providers.
ODP - BW (ODP_BW):
SAP BW/4HANA exposes its data as Operational Data Providers for extraction purposes. All relevant SAP BW/4HANA InfoProviders are supported as sources.
ODP - SLT Queue (ODP_SLT):
Utilizing the ODP infrastructure and the trigger-based replication of the SAP Landscape Transformation Replication Server (SLT), data can be transferred in real-time from multiple source systems to one or more target systems. The SLT Replication Server serves as a provider for the ODP infrastructure, making tables from SAP sources available as delta queues.
How do I expose an ODP Data Source via OData
To capture deltas, the source data is written to an Operational Delta Queue through an update process. The process supports package-based full extraction for very large volumes using the operational delta queue. After the initial full extraction, it can also capture and extract deltas for changes using the operational delta queue. However, the ODP data source must support deltas for this functionality. SnapLogic can then retrieve data from this delta queue, which also aids in monitoring the data extraction and replication process. Exposure of the data is achieved using the SAP Gateway Foundation and by generating an SAP OData service with the SAP Gateway Service Builder for the ODP Data Source. This service (URL) can be accessed from an external OData client application via OData/HTTP.
Generating the OData Service in the Gateway Service Builder
The process is entirely non code, everything that is required in SAP will be generated including the OData service in the Gateway Service Builder. Simply navigate to transaction SEGW in SAP. Start by creating a new Project using the first icon in the menu bar marked in red in the below screenshot. In the popup give your project a name and description, choose Service with SAP Annotations and select the package you want the generated code to be stored under. If you do not need to transport the generated code to other SAP systems you can choose $TMP as the package
Once the project is created select Data Model and choose Redefine -> ODP Extraction from the context menu.
A wizard will now guide you through the steps in which you can select from all possible ODP data sources. Go through the wizard by choosing the Next button until the wizard finishes by generating the Model.
When the Model is generated you continue to generate the OData Services by choosing the Generate Runtime Objects button.
Choose the Defaults for all settings and take note of the Technical Service Name. This is the name that your OData service will have once it is generated.
Activate the OData Service
To activate the newly created service head to transaction /IWFND/MAINT_SERVICE in SAP. Choose the Add Service button.
Enter the Technical Service Name and hit Enter. Then select the Technical Service Name from the List at the bottom and click the Add Selected Services button.
Once activated you can find the service in the Service Catalog by filtering by Technical Service Name
How do I consume the OData Service in SnapLogic
On the SnapLogic side you use the SAP S/4HANA Read Snap to read data from the exposed OData Service. The OData Service uses an account that points to the OData Version 2 catalog containing all services as show in the screenshot blow.
On the Settings screen of the SAP S/4HANA Read Snap, search for the OData Service in the API Object or select the service from the dropdown list. The service creation from the extractor gives you two entities to use:
- One with a name starting with EntityOf<objectName>, FactsOf<objectName> or AttrOf<objectName> depending on the type of extractor or view that represents the data source model.
- One one that starts with DeltaLinksOf<objectName> that exposes information about current and past delta tokens.
By default, if you send a request to the first service, you will retrieve a full dataset, just like you’d work with any other OData service. However, if you add the Prefer: odata.track-changes special request header as show in the screenshot below the OData Service will subscribe to the Operational Delta Queue (ODQ) and return __delta links that allow you to request the delta that accumulated. Be sure that you removed the default entry in the Output Entry Limit field that represents the top=X URL attribute in OData in the Snap. If I kept the default entry during my tests, our S/4HANA system would not register the request in the ODQ at all. Lastly make sure that you check the Raw response checkbox for the Snap to add the __delta link to the output for downstream snaps.
Retrieving ODP data in packages
Should you want the ODP data to be retrieved in packages, you can specify an additional value in the Prefer header called odata.maxpagesize=<N> where <N> is the number of packages that should be prepared for retrieval, for example, odata.maxpagesize=50. This starts a background job in SAP for paging and the data is cached in ODQ. Using the odata.maxpagesize value you can specify the quantity of packages to be retrieved from the delta queue. The end of the response contains a relative link in the attribute __next, which can be used to retrieve the next package. This link can be fed into the Has Next and Next URL fields to make the S/4HANA Read Snap page through the packages stored in ODQ by the background job.
How Can I make use of the Deltatoken in a SnapLogic Pipeline
If subscribed successfully to ODQ the output of the S/4HANA Read snap will contain the __delta link. This link contains the query parameter !deltatoken. This URL can be used in subsequent calls to retrieve the delta update from the point in time when the token was issued to your subscription. You can either store the URL or only the token and use it in subsequent runs of the pipeline or alternatively use the DeltaLinksOf<objectName> described above to retrieve all tokens that have been issued for your subscription and then use the last token in the list to read the delta.
Understanding the information received with a delta update
With every row that you receive from a delta you also get two additional fields named ODQ_ENTITIYCNTR and ODQ_CHANGEMODE. The two fields will need to be considered in the target to understand if they are changed or new records.
Tracking and Terminating a Subscription.
Every OData Service generated for ODP also contains two additional Entities, SubscribedToFactsOf<objectName> to determine whether a delta subscription exists for the user and TerminateDeltasForFactsOf<objectName> to terminate delta extraction for the user. In our S/4HANA system the entities where not exposed through the ODP catalog but could be called without issues. The attached pipeline makes use of the SubscribedToFacts service to check if we have subscribed in a previous run to ODQ.
