Create Box folders for Salesforce accounts
Created by @skatpally For an account created in Salesforce, SnapLogic creates a folder in Box with the Salesforce account name as the folder name, if it doesn’t already exist. SnapLogic creates a shared link to the Box folder. Configuration Sources: Salesforce account ID Targets: Box folder Snaps used: Mapper, Box Add Folder, REST Put DownloadsConvert data into various file formats and write to HDFS
Created by @pkona This pipeline pattern has a total of two pipelines: Setup and HDFS RW (Standard), which contains the datasets and writes to HDFS Hadoop file formats (Standard) that converts the CSV dataset into file formats that are commonly used for Hadoop and Big Data use cases. File formats include: CSV JSON Parquet ORCFile Avro SequenceFile Configuration Specify values for the following pipeline parameters: hdfs_base_uri hdfs_folder_path Sources: CSV File on HDFS Targets: Avro, Parquet, CSV, JSON, ORCFile, SequenceFile files on HDFS Snaps used: DownloadsWorkday to Oracle EBS Employees
Created by @ckonduru When there is a new or updated object in Workday, execute operation in Oracle E-business Suite Screenshot of pipeline Configuration Sources: Workday Targets: Oracle EBS Snaps used: JSON Generator, XML Parser, JSON Formatter, JSON Parser, JSON Splitter, Group By N, Mapper, Workday Read Downloads Attach pipelines and any necessary resources such as expression libraries or source files.Scenario Detection Stage
Submitted by @stodoroska from Interworks Based on the source data, we are creating action flags that later will be used in the Action stage pipeline. Screenshot of pipeline Configuration Because we are working with a large scale of data, the target records are grouped in the chunks of 2000 records. This makes the pipeline faster and less memory is used. Sources: CSV file Targets: CSV file and PGP-encrypted file Snaps used: Mapper, Join, Group by N, Splitter, CSV Parser, File Writer, File Reader, Union, Copy, PGP Encrypt Downloads Attach pipelines and any necessary resources such as expression libraries or source files.EDWH to Azure Blob Storage and Azure Data Lake
Submitted by @stodoroska from Interworks___ Selecting from Data warehouse data from an on-premise Oracle table, converting into CSV, storing it in Azure Blob Storage and storing in Azure Data Lake. Screenshot of pipeline Configuration In order to make this pipeline work you, need to configure the child backlog new table, which is running on premise and that is why it is put in a Pipeline Execute Snap. Also you need to configure Azure accounts and Azure paths for Blob Storage and Data Lake. Sources: Oracle table Targets: Azure Blob Storage and Azure Data Lake Snaps used: Oracle Select, Pipeline Execute, CSV Formatter, File Writer, File Reader Downloads Attach pipelines and any necessary resources such as expression libraries or source files.Validation Stage 3 (With Workday)
Submitted by @stodoroska from Interworks Based on the 2 sources we determine what is changed and call the corresponding action. Screenshot of pipeline Configuration Respective files and Workday account configurations should be applied. Sources: CSV file and Workday rest service Targets: Multiple Workday objects Snaps used: Mapper, Join, Group by N, Splitter, CSV Parser, File Writer, File Reader, Union, Copy, Workday Read, Filter, Data Validator Downloads Attach pipelines and any necessary resources such as expression libraries or source files.CDC Delta Load Oracle
Submitted by @stodoroska from Interworks Delta Load from a staging to Oracle DWH table. Pipeline should detect if the record is an Update, Delete or Insert according to a flag in the Mlog table which is Oracle native table for storing logs of transactions. According to those flags and the id of the records, the pipeline is deciding which action it should perform, update, delete or insert. Screenshot of pipeline Configuration In each of the Snaps you should define your respective queries and DB accounts. Sources: Oracle DB table Targets: Oracle DB table Snaps used: Oracle Execute, Router, Mapper, Aggregate, Unique, Join, Copy Downloads Attach pipelines and any necessary resources such as expression libraries or source files.
Complex JSONPATH example - extracting values from a deep branch underneath
Here is an example of a very complex json tree and how to extract values from the middle of the tree. If you look at this sample json, we need to extract the TenderType and AuthorizationCode under the Tender Element and also extract the ab1:BeginDateTime where the @EventName": "First Item Time" under ab1:Event . You can see immediately that the both the Tender and SequenceNumber are at different levels under the tree. The following expression will get the authorization code. > jsonPath($, “$RetailTransaction.LineItem[?(@.hasOwnProperty(‘Tender’) == true)].Tender.Authorization.AuthorizationCode”) As you can see, LineItem is an array and within the array, instead of [*], @ gets the current node and we are checking if the current node has Tender Element. If the node has Tender element then get the AuthorizationCode under Authorization. The above is one level nested deep. The following expression will get the BeginDateTime under ab1:Event where the eventname = ‘First Item Time’ > jsonPath($, “$RetailTransaction.LineItem[?(@.hasOwnProperty(‘ab1:TransactionExtensions’) == true)].[‘ab1:TransactionExtensions’][‘ab1:SpeedOfService’][‘ab1:OrderSpeedOfService’][‘ab1:SOSTransaction’][‘ab1:Event’][?(@[‘ab1:EventID’][‘@EventName’]==‘First Item Time’)][‘ab1:BeginDateTime’]”) As you can see, in the LineItem array, first find if the element has ‘ ab1:TransactionExtensions ’ then, go deep unto the ab1:Event array and within that array again first if the eventname equals First Item Time , and then get the element ab1:BeginDateTime I hope this helps in resolving complex JSON PATH expressions. Enclosed is the pipeline too that demonstrates and make use of this. Complex-JsonPath.slp (11.0 KB)
