Decompressing and reading WINDOWS zip files
Here is a pipeline that reads a .zip file, decompresses (unzips) it and parses it contents. Please note that our current “Decompress” Snap only supports BZIP2, GZIP and DEFLATE. Binary to Document Snap Encode or Decode property is set to NONE Mapper configuration Zip file read Snap configuration This will return a binary stream and based on content-type, in this case we have an Excel xls file so we snap-in Excel Parser. Pipeline is attached, you need to pass zip file to read BK new pipeline 0_2017_02_21.slp (6.3 KB)5.4KViews1like3CommentsReturn custom HTTP Header
I’m trying to return a hash of a CSV document I’m returning in a triggered task and I want to return it in a HTTP header. I would have thought i can do it the same way i can manipulate the status code by setting a binary header, but the headers aren’t coming through. Does anyone have an idea how to achieve this? (Ideally without buying any additional subscription services, thanks) Here’s an example of the output of the csv formatter: Best regards ThomasSolved4.6KViews0likes3CommentsReading and Writing Data to Azure Blob Storage and Azure Data Lake Store
Organizations have been increasingly moving towards and adopting cloud data and analytics platforms like Microsoft Azure. In this first in a series of Azure Data Platform blog posts, I’ll get you on your way to making your adoption of the cloud platforms and data integration easier. In this post, I focus on ingesting data into the Azure Cloud Data Platform and demonstrate how to read and write data to Microsoft Azure Storage using SnapLogic. For those who want to dive right in, my 4-minute step-by-step video “Building a simple pipeline to read and write data to Azure Blob storage” shows how to do what you want, without writing any code. What is Azure Storage? Azure Storage enables you to store terabytes of data to support small to big data use cases. It is highly scalable, highly available, and can handle millions of requests per second on average. Azure Blob Storage is one of the types of services provided by Azure Storage. Azure provides two key types of storage for unstructured data: Azure Blob Storage and Azure Data Lake Store. Azure Blob Storage Azure Blob Storage stores unstructured object data. A blob can be any type of text or binary data, such as a document or media file. Blob storage is also referred to as object storage. Azure Data Lake Store Azure Data Lake Store provides what enterprises look for in storage today and it: Provides additional enterprise-grade security features like encryption and uses Azure Active Directory for authentication and authorization. Is compatible with Hadoop Distributed File System (HDFS) and works with the Hadoop ecosystem including Azure HDInsight. Includes Azure HDInsight clusters, which can be provisioned and configured to directly access data stored in Data Lake Store. Allows data stored in Data Lake Store to be easily analyzed using Hadoop analytic frameworks such as MapReduce, Spark, or Hive. How do I move my data to the Azure Data Platform? Let’s look at how you can read and write to Azure Data Platform using SnapLogic. For SnapLogic Snaps that support Azure Accounts, we have an option to choose one of Azure Storage Account or Azure Data Lake Store: Configuring the Azure Storage Account in SnapLogic can be done as shown below using the Azure storage account name and access key you get from the Azure Portal: Configuring the Azure Data Lake Store Account in SnapLogic as shown below, uses the Azure Tenant ID, Access ID, and Secret Key that you get from the Azure Portal: Put together, you’ve got a simple pipeline that illustrates how to read and write to Azure Blob Storage: In the next post in this series, I will describe the approaches to move data from your on-prem databases to Azure SQL Database.7.8KViews0likes2CommentsAn XML Generator & XML Formatter Snaps example
Here’s an example illustrating the use case of the snaps XML Generator, XML formatter and File writer. The 1st snap “XML Generator” of the pipeline generates a document format data with the given xml input configured in the snap on “Edit XML *” in snap settings. The 2nd snap “XML Formatter” accepts the document format and encapsulates the data into XML formates and provides the the binary output which can be an input to the File writer snap which accepts the binary input. Attached is the .slp for pipeline reference.XMLGenerator_Formatter_2017_03_1.slp (4.1 KB)4.6KViews1like1CommentHow to create an AES key for the AES Snaps?
Step 1: use openssl to create a new AES key. Below is an example of a command and the result. The iv string can be copy and pasted into the Initialization vector field of an AES encryption/decryption snap. $ openssl enc -aes-256-cbc -base64 -k secret -P -md sha1 salt=18F3973C70475135 key=1B76BAE6805E9EEF5C4A34AB26B1D6F68A5CA6965DD1B157AFAD25AE029C4874 iv =8E204167430AA0CD292C35DFB52D86F8 Step 2: The AES encryption/decryption snap required the AES key to be a base64 encoded string. Below is an example of a command and the result of base64-encoding the binary AES key: $ echo "1B76BAE6805E9EEF5C4A34AB26B1D6F68A5CA6965DD1B157AFAD25AE029C4874" | xxd -r -p | base64 G3a65oBenu9cSjSrJrHW9opcppZd0bFXr60lrgKcSHQ= Here’s an screenshot of a settings:3.4KViews1like0Comments