hkaplan
Former Employee
Joined 9 years ago
User Profile
User Widgets
Contributions
SnapLogic Before and After Control-Break Pipeline
Created by @hkaplan This pipeline pattern shows how to use SnapLogic’s snap.out.totalCount and the Join Snap to compare current and/or previous incoming records or current and/or next incoming records and set a Control-Break Flag to identify when a Control-Break occurs. Configuration The self-contained data has 2 outputs: one comparing current record with next record and the second comparing current record with previous record. Sources: JSON Generator; ID, Name, and InvoiceAmount Targets: Mapper, Input elements with Control-Break Flag Snaps used: JSON Generator, Mapper, Copy, Join Downloads Control-Break using Join on Next Record Pattern.slp (13.9 KB)2.5KViews0likes0CommentsConvert Excel Columns to JSON Arrays
Contributed by @hkaplan This pipeline reads in an Excel file with multiple columns, 1 to n number of rows with null cells and converts each Excel column into a JSON array with all of the null rows removed. Configuration The trick here is SnapLogic’s Aggregator Snap’s Concat function. Source: Excel spreadsheet Columnar data Target: Excel columnar data as an array Snaps used: File Reader, Excel Parser, Aggregate, Mapper, The sample Excel file looks like this: The resulting array looks like this: Downloads Convert Excel Columns to Arrays.slp (12.6 KB) ExcelColumnToArray.xlsx (8.7 KB)8.6KViews1like0CommentsMARC21 custom snap
MARC21 (MAchine-Readable Cataloging) standards are a set of digital formats for the description of items such as books developed in the 1960s. The latest MARC standard, MARC 21, was created in 1999 and added formats for authority records, holdings records, classification schedules, and community information, in addition to bibliographic. EachMARC21 record comprises of 3 aspects: the field designations within each record the structure of the record the actual content of the record itself Then MARCXML was created as an XML schema based on the common MARC21 standards and is parsable by SnapLogic’s XML parser. This MARC21 custom snap converts MARC21 to MARCXML using open-source java code available on github using the following URL. GitHub - marc4j/marc4j: Git repository initially imported from the Tigris CVS repo as of 2013.01.25 Once the MARC21 data is converted to MARCXML, SnapLogic can then read in the XML using the XML parser just like any other XML input. Below is a link to download the MARC21 custom snap as well as a sample MARC21 record to illustrate the complexity of the MARC21 data structure. [MARC21custom snap download link] https://snaplogic.box.com/s/25lwqm6ccm4t22y09a19ksi5b0z9t52d [MARC21 sample record] 02672ngm 2200553 a 4500001001300000003000400013005001700017006001200034007001000046007001500056008004100071028002800112040002100140245024600161246001400407260005600421300005700477490005700534505034800591506004600939511003400985520011801019538005401137546002201191600002701213600003901240600003901279600004901318650003001367651004901397651005601446651003501502651002901537655002901566655003101595700002101626700002301647700001801670710002701688710003001715740002601745740003701771740004401808740006001852740005401912740004701966830005802013856004702071mfm00108212 MFM20140731134835.0m go c vz||z|zu|cr||n|—||a|a080716s2003 nyu027 g s vleng d41a16th-001bAaaaaaa Video aAzPhLCLcAzPhLCL00a16th & 17th century turning points in U.S. history ;v1,p1500-1585h[electronic resource] /cproduced by Centre Communications and People Productions for Aaaaaaa Video Publishing written by Mark Reeder and Ron Meyer ; directed by Ron Meyer.32a1500-1585 aNew York, N.Y. :bAaaaaaa Video Publishing,cc2003. aStreaming video file (27 min.) :bdigital, sd., col.1 a16th and 17th century turning points in U.S. history0 a1502: Christopher Columbus (3 min.) – 1513: Ponce de Leon discovers Florida (4 min.) – 1539: Hernando De Soto explores Southern U.S. (6 min.) – 1540: Francisco Vasques de Coronado maps America’s Southwest (5 min.) – 1565: Spanish found first American City: St. Augustine (4 min.) – 1585: Sir Walter Raleigh and the Roanoke Colony (4 min.) aAccess restricted by licensing agreement.0 aNarrated by Alphonse Keasley. aChaptered programs chronicle the decisive events and people in the exploration and early settlement of America. aMode of access: World Wide Web ; Flash streaming. aClosed captioned.10aColumbus, Christopher.10aPonce de Le‚on, Juan,d1460?-1521.10aSoto, Hernando de,dca. 1500-1542.10aV‚azquez de Coronado, Francisco,d1510-1549.2.4KViews0likes0CommentsBest way to filter out zip file __MACOSX folder entries
I have a zip file I need SnapLogic to source that was created on a Mac using Apple’s compress utility. To read the zip file, I used a ZipFile Reader snap which had no problem reading the zip file. Next I connected a XML parser snap to the ZipFile Reader snap and ran into an issue about the XML data not being well formed. “Failure: Failed to convert xml to json, Reason: Invalid UTF-8 start byte 0xad (at char #38, byte #-1), Resolution: Please check if the xml data is well formed” When I do a ‘view data’ on the ZipFile Reader’s output, I see there is an extra 223 byte __MACOSX entry for every file contained in the zip file. What is the best way to remove these extra _MACOSX entries from be passed to the XML parser snap?2.3KViews0likes1CommentIdentifying when a Control Break occurs in a pipeline
Have you ever needed to identify when a control break occurs in your incoming data? Having this ability can save you considerable processing resources in cases where your incoming data is sorted on last names and you only need to perform calculations or lookups when a change occurs on last name. Here is the pipeline i put together to demonstrate how to identify control breaks in your incoming data. the first snap generates the following input data: Using a sort snap i then sort the incoming records by lastname In order to identify when a control break occurs, we need to compare one record to another and to do this we need to join one record to it’s previous record. To accomplish this we will use one of the snap functions accessible in as an expression in a mapper snap. Here you can see where i am moving snap.in.totalCount out as $SnapInTotalCount. Next i use a copy snap to make two copies of the incoming data which i will then join back together using a join snap. I now need to join the current record from one input stream to the previous record from the other input stream and the following trick below is how i accomplish this. I subtract one from the left path’s $SnapInTotalCount and join it to right path’s $SnapInTotalCount. *note i subtracted 0 to keep the data formats aligned. i now have record 1 on the same line as record 0, record 2 on the same line as record 1 and so on. Now all i have to do to set my flag is compare LastName to input1_LastName. Using a condition snap and the following conditional expressions, i am able to set my control break flag ($CD_FLAG) to either T or N. Below are the final results and as you can see CB_FLAG is set to ‘Y’ anytime there is a change in LastName. I can now use CB_FLAG downstream in my pipeline in a router or filter snap to avoid unneeded processing. Control Break Pipeline Example_2017_02_28.slp (12.4 KB)2.5KViews0likes0Comments