Forum Discussion

ryan's avatar
ryan
New Contributor
4 years ago

REST API returning tar.gz

I am attempting to make an API call that returns a tar.gz file. I want to convert it to a csv and then upload it to an S3 bucket.
Note: S3 and aws is not set up yet but would like to be able to read it in and get that aspect down before they our team has our aws accounts applied.

Currently I have tried a few different methods but am not getting the expected results.

Rest Get >>> Mapper (entity/what I would think is the content data)>>>CSV Formatter>>>File Writer

I am obviously doing something wrong so any suggestions would be appreciated.

1 Reply

  • koryknick's avatar
    koryknick
    Employee
    4 years ago

    Hi @ryan - welcome to the Community! Great first question! SnapLogic does not have a snap to read the TAR file, so you will need to use a Script snap to handle that portion of this request.

    I came across this a while back and have a simple pipeline example using a Script snap to explode out the TAR data. This solution works fine for smaller TAR files but since it does everything in memory, it may consume large amounts of your node memory if the input file is large. This is attached here:
    Read tar.gz file_2022_01_31.slp (12.3 KB)

    One of my co-workers created a slightly different design that pushes the TAR file to the snaplex local node file system and uses Python to explode out the files within the file system then reads the contents back in. Hopefully this link works for you: Box