Forum Discussion

maahutch's avatar
maahutch
New Contributor
2 years ago
Solved

Backing up two database tables to an S3 bucket as one archive

I'm trying to develop a pipeline that queries two (or more) database tables, converts them to CSV, compresses them to a single archive and writes them to and S3 bucket. I have a working pipeline for a single table:

But what I'd like to do is have multiple select queries that pull in different tables and compresses them to a single archive. Is this possible? I looked at "ZipFile Write" but there didn't seem to be a way to write to S3.

  • koryknick's avatar
    koryknick
    2 years ago

    maahutch - update the input views in the ZipFile Write snap to accept multiple inputs so your pipeline looks like this:

    Note that you will want to name your input views based on the filename you want to see in the archive and make sure you include the extension for the filename as part of the view name.  

    Note the 3 slashes following the s3: protocol designation before your S3 bucket name and make sure you check the "Use input view label" option.

    Hope this helps!

3 Replies

  • koryknick's avatar
    koryknick
    Employee
    2 years ago

    maahutch - update the input views in the ZipFile Write snap to accept multiple inputs so your pipeline looks like this:

    Note that you will want to name your input views based on the filename you want to see in the archive and make sure you include the extension for the filename as part of the view name.  

    Note the 3 slashes following the s3: protocol designation before your S3 bucket name and make sure you check the "Use input view label" option.

    Hope this helps!

  • maahutch's avatar
    maahutch
    New Contributor
    2 years ago

    That's great. I didn't realize you could write to S3 with the ZipFile Write snap. As a bonus, is it possible to make the file name today's date? I tried:

    's3:///my-data-warehouse-backup@s3.us-west-1.amazonaws.com/' +Date.now().toString().replaceAll(":","_")+'.gz'

    but that didn't work, got 'Unsupported protocol or URL syntax error '