Backing up two database tables to an S3 bucket as one archive

[maahutch]

I'm trying to develop a pipeline that queries two (or more) database tables, converts them to CSV, compresses them to a single archive and writes them to and S3 bucket. I have a working pipeline for a single table:

But what I'd like to do is have multiple select queries that pull in different tables and compresses them to a single archive. Is this possible? I looked at "ZipFile Write" but there didn't seem to be a way to write to S3.

[AleksandarAngel]

Hello @maahutch,

According to the documentation, writing zip files using ZipFile Write Snap to S3 is supported.

Please refer to the official documentation. [ZipFile Write Snap]

Regards,

Aleksandar.

[koryknick · Accepted solution]

@maahutch - update the input views in the ZipFile Write snap to accept multiple inputs so your pipeline looks like this:

Note that you will want to name your input views based on the filename you want to see in the archive and make sure you include the extension for the filename as part of the view name.  

Note the 3 slashes following the s3: protocol designation before your S3 bucket name and make sure you check the "Use input view label" option.

Hope this helps!

[maahutch]

That's great. I didn't realize you could write to S3 with the ZipFile Write snap. As a bonus, is it possible to make the file name today's date? I tried:

's3:///my-data-warehouse-backup@s3.us-west-1.amazonaws.com/' +Date.now().toString().replaceAll(":","_")+'.gz'

but that didn't work, got 'Unsupported protocol or URL syntax error '