cancel
Showing results for 
Search instead for 
Did you mean: 

Backing up two database tables to an S3 bucket as one archive

maahutch
New Contributor

I'm trying to develop a pipeline that queries two (or more) database tables, converts them to CSV, compresses them to a single archive and writes them to and S3 bucket. I have a working pipeline for a single table:

maahutch_0-1707514902868.png

But what I'd like to do is have multiple select queries that pull in different tables and compresses them to a single archive. Is this possible? I looked at "ZipFile Write" but there didn't seem to be a way to write to S3.

1 ACCEPTED SOLUTION

koryknick
Employee
Employee

@maahutch - update the input views in the ZipFile Write snap to accept multiple inputs so your pipeline looks like this:

koryknick_0-1707744131103.png

Note that you will want to name your input views based on the filename you want to see in the archive and make sure you include the extension for the filename as part of the view name.  

koryknick_1-1707744155629.png

Note the 3 slashes following the s3: protocol designation before your S3 bucket name and make sure you check the "Use input view label" option.

koryknick_3-1707744351840.png

Hope this helps!

View solution in original post

3 REPLIES 3

Aleksandar_A
Contributor III

Hello @maahutch,

According to the documentation, writing zip files using ZipFile Write Snap to S3 is supported.

Please refer to the official documentation. [ZipFile Write Snap]

Regards,

Aleksandar.

koryknick
Employee
Employee

@maahutch - update the input views in the ZipFile Write snap to accept multiple inputs so your pipeline looks like this:

koryknick_0-1707744131103.png

Note that you will want to name your input views based on the filename you want to see in the archive and make sure you include the extension for the filename as part of the view name.  

koryknick_1-1707744155629.png

Note the 3 slashes following the s3: protocol designation before your S3 bucket name and make sure you check the "Use input view label" option.

koryknick_3-1707744351840.png

Hope this helps!

maahutch
New Contributor

That's great. I didn't realize you could write to S3 with the ZipFile Write snap. As a bonus, is it possible to make the file name today's date? I tried:

's3:///my-data-warehouse-backup@s3.us-west-1.amazonaws.com/' +Date.now().toString().replaceAll(":","_")+'.gz'

but that didn't work, got 'Unsupported protocol or URL syntax error '