S3 File as Email Attachment
Hello, As part of an error handling pipeline, I am attempting to write a detailed error file to an S3 bucket, and then attach that file to the email notification at the end of the error pipeline to alert users of the issue. I was able to do this successfully when writing the file to snaplogic’s native file system, but the email sender snap is failing consistently when attempting to grab the S3 file. The error message is: Cannot find AWS access key ID in URL Resolution: Please enter valid URI path. Reason: URI syntax error. For the attachments field, I have something like this: s3:///buistintegration/test_errors_12082017.json I tried adding the AWS Access Key and Secret in a few different combinations before the bucket name, with no luck. Has anyone attempted this before? Is it possible? Any input is appreciated. Thanks, Jack SkrableWriting Zip Files to S3
I am encountering some severe sluggishness in writing zip files to S3. When writing a 76 MB file, it takes 12 minutes to complete, versus 16 second when writing to a local destination. I think the problem is in transferring from ground to cloud. This process is part of a generic file transport solution, so the read file snap is being executed on a groundplex, and the write file snap is being executed on the cloudplex. This switch is done by a pipeline execute snap specifying execution on the cloudplex. I’m thinking it is possible the issues are cause by the conversion from binary to document and then back to binary once the document stream is passed into the child pipeline. Has anyone else run into similar issues? I am happy to provide an outline of the pipeline if that helps. Thanks.
AWS S3 multi-part file upload?
The S3 File Writer Snap doesn’t appear to support multi-part file uploads. We have large files to upload and want to improve performance. Has anyone been able to implement multi-part uploads to S3 in SnapLogic or have any tuning tricks for improving the performance of the S3 File Writer Snap?
How to enable Server-side Encryption with IAM Role support for Amazon S3
To enable Server-side Encryption support in UAT, the following steps must be followed: Include the following directive in global.properties jcc.jvm_options = -DIAM_CREDENTIAL_FOR_S3=TRUE Note : JCC must be restarted if properties file is updated In Designer or Manager, create a new S3 Account as follows : a) Leave Access-key ID and Secret key properties blank b) Enable the Server-side encryption checkbox c) Enable the IAM role checkbox 3 Bind the account created in Step 2 to all applicable Snaps writing to S3. NOTE: This will only work if the IAM role assigned to the JCC is assigned the correct role at the time it was provisioned. We do not support referencing IAM roles otherwise. If server-side encryption is not required, an account is not necessary.
Updated S3 Snap Pack and Selecting Accounts
I’m playing around with some of the new S3 snaps that were recently released. They are not allowing me to use existing S3 accounts. Do we have to create new S3 accounts for these snaps every time? For instance, when I open the S3 Upload snap and click on the accounts tab, when I click to add an existing account, nothing shows up, but I know that I have S3 accounts that should be available in my project space and project. When I click on the add account button and click continue, the dialogue box opens to add a new account. I’m on the latest snap pack version.
SE Server-side encryption
We have tightened up our bucket policy and specified AES256 as our server side encryption. As a result our pipeline now fails. Reason: You may not have access right to bucket: {bucketname}, detail: Access Denied Resolution: Check for valid bucket name, AWS credential and permission. This is using a snaplogic cross account IAM role. Any suggestions on what we can do to make this work?
