Updated S3 Snap Pack and Selecting Accounts
I’m playing around with some of the new S3 snaps that were recently released. They are not allowing me to use existing S3 accounts. Do we have to create new S3 accounts for these snaps every time? For instance, when I open the S3 Upload snap and click on the accounts tab, when I click to add an existing account, nothing shows up, but I know that I have S3 accounts that should be available in my project space and project. When I click on the add account button and click continue, the dialogue box opens to add a new account. I’m on the latest snap pack version.
AWS Gov Cloud and the S3 File Reader
My company uses sub accounts under a master account to control access and track resources in AWS Gov Cloud. I am trying to get an account configured for the S3 File Reader that works in this configuration. Seems simple enough, I specify my access key and secret key and set the Cross Account IAM Role’s Role ARN field but the validation is failing: Failed to validate account: The security token included in the request is invalid. (Service: AWSSecurityTokenService; Status Code: 403; Error Code: InvalidClientTokenId; Request ID: 4961a29f-a875-4c69-9665-e640610bc840) (Reason: No reason provided by the snap; Resolution: No resolution provided by the snap) On my local machine, my ~/.aws/config file looks something like this: [default] region = us-gov-west-1 [profile engineer] role_arn = arn:aws-us-gov:iam::############:role/Sandbox_Administrator source_profile = default region = us-gov-west-1 and my ~/.aws/credentials file looks something like this: [default] aws_access_key_id = <redacted> aws_secret_access_key = <redacted> When I run AWS CLI commands, I have to add “–profile engineer” to the command line but everything works properly. Any clues as to what I need to do to make this work with the S3 File Reader?
Writing Zip Files to S3
I am encountering some severe sluggishness in writing zip files to S3. When writing a 76 MB file, it takes 12 minutes to complete, versus 16 second when writing to a local destination. I think the problem is in transferring from ground to cloud. This process is part of a generic file transport solution, so the read file snap is being executed on a groundplex, and the write file snap is being executed on the cloudplex. This switch is done by a pipeline execute snap specifying execution on the cloudplex. I’m thinking it is possible the issues are cause by the conversion from binary to document and then back to binary once the document stream is passed into the child pipeline. Has anyone else run into similar issues? I am happy to provide an outline of the pipeline if that helps. Thanks.
