Writing Zip Files to S3
I am encountering some severe sluggishness in writing zip files to S3. When writing a 76 MB file, it takes 12 minutes to complete, versus 16 second when writing to a local destination. I think the problem is in transferring from ground to cloud. This process is part of a generic file transport solution, so the read file snap is being executed on a groundplex, and the write file snap is being executed on the cloudplex. This switch is done by a pipeline execute snap specifying execution on the cloudplex. I’m thinking it is possible the issues are cause by the conversion from binary to document and then back to binary once the document stream is passed into the child pipeline. Has anyone else run into similar issues? I am happy to provide an outline of the pipeline if that helps. Thanks.
Remote file renaming/moving
Hi, I have a requirement to push a file to SFTP server. But since the file is quite large and to avoid race condition (we writing and remote function reading at the same time) we like to write the file with different extension or to a temp folder in remote server and them rename or move the file at the server. None for the File snaps support these feature. Could you please help me identify a way around to address above problem.