Redshift Bulk Load on Cloudplex
We have a Redshift cluster that we would like to load data into. Using the Redshift - Insert snap is simple, but prohibitively slow and isn’t the recommended way to load data into a Redshift. We would like to use the Redshift - Bulk Load snap, but are running into a few issues when setting up the Redshift Account in SnapLogic. Our understanding is that using IAM Roles for authentication is NOT possible on a Cloudplex. Is this true? If so, this is a huge issue. If we can’t use an IAM Role for authentication, the only other option is an AWS Access Key with its secret and token. The main issue with this approach is that the tokens are temporary and only last a few hours at a maximum. How can we use an AWS Access Key with its secret and token without having to refresh the token every 15 minutes? This doesn’t seem useful. Any help would be great. Thanks!Solved7.4KViews0likes9CommentsSnaplogic Redshift Copy Command
Hi Team, When I read data from a file/csv and do a bulk load, into redshift I know that at the backend it’s trying to copy the data and tries to do the bulk load. Here…I need to know what was the full set of options the Snaplogic code used for the Redshift Copy command at its backend. Where I can get that details… Thanks in advance. MakeshSolved2.8KViews0likes2CommentsRedshift bulk load
Hi team, Looking for a suggestion/enhancement to achieve the below senario. when i read a csv file (it might have \n\r (windows) & \n(unix) ). Who to fix this … Today I read the file and when I do the bulk load using redshift bulk load snap and the data gets loaded with the \n\r . How can I escape these characters? When i look at the properties of the snap , i could see the below. but it’s not working.Solved4.2KViews0likes5CommentsCalling a Redshift Stored procedure in Snaplogic
Hi All, Has anyone been able to call a redshift stored proc using Snaplogic, couple of items which I read on the community mentioned they were not able to do so and there is an enhancement request for the same. If anyone one has an idea regarding how to achieve this, please let me know in the comments. Thanks and Regards Anubhav3.6KViews0likes3CommentsUnable to execute Redshift Script with comments in it using Redshift Execute
Hi All, I am using redshift execute to run a redshift script . While I am able to run the script when every redshift query is enclosed in double quote, but the script is failing when I am adding comments to that. Would like to know how to write comment in redshift script so that it can be acceptable in “Redshift Execute”.2.9KViews0likes2CommentsRedshift Unload Directly to Parquet
Hi, Has anyone figured out a way to Unload data directly to parquet format from Redshift? The Redshift Unload snap does not have parquet format as an option (just delimited and fixed width files). I have tried to circumvent this by using a Redshift Execute snap and then writing an unload command. The process seems to work on a limited data set but I get an error at the end of the Unload like this (data in brackets redacted from actual values): ERROR: Mismatch in returned vs expected bucket in CompleteMultipartUpload. Expected: returned: [my-bucket-name-only]. Mismatch in returned vs expected key in CompleteMultipartUpload. Expected: [my-bucket-name]/[mypath]/[my-prefix] The path I entered into the redshift unload command was: [my-bucket-name]/[mypath]/[my-prefix]1.9KViews0likes0CommentsRedshift and SnapLogic
We are trying to stand up a new Redshift environment and want to be able to connect and run pipelines with SnapLogic. Corporate security rules require us to launch the Redshift cluster in a private subnet, no NAT or Internet Gateway, all access through firewall rules. Do we absolutely have to use a Groundplex, can we do it in a cloudplex? If we have to use a Groundplex, can we use an existing Groundplex already running in our SnapLogic account? Thanks!3.3KViews0likes3CommentsUnable to execute stored procedure (Redshift) using Redshift Execute
SQL: CALL get_result_set(1, ‘mycursor’); FETCH ALL FROM mycursor; Failure: SQL operation failed, Reason: A result was returned when none was expected., error code: 0, SQL state: 0100E, Resolution: Please check for valid Snap properties and input data.3.2KViews0likes2Comments