Redshift bulk load preview showing no failed record counts but not loading records to tables

[amardeep2021]

Hi All,

I am trying to load data to Redshift table using Redshift bulk load snap. Upstream documents has preview with proper data and bulk load snap showing no error view and showing success count as 50 and failed count as 0 with proper table and schema information in preview during validation. While executing the pipeline, it ends successfully with out error, but no new records loaded to table. Does anyone know what could be the reason for this and how to over come this problem of not loading data even though pipeline execution successfully.

Regards,
Amar

[mbowen]

Hi @amardeep2021

If you have access to the Redshift Cluster console, you may be able to get some insight from inspecting the queries submitted to the database. For example, here is the SQL issued for execution my pipeline loading just one row of data into a Redshift table using the BulkLoad snap

BulkLoad snap will spool data in S3, and then execute Redshift COPY SQL sourcing from S3.

COPY "public"."swat30322" ("id", "name", "price")
FROM 's3://robin-swat-3032/Redshift_load_temp/5eb168b4-9032-438f-9345-1a744daae84d.csv.gz' 
CREDENTIALS '' DELIMITER '' ESCAPE ACCEPTANYDATE ACCEPTINVCHARS MAXERROR 100 GZIP 
TRUNCATECOLUMNS EMPTYASNULL BLANKSASNULL DATEFORMAT AS 'auto' TIMEFORMAT AS 'auto'

Here’s pipeline execution. Observe output of 1 doc for BulkLoad snap.

The preview document count is a user configurable setting which defaults to 50. You can change this by clicking the gear-icon in the toolbar of the Designer. I’m not sure how many records you are expecting to be inserted into Redshift (well, at least 50).

[amardeep2021]

Thanks Mathew.