I’m trying to make this pattern work for MSSQL to Snowflake. Do you have a working implementation to Snowflake. I cannot seem to get the bulk load to work from the MSSQL->Select snap to the Snowflake Bulk Loader. Is there functionality in the Oracle bulk loader that this needed for this pattern?
Lastly, I can make this work if I have a specific mapper and bulk loader for each table but that seems like a lot of redundant work (100+ tables worth).
Let me know.
The oracle extract uses the limit offset to help with shrading the data. If you plan to ignore shrading the data then you can easily convert the existing pattern to support MSSQL and target as Snowflake… Set shrading as no in the pipeline parameters on the master pipeline.
Here is an example of a pattern that might also help that gets files on s3 and insert into snowflake .
Shading is not the issue.
The issue is that there are data type mismatches between MSSQL and Snowflake. Which implies we need a mapper. But mappers are not generic so the pattern breaks down.
Try the pattern with MS and Snowflake. Also use Bit fields. I suspect date field I miss you as well.
Thanks,
Sonny
“This communication (including all attachments) is intended solely for the use of the person(s) to whom it is addressed and should be treated as a confidential AAA communication. If you are not the intended recipient, any use, distribution, printing, or copying of this email is strictly prohibited. If you received this email in error, please immediately delete it from your system and notify the originator. Your cooperation is appreciated.”
here is a DB datatype cross reference I build which could be handy. Have you already done some experiment or are you suspecting it will be an issue?
Yes there is a data type issue. I’ve seen it with BIT vs Boolean and I made that work. But I also suspect that some others have a similar issue (datetime , timestamp ntz, etc).
I’m not clear on why the reference architecture works.
Thanks for the input. I’m not really concerned about sharding (at the moment) but it’s the bulk loader and data types that concern me. The reference architecture connects the bulk loader directly to the MSSQL Select but the data types won’t convert properly (BIT to BOOLEAN, etc)
Can you explain why it work in Oracle to Redshift. Maybe that will help me fix /understand the MSSQL to Snowflake approach.
