Snowflake bulk load snap is executing COPY INTO for each input document
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-11-2023 11:20 AM
I am working on building a pipeline to pull data from an OAUTH 2.0 API and load the data into snowflake using snowflake bulk load. I observed that for each output from REST Get snap, the bulk load is triggering a COPY INTO command in snowflake. So for 20 outputs from Rest Get triggers 20 COPY INTO command in snowflake. This is not efficient. How can i make sure that the bulk load moves data using a single COPY INTO command? Thanks!
1 REPLY 1
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-31-2023 09:17 AM
Can you pls share your pipeline design to have more clarity ?
