Forum Discussion

sasikanth's avatar
sasikanth
New Contributor
5 years ago

Does Snowflake snappack 'Bulk Load' snap support ingesting data on to snow flake managed internal stage?

Snowflake has three kinds of stages before loading data on to snowflake dbs.
1/ Supported cloud storage, 2 / Internal snowflake managed named stage 3/ default table/user stage

Bulk Load Snap’s documentation says, it does support PUT command. However I found it hard to make the snap work. It is not clear, if the snap does both operations ( PUT first, and followed by COPY INTO) or the pipeline needs two Buld Load snaps in sequence ( PUT snap followed by a COPY INTO snap)

The bulk load snap worked (COPY INTO ) if data is already placed in the internal stage.

Pls let us know if there is a writer snap that does snowflake PUT operation on to snowflake internal stage.

Snowflake

3 Replies

  • mbowen's avatar
    mbowen
    Employee
    5 years ago

    Hi sasikanth:

    I’m not a Snowflake expert, but know some who are. Let me try to find an answer to your question. Some others in community may respond too.

    • mbowen's avatar
      mbowen
      Employee
      5 years ago

      One Snowflake snap (Bulk Load/Upsert) should be sufficient to load data into Snowflake.

      If snap is configured to use Input View as the data source with Internal staging, data will be written to a temporary file on the local Snaplex. Snap will then execute a PUT command to copy the temp file to Snowflake’s staging area; it will then write data from staging area to Snowflake target table via a COPY command.

      If the incoming data size is expected to be larger than the available size in the local Snaplex, you should use your own external staging (S3, Azure Blob, or Google Cloud Storage).

      You mentioned that it was difficult to make Bulk snap work for you. Were you getting an error?

  • sasikanth's avatar
    sasikanth
    New Contributor
    5 years ago

    Hello, mbowen. Thank you for the reply. It is in line with my expectation. Let me prepare a sample data and update here.
    Earlier, Bulk load snap did not stage files. But it worked for loading data if it is already present in stage. the bulk load snap copied ( COPY INTO) data from internal named staging to snowflake table after I manually staged files ( manual PUT)