Forum Discussion

arvindnsn's avatar
arvindnsn
Contributor
2 years ago

$original values thru Snowflake Bulk Load

Below is the snaplogic process to load data from Redshift to Snowflake. The Map execution mapper has a output variable named "$last_execution_datetime". The data is sourced from Redshift Select Snap and Snowflake Bulk load is used to load the data. The output of the Snowflake Bulk Load snap does not bring the $original values that would include the "$last_execution_datetime" from the mapper as shown in the below pictures.

Is there a way to get the $original values from the output of the Snowflake Bulk Load 

 

As the above picture shows, there is no $original being carried from the previous mapper even though I have "Pass through" checked.

Is there a way to achieve the above mentioned use case.

Thanks

Aravind N

 

5 Replies

  • Imran's avatar
    Imran
    New Contributor
    2 years ago

    arvindnsn - I did face the similar issue, but I got it resolved using copy snap, where you can get all the $original documents and pass it to required snap.

    • arvindnsn's avatar
      arvindnsn
      Contributor
      2 years ago

      Imran , If i used the COPY Snap and if the bulk load fails, I don't want the copy output to move forward. If the Bulk Load failed, I don't want the $Original values in its output. The only time i need the $Original values are if the Bulk load is successful.

  • Aleksandar_A's avatar
    Aleksandar_A
    Contributor III
    2 years ago

    Hello arvindnsn,

    My assumption here is that you want to pass the $last_execution_datetime from the first Mapper.

    To achieve that you can use a Copy Snap and a Join Snap that will wait for the Bulk Load to finish and then you can use the $last_execution_datetime field.

    Please refer to the photo below:

    and the Join Snap configuration:

    Please let me know if this helps you.

    Regards,

    Aleksandar.

    • arvindnsn's avatar
      arvindnsn
      Contributor
      2 years ago

      Aleksandar_A In the solution, you provided above, If the Bulk Load failed, I don't want the JOIN-WAIT snap to bring the $Original values in its output. The only time i need the $Original values are if the Bulk load is successful.

      • Aleksandar_A's avatar
        Aleksandar_A
        Contributor III
        2 years ago

        Yes, if the Bulk Load fails then you will have no output documents from the Join Snap.