Forum Discussion

pranjbhatt's avatar
pranjbhatt
New Contributor III
12 months ago

Compare data in Snowflake tables before insertion

Hi,

We have a requirement to build a pipeline to identify if the data from source is updated in Snowflake table or not. Essentially, I have to compare, pre and post results from the tables using SnapLogic pipeline and set-up an email alert that provides details associated with the newly loaded data (whether it is updated or not, timestamp, new record count, etc). 

Any material or source to look into ? 

  • tstack's avatar
    tstack
    Former Employee
    7 years ago

    I think you’re looking for the GroupByN snap. You can set the Group Size to zero to have it consume all of the documents.

    • matt_bostrom's avatar
      matt_bostrom
      New Contributor II
      7 years ago

      can you share how i would use this snap with the data from above?

  • tlikarish's avatar
    tlikarish
    Employee
    7 years ago

    Here’s a full example of what @tstack described.

    First group all the documents into a single document.

    Then select all the values into a single list using a mapper. Here is an expression that uses the jsonPath function to select all the value from the document.

    jsonPath($, '$.group[*].value')

    Here is a copy of the pipeline if you want to experiment with it further.

    groupby_2018_10_16.slp (5.6 KB)