Forum Discussion

darshthakkar's avatar
darshthakkar
Valued Contributor
4 years ago
Solved

Fetching 1k records from table using "Snowflake select snap"

I’m planning to fetch the data from a table using “Snowflake select snap” while putting the condition of getting 1k records ONLY in WHERE clause however it is not getting any results. I can put the...
  • darshthakkar's avatar
    darshthakkar
    4 years ago

    Wanted to share the solution here. The query was right, no need to change the “Fetch-size” as well.
    The only thing that was missing was ’ ’ for the IDs that we were passing in the query. Thus, the query would look like -
    WHERE NAME IN (‘102345’, ‘1701’, ‘5604878’)

darshthakkar's avatar
darshthakkar
Valued Contributor
4 years ago
Copy LinkReport

@robin @bojanvelevski: Would you recommend using Join Snap instead of snowflake lookup & execute for a faster execution of the pipeline?

Please feel free to provide any other suggestions, trust me, with those suggestions of yours, I would be exposed to something different (I’m still new to snapLogic and learning my way out)

akidave's avatar
akidave
Employee
4 years ago
Copy LinkReport

The database lookup pattern would work well with an OLTP database, which has low query startup costs and is optimized for single record operations. With a data warehouse like Snowflake which is optimized for analytical queries, the query startup cost and id based lookup will not be as performant as an OLTP database.

You could fetch the required column of the data set from Snowflake using a select query and then use the In-Memory Lookup snap. If the lookup table fits in memory, that would be more performant. If the data set is too large, using a OLTP database would be better for such an use case.