@robin @bojanvelevski: Would you recommend using Join Snap instead of snowflake lookup & execute for a faster execution of the pipeline?
Please feel free to provide any other suggestions, trust me, with those suggestions of yours, I would be exposed to something different (I’m still new to snapLogic and learning my way out)
The database lookup pattern would work well with an OLTP database, which has low query startup costs and is optimized for single record operations. With a data warehouse like Snowflake which is optimized for analytical queries, the query startup cost and id based lookup will not be as performant as an OLTP database.
You could fetch the required column of the data set from Snowflake using a select query and then use the In-Memory Lookup snap. If the lookup table fits in memory, that would be more performant. If the data set is too large, using a OLTP database would be better for such an use case.
You can use a Mapper with Passthrough enabled, set the Expression column to the path to be deleted, and leave the Target Path empty:
The “Leave blank to delete source path” text in the target field is supposed to guide the reader in this direction. I guess it’s not working too well…
@tstack, this worked now, but I know i previously got errors saying I had to have a target path.
Just don’t put the field at all in the list. E.g. where $unwantedField is, make that blank and leave Target Path blank.
