You can use the filter function of the Directory Browser snap to only look for specific file names. In your case of matching test1.csv and test2.csv, but not matching test3.csv or any other value, you could use the filter expression :
test[1-2].csv
This will search the directory for all files with the value of test1.csv and test2.csv, but no other values. If there are multiple files with the test1.csv/test2.csv name, then all those files will be returned.
That looks promising …looks like ive used names with same name (test) …but how to build expression if one file is “mama mia.csv” and another one is “snaplogic.csv”?
is there a chance to use regex? to solve it with something like this?
‘^mamma mia.csv$|^snaplogic.csv$’
Tagging the experts to get some help on this one.
I defer to the developers on this one. I avoid Validate and Execute.
Sorry, I’m finding this a very confusing post.
Is that related to the paragraph before it or the one after it? I’m not seeing anything else about joins.
What do you mean by “going blank”? What’s supplying that data?
We need more context to make sense of this. What’s a brand new record? From where? It sounds like you might want to change the snap that updates Salesforce to Execute only so that validation doesn’t perform updates that you only want to happen during execution.
Please use “validate” and “execute”, not “compile” and “run”. There’s nothing in SnapLogic that’s really equivalent to “compiling” so that’s a confusing term.
Sincere apologies @ptaylor for the confusion. How I understand the 3 buttons on snapLogic is as below:
I will not use Compile and Run time to avoid confusion henceforth. Let me explain again what the issue is and what I’m trying to achieve.
Upstream system: Snowflake
Downstream system: Salesforce
Issue: Data is different in output preview of a snap when the pipeline is saved or validated. I did a sanity test and the ID record (i.e. col A) that should have some data (i.e. in col H) is coming as “Null” values. This data that I’m expecting is coming from Joins though.
The above behavior is not the same when the pipeline is executed. With execution in place, the data comes as expected.
Definition of brand new records: With the help of Inner Join, I’m considering IDs from snowflake that are not in Salesforce and I’m ingesting those NEW records to Salesforce with this pipeline.
Salesforce Create and Salesforce Update doesn’t work efficiently (from what I’ve observed) so I’ve been using Salesforce Upsert for this operation.
The concerning piece is that, the last snap of my pipeline is a Salesforce Upsert with snap execution as “Validate & Execute”; whenever I make some minute changes and save my pipeline, those changes flow to downstream system (i.e. Salesforce) and that’s an expected behavior however with Saving/Validating pipeline, the data is NOT consistent for the joins (as explained before) and this inserts the records (if new record) or updates the record (if existing record) as we are using an Upsert.
When the pipeline is executed, the same record won’t be inserted as it’s no longer a NEW record and it won’t be updated too as that record hasn’t received any update from the upstream systems. This is basically the data snapLogic should have calculated during saving/validating pipeline but it wasn’t able to!
What I’m trying to achieve: Consistent data flow from Snowflake to Salesforce (with the help of Joins as those would be needed anyhow)
Happy to clarify further questions if any!
Solution: For now, I’ve already changed my Salesforce Upsert snap’s settings to “Execute ONLY” so that even when I save/validate my pipeline, those updates don’t flow downstream. I would have expected joins to work in all the different scenarios like save/validate/execute. Is this a limitation of the tool or am I doing something wrong over here? I wouldn’t be surprised if I’m missing a crucial step, I’m still learning snapLogic so by all means, feel free to give me right directions (I wouldn’t be offended).
Apologies again for the confusion and looking forward to your thoughts on this one.
Best Regards,
Darsh
