cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Distinct results while saving/validating pipeline vs executing pipeline

darshthakkar
Valued Contributor

Hi Team,

Lately, I observed distinct results getting generated during compile time (i.e. saving a pipeline) vs run time (i.e. executing a pipeline). Has anyone observed this behavior?

For instance, consider a simple example of reading a flat file, doing some transformations, using joins and exporting it as a csv/excel file. When we save the pipeline and the Snap Execution has been set to Validate & Execute, a flat file will be generated in Manager with default 50 records (this may be ranging from 1-2000 depending on each userโ€™s preview count settings). The file generated during saving a pipeline is different than one generated after executing a pipeline.

What I observed was, the joins didnโ€™t work during compile time whereas they worked during run time.

My use case doesnโ€™t need to export a file but to update new records in Salesforce, whatโ€™s happening over here is that, during compile time, a vital field data is going blank and while running the pipeline, the data is going as is. The concern here is "As I'm considering brand new records ONLY ", compile time execution ingests wrong data and run time wouldnโ€™t be able to ingest the correct data as the record was already ingested (new fields only, run time wouldnโ€™t fetch that data as itโ€™s no longer a new record)

Any help/thoughts on this would be highly appreciated.
Thanks!

Best Regards,
Darsh

11 REPLIES 11

darshthakkar
Valued Contributor

Tagging the experts to get some help on this one.

CC: @robin @koryknick @dmiller @bojanvelevski @ptaylor

I defer to the developers on this one. I avoid Validate and Execute.


Diane Miller
Community Manager

Sorry, Iโ€™m finding this a very confusing post.

Is that related to the paragraph before it or the one after it? Iโ€™m not seeing anything else about joins.

What do you mean by โ€œgoing blankโ€? Whatโ€™s supplying that data?

We need more context to make sense of this. Whatโ€™s a brand new record? From where? It sounds like you might want to change the snap that updates Salesforce to Execute only so that validation doesnโ€™t perform updates that you only want to happen during execution.

Please use โ€œvalidateโ€ and โ€œexecuteโ€, not โ€œcompileโ€ and โ€œrunโ€. Thereโ€™s nothing in SnapLogic thatโ€™s really equivalent to โ€œcompilingโ€ so thatโ€™s a confusing term.

Sincere apologies @ptaylor for the confusion. How I understand the 3 buttons on snapLogic is as below:

  1. Validating a pipeline: Compile time
  2. Executing a pipeline: Run time
  3. Saving a pipeline with the last snap having snap execution as โ€œValidate & Executeโ€: Compile time

I will not use Compile and Run time to avoid confusion henceforth. Let me explain again what the issue is and what Iโ€™m trying to achieve.

Upstream system: Snowflake
Downstream system: Salesforce

Issue: Data is different in output preview of a snap when the pipeline is saved or validated. I did a sanity test and the ID record (i.e. col A) that should have some data (i.e. in col H) is coming as โ€œNullโ€ values. This data that Iโ€™m expecting is coming from Joins though.

The above behavior is not the same when the pipeline is executed. With execution in place, the data comes as expected.

Definition of brand new records: With the help of Inner Join, Iโ€™m considering IDs from snowflake that are not in Salesforce and Iโ€™m ingesting those NEW records to Salesforce with this pipeline.

Salesforce Create and Salesforce Update doesnโ€™t work efficiently (from what Iโ€™ve observed) so Iโ€™ve been using Salesforce Upsert for this operation.

The concerning piece is that, the last snap of my pipeline is a Salesforce Upsert with snap execution as โ€œValidate & Executeโ€; whenever I make some minute changes and save my pipeline, those changes flow to downstream system (i.e. Salesforce) and thatโ€™s an expected behavior however with Saving/Validating pipeline, the data is NOT consistent for the joins (as explained before) and this inserts the records (if new record) or updates the record (if existing record) as we are using an Upsert.

When the pipeline is executed, the same record wonโ€™t be inserted as itโ€™s no longer a NEW record and it wonโ€™t be updated too as that record hasnโ€™t received any update from the upstream systems. This is basically the data snapLogic should have calculated during saving/validating pipeline but it wasnโ€™t able to!

What Iโ€™m trying to achieve: Consistent data flow from Snowflake to Salesforce (with the help of Joins as those would be needed anyhow)

Happy to clarify further questions if any!

Solution: For now, Iโ€™ve already changed my Salesforce Upsert snapโ€™s settings to โ€œExecute ONLYโ€ so that even when I save/validate my pipeline, those updates donโ€™t flow downstream. I would have expected joins to work in all the different scenarios like save/validate/execute. Is this a limitation of the tool or am I doing something wrong over here? I wouldnโ€™t be surprised if Iโ€™m missing a crucial step, Iโ€™m still learning snapLogic so by all means, feel free to give me right directions (I wouldnโ€™t be offended).

Apologies again for the confusion and looking forward to your thoughts on this one.

Best Regards,
Darsh