Distinct results while saving/validating pipeline vs executing pipeline

[darshthakkar]

Hi Team,

Lately, I observed distinct results getting generated during compile time (i.e. saving a pipeline) vs run time (i.e. executing a pipeline). Has anyone observed this behavior?

For instance, consider a simple example of reading a flat file, doing some transformations, using joins and exporting it as a csv/excel file. When we save the pipeline and the Snap Execution has been set to Validate & Execute, a flat file will be generated in Manager with default 50 records (this may be ranging from 1-2000 depending on each user’s preview count settings). The file generated during saving a pipeline is different than one generated after executing a pipeline.

What I observed was, the joins didn’t work during compile time whereas they worked during run time.

My use case doesn’t need to export a file but to update new records in Salesforce, what’s happening over here is that, during compile time, a vital field data is going blank and while running the pipeline, the data is going as is. The concern here is "As I'm considering brand new records ONLY ", compile time execution ingests wrong data and run time wouldn’t be able to ingest the correct data as the record was already ingested (new fields only, run time wouldn’t fetch that data as it’s no longer a new record)

Any help/thoughts on this would be highly appreciated.
Thanks!

Best Regards,
Darsh

[darshthakkar]

Sure @ptaylor, I can share some snaps but it will take some time as I have to replicate the pipeline and disable the Salesforce snaps. Reason being; we have a workaround as mentioned earlier on changing the snap execution to “Execute Only” and just executing the pipeline instead of saving/validating pipeline. Also, currently we are testing it and dev has been marked complete so I will have to branch out the pipeline in such a way that it doesn’t affect the ongoing testing and dev that has been marked complete.

You had a valid point on datatype; where one could be a number and other would be a string, I double checked this morning as I wasn’t confident whether it was string vs number for this pipeline and the data type is string so we are covered in there. I’ll share the snaps as soon as I’m able to.

Appreciate your patience and help on this one.

[darshthakkar]

@ptaylor, I have observed this behavior again where saving a pipeline even though the Salesforce snaps’ settings were “Execute Only”, ingested Null values to Salesforce.

[Q-1]: Would Salesforce upsert snap insert/update records while saving a pipeline even though its snap execution has been set to “Execute Only”

[Q-2]: Why would Joins output Null data when we clearly know that the data IS NOT NULL? Again this happened during saving a pipeline, I forgot to disable the salesforce upsert snap while making a minute change in other snap and 2k records have been affected now ☹️