Distinct results while saving/validating pipeline vs executing pipeline
Hi Team,
Lately, I observed distinct results getting generated during compile time (i.e. saving a pipeline) vs run time (i.e. executing a pipeline). Has anyone observed this behavior?
For instance, consider a simple example of reading a flat file, doing some transformations, using joins and exporting it as a csv/excel file. When we save the pipeline and the Snap Execution
has been set to Validate & Execute, a flat file will be generated in Manager with default 50 records (this may be ranging from 1-2000 depending on each user’s preview count settings). The file generated during saving a pipeline is different than one generated after executing a pipeline.
What I observed was, the joins didn’t work during compile time whereas they worked during run time.
My use case doesn’t need to export a file but to update new records in Salesforce, what’s happening over here is that, during compile time, a vital field data is going blank and while running the pipeline, the data is going as is. The concern here is "As I'm considering brand new records ONLY "
, compile time execution ingests wrong data and run time wouldn’t be able to ingest the correct data as the record was already ingested (new fields only, run time wouldn’t fetch that data as it’s no longer a new record)
Any help/thoughts on this would be highly appreciated.
Thanks!
Best Regards,
Darsh