I’m having an issue with a pipeline, and I’m hoping someone can help me with a solution
The pipeline takes data from 2 different databases, and joins them, looks up from a third system, creates a new field, and thenupdates a field in CRM with that newly created value.
Source data is our CRM system, and another core system, and a history table we create weekly
This results in 50k+ records that need to be updated in CRM
The whole pipeline takes 6+ hours to run
The issue is:
We select from CRM, because this is the record we will be updating to at the end of the pipeline
However the error that comes up when I try updating is that the GUID field is null, which isn’t possible
It’s a field that cannot be null, and I have investigated the pipeline and confirmed there is nothing that could make it null
my best guess, is that the data I selected from CRM is timeing out someone, and becoming all nulls, while I am in the process of updating
I’ve confirmed that some records were updated, whenever this has happened
The pipeline usually works, this issue only occurs sometimes, and usually when I run it the next day it works fine
I’ve had the pipeline time out anywhere from 4 to 9 hours
When it runs without an issue, it usually takes 5-6 hours
Has anyone else experienced something like this?
Any suggestions or ideas on what is happening or how to fix it
Thanks
