dwhansen-cbg
6 years agoContributor
Slow PostgreSQL Update Snap
I have a PostgreSQL Update snap at the end of a pipeline. It’s job is to update a column in a row if nothing in an inbound row (matching that data) has been modified. This means that most of the time I will need to update all rows in my table everyday since the data is usually mostly unchanged from one day to the next.
I’ve noticed that running a PostgreSQL Update step using a WHERE clause “primary_key = $primary_key” makes it run very slow. I have also left the WHERE clause empty to see if that speeds things up but I always end up getting a primary key error.
Is there a way to speed up Update snaps?