Create reusable lookup that we call multiple times in the same pipeline
We have requirement where we have multiple data quality rules configured in a Oracle table for each pipeline that loads different tables. Each rule also has a severity associated with it. Ex: If when are reading Employee data we want to check if Employee number is null(High Severity issue) and also Department Code is null(Low Severity Issue). Questions
-
How do we make these null-ability checks or other generic checks reusable so that we don’t
need to re-code it every single time -
Can we make a lookup reusable so that in the same pipeline we can lookup on our rue table and fetch the severity for different rules. So if we have 5 rules, want to lookup 5 times
