Dynamic Data Validation
I want to validate the incoming data which will be in CSV format. I have Target table defined for each file. Now I want to develop a common pipeline which should validate the data based on the rule f...
Forum Discussion
Thanks for your quick response. Here is the example - “{SOURCE_COLUMN_NAME: COLUMN_NAME1:[{DATATYPE: NUMBER, MAX_LENGTH: null,PRECISION :25,NUMERIC_SCALE:7,NULLABLE:NO}] }”
“{SOURCE_COLUMN_NAME: COLUMN_NAME2:[{DATATYPE: VARCHAR, MAX_LENGTH: 50 ,PRECISION :null ,NUMERIC_SCALE:null ,NULLABLE:NO}] }”
We could change this to different format if required. I have challenge in joining this with actual data.
The simplest thing would be to collect all the schema documents together with a GroupByN snap and then use a Join snap to add the schema to all the documents.
Can I ask why you want to do the validation yourself instead of letting the database do the validation? If the tables in the DB are already setup with the appropriate constraints, I would think that would be the most straightforward approach.
