userg
2 years agoNew Contributor II
Mapping dynamically for multiple objects using one pipeline(without a mapper for each object)
Hello,
My input is S3, where we are receiving files of different objects. Our target is to read these files and insert data into postgres DB where we have created target tables for all objects. How can we dynamically map all the objects using one control table that will hold input column name and target column name. So if we add a new column in the control table and in target_table in future the pipeline should be able to map and insert the data into postgres DB.
Control_table :
object_name(target_table_name) | Input_column_name(column name from file) | Target_column_name(column name in postgres target table) |
Thanks in advance.