How to read multiple files with different structure and use a generic pipeline to load them into tables based on parameter?
I can use the parameter to load them into tables based on respective file names. But I am facing challenge in reading different structure in a same file reader.
Can anyone suggest some option?
The best approach here would be to use a child pipeline to execute on each file to be read. That way the pipeline is only ever dealing with a single structure.
Where is this parameter coming from? Can you describe the scenario a little more?
Thank you Craig, for your response!
The parameter is pipeline parameter to pass the table name and I will put file names accordingly so that they can be identified with single param.
But the metadata for different files would be different, can I use schema files in that case? If yes, how is it mapped in a file reader or any other snap? Or is it not feasible in snaplogic?
In SnapLogic the File Reader just reads the binary stream. After the stream has started, this is passed to the next Snap, often a parser. Another option might be to do a file reader, followed by a Binary Router, in which you use an expression (based on pipeline parameter, file name or something) to route the input to a parser (and the downstream) to load into the various different tables. How many different formats are you potentially routing to?
Hi Swatisree,
We have implemented the solution for one of our client. and we process million of record each day. Here is the solution we design-
Hope it helps.
Thank you Supratim for your reply!
I have implemented something similar. But there is a lot of transformation involved so the mapper part and cannot be made generic, that is specific per table. The source and target parts are generic and running fine.
