Transpose column into row csv file

[fernandezso]

I have a pipeline that reads two csv files from different servers, i want to put together the data of both csv files into one. I used a union snap to put the data together but if one server is down it’s only going to bring me the data of only one server. I put a flag in each section of the file reader, if that flag its true after the join it will check that both flags are true, if yes then write one csv file if not don’t write. But after I configure my join snap i notice that the final csv has more columns, new column names with the prefix “input1_ColumnName”. How can I make that data move from the right to the bottom of the csv file?

[koryknick]

Try adding an Exit snap after the Emailer with threshold at 0 to fail the pipeline if either source fails. You may also have to set the batch size property in the Emailer to 1 to prevent record batching.

This should allow you to continue to use the Union snap and avoid the “added columns” issue.

[fernandezso]

I just put the exit snap but since one server stills up it is sending the information to the file writers, and creating the csv file with half of the information:

[koryknick · Accepted solution]

Here is a sample pipeline using Gate and Router to verify the existence of data from both input streams. Note that I’m not doing the email off the read fail, but it will still work the same way.

Example_Missing_Input_2020_07_09.slp (17.4 KB)

Input file used: directory.txt (2.1 KB)

[fernandezso]

It worked!! Thanks!

This is the final pipeline 😀