Assistance requested creating pipeline for SQL query-->mapper-->text file

[wahlborn]

My goal is to execute a SQL query, return data, and format it so that it is accepted by a custom app we run here. So far, I have extracted the necessary data and formatted it using a mapper. The mapper output contains three columns that I need to move into a text file. For example:

ColumnA ColumnB ColumnC ColumnD ColumnE
DataA1 DataB1 DataC1 DataD1 DataE1
DataA2 DataB2 DataC2 DataD2 DataE2
DataA3 DataB3 DataC3 DataD3 DataE3

I need to output the data from the last three columns in the following format and create a text file:

DataC1
DataC2
DataC3
DataD1
DataD2
DataD3
DataE1
DataE2
DataE3

I’ve tried a handful of different snaps, and I haven’t succeeded in getting it to work. Anyone have tips on how to implement this?

[cjhoward18]

Hi @wahlborn

I’ve attached a pipeline that has generated documents with 5 columns like your example above, while extracting the last 3 columns of values, and writing them to a file with each value on it’s own line.

Please note that the order of values is not guaranteed, and that this is a quick solution to show you a POSSIBLE way of achieving this formatted output file.

The pipeline could most likely be better optimized.

last3ColumnsToFile.slp (25.6 KB)

[wahlborn]

Thanks so much! This is very helpful.

[cjhoward18 · Accepted solution]

@wahlborn

Here is a new slp which includes two new solutions to your problem, more optimized than the one given above earlier.

The solution using the pivot snap sorts by the column names, which might not result in the same order as coming in. It works now because the columns are C, D, E. But, if the last three columns in the real thing are not alphabetical, it might not work like you want.

The solution using the mapper snaps does not have the problem listed above. So, depending on your desired behavior you can choose and play with the solutions to fit your use case.

Also note, these solutions are using the actual column names themselves that are to be collected and written out, so if those plan to change this will need to be edited accordingly to handle that.

PivotQuestion_2019_08_29.slp (14.6 KB)

[cjhoward18]

@wahlborn

Lastly, here is a final solution you may find useful that collects the ‘last 3 columns’ generically without using the key names specifically. This solution also retains order.

Last3ColumnsGenericRetainOrder.slp (6.8 KB)