Forum Discussion

jcampion's avatar
jcampion
New Contributor III
8 years ago

Dynamically set order of fields for csv formatter

I am reusing a pipeline for many applications. The pipeline basically takes data, does a small transformation on one field, and then writes a csv file.

Each application has its own set of fields.
The fields must be in a specific order, which I can determine programmatically.
I want to write them out in this order, but the header option on the csv formatter requires hard-coded field values.

Is there a way to specify the order (or even sort the fields alphabetically) dynamically?
Thanks,
Judy

sample_pipeline

2 Replies

  • tstack's avatar
    tstack
    Former Employee
    8 years ago

    If the Header Fields property in the snap is not set, the headers will be pulled from the incoming document. Also, the Snaps try to preserve the order of the fields in the document. So, it’s possible things will “just work” right now for you.

    If you do need to change the order of the fields in the document, you should be able to do it in the mapper using the expression language. You’ll need to get the list of fields, sort it by the key, and then reconstruct the object.
    You can get the list of fields in the document using the entries() method and then sort() that list. The sorted list of key/value pairs can then be fed into the extend() method to produce a new object. The following expression does that and does a reverse alphabetical sort:

    {}.extend($.entries().sort((left, right) => -left[0].localeCompare(right[0])))

    I’m attaching a pipeline that demonstrates both scenarios, using the natural order of the fields in the doc and doing the above sort:

    CSVColumnOrder_2018_05_10.slp (11.8 KB)

    • jcampion's avatar
      jcampion
      New Contributor III
      8 years ago

      Thank you so much! This example pipeline was extremely helpful.