Forum Discussion

Tuomas's avatar
Tuomas
New Contributor II
3 years ago
Solved

Mapper dynamic content

Hello! I have REST API source from which I ingest data and load it to BigQuery. I am executing this dynamically with parameters as there are multiple endpoints to query. I am using mapper to define...
dynamic
mapper
  • Aleksandar_A's avatar
    Aleksandar_A
    3 years ago

    Hi @Tuomas,

    One way that you can try is to define the columns for each schemas and to append that object to the response documents.

    Schemas columns object example:

    [
    {
        "schema1":['a','b'],
        "schema2":['c','d'],
        "schema3":['e','f']
    }
]

    Also, you can make this as part of a config file.

    Then you can append this object to every document using the Join Snap with the following configuration:

    So, finally you can dynamically filter out the columns from the root object using the following expression in a Mapper Snap:

    $.filter((v,k) => $.get(_schema).indexOf(k) != -1)

    Sample pipeline:
    –sl-comm-dynamic-columns-parameter_2023_05_12.slp (6.8 KB)

    Let me know if this helps you @Tuomas.

    BR,
    Aleksandar.

Aleksandar_A's avatar
Aleksandar_A
Contributor III
3 years ago

Hi @Tuomas,

One way that you can try is to define the columns for each schemas and to append that object to the response documents.

Schemas columns object example:

[
    {
        "schema1":['a','b'],
        "schema2":['c','d'],
        "schema3":['e','f']
    }
]

Also, you can make this as part of a config file.

Then you can append this object to every document using the Join Snap with the following configuration:

So, finally you can dynamically filter out the columns from the root object using the following expression in a Mapper Snap:

$.filter((v,k) => $.get(_schema).indexOf(k) != -1)

Sample pipeline:
–sl-comm-dynamic-columns-parameter_2023_05_12.slp (6.8 KB)

Let me know if this helps you @Tuomas.

BR,
Aleksandar.

Tuomas's avatar
Tuomas
New Contributor II
3 years ago

Seems to work like a charm 🙂 Thank you!