Forum Discussion

mmindenhall's avatar
mmindenhall
New Contributor
4 years ago

Mapper: map to target path only if input path is not null

I’m building a pipeline to update some existing data, and I have a Mapper mapping from source input fields to destination target fields. The input data is incomplete – i.e., there are a lot of fields that can be null on any given input document. For some fields in the input document, if the value is null, I want to ignore it, but if the value is non-null I want to write it to the corresponding target field. The goal here is to avoid overwriting non-null values in the target data with null.

Examples:

Input $first_name == null → ignore, do not write to $First_Name field in target
Input $first_name == 'Bob' → write to $First_Name field in target

It would be really cool if there was a way to configure mappings to ignore null values directly in the Mapper, but I’m not seeing how that can be done. Any suggestions for how to do this (either with something I’m missing in the Mapper or with a combination of other snaps)?

4 Replies

  • tstack's avatar
    tstack
    Former Employee

    You’re looking for the toObject() method on arrays. The method takes two callbacks, one to turn the array element into the key and one to turn the element into the value. I think the following will do what you want:

    $BrandData.toObject(elem => elem.Source + "-" + elem.SourceBrand, elem => elem.Brand)
    

    I’m also attaching an example pipeline with this expression.
    ArrayToObject_2019_07_26.slp (3.9 KB)

    • SriVinayaga's avatar
      SriVinayaga
      New Contributor II

      Didnt expect it to be so simple. Thank you very much @tstack. Appreciate your help.

  • SriVinayaga's avatar
    SriVinayaga
    New Contributor II

    Hi @tstack,

    There is a catch in my requirement.

    You can see there is a flower bracket at the open and end of the output instead of square bracket. Currently your solution helped to form the data, but still this issue is lingering, Any thoughts on how to transform them? I tried JSON replace, and other options etc. Did not progress further.

    • tstack's avatar
      tstack
      Former Employee

      I’m not quite clear on what you’re asking for here. What does the rest of your pipeline look like? Are you trying to write the document to a file?

      Property names in JSON are quoted, so they’re necessary. Are you trying to write to a file?

  • SriVinayaga's avatar
    SriVinayaga
    New Contributor II

    Also BrandData in the output shouldnt be within quotes.

  • SriVinayaga's avatar
    SriVinayaga
    New Contributor II

    Yes Tim, we will write it to a file.

    The idea is to use the file as an expr library for transformation. This file is expected to be updated once a day and other pipelines will use this expr library file like a lookup. If the left value in the exprlib file matches with the input field value, it is expected to transform to the right value.

    • tstack's avatar
      tstack
      Former Employee

      Ah, okay… In that case, it’s okay if the property name is in quotes, either way is acceptable.

      As for writing out just the document and not an array of docs, try selecting the “Format each document” option in the JSONFormatter.

      • abinayad's avatar
        abinayad
        New Contributor

        Hi @tstack

        I have a similar requirement here. But my input to JSON formatter snap is the output from union snap and it has many segments and not one. In that case with multiple segments how can I remove square bracket and an extra curly bracket.

        Thanks,
        Abinaya