Mapping optional arrays of objects to arrays of objects

Designing Pipelines
#1

I have an input schema that contains some optional arrays of objects in one format that I need to map to a target schema with an array of objects in another format.

The problem I am having is how to get the Mapper to map the array if it is present and ignore it if it is not. For example, my mapping table expression for one field below works fine for documents where the Address element is present but causes an error if it is missing.

jsonPath($, "$MatchUpdatePartyPayload.Address[*].NonStandardUS.Line1")

Thanks,
Robert

#2

Checking the ‘Null-safe access’ property in the mapper will allow for accessing a null field and will give you a null value instead, which can be dealt with downstream possibly?

#3

While that gets rid of the error, it just creates more problems downstream with my custom snap’s validation code. Is there some other way to accomplish this sort of mapping?

#4

What would you like the value to be when it is not present?

#5

I would like the mapper to simply ignore the entry and move to the next one…

If I have 10 lines in my mapping table and the input field(s) are not present for 5 of those lines, then the mapper should only map the 5 lines that it has input data for…

#6

And if there are none then you’d want an empty array ?

#7

The reason I ask is I am assuming these are all separate documents coming in to the mapper that have a field $MatchUpdatePartyPayload, which may or may not contain an array at the Address key.

For instances in which a document does not contain the specified array, what would you expect the expression to evaluate to in that instance?

It is not possible to simply skip the evaluation/mapping of that expression based on that key not being present. Some value needs to be used for the output of the expression and map to the target field. If you can group your documents, you could filter them based on the presence of this array and extract all values where applicable. Hope this helps in some way.

#8

If you enable the Null-safe evaluation, then you could follow this Mapper with a second Mapper to prune out the null or blank fields with a mapping like this:

Expression:    $.filter((value, key) => value && value.trim())
Target path: $

See the Preview panels in this example:

image
image3116×1532 265 KB

#9

The actual payload is a little more complex than that.

The payload really looks like this (even this is somewhat simplified):

{
    "AddAttorneyStandingPayload": {
        "AttorneyID": <integer>,
        "CurrentStanding": <string>,
        "Date": <string>
    },
    "MatchUpdatePartyPayload": {
        "PartyID": <integer>,
        "Name": {
            "Title": <string>,
            "First": <string>,
            "Middle": <string>,
            "Last": <string>,
            "Suffix": <string>
        },
        "Address": [
            {
                "NonStandardUS": {
                    "Line1": <string>,
                    "Line2": <string>,
                    "Line3": <string>,
                    "City": <string>,
                    "State": <string>,
                    "Zip": <string>
                },
                "Foreign": {
                    "Line1": <string>,
                    "Line2": <string>,
                    "Line3": <string>,
                    "Line4": <string>
                }
            }
        ],
        "Phones": {
            "Phone": [
                {
                    "Number": <string>,
                    "Extension": <string>,
                    "Type": <string>
                }
            ]
        },
        "Emails": {
            "Email": [
                {
                    "EmailAddress": <string>,
                    "IsCurrent": <boolean>
                }
            ]
        }
    }
}

In this payload, the MatchPartyUpdatePayload element itself is optional (if the name, address, phone, and emails hasn’t changed, no need to specify the payload). Within the MatchPartyUpdatePayload element, Name, Address, Phones, and Emails are all optional (they are only needed if the data is being changed). Within some of those arrays, the objects themselves have optional fields (e.g., Line2 and Line3 for the NonStandardUS object).

The problem with the null-safe access is that it is putting the null at the lowest level. For example, let say that I have no Address element, what I am ending up with is a fully populated Address element with all values set to null. This doesn’t work for me because the NonStandardUS address element, if present, has 4 required fields and 2 optional fields. And the Foreign and NonStandardUS fields are mutually exclusive. My validation code verifies that any Address element has at least the 4 fields set to non-null/non-empty values. What would be better for me would be for the entire address list to be either empty or set to null without creating the nested address object and setting all 6 fields to null.

My company has some very large payloads (100s of fields with optional elements at various points in the message hierarchies, some of which are objects or arrays of objects). Having the Mapper set dozens of leaf node fields to null quickly becomes intractable since I would have to write a script to walk the entire tree and prune off the elements where all fields are set to null, starting at the leaf nodes and working my way back up. If I have to do that, I might as well just write a script to do the mapping the way I need it in the first place, right?

Hope this helps,
Robert

#10

Unfortunately, you’re correct. Currently, every mapping defined by a Mapper will result in a value of some kind for the mapped field. There’s no way to conditionally omit a mapping. I noticed this deficiency a few weeks ago and raised it internally but we don’t have a solution yet. I’m going to link to your post in that discussion to underscore the need for this.

For now, I would agree that a Script would probably be the most straightforward means to your end.

#11

I can see several possible solutions, depending on how fancy you want to get. The simplest solution would be similar to Null Safe Access option except that it would skip writing the null to the target field (and skip creating any intermediate object structure). Not sure what the mapper snap’s code looks like to know how hard this is but this seems like a reasonably simple solution.

#12

You’ve probably already noticed this, but I should note for whoever else might be following along that both the label and the description of the “Null-safe access” checkbox are confusing. What it’s meant to address is data that missing from the input, not data that’s present but whose value is null – you don’t need to check this checkbox just to map a present-but-null value to the output.

The description (tool tip) for this setting is a bit more accurate:

Enable this to ignore missing data when accessing the source path, such as $a but a does not exist in the source data

The problem with that description is that “ignore” implies it will simply omit any value from the output. Instead, it maps it to a null value. What we’re looking for here is a way to actually do what this says – if a mapped value is missing from the input, don’t put anything in the output for it.

So, yes, there are a number of ways to approach a solution, depending on how flexible you want to get. A simple way would be to replace the “Null-safe access” checkbox with a 3-way choice:

Missing Data Policy:

  1. Error (equivalent to “Null-safe access” = false now)
  2. Map to null (equivalent to “Null-safe access” = true now)
  3. Omit (not possible now)

What I like about this approach is its simplicity. But it’s lacking some flexibility:

  1. One setting applies to all of the mappings for that Mapper. What if you want different policies for different mappings within the same Mapper?
  2. The only condition where this will omit data from the output is when the value is missing from the input. What if you want to omit for other reasons, such as when the value is present but is a blank string? For that, I think you’d need some sort of support in the expression language. Something like .get("path", omit()) or .getOrOmit(“path”). That adds more flexibility at the cost of relying on EL tricks that some users might find somewhat intimidating.
#13

Agreed. I had thought about the custom function in the expression language. The only issue I have with that one is that it is also, if ever so slightly, non-intuitive that you change behavior of the entire mapping operation for that row with a function in the expression (which typically just manipulates what data gets assigned).

If you really wanted to be able to set different policies for each row in the mapping table, you could add some sort of per-row setting which would clutter up the UI but might be a little more intuitive. For example, you could set the behavior at the mapper level and then override a particular row with the per-row setting…

Anyway, I will leave it to you to decide how you want to surface the functionality. I am just happy that you see the need for this.