Mapping fields from objects in an array to fields in objects in an array

So I have an input schema that has a structure similar to the one shown here:

{
    "MatchUpdatePartyPayload:" {
        "Address": [
            {
                "NonStandardUS": {
                    "Line1": <String>,
                    "Line2": <String>,
                    "Line3": <String>,
                    "City": <String>,
                    "State": <String>,
                    "Zip": <String>
                },
                "Foreign": {
                    "Line1": <String>,
                    "Line2": <String>,
                    "Line3": <String>,
                    "Line4": <String>,
                }
            }
        ]
    }
}

I need to map these address fields to an output structure that is similar, but not quite the same. I have tried using the Mapper to do this and it is creating mapping table entries that look like this:

Expression => jsonPath($, "$MatchUpdatePartyPayload.Address[*].NonStandardUS.Line1")
Target Path => jsonPath($, "$MatchUpdateParty.Address[*].NonStandardUS.AddressLine1")

The input documents to the mapper all haver Line1 as a String. Unfortunately, the Mapper output all have AddressLine1 as an array of strings–which is not what I need. What do I need to do to fix the mapping to assign input String fields to output String fields for each element in the array?

You need to split the Array using the JSON Splitter and work on the Transformations and Group them back.

Attached is a sample pipelinesample_2020_09_28.slp (6.2 KB)

1 Like

Thanks for the reply. This is going to be painful since some of our payloads have a dozen or more optional fields that are arrays…

@skatpally I am not sure how this sample would work. In my case, most arrays have 1 to unbounded elements. The Group By N is asking for the number of elements to put back into the array that was split. Since each document will have different numbers of array elements, I don’t see how your approach can work. What am I missing?

Here is the start of a sample pipeline just to handle a single mapping. Mapping Conundrum_2020_09_28.slp (29.7 KB)

A value of 0 instructs the Snap to group all input documents into a single document.

@skatpally, is that all in the “pipeline’s current document” or all (meaning across all documents)? In my sample, I have 5 separate documents in the JsonGenerator. Will 0 cause it to put the values of all 5 documents in it or only the ones from the currently processing document?

It will group all the incoming docs.

For your scenario, you need to use a unique ID and do a split and work on the transformations and use a Group by field snap to merge the data back. Attaching the sample to get an idea. Mapping Conundrum_2020_09_29 (1).slp (16.9 KB)

@skatpally Great, we are getting closer. Now, I just need to figure out how to recombine the mapped split components back to form a message that reflects the original with the mappings applied. Here is what I am trying but it isn’t doing what I need…

Mapping Conundrum_2020_09_29 (2).slp (47.6 KB)

You just need to JOIN them back on the Unique ID

@rpatrick00 if it’s one to one try this in mapper - jsonPath($, “$MatchUpdatePartyPayload.Address[*].NonStandardUS.Line1”).toString() also make sure check null safe option in mapper.

@skatpally, A couple of problems:

  1. Group By Field works but I end up with extra stuff inside each element of the array, which will not work for me. The problem is that in order for Group By Field to work, I need to somehow pass the unique ID field value to use for grouping through the Mapper that is working on the individual array elements. If I explicitly map that field in the Mapper, the output from Group By Field has the unique ID field inside each array element. If I do not map the field and use Pass Through, I end up with all of the pass through fields inside each array element. If I do neither, Group By Field errors out because the payload does not have access to the unique ID field. There probably needs to be a “Remove Group By Field from elements” option in the Group By Field snap.

  2. The Join snap does not work because not all messages have all 5 elements that are being joined by unique ID so the snap errors out if I use any of the join types other than Merge. Merge doesn’t do what I need since the reassembled messages have pieces from various messages combined into a single message. I figured Null Safe Access might be the solution but the Join snap still fails to do the join.

What am I missing?

Mapping Conundrum_2020_09_30.slp (48.1 KB)

You should do a LEFT JOIN instead of Merge. After the JOIN you have to clean up the extra keys. Attaching a sample.

Mapping Conundrum_2020_09_30 (1).slp (51.9 KB)

@skatpally Your join is not working properly even with only two of the 5 streams. Notice on the third document that the MatchUpdateParty element only contains the Address field and the Name field is stashed in the input1_MatchUpdateParty element. So now I have an array mapping problem all over again? :frowning:

Its not an issue with the design, you have to change the name on the Group by to solve it. Also make sure you do the clean up. Mapping Conundrum_2020_09_30 (2).slp (53.3 KB)

Yeah, I finally realized that Join snap cannot handle joining fields into same top-level structure so I eliminated the top level structure for the join, and then used a mapper to add the top-level structure back. I had to write a custom snap (not included) to prune off all the null fields/elements created by the Null Safe Access mappers…

I must say that was a lot more painful than I think it should be. If I were to use this in an actual business process, the logic of the business process flow would get lost due to this complex mapping flow…

Here is the final working example (with all of the nulls still present).

Mapping Conundrum_2020_09_30 (3).slp (56.6 KB)

Thanks for the help,
Robert