Tuomas - thanks for the clarification. As SnapLogic is a streaming platform, it uses the first record to build the list of columns used for all subsequent records for many endpoints. Attached is a pipeline I created for you to ensure all records have all fields contained in the "labels" field of the input documents. Note that this does not do a deep copy of the structure, so if you have nested objects within "labels", you could wind up with the same issue depending on how your target system functions. Please download the zip file, decompress it, and import the pipeline to view the solution in full.
There are a few sections here that I want to provide some description.
For "Map fieldNames", the "$lables.keys()" expression simply returns the list of field names from the $labels sub-object of your input document. The rest of that bottom path is simply gathering up a complete list of field names that are discovered across all of your input documents.
In the "Map all fieldNames", let me break down that expression:
jsonPath($fieldNames, "$[*].fieldName")
.toObject(cur=> cur, cur=> null)
.merge($labels)jsonPath let's you reference object sub-elements, returning an array of those elements within the input document. I recommend following the link to external documentation for even more information.
The Array.toObject() method converts the entire array into an Object. Since our array is a list of field names, this call is simply creating an object with all field names with a null value for each.
Finally, the Object.merge() method does a deep merge of the original input document with the new "nulled Object" we just created from the list of field names.
Hope this helps!
