We built a custom snap that performs a transformation. You will notice in the output of the custom snap in the screenshot below that “medicalEnrollment” is a json array.
In a downstream mapper, we attempted a transformation on this array. We have tried (1) using the sl.ensureArray() method and (2) another downstream mapper performing a transformation in the array. However, we notice that in both the #1 and #2 attempts the mappers do not detect the array, thus causing issues with the transformation.
Below is a snippet of the preview of what we expect vs actual preview.
We also notice that chaining a Json Formatter and a Json Parser right after the custom snap does indeed makes the mappers work correctly. Pattern shown below:
So my question is, is there something hidden in the JSON parser snap that allows strong typing of the array that we are missing? Anything in between?
@robin , Tagging you since I was told you might be able to help out. Thanks!
1 ACCEPTED SOLUTION
Thanks. I installed the binary snap pack and set a breakpoint in the Mapper to see the Java types used in the input, which is the output of your snap. I can see that medicalEnrollment is still a HashSet, not an ArrayList:
I haven’t studied your source code but I can see that you’re still using HashSet in a lot of places. Don’t use that type anywhere in your Document data. It’s not compatible with any of the JSON types. Use ArrayList everywhere you need an array.
@ptaylor Thanks for the prompt response. Let me know what other information you might need.
Is your custom snap the one labeled “Excel Structure” in the image?
Correct.
It looks like it takes an XLS file as binary input, correct?
Correct.
What sort of transformation does it perform?
In essence, it’s restructuring a flat excel structure into a more complex JSON that we need. For example, we group rows by specific criteria - some pieces of the data ends up in different arrays.
Does the custom snap sometimes generate output where
medicalEnrollmentis not an array? If so, isn’t this a bug that you could fix in the custom snap’s code? That seems preferable to fixing it using a subsequent Mapper withsl.ensureArray.
We have tried both in the custom snap 1) having the output be simple where only necessary data is in the output. The “medicalEnrollment” array might be missing from the output, however, if it’s present, it’s always an array 2) producing a medicalEnrollment array for every document.
Can you please show/say more about this transformation? Are you using the Mapping Root functionality of the Mapper?
Correct. We are using the Mapping Root functionality. The three screenshots below show the input, the root, and the output of the downstream mapper respectively. I’m trying to show here that the mapper did not detect the Benefit_Election_Data Array and no transformation was performed.
Input:
Mapper:
Output:
Frankly, this is a bit hard to follow. You were talking about medicalEnrollment and then suddenly switched to Benefit_Election_Data halfway through that last reply. The content of each array looks similar but the surrounding content looks different. But let me focus on your new examples involving Benefit_Election_Data.
It’s odd that the Mapper is showing the type of Benefit_Election_Data as object rather than array, despite the fact that it’s clearly an array in the ROOT output0 screenshot. It seems that’s the core issue you’re posting about. But does this happen consistently? Your first post seemed to show that sometimes it correctly shows as an array rather than as an object, at least when you were talking about medicalEnrollment. Is the problem that the behavior is not consistent?
Can you please post a screenshot of the Mapper that includes the Input Preview panel at the bottom? I tried to reproduce the issue you’re seeing using an abbreviated version of your data in a JSON Generator, but it’s working fine for me…
Here’s the Mapper before setting the Mapping root, where the Benefit_Election_Data is correctly shown as an array:
And here’s the same Mapper after setting the Mapping root to focus on the elements of the Benefit_Election_Data array, which changes the Input Schema:
Let me ask you this: In your custom snap, what Java class are you using for the array? We typically use ArrayList.
Frankly, this is a bit hard to follow. You were talking about
medicalEnrollmentand then suddenly switched toBenefit_Election_Datahalfway through that last reply. The content of each array looks similar but the surrounding content looks different. But let me focus on your new examples involvingBenefit_Election_Data.
Apologies for the confusion. I’ll try to replicate the data flow to make it clearer
- “Excel Structure” Custom Snap is transforming an excel file. “medicalEnrollment” Array is a part of the the output
- “ROOT” Mapper. Transforms the base root of the data. In here, “medicalEnrollment” is transformed to $Change_Benefits_Data.Benefit_Election_Data. We expect “Benefit_Election_Data” to be the array containing the information medicalEnrollment previously had.
- “BENEFIT ELECTION DATA” Mapper. Transforms the “Benefit_Election_Data”.
But does this happen consistently? Your first post seemed to show that sometimes it correctly shows as an
arrayrather than as anobject, at least when you were talking aboutmedicalEnrollment. Is the problem that the behavior is not consistent?
Correct, that this is the core issue of this post. In this pipeline, it’s consistently happening whenever a mapper is chained downstream after the custom snap. If I introduce a JSON formatter and JSON Parser right after the custom snap as shown in the original post, the downstream mapper consistently behaves as we expect it to and transforms the array. In the second screenshot of the original post, I tried to show expect vs. actual behavior.
Can you please post a screenshot of the Mapper that includes the
Input Previewpanel at the bottom?
Below is the screenshot.
Let me ask you this: In your custom snap, what Java class are you using for the array? We typically use ArrayList.
We are using a HashSet. We will try converting to ArrayList and post an update tomorrow.
