cclaudio
New Contributor II
Joined 7 years ago
User Profile
User Widgets
Contributions
Re: Issue with Arrays in Downstream Mappers after a Custom Snap Transformation
I suggest creating a support ticket and providing us with a link to a test pipeline that we can execute. @ptaylor I already have created a ticket with support (Ticket #43346). I will follow up there with the details.1.3KViews0likes11CommentsRe: Issue with Arrays in Downstream Mappers after a Custom Snap Transformation
@ptaylor, We were not able to make it work by switching to ArrayList. We will create another custom snap simplified to bare bones and use an ArrayList. Are you aware of any other requirements that the Mapper might need to interpret the Arrays correctly?1.3KViews0likes14CommentsRe: Issue with Arrays in Downstream Mappers after a Custom Snap Transformation
Frankly, this is a bit hard to follow. You were talking about medicalEnrollment and then suddenly switched to Benefit_Election_Data halfway through that last reply. The content of each array looks similar but the surrounding content looks different. But let me focus on your new examples involving Benefit_Election_Data . Apologies for the confusion. I’ll try to replicate the data flow to make it clearer “Excel Structure” Custom Snap is transforming an excel file. “medicalEnrollment” Array is a part of the the output “ROOT” Mapper. Transforms the base root of the data. In here, “medicalEnrollment” is transformed to $Change_Benefits_Data.Benefit_Election_Data. We expect “Benefit_Election_Data” to be the array containing the information medicalEnrollment previously had. “BENEFIT ELECTION DATA” Mapper. Transforms the “Benefit_Election_Data”. But does this happen consistently? Your first post seemed to show that sometimes it correctly shows as an array rather than as an object , at least when you were talking about medicalEnrollment . Is the problem that the behavior is not consistent? Correct, that this is the core issue of this post. In this pipeline, it’s consistently happening whenever a mapper is chained downstream after the custom snap. If I introduce a JSON formatter and JSON Parser right after the custom snap as shown in the original post, the downstream mapper consistently behaves as we expect it to and transforms the array. In the second screenshot of the original post, I tried to show expect vs. actual behavior. Can you please post a screenshot of the Mapper that includes the Input Preview panel at the bottom? Below is the screenshot. Let me ask you this: In your custom snap, what Java class are you using for the array? We typically use ArrayList. We are using a HashSet. We will try converting to ArrayList and post an update tomorrow.3.1KViews1like1CommentRe: Issue with Arrays in Downstream Mappers after a Custom Snap Transformation
@ptaylor Thanks for the prompt response. Let me know what other information you might need. Is your custom snap the one labeled “Excel Structure” in the image? Correct. It looks like it takes an XLS file as binary input, correct? Correct. What sort of transformation does it perform? In essence, it’s restructuring a flat excel structure into a more complex JSON that we need. For example, we group rows by specific criteria - some pieces of the data ends up in different arrays. Does the custom snap sometimes generate output where medicalEnrollment is not an array? If so, isn’t this a bug that you could fix in the custom snap’s code? That seems preferable to fixing it using a subsequent Mapper with sl.ensureArray . We have tried both in the custom snap 1) having the output be simple where only necessary data is in the output. The “medicalEnrollment” array might be missing from the output, however, if it’s present, it’s always an array 2) producing a medicalEnrollment array for every document. Can you please show/say more about this transformation? Are you using the Mapping Root functionality of the Mapper? Correct. We are using the Mapping Root functionality. The three screenshots below show the input, the root, and the output of the downstream mapper respectively. I’m trying to show here that the mapper did not detect the Benefit_Election_Data Array and no transformation was performed. Input: Mapper: Output:3.1KViews0likes1CommentIssue with Arrays in Downstream Mappers after a Custom Snap Transformation
We built a custom snap that performs a transformation. You will notice in the output of the custom snap in the screenshot below that “medicalEnrollment” is a json array. In a downstream mapper, we attempted a transformation on this array. We have tried (1) using the sl.ensureArray() method and (2) another downstream mapper performing a transformation in the array. However, we notice that in both the #1 and #2 attempts the mappers do not detect the array, thus causing issues with the transformation. Below is a snippet of the preview of what we expect vs actual preview. We also notice that chaining a Json Formatter and a Json Parser right after the custom snap does indeed makes the mappers work correctly. Pattern shown below: So my question is, is there something hidden in the JSON parser snap that allows strong typing of the array that we are missing? Anything in between? @robin , Tagging you since I was told you might be able to help out. Thanks!Solved9.6KViews0likes20Comments