โ08-04-2021 01:01 PM
Hi Team,
I am trying to achieve below. I would need directions on how to achieve below in a mapper or any other way.
Source :
{
โUBER_IDโ:[
โ1โ,
โInteger_patternโ
],
โFirst_nameโ:[
โMajidโ,
โTextOnly_patternโ
],
โLast_nameโ:[
โโ,
โTextOnly_patternโ
]
}
Target :
I would like to rename the above fields based on the value in the array. Below is the output I am looking at.
{
โInteger_patternโ:[
โ1โ,
],
โTextOnly_patternโ:[
โMajidโ,
],
" TextOnly_pattern":[
โโ,
]
}
โ08-05-2021 01:30 PM
Attached is a sample pipeline with a couple of options using the Pivot Snap to produce results similar to what your sample pipeline appears to be attempting. Both options use the same logic - the second just brings the grouping back together if desired.
It may not be your target solution, and my expressions could need a little work, but I hope it helps some with the direction you want to go.
Community.10510_2021_08_05.slp (30.4 KB)
โ08-05-2021 02:10 PM
Thank You @del โฆ This works but in my case the number and name of fields will be different from one file to other as I am creating a generic pipeline. Is there anyway to make the Pivot snap number of fields and field names dynamic.
I am also not sure about performance as this method is going to divide each record into x records based on the number of fields.
I appreciate you looking into the use case and providing directions.
โ08-05-2021 02:48 PM
Here is a version 2 of the pipeline that uses a Mapper and Splitter (instead of Pivot snap) for a dynamic pivot of the data. Community.10510_2021_08_05 (v2).slp (32.8 KB)
Iโm not sure about performance, either, but I donโt know how you use your validator snap as-is without splitting/pivoting the data - because of the duplicate key issues.
As this is to be a generic pipeline, I think you might be better off using an Expression Library
in place of (or in conjunction with) your Validator. I think you could avoid the pivot, then.
โ08-06-2021 05:32 AM
@del Thank you so muchโฆ this is what I was exactly looking for. I will verify the performance and update youโฆ
Can you provide example if possible about the way it can be done with expression library. As data validator does not allow any expression library or parameters used I am not sure how this can be achieved in a mapper using expression library.
โ08-06-2021 12:02 PM
@Majid Iโm glad the above helped.
I think I may be derailing too much by suggesting the expression library. It was a creative thought, but would require readjusting downstream logic to reach desired end result.
But, for an exercise, I put this together to show where my thought was leaning.
It basically changes your source:
{
"UBER_ID": [
"1",
"Integer_pattern"
],
"First_name": [
"Majid",
"TextOnly_pattern"
],
"Last_name": [
"",
"TextOnly_pattern"
]
}
to this:
{
"UBER_ID": [
"1",
"Integer_pattern",
"valid"
],
"First_name": [
"Majid",
"TextOnly_pattern",
"valid"
],
"Last_name": [
"",
"TextOnly_pattern",
"invalid"
]
}
But youโd still have to handle this creatively in downstream snaps to get your desired results.
Community.10510.v3_2021_08_06.slp (4.9 KB)
community10510v3.expr.txt (376 Bytes)