โ12-07-2017 08:14 AM
I have a case where many fields may have to be excluded from mapper output, as if they never existed, selectively during a run. I am going to a cloud instance of a routine that will take null values in, and display them, or may delete wanted values, if this is not implemented correctly. Is there some way of implementing a way to selectively disable/enable the โtarget pathโ declaration on a mapper, while it is running? I even tried passing all nulls, and it doesnโt work.
โ12-07-2017 08:39 AM
Can you give a concrete example, that might help to clarify things.
Hereโs a few ways to delete fields in a document:
$.filter((val, key) => key != 'foo')
to create a new document and then set the target path to โ$โ to write out the new object.โ12-07-2017 11:38 AM
I know I can do that manually. That is great if you never want the field and know which portions you donโt want. Neither is the case for me.
If I have:
"Address": [{
"value": {
"AddressType": {
"value": "Mailing"
},
"AddressLine1": {
"value": "3740 Turtle ST"
},
"City": {
"value": "DALLAS"
},
"StateProvince": {
"value": "TX"
},
"Zip": {
"value": "75215"
},
"Country": {
"value": "US"
}
}
}, {
"value": {
"AddressType": {},
"AddressLine1": {},
"City": {},
"StateProvince": {},
"Zip": {},
"Country": {}
}
}]
I want it to be changed to:
โAddressโ: [{
โvalueโ: {
โAddressTypeโ: {
โvalueโ: โMailingโ
},
โAddressLine1โ: {
โvalueโ: โ3740 Turtle STโ
},
โCityโ: {
โvalueโ: โDALLASโ
},
โStateProvinceโ: {
โvalueโ: โTXโ
},
โZipโ: {
โvalueโ: โ75215โ
},
โCountryโ: {
โvalueโ: โUSโ
}
}
}
}]
Of course, there are other parts that may or may not be null. I want any null part to be ignored.
Do you have any ideas how to use such functionality for a general case such as this? The models may change, and there are several.
Steve
โ12-07-2017 02:36 PM
You can do quite a lot with JSON-Path, itโs not limited to statically defined paths like $foo.bar
. The filtering functionality makes it possible to quite a lot of stuff dynamically.
That being said, it sounds like you need to be able to recursively remove objects that become empty as their descendants are removed. That kind of a task is out-of-scope for JSON-Path since it works on the full object tree at once. I think you can do what you want in the expression language, though. Since recursion is required, youโll want to use an expression library to do the job. Iโll attach a library that contains a cleanupTree
function that should do what you want. The function takes a value and, if itโs an object, calls itself for each field in the object and then filters out any fields that are empty objects. In other words, it will descend to the leaves of the object hierarchy, remove any fields that have empty objects and, if that empties out an object, that field will be removed.
Iโll paste the library code in here since itโs pretty small:
{
// Returns true if the value is not an object or the object has fields.
isNotEmpty: val => !(val instanceof Object) || !val.isEmpty(),
// Recursively removes empty fields in an object.
cleanupTree: root =>
(root instanceof Object ?
root.mapValues(this.cleanupTree).filter(this.isNotEmpty) :
(root instanceof Array ?
root.map(this.cleanupTree).filter(this.isNotEmpty) : root))
}
Youโll want to unzip the attachment on your computer and then import it into your pipeline via the Pipeline Properties dialog. You can then call the function in a Mapper using lib.objutil.cleanupTree($)
.
objutil.expr.zip (325 Bytes)
โ12-08-2017 08:24 AM
THANKS for the help. On the recursive one, with the expression library, I installed the library in the pipeline profile expressions section. I did it as objutil, though I donโt think that matters. I tried the lib.objutil.cleanupTree($) in a mapper connected to the original mapper. It was put in the source section. I didnโt put anything in the target section. When I put it in the source section, it did show a preview of what was there, with โฆ for the values. I even tried to change the name, to see if it gave an error, as I would expect. It did. So it looks like it is installed right. What am I doing wrong?
The reason I want to do this is simply to exclude fields from being updated in the target, selectively. I donโt expect any special processing to be done between it and the target. Between this, and the option del specified, I think this would really help snaplogicโs customers on this type of problem.
Delโs solution would probably work on 80% of the original concern, but the recursive option should work on just about everything.
Steve