Is there a way to selectively exclude fields output from a mapper snap?

[stephenknilans]

I have a case where many fields may have to be excluded from mapper output, as if they never existed, selectively during a run. I am going to a cloud instance of a routine that will take null values in, and display them, or may delete wanted values, if this is not implemented correctly. Is there some way of implementing a way to selectively disable/enable the “target path” declaration on a mapper, while it is running? I even tried passing all nulls, and it doesn’t work.

[tstack]

stephenknilans:

I have a case where many fields may have to be excluded from mapper output, as if they never existed, selectively during a run.

Can you give a concrete example, that might help to clarify things.

Here’s a few ways to delete fields in a document:

• In the Mapper, if you leave the “Target Path” property for a row empty it will treat the expression as a JSON-Path and delete whatever fields are on the path.
• Through the expression language, the Object.filter() method will return a new object with only the fields that match the callback passed into the method. So, in a Mapper, you can write an expression like $.filter((val, key) => key != 'foo') to create a new document and then set the target path to ‘$’ to write out the new object.

[stephenknilans]

tstack:

In the Mapper, if you leave the “Target Path” property for a row empty it will treat the expression as a JSON-Path and delete whatever fields are on the path.

I know I can do that manually. That is great if you never want the field and know which portions you don’t want. Neither is the case for me.

If I have:

		"Address": [{
				"value": {
					"AddressType": {
						"value": "Mailing"
					},
					"AddressLine1": {
						"value": "3740 Turtle ST"
					},
					"City": {
						"value": "DALLAS"
					},
					"StateProvince": {
						"value": "TX"
					},
					"Zip": {
						"value": "75215"
					},
					"Country": {
						"value": "US"
					}
				}
			}, {
				"value": {
					"AddressType": {},
					"AddressLine1": {},
					"City": {},
					"StateProvince": {},
					"Zip": {},
					"Country": {}
				}

}]

I want it to be changed to:

“Address”: [{
“value”: {
“AddressType”: {
“value”: “Mailing”
},
“AddressLine1”: {
“value”: “3740 Turtle ST”
},
“City”: {
“value”: “DALLAS”
},
“StateProvince”: {
“value”: “TX”
},
“Zip”: {
“value”: “75215”
},
“Country”: {
“value”: “US”
}
}
}
}]

Of course, there are other parts that may or may not be null. I want any null part to be ignored.

tstack:

Through the expression language, the Object.filter() method will return a new object with only the fields that match the callback passed into the method. So, in a Mapper, you can write an expression like $.filter((val, key) => key != ‘foo’) to create a new document and then set the target path to ‘$’ to write out the new object.

Do you have any ideas how to use such functionality for a general case such as this? The models may change, and there are several.

Steve

[tstack]

stephenknilans:

I know I can do that manually. That is great if you never want the field and know which portions you don’t want. Neither is the case for me.

You can do quite a lot with JSON-Path, it’s not limited to statically defined paths like $foo.bar. The filtering functionality makes it possible to quite a lot of stuff dynamically.

That being said, it sounds like you need to be able to recursively remove objects that become empty as their descendants are removed. That kind of a task is out-of-scope for JSON-Path since it works on the full object tree at once. I think you can do what you want in the expression language, though. Since recursion is required, you’ll want to use an expression library to do the job. I’ll attach a library that contains a cleanupTree function that should do what you want. The function takes a value and, if it’s an object, calls itself for each field in the object and then filters out any fields that are empty objects. In other words, it will descend to the leaves of the object hierarchy, remove any fields that have empty objects and, if that empties out an object, that field will be removed.

I’ll paste the library code in here since it’s pretty small:

{
    // Returns true if the value is not an object or the object has fields.
    isNotEmpty: val => !(val instanceof Object) || !val.isEmpty(),
    // Recursively removes empty fields in an object.
    cleanupTree: root =>
      (root instanceof Object ?
         root.mapValues(this.cleanupTree).filter(this.isNotEmpty) :
       (root instanceof Array ?
       	 root.map(this.cleanupTree).filter(this.isNotEmpty) : root))
}

You’ll want to unzip the attachment on your computer and then import it into your pipeline via the Pipeline Properties dialog. You can then call the function in a Mapper using lib.objutil.cleanupTree($).

objutil.expr.zip (325 Bytes)

[stephenknilans]

THANKS for the help. On the recursive one, with the expression library, I installed the library in the pipeline profile expressions section. I did it as objutil, though I don’t think that matters. I tried the lib.objutil.cleanupTree($) in a mapper connected to the original mapper. It was put in the source section. I didn’t put anything in the target section. When I put it in the source section, it did show a preview of what was there, with … for the values. I even tried to change the name, to see if it gave an error, as I would expect. It did. So it looks like it is installed right. What am I doing wrong?

The reason I want to do this is simply to exclude fields from being updated in the target, selectively. I don’t expect any special processing to be done between it and the target. Between this, and the option del specified, I think this would really help snaplogic’s customers on this type of problem.

Del’s solution would probably work on 80% of the original concern, but the recursive option should work on just about everything.

Steve