Forum Discussion

stephenknilans's avatar
stephenknilans
Contributor
8 years ago

Is there a way to selectively exclude fields output from a mapper snap?

I have a case where many fields may have to be excluded from mapper output, as if they never existed, selectively during a run. I am going to a cloud instance of a routine that will take null values in, and display them, or may delete wanted values, if this is not implemented correctly. Is there some way of implementing a way to selectively disable/enable the “target path” declaration on a mapper, while it is running? I even tried passing all nulls, and it doesn’t work.

transform

9 Replies

  • tstack's avatar
    tstack
    Former Employee
    8 years ago

    Can you give a concrete example, that might help to clarify things.

    Here’s a few ways to delete fields in a document:

    • In the Mapper, if you leave the “Target Path” property for a row empty it will treat the expression as a JSON-Path and delete whatever fields are on the path.
    • Through the expression language, the Object.filter() method will return a new object with only the fields that match the callback passed into the method. So, in a Mapper, you can write an expression like $.filter((val, key) => key != 'foo') to create a new document and then set the target path to ‘$’ to write out the new object.
    • stephenknilans's avatar
      stephenknilans
      Contributor
      8 years ago

      I know I can do that manually. That is great if you never want the field and know which portions you don’t want. Neither is the case for me.

      If I have:

      		"Address": [{
				"value": {
					"AddressType": {
						"value": "Mailing"
					},
					"AddressLine1": {
						"value": "3740 Turtle ST"
					},
					"City": {
						"value": "DALLAS"
					},
					"StateProvince": {
						"value": "TX"
					},
					"Zip": {
						"value": "75215"
					},
					"Country": {
						"value": "US"
					}
				}
			}, {
				"value": {
					"AddressType": {},
					"AddressLine1": {},
					"City": {},
					"StateProvince": {},
					"Zip": {},
					"Country": {}
				}

      }]

      I want it to be changed to:

      “Address”: [{
      “value”: {
      “AddressType”: {
      “value”: “Mailing”
      },
      “AddressLine1”: {
      “value”: “3740 Turtle ST”
      },
      “City”: {
      “value”: “DALLAS”
      },
      “StateProvince”: {
      “value”: “TX”
      },
      “Zip”: {
      “value”: “75215”
      },
      “Country”: {
      “value”: “US”
      }
      }
      }
      }]

      Of course, there are other parts that may or may not be null. I want any null part to be ignored.

      Do you have any ideas how to use such functionality for a general case such as this? The models may change, and there are several.

      Steve

      • tstack's avatar
        tstack
        Former Employee
        8 years ago

        You can do quite a lot with JSON-Path, it’s not limited to statically defined paths like $foo.bar. The filtering functionality makes it possible to quite a lot of stuff dynamically.

        That being said, it sounds like you need to be able to recursively remove objects that become empty as their descendants are removed. That kind of a task is out-of-scope for JSON-Path since it works on the full object tree at once. I think you can do what you want in the expression language, though. Since recursion is required, you’ll want to use an expression library to do the job. I’ll attach a library that contains a cleanupTree function that should do what you want. The function takes a value and, if it’s an object, calls itself for each field in the object and then filters out any fields that are empty objects. In other words, it will descend to the leaves of the object hierarchy, remove any fields that have empty objects and, if that empties out an object, that field will be removed.

        I’ll paste the library code in here since it’s pretty small:

        {
    // Returns true if the value is not an object or the object has fields.
    isNotEmpty: val => !(val instanceof Object) || !val.isEmpty(),
    // Recursively removes empty fields in an object.
    cleanupTree: root =>
      (root instanceof Object ?
         root.mapValues(this.cleanupTree).filter(this.isNotEmpty) :
       (root instanceof Array ?
       	 root.map(this.cleanupTree).filter(this.isNotEmpty) : root))
}

        You’ll want to unzip the attachment on your computer and then import it into your pipeline via the Pipeline Properties dialog. You can then call the function in a Mapper using lib.objutil.cleanupTree($).

        objutil.expr.zip (325 Bytes)