An Elastic Mapper which outputs JSON that only contains keys that have non-null values
Background: The use case came while trying to map fields from a Salesforce Outbound Message. This message would send a varying number of fields, from each record, each having non-null values. For example, the SFDC Object had 50 fields in total and one record would return a document with 35 of the 50 possible fields populated and the next record would send 25 and so on; the data is then moved to another database that is pre-populated with data for each record. The problem was that using a Mapper, all 50 fields would need to be mapped in advance, and therefore those fields with NULL values would potentially overwrite data when updating the database.
Our Mapper is static. Target Schemas need to be defined up front. The problem to solve was how to create a Mapper that filters out the non-null fields from each input document so that the update can occur without overwriting data with the null values.
The Magic:
All you need is ONE line of code added to the ‘try’ block of the script snap (javascript in this case) that removes from the input map any elements that have null values:
…
this.log.info("Executing Transform Script");
while (this.input.hasNext()) {
try{
// Read the next document, wrap it in a map and write out the wrapper
var doc = this.input.next();
while (doc.values().remove(null));
var wrapper = new java.util.HashMap();
wrapper.put("original", doc);
this.output.write(doc, doc); //ß Optional note: I modified this to flatten the output
this.log.info("Transform Script finished");
}
…
** In Python you can use this construct in two lines (please note the indentation required in python):
while (in_doc.values().remove(None)):
None
See attached test case pipeline:
Limitations:
This was created to work only with flattened, non-hierarchical structures.
**Elastic Mapping_2017_03_01.slp (9.6 KB)
