Ability to dynamically reduce the fields in an output document based on a comma delimited list

[arvindnsn]

Topic: Looking for a best practice and/or Library Expression to provide the ability to dynamically reduce the fields in an output document based on a comma delimited list.

Description: We are using snaplogic as an information provisioning layer and need to provide the ability to reduce the output document set of fields based on a comma delimited string. We have created a mapper expression to take a delimited string and filter the output document fields based on a match expression. We are looking for help to make this more modular via library function or any other suggested way. We also ran into a situation where we could not figure out how to handle an Array of documents using the Filter expression. Below I have outline a simple example with 3 scenarios, if anyone would be so kind to review and make some suggestion on how to solve #3 and or write options 1&2 more efficiently.

The Pipeline searches input document for all occurrences of fields in “fields” string object using Array functions. There are 3 different snaps in the pipeline.

1. First snap has “fields” variable as columns to be searched as string. The field reducer snap uses the Array functions in the expression to search the columns mentioned in the “fields”.
2. Second snap has “fields” variable as columns to be searched as Array (no specific bound is set). The field reducer snap uses the Array functions in the expression to search the columns mentioned in the “fields”.
3. Third snap has “fields” variable as columns to be searched as Array (with specific bounds). The field reducer snap uses the Array functions in the expression to search the columns mentioned in the “fields”. I am trying for an expression to give output for n number of bounds of an array.

The issue with the third snap is, unless the expression is specified with an array bound, it doesn’t validate. For example, the expression $input[0].filter((value, key) => ($fields.split(‘,’)).indexOf(key) >= 0) is valid, where as $input[*].filter((value, key) => ($fields.split(‘,’)).indexOf(key) >= 0) is not valid. The input array has input[0] and input[1] values and the expression should be able to execute both the bounds of the array.

Also, I am trying to find a way to write this expression as a common library so as to use this in any other API’s. I have also uploaded the pipeline.

Field_Reducer_2018_01_22.slp (10.8 KB)

[tstack]

arvindnsn:

First snap has “fields” variable as columns to be searched as string. The field reducer snap uses the Array functions in the expression to search the columns mentioned in the “fields”.

If you’re really concerned about performance, you might want to split the $fields property in an upstream snap. Otherwise, it will be split for every property in the object when it is done in the filter() callback function, like here:

$.filter((value, key) => ($fields.split(',')).indexOf(key) >= 0)

(Really, the compiler should figure this out optimize away, but that’s not done right now.)

arvindnsn:

The issue with the third snap is, unless the expression is specified with an array bound, it doesn’t validate.

The [*] syntax is for JSON-Paths and is not usable in the expression language, which is why this won’t work:

$input[*].filter((value, key) => ($fields.split(’,’)).indexOf(key) >= 0)

Instead, you’ll need to use the map() method to transform the elements of the $input array. So, something like:

$input.map(x => x.filter((value, key) => ($fields.split(’,’)).indexOf(key) >= 0))