Hi Team,
The community is amazing and very helpful. Kudos to everyone behind it.
I'm working on encrypting certain fields(PII) coming from SFDC sources and using encrypt field snap for the same. For that I'm having to add the fields manually in the snap. But I want to develop a generic pipeline for multiple source objects from SFDC where the fields to be encrypted would be varying(I can store those details of objects and fields in a file or config table to identify them dynamically). I'm not able to find an efficient way for the same as the fields tab in encrypt field snap is not having option to create an expression, leaving with no option except adding them manually. I'd appreciate if anyone could help me understand if there's a way I can achieve that.
Thanks in advance !
3 ACCEPTED SOLUTIONS
@manichandana_ch - The Encrypt Field snap allows you to send any object element: even a sub-object. So you can move all the fields you want to encrypt into a sub-object and have the encrypt snap work on that whole element. For example, I have my PII elements in the "sensitive" sub-object and pass that to the snap:
If you want to see my whole pipeline, download and decompress the attached zip and import the pipeline (SLP) and expression library (EXPR) files into your project.
I hope this helps!
@manichandana_ch - I'm glad that seems to work for your use case! Let me explain a couple things in the more generic solution. The first thing to explain is the use of the Expression Library (encrypt.expr) that is configured in the Pipeline Properties.
I'm a big advocate for the use of Expression Libraries, especially since you can store static variables and create inline functions to replace complex or re-usable code in your pipelines. In this case, I'm simply storing the list of sensitive fields that I want to encrypt and referencing it generically in the first Mapper (Move fields to encrypt):
{}.extend($).filter((val,key)=> lib.encrypt.sensitive.indexOf(key) < 0)
.extend({ "sensitive" : $.filter((val,key)=> lib.encrypt.sensitive.indexOf(key) >= 0) })The above statement is worth unpacking a bit. This uses object and array methods to move the sensitive fields to a sub-element of the root object.
- The "{}.extend($)" copies the entire incoming document to a new object so the ".filter()" method doesn't affect the original document
- The ".filter((val,key)=>" is the start of the object filter method, using parameters of the current value and key for each field in the object (that we just copied)
- "lib.encrypt.sensitive.indexOf(key) < 0)" is the rest of the filter() method call that is checking to see if the object field name exists in the list of fields stored in the encrypt.expr under the "senstiive" variable name; if the field name is listed as "sensitive", it will be removed from the copied object
- Now that we have copied the original document and filtered out the sensitive fields, we can ".extend()" the new object and add a new field called "sensitive" by using the original object and filtering it to contain only the list of fields listed in the encrypt.expr as sensitive
In the last Mapper in the pipeline (Move sensitive fields to root), we are simply moving the "sensitive" sub-object fields up to the root level of the document.
Hope this helps!
@manichandana_ch - One way I can see to do this pretty simply is by creating a child pipeline for each type of object you want to encrypt and call that dynamically from your main pipeline that is reading and writing to your endpoints. Assuming that you are reading from your source generically using pipeline parameters (or expression library reference), you could create a set of "encryption" child pipelines: one pipeline for each source object. These child pipelines only need to contain the Encrypt Field snap, configured with the appropriate fields to be encrypted and named using a standard convention, such as "Encrypt SFDC Account", "Encrypt SFDC Contact", etc. Then in your main pipeline, use a Pipeline Execute snap configured as follows:
- The pipeline name can be an expression - in our case, we're going to use the standard prefix of our child pipelines "Encrypt SFDC " concatenated with the pipeline parameter "object" that we're using to dynamically read from SFDC.
- Use "Execute On" of "LOCAL_NODE" because we're sending every input document to the child pipeline and want to keep the data local on the execution node to prevent unnecessary network traffic.
- Enable "Reuse executions..." checkbox to bring up the child pipeline and keep it active to process all incoming documents.
Hope this helps!
@manichandana_ch - The Encrypt Field snap does allow you to encrypt any element of the object, so you can simply move the fields you want to encrypt to a single sub-object and encrypt that element. For example, here is my input to the Encrypt Field snap:
Then the Encrypt Field settings
Hope this helps!
@manichandana_ch - The Encrypt Field snap allows you to send any object element: even a sub-object. So you can move all the fields you want to encrypt into a sub-object and have the encrypt snap work on that whole element. For example, I have my PII elements in the "sensitive" sub-object and pass that to the snap:
If you want to see my whole pipeline, download and decompress the attached zip and import the pipeline (SLP) and expression library (EXPR) files into your project.
I hope this helps!
@manichandana_ch - I'm glad that seems to work for your use case! Let me explain a couple things in the more generic solution. The first thing to explain is the use of the Expression Library (encrypt.expr) that is configured in the Pipeline Properties.
I'm a big advocate for the use of Expression Libraries, especially since you can store static variables and create inline functions to replace complex or re-usable code in your pipelines. In this case, I'm simply storing the list of sensitive fields that I want to encrypt and referencing it generically in the first Mapper (Move fields to encrypt):
{}.extend($).filter((val,key)=> lib.encrypt.sensitive.indexOf(key) < 0)
.extend({ "sensitive" : $.filter((val,key)=> lib.encrypt.sensitive.indexOf(key) >= 0) })The above statement is worth unpacking a bit. This uses object and array methods to move the sensitive fields to a sub-element of the root object.
- The "{}.extend($)" copies the entire incoming document to a new object so the ".filter()" method doesn't affect the original document
- The ".filter((val,key)=>" is the start of the object filter method, using parameters of the current value and key for each field in the object (that we just copied)
- "lib.encrypt.sensitive.indexOf(key) < 0)" is the rest of the filter() method call that is checking to see if the object field name exists in the list of fields stored in the encrypt.expr under the "senstiive" variable name; if the field name is listed as "sensitive", it will be removed from the copied object
- Now that we have copied the original document and filtered out the sensitive fields, we can ".extend()" the new object and add a new field called "sensitive" by using the original object and filtering it to contain only the list of fields listed in the encrypt.expr as sensitive
In the last Mapper in the pipeline (Move sensitive fields to root), we are simply moving the "sensitive" sub-object fields up to the root level of the document.
Hope this helps!
