cancel
Showing results for 
Search instead for 
Did you mean: 

Add fields dynamically in Encrypt field snap

manichandana_ch
New Contributor III

Hi Team,

The community is amazing and very helpful. Kudos to everyone behind it.

I'm working on encrypting certain fields(PII) coming from SFDC sources and using encrypt field snap for the same. For that I'm having to add the fields manually in the snap. But I want to develop a generic pipeline for multiple source objects from SFDC where the fields to be encrypted would be varying(I can store those details of objects and fields in a file or config table to identify them dynamically). I'm not able to find an efficient way for the same as the fields tab in encrypt field snap is not having option to create an expression, leaving with no option except adding them manually. I'd appreciate if anyone could help me understand if there's a way I can achieve that.

Thanks in advance !

manichandana_ch_0-1695964908945.png

3 ACCEPTED SOLUTIONS

koryknick
Employee
Employee

@manichandana_ch - The Encrypt Field snap allows you to send any object element: even a sub-object.  So you can move all the fields you want to encrypt into a sub-object and have the encrypt snap work on that whole element.  For example, I have my PII elements in the "sensitive" sub-object and pass that to the snap:

koryknick_0-1696016741675.png

koryknick_1-1696016767179.png

If you want to see my whole pipeline, download and decompress the attached zip and import the pipeline (SLP) and expression library (EXPR) files into your project.

I hope this helps!

 

 

View solution in original post

koryknick
Employee
Employee

@manichandana_ch - I'm glad that seems to work for your use case!  Let me explain a couple things in the more generic solution.  The first thing to explain is the use of the Expression Library (encrypt.expr) that is configured in the Pipeline Properties.

koryknick_0-1696245239008.png

I'm a big advocate for the use of Expression Libraries, especially since you can store static variables and create inline functions to replace complex or re-usable code in your pipelines.  In this case, I'm simply storing the list of sensitive fields that I want to encrypt and referencing it generically in the first Mapper (Move fields to encrypt):

{}.extend($).filter((val,key)=> lib.encrypt.sensitive.indexOf(key) < 0)
.extend({ "sensitive" : $.filter((val,key)=> lib.encrypt.sensitive.indexOf(key) >= 0) })

The above statement is worth unpacking a bit.  This uses object and array methods to move the sensitive fields to a sub-element of the root object. 

  • The "{}.extend($)" copies the entire incoming document to a new object so the ".filter()" method doesn't affect the original document
  • The ".filter((val,key)=>" is the start of the object filter method, using parameters of the current value and key for each field in the object (that we just copied)
  • "lib.encrypt.sensitive.indexOf(key) < 0)" is the rest of the filter() method call that is checking to see if the object field name exists in the list of fields stored in the encrypt.expr under the "senstiive" variable name; if the field name is listed as "sensitive", it will be removed from the copied object
  • Now that we have copied the original document and filtered out the sensitive fields, we can ".extend()" the new object and add a new field called "sensitive" by using the original object and filtering it to contain only the list of fields listed in the encrypt.expr as sensitive

In the last Mapper in the pipeline (Move sensitive fields to root), we are simply moving the "sensitive" sub-object fields up to the root level of the document.

Hope this helps!

 

 

View solution in original post

koryknick
Employee
Employee

@manichandana_ch - One way I can see to do this pretty simply is by creating a child pipeline for each type of object you want to encrypt and call that dynamically from your main pipeline that is reading and writing to your endpoints.  Assuming that you are reading from your source generically using pipeline parameters (or expression library reference), you could create a set of "encryption" child pipelines: one pipeline for each source object.  These child pipelines only need to contain the Encrypt Field snap, configured with the appropriate fields to be encrypted and named using a standard convention, such as "Encrypt SFDC Account", "Encrypt SFDC Contact", etc.  Then in your main pipeline, use a Pipeline Execute snap configured as follows:

koryknick_0-1697128856275.png

  • The pipeline name can be an expression - in our case, we're going to use the standard prefix of our child pipelines "Encrypt SFDC " concatenated with the pipeline parameter "object" that we're using to dynamically read from SFDC.
  • Use "Execute On" of "LOCAL_NODE" because we're sending every input document to the child pipeline and want to keep the data local on the execution node to prevent unnecessary network traffic.
  • Enable "Reuse executions..." checkbox to bring up the child pipeline and keep it active to process all incoming documents.

Hope this helps!

 

View solution in original post

10 REPLIES 10

koryknick
Employee
Employee

@manichandana_ch - The Encrypt Field snap does allow you to encrypt any element of the object, so you can simply move the fields you want to encrypt to a single sub-object and encrypt that element.  For example, here is my input to the Encrypt Field snap:

koryknick_0-1696006370358.png

Then the Encrypt Field settings

koryknick_1-1696006412420.png

Hope this helps!

koryknick
Employee
Employee

@manichandana_ch - The Encrypt Field snap allows you to send any object element: even a sub-object.  So you can move all the fields you want to encrypt into a sub-object and have the encrypt snap work on that whole element.  For example, I have my PII elements in the "sensitive" sub-object and pass that to the snap:

koryknick_0-1696016741675.png

koryknick_1-1696016767179.png

If you want to see my whole pipeline, download and decompress the attached zip and import the pipeline (SLP) and expression library (EXPR) files into your project.

I hope this helps!

 

 

Hi @koryknick 

Thanks for the response. I tested the attached pipeline and tried understanding it. I need to tweak my existing pipeline a bit to implement this solution and I'll work on it. Shall let you know in case of any discrepancy.  This is really helpful !

Thanks,

Mani Chandana Chalasani

koryknick
Employee
Employee

@manichandana_ch - I'm glad that seems to work for your use case!  Let me explain a couple things in the more generic solution.  The first thing to explain is the use of the Expression Library (encrypt.expr) that is configured in the Pipeline Properties.

koryknick_0-1696245239008.png

I'm a big advocate for the use of Expression Libraries, especially since you can store static variables and create inline functions to replace complex or re-usable code in your pipelines.  In this case, I'm simply storing the list of sensitive fields that I want to encrypt and referencing it generically in the first Mapper (Move fields to encrypt):

{}.extend($).filter((val,key)=> lib.encrypt.sensitive.indexOf(key) < 0)
.extend({ "sensitive" : $.filter((val,key)=> lib.encrypt.sensitive.indexOf(key) >= 0) })

The above statement is worth unpacking a bit.  This uses object and array methods to move the sensitive fields to a sub-element of the root object. 

  • The "{}.extend($)" copies the entire incoming document to a new object so the ".filter()" method doesn't affect the original document
  • The ".filter((val,key)=>" is the start of the object filter method, using parameters of the current value and key for each field in the object (that we just copied)
  • "lib.encrypt.sensitive.indexOf(key) < 0)" is the rest of the filter() method call that is checking to see if the object field name exists in the list of fields stored in the encrypt.expr under the "senstiive" variable name; if the field name is listed as "sensitive", it will be removed from the copied object
  • Now that we have copied the original document and filtered out the sensitive fields, we can ".extend()" the new object and add a new field called "sensitive" by using the original object and filtering it to contain only the list of fields listed in the encrypt.expr as sensitive

In the last Mapper in the pipeline (Move sensitive fields to root), we are simply moving the "sensitive" sub-object fields up to the root level of the document.

Hope this helps!