cancel
Showing results for 
Search instead for 
Did you mean: 

Add fields dynamically in Encrypt field snap

manichandana_ch
New Contributor III

Hi Team,

The community is amazing and very helpful. Kudos to everyone behind it.

I'm working on encrypting certain fields(PII) coming from SFDC sources and using encrypt field snap for the same. For that I'm having to add the fields manually in the snap. But I want to develop a generic pipeline for multiple source objects from SFDC where the fields to be encrypted would be varying(I can store those details of objects and fields in a file or config table to identify them dynamically). I'm not able to find an efficient way for the same as the fields tab in encrypt field snap is not having option to create an expression, leaving with no option except adding them manually. I'd appreciate if anyone could help me understand if there's a way I can achieve that.

Thanks in advance !

manichandana_ch_0-1695964908945.png

3 ACCEPTED SOLUTIONS

koryknick
Employee
Employee

@manichandana_ch - The Encrypt Field snap allows you to send any object element: even a sub-object.  So you can move all the fields you want to encrypt into a sub-object and have the encrypt snap work on that whole element.  For example, I have my PII elements in the "sensitive" sub-object and pass that to the snap:

koryknick_0-1696016741675.png

koryknick_1-1696016767179.png

If you want to see my whole pipeline, download and decompress the attached zip and import the pipeline (SLP) and expression library (EXPR) files into your project.

I hope this helps!

 

 

View solution in original post

koryknick
Employee
Employee

@manichandana_ch - I'm glad that seems to work for your use case!  Let me explain a couple things in the more generic solution.  The first thing to explain is the use of the Expression Library (encrypt.expr) that is configured in the Pipeline Properties.

koryknick_0-1696245239008.png

I'm a big advocate for the use of Expression Libraries, especially since you can store static variables and create inline functions to replace complex or re-usable code in your pipelines.  In this case, I'm simply storing the list of sensitive fields that I want to encrypt and referencing it generically in the first Mapper (Move fields to encrypt):

{}.extend($).filter((val,key)=> lib.encrypt.sensitive.indexOf(key) < 0)
.extend({ "sensitive" : $.filter((val,key)=> lib.encrypt.sensitive.indexOf(key) >= 0) })

The above statement is worth unpacking a bit.  This uses object and array methods to move the sensitive fields to a sub-element of the root object. 

  • The "{}.extend($)" copies the entire incoming document to a new object so the ".filter()" method doesn't affect the original document
  • The ".filter((val,key)=>" is the start of the object filter method, using parameters of the current value and key for each field in the object (that we just copied)
  • "lib.encrypt.sensitive.indexOf(key) < 0)" is the rest of the filter() method call that is checking to see if the object field name exists in the list of fields stored in the encrypt.expr under the "senstiive" variable name; if the field name is listed as "sensitive", it will be removed from the copied object
  • Now that we have copied the original document and filtered out the sensitive fields, we can ".extend()" the new object and add a new field called "sensitive" by using the original object and filtering it to contain only the list of fields listed in the encrypt.expr as sensitive

In the last Mapper in the pipeline (Move sensitive fields to root), we are simply moving the "sensitive" sub-object fields up to the root level of the document.

Hope this helps!

 

 

View solution in original post

koryknick
Employee
Employee

@manichandana_ch - One way I can see to do this pretty simply is by creating a child pipeline for each type of object you want to encrypt and call that dynamically from your main pipeline that is reading and writing to your endpoints.  Assuming that you are reading from your source generically using pipeline parameters (or expression library reference), you could create a set of "encryption" child pipelines: one pipeline for each source object.  These child pipelines only need to contain the Encrypt Field snap, configured with the appropriate fields to be encrypted and named using a standard convention, such as "Encrypt SFDC Account", "Encrypt SFDC Contact", etc.  Then in your main pipeline, use a Pipeline Execute snap configured as follows:

koryknick_0-1697128856275.png

  • The pipeline name can be an expression - in our case, we're going to use the standard prefix of our child pipelines "Encrypt SFDC " concatenated with the pipeline parameter "object" that we're using to dynamically read from SFDC.
  • Use "Execute On" of "LOCAL_NODE" because we're sending every input document to the child pipeline and want to keep the data local on the execution node to prevent unnecessary network traffic.
  • Enable "Reuse executions..." checkbox to bring up the child pipeline and keep it active to process all incoming documents.

Hope this helps!

 

View solution in original post

10 REPLIES 10

@koryknick  Thanks for such a detailed and clear explanation. It's very kind of you to not just provide a solution, but explain it clearly. I'm working on my pipeline to implement the same.

I need to try exploring the use of expression library and the functions more , as they seem to provide solution to any/most of the use cases 🙂 

manichandana_ch
New Contributor III

Hi @koryknick 

I'm not sure why but facing issues recently with the pipeline you shared, even though nothing's been changed. Could you please help me understand the issue and resolve it. Thanks in Advance !

manichandana_ch_0-1696868966165.png

 

koryknick
Employee
Employee

I believe the error means you are missing an account on the Encrypt snap.

koryknick_0-1696876454707.png

 

manichandana_ch
New Contributor III

Hi @koryknick ,

The solution you suggested worked fine, but now the requirement is to just encrypt and load them to parquet files. Downstreams are going to decrypt it (probably it is redshift and decrypts using redshift decrypt function). I tried doing that in redshift via DBeaver, but not able to decrypt the sensitive column group that we encrypted. Now I need to only encrypt the fields and individually. I need to use encrypt field snap. Would you be able to suggest a way for that?
Thanks in Advance !

koryknick
Employee
Employee

@manichandana_ch - One way I can see to do this pretty simply is by creating a child pipeline for each type of object you want to encrypt and call that dynamically from your main pipeline that is reading and writing to your endpoints.  Assuming that you are reading from your source generically using pipeline parameters (or expression library reference), you could create a set of "encryption" child pipelines: one pipeline for each source object.  These child pipelines only need to contain the Encrypt Field snap, configured with the appropriate fields to be encrypted and named using a standard convention, such as "Encrypt SFDC Account", "Encrypt SFDC Contact", etc.  Then in your main pipeline, use a Pipeline Execute snap configured as follows:

koryknick_0-1697128856275.png

  • The pipeline name can be an expression - in our case, we're going to use the standard prefix of our child pipelines "Encrypt SFDC " concatenated with the pipeline parameter "object" that we're using to dynamically read from SFDC.
  • Use "Execute On" of "LOCAL_NODE" because we're sending every input document to the child pipeline and want to keep the data local on the execution node to prevent unnecessary network traffic.
  • Enable "Reuse executions..." checkbox to bring up the child pipeline and keep it active to process all incoming documents.

Hope this helps!