Designing pattern pipelines to support upgrades while retaining customizations

Hi all,

We have published a number of SnapLogic pattern pipelines implementing two-way integration between Salesforce Contacts and a Java web application (via SOAP API calls)

At its core, we’re taking payload sent from the source system, converting it via mappers and script snaps and integrating it with the destination system.

Most of our mappings are simple property remappings, with occasional flattening of relationships
image

Occasionally we need to invoke expression library functions or do simple lookups as well:

As part of ongoing development we occasionally need to make bugfixes and enhancements. Additionally, our pattern pipeline end-users also need to make customizations, e.g. to map custom Salesforce Contact fields.

We are looking into ways to design our pipelines in a way that allows our pattern pipeline end-users to customize their data mappings, but not have to redo their data mappings every time they upgrade pipeline versions.

I’ve identified two possible ways this could be achieved:

  • Extract the portion of the pipeline implementing data mappings containing mapper snaps into a subpipeline and publish it as a “Default Data mappings” pattern pipeline . Pattern pipeline end-users can customize this snap with their own custom mappings.

  • Introduce an expression library that defines the data mappings and have a script snap parse the expression library mappings and evaluate them to add mappings to the destination document… e.g. the expression library might contain the following:

      {
        mappings: {
          "student.Home.street1": "MailingStreet",
          "student.Home.city" : "MailingCity",
          '{"M":"Male","F":"Female"}[$student.genderCode]': "one_Gender__c"
      }
    

    }

I’m guessing that we would need to make use of the eval function to evaluate expressions as JS statements to be able to effect this within the script snap.

Are either of these approaches viable?
Is one better than the other?
Or is there a different approach that the community would recommend?

Thanks

Both the approaches you mentioned would be possible.

For the second approach (using expression library), it is possible to dynamically call expression library functions without using eval(), see Lookup on an array in the input document. eval() has a significant performance penalty because the expression has to be parsed for each document being processed.