Loading configuration file to a pipeline

#1

Hi,

It is very common in most integration flow design we need to refer to configuration files for mapping information, file path info, lookup values etc. As of today we keep these configuration files in SLDB or SFTP server and load them to pipeline using FileReader snap. This is not very ideal, reading the configuration file is not part of my integration flow design but the configuration is needed for the integration flow, loading of configuration should be outside of my flow design. This is making pipeline design complicated with steps involved in reading configuration file. Also the read of file is considered in document count, resulting is not so accurate document count in the dashboard.

Solution: Configuration files (lets limit to JSON) should be part of pipeline settings and they should be loaded as part of pipeline preparation and available throughout the pipeline. Either a new section to load configuration files or enhance parameters to load value (JSON) from a file.

image
image.png739Ă—122 4.91 KB

Looking forward to hear from SnapLogic team.

Regards,
Sripathi

#2

You can try to use “Expression Library” . Hope you are looking for the same thing. Here you can build your json and pass to pipeline like below in pipeline properties.

image
image.png757Ă—136 4.1 KB

1 Like
#3

Thanks @Surendar. This looks promising, we will check if it meets all our need.

Regards,
Sripathi

#4

My approach to this has been to use a 2nd pipeline with a FileReader and PipelineExecute.

The FileReader reads in the config file, then the PipelineExecute takes the input variables and makes them Parameters for the pipeline that’s actually running your workflow. Then the fields are available throughout as parameters.

Also, just a consideration, if you use Expression Libraries it looks like you need to edit the pipeline to make a change. If you don’t have Write access to your prod environment, then you’ll have to go through change management if your mappings, file paths, values, etc change. If you use this approach to pass in the parameters, you’ll keep control of these changes via the config file on the SFTP.

#5

Thanks Chris. For Expression libraries are loaded to file system, you do not need to edit the pipeline, unless you are changing the file location. Your approach is OK for simple configuration, in most case we have very complex configuration and these cases Expression libraries comes in very handy.

Regards,
Sripathi

#6

Hi Sripathi

I have been working on a solution for loading complex config files, I would be interested in seeing how you are using expression libraries to load your config files. Could you share some more details?

Thanks

Trevor

#7

Hi Trevor,

Here’s a sample implementation.

Following config.expr loaded to pipeline,

image

And its values are used in a mapper as below,

image
image.png1205Ă—614 36.2 KB

Hope it gives an idea. Its adequately documented here https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/1438110/Expression+Libraries

Regards,
Sripathi

#8

Hi Sripathi,

Thanks for the sample, it mimics what I am doing and I am glad to see someone else using this technique successfully.

Cheers,

Trevor