How to access the expression library loaded in the parent pipeline from a child pipeline

[patan]

I have a scenario as explained below:

I have an expression library which is 176kb to compare with the static data(we have a performance issue with JSON Generator and then performing the join). Which needs to be accessible from the child pipeline which will be executed multiple times, since I have to pass the pipeline properties I cannot configure to reuse executions.

Since the child pipeline will be launching individual pipeline instances which is resulting in loading the expression library every time and causing the performance of Memory usage(avg of 40%).

Expression Library → compareandfetchfilter_new.expr (433 Bytes)
Parent pipeline → invoke-child-pipe-lib-with-170kb_2023_01_02.slp (4.2 KB)
Child Pipeline → compare and get result from Expression Library_2023_01_02.slp (208.9 KB)

Any suggestions would be appreciated.

[AleksandarAngel]

@patan Another approach that I can think of is to introduce separate Comparator Parameter in the child pipeline as following:

And in the child pipeline in the Mapper Snap use the following expression:

lib.arrayLib.getorgId($, JSON.parse(_comparator)).

Parent →
invoke-child-pipe-lib-with-170kb_2023_01_03.slp (7.5 KB)

Child →
compare and get result from Expression Library_2023_01_03.slp (212.7 KB)

Expression Library →
compareandfetchfilter_new.expr (148 Bytes)

And this way you can exclude the Comparator object from the expression library.

Let me know if this approach is applicable in your scenario.

BR,
Aleksandar.

[patan]

@AleksandarAngelevski
Right now we are using the JSONGenerator snap with in the child pipeline but it is causing the performance issues.
I used another library which contains only comparator array, and pass it as pipeline parameter:

And used the expression library in the child pipeline to perform the compare as suggested by you. but the performance is still not improved.

[koryknick]

@patan - are you simply trying to lookup the details associated with a “lookup” identifier? If so, skip the child pipeline altogether and use an In-Memory Lookup snap.

Here is an example that I think meets your needs.
Community 15256 - compare and get result_2023_01_03.slp (206.3 KB)

Hope this helps!

[alchemiz]

Hi @patan ,

Good day, you can always have the option to stream the static reference by enabling the pass through option in a JSON generator snap, see updated snap for reference

compare and get result from Expression Librarys_2023_01_05.slp (213.0 KB)

Thanks,
Alchemiz