How to access the expression library loaded in the parent pipeline from a child pipeline

Question

I have a scenario as explained below:
I have an expression library which is 176kb to compare with the static data(we have a performance issue with JSON Generator and then performing the join). Which needs to be accessible from the child pipeline which will be executed multiple times, since I have to pass the pipeline properties I cannot configure to reuse executions.
Since the child pipeline will be launching individual pipeline instances which is resulting in loading the expression library every time and causing the performance of Memory usage(avg of 40%).
Expression Library → compareandfetchfilter_new.expr (433 Bytes)
Parent pipeline → invoke-child-pipe-lib-with-170kb_2023_01_02.slp (4.2 KB)
Child Pipeline → compare and get result from Expression Library_2023_01_02.slp (208.9 KB)
Any suggestions would be appreciated.

aleksandar_a · Answer

Hi @patan,
I’ve managed to make it run using the Pipeline Parameter but I had to make small modification in the expression library itself.
Expression Library → 
compareandfetchfilter_new.expr (553 Bytes)
Next, in the child pipeline in the mapper snap swap lib.arrayLib.getorgId($) with eval(JSON.parse(_expLib).getorgId($))
Child pipeline → 
compare and get result from Expression Library_2023_01_03.slp (210.3 KB)
Let me know if this helps you.
BR,
Aleksandar.

patan · Answer

@AleksandarAngelevski
The Comparator object will have more documents to be compared than it has now.

patan · Answer

@AleksandarAngelevski
Right now we are using the JSONGenerator snap with in the child pipeline but it is causing the performance issues.
I used another library which contains only comparator array, and pass it as pipeline parameter:

And used the expression library in the child pipeline to perform the compare as suggested by you. but the performance is still not improved.

patan · Answer

@AleksandarAngelevski Thank you for your feedback. but this solution will not work as I have more than 1000 records to be compared. A ternary operator solution might not fit my scenario.

aleksandar_a · Answer

@patan What you want to say is that the Comparator object will contain more documents than it has right now (2)?

Forum Discussion

How to access the expression library loaded in the parent pipeline from a child pipeline

8 Replies

Recent Discussions

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?

Multipart Reader failure - 'content-type' was not found

Unable to write sldb file to file system