How to access the expression library loaded in the parent pipeline from a child pipeline

Designing Pipelines
,
1

I have a scenario as explained below:

I have an expression library which is 176kb to compare with the static data(we have a performance issue with JSON Generator and then performing the join). Which needs to be accessible from the child pipeline which will be executed multiple times, since I have to pass the pipeline properties I cannot configure to reuse executions.

Since the child pipeline will be launching individual pipeline instances which is resulting in loading the expression library every time and causing the performance of Memory usage(avg of 40%).

Expression Library → compareandfetchfilter_new.expr (433 Bytes)
Parent pipeline → invoke-child-pipe-lib-with-170kb_2023_01_02.slp (4.2 KB)
Child Pipeline → compare and get result from Expression Library_2023_01_02.slp (208.9 KB)

Any suggestions would be appreciated.

2

Hi @patan,

I’ve managed to make it run using the Pipeline Parameter but I had to make small modification in the expression library itself.

Expression Library →
compareandfetchfilter_new.expr (553 Bytes)

Next, in the child pipeline in the mapper snap swap lib.arrayLib.getorgId($) with eval(JSON.parse(_expLib).getorgId($))

Child pipeline →
compare and get result from Expression Library_2023_01_03.slp (210.3 KB)

Let me know if this helps you.

BR,
Aleksandar.

2 Likes
3

@AleksandarAngelevski Thank you for your feedback. but this solution will not work as I have more than 1000 records to be compared. A ternary operator solution might not fit my scenario.

4

@patan What you want to say is that the Comparator object will contain more documents than it has right now (2)?

5

@AleksandarAngelevski

The Comparator object will have more documents to be compared than it has now.

1 Like
6

@patan Another approach that I can think of is to introduce separate Comparator Parameter in the child pipeline as following:

image
image1354×573 68.1 KB

And in the child pipeline in the Mapper Snap use the following expression:

lib.arrayLib.getorgId($, JSON.parse(_comparator)).

Parent →
invoke-child-pipe-lib-with-170kb_2023_01_03.slp (7.5 KB)

Child →
compare and get result from Expression Library_2023_01_03.slp (212.7 KB)

Expression Library →
compareandfetchfilter_new.expr (148 Bytes)

And this way you can exclude the Comparator object from the expression library.

Let me know if this approach is applicable in your scenario.

BR,
Aleksandar.

7

@AleksandarAngelevski
Right now we are using the JSONGenerator snap with in the child pipeline but it is causing the performance issues.
I used another library which contains only comparator array, and pass it as pipeline parameter:

image
image1051×122 12.4 KB

And used the expression library in the child pipeline to perform the compare as suggested by you. but the performance is still not improved.

1 Like
8

@patan - are you simply trying to lookup the details associated with a “lookup” identifier? If so, skip the child pipeline altogether and use an In-Memory Lookup snap.

Here is an example that I think meets your needs.
Community 15256 - compare and get result_2023_01_03.slp (206.3 KB)

Hope this helps!

9

Hi @patan ,

Good day, you can always have the option to stream the static reference by enabling the pass through option in a JSON generator snap, see updated snap for reference

compare and get result from Expression Librarys_2023_01_05.slp (213.0 KB)

image
image1581×603 60.4 KB

Thanks,
Alchemiz