Using Expression library in pipeline consumes more CPU than expected

[abinayad]

Hi,

We have implemented a pipeline using expression library file to do a lookup to get the transformed values. The expression library file is loaded from server. It is observed that after using expression library file it consumes more CPU than expected. Hence, please suggest is this the expected behavior of using expression libraries.

Thanks,
Abinaya

[tstack]

abinayad:

It is observed that after using expression library file it consumes more CPU than expected.

What makes you say that?

Did you try some other approach that behaved differently?

[abinayad]

Hi @tstack

Yes. Previously we used In-Memory Lookup snap to do the lookup. To improve the performance we switched the design to use Expression libraries to achieve the same task. But after using expression library it is observed that it consumes more CPU comparatively though it helped in improving the performance.

[tstack]

abinayad:

But after using expression library it is observed that it consumes more CPU comparatively though it helped in improving the performance.

How are you measuring CPU consumption?

I think what you might be seeing is that both approaches use about the same amount of CPU. But, since the library version finishes earlier, the CPU usage is bunched up in that shorter amount of time and there is less pausing. In other words, if one version took 2 minutes and consumed 50% of the CPU over that time, while the other took 1 minute and consumed 100% of the CPU, they used the same amount of CPU in the end.

[SriVinayaga]

Thanks @tstack,

I can see this is common problem not only for Expr library, we have XSD validation for which, initially we were loading the complex schema from sldb which was making a pipeline to run over 40 to 50 secs, and that wasnt consuming server resources much. But when we changed that to load the schema from the server using file protocol, The performance was improvised drastically where time taken came down to 6 to 8 secs, but the CPU utilization bumped up from 6 to 7 % to 50 to 60%.

Is this the expected behavior of SL nodes?

Thanks and regards,
Sri