Subpipeline - Lookup

[IgnatiusN]

We have a requirement to do a lookup on a db table and get additional attributes. This lookup is being done in a sub-pipeline. We would ideally like to load the lookup data into memory and do an in-memory lookup instead of querying the database for each record.

Since the lookup happens in the sub-pipeline, everytime we do the lookup the data is populated again, is there a way to load reference data globally once for a session and do a in-memory lookup instead of reloading it for each record?

[nganapathiraju]

Did you try the In-Memory lookup snap?

[IgnatiusN]

Yes, we tried the in-memory lookup, but it looks like every time the sub-pipeline is executed the data in memory is being refreshed. Since the sub-pipeline is being called from the parent for each record, this is taking a long time.

[Piyush]

instead of sub-pipeline do a lookup in main pipeline and re-use the values e.g.

Here you can see I am doing lookup once (My SQl- Select). I am using this lookup for all records via In-Memory lookup snap