Forum Discussion

IgnatiusN's avatar
IgnatiusN
New Contributor II
8 years ago

Subpipeline - Lookup

We have a requirement to do a lookup on a db table and get additional attributes. This lookup is being done in a sub-pipeline. We would ideally like to load the lookup data into memory and do an in-memory lookup instead of querying the database for each record.

Since the lookup happens in the sub-pipeline, everytime we do the lookup the data is populated again, is there a way to load reference data globally once for a session and do a in-memory lookup instead of reloading it for each record?

3 Replies

  • IgnatiusN's avatar
    IgnatiusN
    New Contributor II
    8 years ago

    Yes, we tried the in-memory lookup, but it looks like every time the sub-pipeline is executed the data in memory is being refreshed. Since the sub-pipeline is being called from the parent for each record, this is taking a long time.

    • Piyush's avatar
      Piyush
      New Contributor III
      8 years ago

      instead of sub-pipeline do a lookup in main pipeline and re-use the values e.g.

      Here you can see I am doing lookup once (My SQl- Select). I am using this lookup for all records via In-Memory lookup snap