Caching in Snaplogic

[AdityaReja]

I’m designing a pipeline to retrieve data from oracle database. Hence i want to implement caching, in order to increase the performance in subsequent executions. Is there any way to do this ?

[aleung]

AdityaReja:

esigning a pipeline to retrieve data from oracle database. Hence i want to implement caching, in order to increase the performance in subsequent executions. Is there any wa

Depends on your use case. Do you mind providing more details?

[AdityaReja]

The pipelines will extract data from source databases need not be oracle and put into target tables. Now this process will be repeated numerous time to keep the target up to date. Hence we are planning to implement caching (sort of middle ware logic), which will speed up this process and also we don’t have to hit the source databases every time. Overall goal is to speed up the process of extracting and loading data !!

[apham]

We have similar need to improve performance - in our case, the data can come from various applications.

We have developed a group of shared pipelines to implement the caching functionality. In-memory Derby database is used to store the data. Users don’t see the database however - they store/retrieve JSON objects, 1 or more of whose elements are registered as keys for lookup.

[AdityaReja]

would you mind sharing a copy of one of ur pipeline