โ01-03-2018 12:58 AM
Iโm designing a pipeline to retrieve data from oracle database. Hence i want to implement caching, in order to increase the performance in subsequent executions. Is there any way to do this ?
โ01-03-2018 10:37 AM
Depends on your use case. Do you mind providing more details?
โ01-03-2018 09:05 PM
The pipelines will extract data from source databases need not be oracle and put into target tables. Now this process will be repeated numerous time to keep the target up to date. Hence we are planning to implement caching (sort of middle ware logic), which will speed up this process and also we donโt have to hit the source databases every time. Overall goal is to speed up the process of extracting and loading data !!
โ01-11-2018 04:11 PM
We have similar need to improve performance - in our case, the data can come from various applications.
We have developed a group of shared pipelines to implement the caching functionality. In-memory Derby database is used to store the data. Users donโt see the database however - they store/retrieve JSON objects, 1 or more of whose elements are registered as keys for lookup.
โ01-11-2018 09:11 PM
would you mind sharing a copy of one of ur pipeline