โ08-26-2020 01:20 PM
I have a pipeline that extracts data from Oracle table and write data to destination. Oracle select snap will read current dayโs date based on a time field. When I am migrating this pipeline to PROD environment in the first run it should read entire data(historical data) and from second run it should read current day data. how to design this ?Thanks
โ08-27-2020 12:43 AM
@Siva_Venna You should have one file with date filed in SLDB. your pipeline should read the file and pass the date value to oracle snap and do the comparison like- date_field > oracle_date_field . After completion of your pipeline you should calculate the max value of oracle_date_field and save to the same file in SLDB.It will help you to extract the delta record from Oracle any time you want.
โ09-24-2020 04:54 AM
thank you @Supratim, as you have suggested thatโs doable in DEV. I am concerned how do I achieve this in PROD environment where I wonโt be able to modify the file or snap settings . I am looking for some configuration/design that doesnโt need manual intervention.
โ09-24-2020 06:06 AM
You donโt need to change the manually for every single execution.
When you migrate to prod first time also move this file with some old dates , rest of the time pipeline will update the latest time stamp on this file.
โ09-25-2020 01:51 AM
@Supratim attached pipeline.
full_incremental_load_2020_09_25.slp (16.5 KB)