I have a pipeline that extracts data from Oracle table and write data to destination. Oracle select snap will read current day’s date based on a time field. When I am migrating this pipeline to PROD environment in the first run it should read entire data(historical data) and from second run it should read current day data. how to design this ?Thanks
@Siva_Venna You should have one file with date filed in SLDB. your pipeline should read the file and pass the date value to oracle snap and do the comparison like- date_field > oracle_date_field . After completion of your pipeline you should calculate the max value of oracle_date_field and save to the same file in SLDB.It will help you to extract the delta record from Oracle any time you want.
thank you @Supratim, as you have suggested that’s doable in DEV. I am concerned how do I achieve this in PROD environment where I won’t be able to modify the file or snap settings . I am looking for some configuration/design that doesn’t need manual intervention.
You don’t need to change the manually for every single execution.
When you migrate to prod first time also move this file with some old dates , rest of the time pipeline will update the latest time stamp on this file.
@Supratim attached pipeline.
full_incremental_load_2020_09_25.slp (16.5 KB)
[full_incremental_load_2020_09_25.slp|attachment]
Try this way
(upload://mpHeJ7IYL1IB9KyKwa2VlgD2FtY.slp) (19.7 KB)
