How to increment a date

[amit_saroha]

Hi All,

I have a pipeline date parameter with timestamps and based on this date I am processing some records which I change manually as of now but the requirement is to schedule the pipeline.

I need your help in achieving this date change automatically when I schedule the pipeline. How I can put logic to dynamically change this date parameter?

example -

The pipeline is scheduled and the first run starts on at 01-Aug-2022 12:00:00, and the second run starts on at 01-Aug-2022 01:00:00 but in this run date parameter value must be its previous run i.e. 01-Aug-2022 12:00:00, and the thrid run starts on at 01-Aug-2022 02:00:00 but in this run date parameter value must be its previous run i.e. 01-Aug-2022 01:00:00 and so on.

[darshthakkar]

@amit_saroha - If you're staging your data to a database, then add a field in your mapper with Date.now() and before the execution of your main pipeline, read that Date from database, put a filter snap and ensure that Date.now() > Date (from the database)

Lemme know how it goes.

Thank you.

BR,
Darsh