How to increment a date

[amit_saroha]

Hi All,

I have a pipeline date parameter with timestamps and based on this date I am processing some records which I change manually as of now but the requirement is to schedule the pipeline.

I need your help in achieving this date change automatically when I schedule the pipeline. How I can put logic to dynamically change this date parameter?

example -

The pipeline is scheduled and the first run starts on at 01-Aug-2022 12:00:00, and the second run starts on at 01-Aug-2022 01:00:00 but in this run date parameter value must be its previous run i.e. 01-Aug-2022 12:00:00, and the thrid run starts on at 01-Aug-2022 02:00:00 but in this run date parameter value must be its previous run i.e. 01-Aug-2022 01:00:00 and so on.

[bojanvelevski]

Hi @amit.saroha,

What would happen if your ‘01-Aug-2022 02:00:00’ run fails? Next one will be at ‘01-Aug-2022 03:00:00’ but what date value should be passed? Again ‘01-Aug-2022 01:00:00’, or ‘01-Aug-2022 02:00:00’? Meaning, if the parameter should always be one hour less than the time of execution, you can use the .minusHours() function and pass 1 as an argument. But if you need more complex logic where you consider failures like the example I gave you than you should include some logging mechanism.

Bojan

[amit_saroha]

Hi @bojanvelevski - Thanks for the inputs. Request you to share some examples if possible. It will help me understand what you are referring to.

[darshthakkar]

@amit.saroha: @bojanvelevski is referring to a complete end to end cycle wherein if for some reason, the pipeline execution fails, what would be the default value that you would like for those new runs after the failed execution as the date will be different.

I believe at this point, you’ll have to tell us what are your different scenarios so that we can help you building a complete pipeline.
Thanks!

[amit_saroha]

Thank you, I want to start the next pipeline from the last successful run onwards. There are no other scenarios.