Forum Discussion

amit_saroha's avatar
amit_saroha
New Contributor III
3 years ago

How to increment a date

Hi All,

I have a pipeline date parameter with timestamps and based on this date I am processing some records which I change manually as of now but the requirement is to schedule the pipeline.

I need your help in achieving this date change automatically when I schedule the pipeline. How I can put logic to dynamically change this date parameter?

example -

The pipeline is scheduled and the first run starts on at 01-Aug-2022 12:00:00, and the second run starts on at 01-Aug-2022 01:00:00 but in this run date parameter value must be its previous run i.e. 01-Aug-2022 12:00:00, and the thrid run starts on at 01-Aug-2022 02:00:00 but in this run date parameter value must be its previous run i.e. 01-Aug-2022 01:00:00 and so on.

5 Replies

  • bojanvelevski's avatar
    bojanvelevski
    Valued Contributor
    3 years ago

    Hi @amit.saroha,

    What would happen if your ‘01-Aug-2022 02:00:00’ run fails? Next one will be at ‘01-Aug-2022 03:00:00’ but what date value should be passed? Again ‘01-Aug-2022 01:00:00’, or ‘01-Aug-2022 02:00:00’? Meaning, if the parameter should always be one hour less than the time of execution, you can use the .minusHours() function and pass 1 as an argument. But if you need more complex logic where you consider failures like the example I gave you than you should include some logging mechanism.

    Bojan

    • amit_saroha's avatar
      amit_saroha
      New Contributor III
      3 years ago

      Hi @bojanvelevski - Thanks for the inputs. Request you to share some examples if possible. It will help me understand what you are referring to.

      • darshthakkar's avatar
        darshthakkar
        Valued Contributor
        3 years ago

        @amit.saroha: @bojanvelevski is referring to a complete end to end cycle wherein if for some reason, the pipeline execution fails, what would be the default value that you would like for those new runs after the failed execution as the date will be different.

        I believe at this point, you’ll have to tell us what are your different scenarios so that we can help you building a complete pipeline.
        Thanks!

  • darshthakkar's avatar
    darshthakkar
    Valued Contributor
    11 months ago

    amit_saroha - If you're staging your data to a database, then add a field in your mapper with Date.now() and before the execution of your main pipeline, read that Date from database, put a filter snap and ensure that Date.now() > Date (from the database)

    Lemme know how it goes.

    Thank you.

     

    BR,
    Darsh