Forum Discussion

sp41832's avatar
sp41832
New Contributor II
8 years ago

Pipeline design

Hi ,

I have requirement

  1. I read a database table and get a max date from the table and pass it on. E.g my max date is 2018-07-02
  2. now i need to process a file for each day bet ween max date from previous snap and today.
    E.g(max date is : 2018-07-02 and today date is 2018-07-10) so i need to process a snap for each date of 2018-07-02 ,2018-07-03,2018-07-04,2018-07-05…till 2018-07-10

i have a solution which is not very effective , I can create CSV file which contains all dates and use it to pass the date as parameter to downstream snaps.

Looking for suggestions.

4 Replies

  • Supratim's avatar
    Supratim
    Contributor III
    8 years ago

    @sp41832 Can you please be more specific why you need to execute each day wise, rather filter data based on max date?

    • sp41832's avatar
      sp41832
      New Contributor II
      8 years ago

      Hi,

      The reason i cannot do as suggested by you is, i need to prepare a table based where i track the performance of a product or a campaign based on each day.

  • sp41832's avatar
    sp41832
    New Contributor II
    8 years ago

    HI,

    Thank you, i have implemented a java script more or less similar to your script and call the down stream snaps for each day.
    Thank you