Forum Discussion

swatifunde's avatar
swatifunde
Contributor
3 years ago
Solved

How to iterate the pipeline

Hello Community,

I want to iterate the pipeline when expression get true,below is my current pipeline design

In this pipeline I am checking count of Null values of column if all values Null then pipeline wait for 20 Sec and again try to fetch data from system, If its not Null then simply load the data into Snowflake.
My requirement is that after 20 Sec pipeline should check if there is full data available or not in continues loop.

Can you please advise me here how can I iterate this pipeline

  • koryknick's avatar
    koryknick
    3 years ago

    @swatifunde - Here is an example that calls a triggered task until a desired result is returned. I’ve also attached the pipeline that it is calling so you can correlate the query results with the Has Next property in the REST Get.
    Community 13497 - Child - Query Snowflake Table_2022_09_28.slp (2.7 KB)
    Community 13497 - Poll Snowflake Table_2022_09_28.slp (6.6 KB)

    Please note that calling a triggered task on an extremely frequent schedule will create a large amount of noise in your Dashboard. It is possible you may also experience some occasional failures calling the task due to network traffic. If your org has Ultra Tasks enabled, this would resolve the additional traffic in Dashboard and has excellent response since it avoids instantiating a new pipeline with each call.

12 Replies

  • koryknick's avatar
    koryknick
    Employee
    3 years ago

    @swatifunde - Here is an example that calls a triggered task until a desired result is returned. I’ve also attached the pipeline that it is calling so you can correlate the query results with the Has Next property in the REST Get.
    Community 13497 - Child - Query Snowflake Table_2022_09_28.slp (2.7 KB)
    Community 13497 - Poll Snowflake Table_2022_09_28.slp (6.6 KB)

    Please note that calling a triggered task on an extremely frequent schedule will create a large amount of noise in your Dashboard. It is possible you may also experience some occasional failures calling the task due to network traffic. If your org has Ultra Tasks enabled, this would resolve the additional traffic in Dashboard and has excellent response since it avoids instantiating a new pipeline with each call.

    • swatifunde's avatar
      swatifunde
      Contributor
      3 years ago

      I tried to implement this solution but somehow I am getting below error

  • koryknick's avatar
    koryknick
    Employee
    3 years ago

    My pleasure! I’m happy you got a solution that works for you!

    • swatifunde's avatar
      swatifunde
      Contributor
      3 years ago

      koryknick when I am trying to test this pipeline with some dummy data then it is not giving expected response. when there is no data then also it get executed within 20 sec. I don’t think so it is looping there

  • koryknick's avatar
    koryknick
    Employee
    3 years ago

    @swatifunde - you could accomplish this using a child pipeline with the Select snap to get a count of null values and create a triggered task for it. Then call that triggered task via REST Get snap using pagination to continue calling until you get the response you want (i.e. Not Null) before completing the rest of the steps to load the data. Note that the REST Get will return documents even for null counts so you need to do something with that output - perhaps a Filter, Router, or Tail snap.

    • swatifunde's avatar
      swatifunde
      Contributor
      3 years ago

      Thank you for response, I am trying this method but I am not sure how to set pagination here, what will be value here

      In my design there is LOGOS column who will get all Null values if there is partial data on source and I have to keep checking for full data so can you advise me here

  • koryknick's avatar
    koryknick
    Employee
    3 years ago

    Did you update the URL expression in the Mapper snap to your triggered task url?

    • swatifunde's avatar
      swatifunde
      Contributor
      3 years ago

      its working now but can you just confirm on thing here

      I used Has Next parameter as per attachment so I want to know that until unless this condition get false it will continuously check the data on source system, right??

      • koryknick's avatar
        koryknick
        Employee
        3 years ago

        Yes - it will continually check until the Not_null value is non-zero and then it will stop calling URL.

  • koryknick's avatar
    koryknick
    Employee
    3 years ago

    @swatifunde - Please verify that your child pipeline returns the results you are expecting during your testing. Remember that the REST Get will complete as soon as the “Has next” condition is true; however, it will return whatever value the query returns every Pagination interval. If you don’t want to do anything until Not_null is 0, then you can use a Tail snap to wait for the last value (i.e. Not_null is zero).