Forum Discussion

sravan's avatar
sravan
New Contributor II
3 years ago
Solved

Looping through Api call

Hi We have a requirement where, If we do an api call , we get response something like below
{
records_remaining : “200”,
records_processed: “100”,
failed: “0”
},
“entity”: [
"rec1: “abc”,
“rec2”:“xyz”
]

Pagination does not support by Api . we need to wait for 10 sec for every api call until we get records_remaining is 0

for example https:abc.com/8080/getfullload?batch=50 -----> wait for 10 sec and do the same call until we get records remainining to 0
@bojanvelevski Any best way we can achieve this? please help!

  • bojanvelevski's avatar
    bojanvelevski
    3 years ago

    Hi @sravan,

    Even though the API does not support pagination, I think we can leverage of the pagination option in the Rest Get snap. There is a Pagination Interval option, which basically means “Wait for X seconds before doing the next request”.

    The logic would be something like:

    HasNext: parseInt($records_remaining) > 0
    NextURL: https:abc.com/8080/getfullload?batch=50

    Let me know if this helps,

    Regards,
    Bojan

3 Replies

  • tlikarish's avatar
    tlikarish
    Employee
    5 years ago

    I don’t think this is documented in the error pipeline documentation, but this is actually the expected behavior. Say you’re processing 500 documents and the 100th document in is munged and will result in an error when processing. A user may still want to execute on the other 499 valid documents and record which document is causing an issue. Additionally teams generally want to standardize on the error handling, so error pipelines allow all pipelines to process errors in a standard way.

    When viewing a pipeline configured with an error pipeline, you can view the errors by looking at the pipeline execution stats. These show how many documents pass through the output and error view respectively.

    As @cjhoward18 mentioned, you can add an Exit snap to the error pipeline if you want the parent pipeline to fail and stop processing. Here’s an example of that if you want to try it out.

    projects_error-pipeline-example.zip (2.5 KB)

    You can also modify the error handling behavior of each snap individually.

    Hope this helps clarify and give you some starting points when building out the error handling in your integrations.

    Best