Forum Discussion

adityamohanty's avatar
adityamohanty
New Contributor II
18 days ago
Solved

Pagination Logic Fails After Migrating from REST GET to HTTP Client Snap

Hello everyone, Three years ago, I developed a pipeline to extract data from ServiceNow and load it into Snowflake. As part of this, I implemented pagination logic to handle multi-page responses b...
  • ssapa's avatar
    ssapa
    13 days ago

    adityamohanty​ First, I’d like to understand the change you mentioned. After adding the condition $headers.get('link') != null, were you able to retrieve all the records? In other words, is your Snap performing pagination and fetching data from all pages?

    From what I understand, with the above change, your Snap might validate successfully, but it likely won’t perform pagination, since $headers.get('link') will always return null.

    As mentioned in my earlier comment, to enable pagination, you need to update the Has Next and Next URL expressions as shown below.

    Pagination → Has Next:

    isNaN($headers['Link'].match(/",<([^;"]*)>;rel="next",/))

    Override URI → Next URL:
    $headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/) ? $headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/)[1].replace(_servicenow_cloud_base_url, _servicenow_b2b_base_url) : null

    Regarding the sysparm_display_value parameter — by default, this value is set to false. If you’d like to return all columns (both displayValue and value), make sure to include the query parameter "sysparm_display_value=all".

    I’ve attached a sample pipeline created using our test ServiceNow endpoint. With this configuration, I was able to run the pipeline successfully with pagination enabled and also able to fetch both displayvalue and value columns.

adityamohanty's avatar
adityamohanty
New Contributor II
13 days ago

Thank You ssapa​ . 

Your approach didn’t work. However, I used SnapGPT, and it suggested the following changes, which resolved the issue:

Pagination → Has Next:

$headers.get('link') != null && $headers.get('link').match(/<[^>]+>;\s*rel="next"/) != null

Override URI → Next URL:

$headers.get('link') != null ? ($headers.get('link').match(/<([^>]+)>;\s*rel="next"/) ? $headers.get('link').match(/<([^>]+)>;\s*rel="next"/)[1].replace(_servicenow_cloud_base_url, _servicenow_b2b_base_url) : null) : null

Now, regarding the data output:

When I limit the number of columns, I receive the expected structure:

"short_description": {

  "display_value": "Test",

  "value": "Test"

}

But when I remove the column limit to retrieve all fields, the display_value and value fields are missing.The response only includes data even though I’ve included sysparm_display_value in the query parameters.

Any insights or suggestions would be greatly appreciated.

"short_description": {"Test"}

ssapa's avatar
ssapa
Employee
13 days ago

adityamohanty​ First, I’d like to understand the change you mentioned. After adding the condition $headers.get('link') != null, were you able to retrieve all the records? In other words, is your Snap performing pagination and fetching data from all pages?

From what I understand, with the above change, your Snap might validate successfully, but it likely won’t perform pagination, since $headers.get('link') will always return null.

As mentioned in my earlier comment, to enable pagination, you need to update the Has Next and Next URL expressions as shown below.

Pagination → Has Next:

isNaN($headers['Link'].match(/",<([^;"]*)>;rel="next",/))

Override URI → Next URL:
$headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/) ? $headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/)[1].replace(_servicenow_cloud_base_url, _servicenow_b2b_base_url) : null

Regarding the sysparm_display_value parameter — by default, this value is set to false. If you’d like to return all columns (both displayValue and value), make sure to include the query parameter "sysparm_display_value=all".

I’ve attached a sample pipeline created using our test ServiceNow endpoint. With this configuration, I was able to run the pipeline successfully with pagination enabled and also able to fetch both displayvalue and value columns.