Forum Discussion

adityamohanty's avatar
adityamohanty
New Contributor II
18 days ago
Solved

Pagination Logic Fails After Migrating from REST GET to HTTP Client Snap

Hello everyone, Three years ago, I developed a pipeline to extract data from ServiceNow and load it into Snowflake. As part of this, I implemented pagination logic to handle multi-page responses b...
  • ssapa's avatar
    ssapa
    13 days ago

    adityamohanty​ First, I’d like to understand the change you mentioned. After adding the condition $headers.get('link') != null, were you able to retrieve all the records? In other words, is your Snap performing pagination and fetching data from all pages?

    From what I understand, with the above change, your Snap might validate successfully, but it likely won’t perform pagination, since $headers.get('link') will always return null.

    As mentioned in my earlier comment, to enable pagination, you need to update the Has Next and Next URL expressions as shown below.

    Pagination → Has Next:

    isNaN($headers['Link'].match(/",<([^;"]*)>;rel="next",/))

    Override URI → Next URL:
    $headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/) ? $headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/)[1].replace(_servicenow_cloud_base_url, _servicenow_b2b_base_url) : null

    Regarding the sysparm_display_value parameter — by default, this value is set to false. If you’d like to return all columns (both displayValue and value), make sure to include the query parameter "sysparm_display_value=all".

    I’ve attached a sample pipeline created using our test ServiceNow endpoint. With this configuration, I was able to run the pipeline successfully with pagination enabled and also able to fetch both displayvalue and value columns.

ssapa's avatar
ssapa
Employee
17 days ago

adityamohanty​ There’s a difference in header naming between the REST and HTTP Client Snaps. The REST Snap converts all header names to lowercase, while the HTTP Client Snap preserves the original case. As a result, the link header appears as Link in the HTTP Client output.

Please replace $headers['link'] with $headers['Link'] in both the Has Next and Next URL fields. This should make the Snap work as expected.

adityamohanty's avatar
adityamohanty
New Contributor II
13 days ago

Thank You ssapa​ . 

Your approach didn’t work. However, I used SnapGPT, and it suggested the following changes, which resolved the issue:

Pagination → Has Next:

$headers.get('link') != null && $headers.get('link').match(/<[^>]+>;\s*rel="next"/) != null

Override URI → Next URL:

$headers.get('link') != null ? ($headers.get('link').match(/<([^>]+)>;\s*rel="next"/) ? $headers.get('link').match(/<([^>]+)>;\s*rel="next"/)[1].replace(_servicenow_cloud_base_url, _servicenow_b2b_base_url) : null) : null

Now, regarding the data output:

When I limit the number of columns, I receive the expected structure:

"short_description": {

  "display_value": "Test",

  "value": "Test"

}

But when I remove the column limit to retrieve all fields, the display_value and value fields are missing.The response only includes data even though I’ve included sysparm_display_value in the query parameters.

Any insights or suggestions would be greatly appreciated.

"short_description": {"Test"}

  • ssapa's avatar
    ssapa
    Employee
    13 days ago

    adityamohanty​ First, I’d like to understand the change you mentioned. After adding the condition $headers.get('link') != null, were you able to retrieve all the records? In other words, is your Snap performing pagination and fetching data from all pages?

    From what I understand, with the above change, your Snap might validate successfully, but it likely won’t perform pagination, since $headers.get('link') will always return null.

    As mentioned in my earlier comment, to enable pagination, you need to update the Has Next and Next URL expressions as shown below.

    Pagination → Has Next:

    isNaN($headers['Link'].match(/",<([^;"]*)>;rel="next",/))

    Override URI → Next URL:
    $headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/) ? $headers['Link'].match(/\",<([^;\"]*)>;rel=\"next\",/)[1].replace(_servicenow_cloud_base_url, _servicenow_b2b_base_url) : null

    Regarding the sysparm_display_value parameter — by default, this value is set to false. If you’d like to return all columns (both displayValue and value), make sure to include the query parameter "sysparm_display_value=all".

    I’ve attached a sample pipeline created using our test ServiceNow endpoint. With this configuration, I was able to run the pipeline successfully with pagination enabled and also able to fetch both displayvalue and value columns.