Order of reponses when using pagination with http client

Question

Hello!I need to fetch a large number of values from an API and retrieve them sorted. The API allows me to specify that the results are sortef and the pagination appears to be working fine. However, I have noticed that I sometimes get errors downstream because the results are in fact not sorted correctly.I have not fully analyzed the problem but one possible theory I have is that the http client does not wait for a reply before sending the request for the next page. This would mean that a page of results could be received out of order.Can anyone confirm whether the paginated responses to the http client are guaranteed to be in the same order as the requests?Thanks very much!

janwyl · Answer

Sorry I should have been clear that I have to use a POST to get the sorted results from the API and specify the sort parameters in the request body.As a result I have been using this model&nbsp;https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/3097822264/PaginationSo it's not really a question about the http client snap I guess. My question is whether this model returns results in the order of the requests.

koryknick · Answer

janwyl&nbsp;- The HTTP Client and REST Post snaps are both synchronous, so they won't call for the next page until the page it receives has been passed to the next snap.&nbsp; My guess is that the API you're calling is using a different collation sequence than the one SnapLogic uses.&nbsp; There are similar known issues when reading from some databases, such as Microsoft SQL Server, which uses a slightly different collation sequence and sorts certain characters in a different order than SnapLogic expects.&nbsp; You are typically able to change the collation sequence at the session or query level in most databases.&nbsp; Unfortunately, you may not be able to do the same with the API.
My recommendation is to throw a Sort snap in after splitting out the API results to ensure proper ordering for SnapLogic.
Hope this helps!

janwyl · Answer

Thanks&nbsp;koryknick&nbsp;Good to have the confirmation on thesynchronous nature of HTTP Client and REST POST snaps.However I would be surprised if sort collation was the problem. One example is where&nbsp;3210448-USM18 came before&nbsp;3210448-NYR6 in a supposedly ascending sort. That would suggest a sort collation that puts U before N which seems unlikely.As I mentioned above, the model I am using is the one given here&nbsp;https://docs-snaplogic.atlassian.net/wiki/spaces/SD/pages/3097822264/Pagination. (Because as per that scenario I need to send sort criteria in the body of the POST for each call.)That model does not really use the pagination functionality of the HTTP Client snap. Instead it creates a list of parameters to send, sends that list to an HTTP Client snap, and then combines the responses using a Union snap.I note here https://community.snaplogic.com/t5/designing-and-running-pipelines/union-preserve-source-data-sequence/m-p/20480 that the Union snap doesn't guarantee preservation of order. Is it possible that in my case the responses are coming back from the API in an order different to the sending order (perfectly possible because of varying latency), and then the response are being presented in that order?If that is the case, then I think the proposed model has a serious flaw. The idea as stated is to be able to specify e.g. sort criteria in the POST body, but if the use of the union means the results aren't necessarily sorted then that's no good.In my case the whole point of specifying the sort in the API criteria is so that I can avoid using a Sort in the pipeline. I need to do a join later on a large volume and I run into memory issues if I have to sort.Any thoughts ? Thanks for your input - much appreciated !

koryknick · Answer

janwyl&nbsp;- Actually the example you gave almost proves to me that it's the sort collation sequence - the API is sorting the hyphen character differently than SnapLogic.&nbsp; One quick check you can perform is to Copy the stream and save the results after the API, then run the other side of the Copy through a Sort snap and save those results and compare for yourself.&nbsp;&nbsp;

janwyl · Answer

The error I had was this :importId is the value I have fetched from the API (in ascending order) and I am joining with other data using inner join and ascending sort order. I interpret this to mean that the error was thrown because the importId of&nbsp;3210448-USM18 came before&nbsp;3210448-NYR6 in the join.I have just twice fetched only the importIds from the API (without doing the join) and both times&nbsp;3210448-NYR6 came before&nbsp;3210448-USM18.In addition, every single value of importId takes the form of 7 numbers followed by a hyphen followed by some other combination of letters and characters. My understanding of sort collation may not be sufficiently advanced so apologies if I'm on the wrong track - given that format I'm not sure how sort collation could have an impact. Even if one collation ignores the hyphen it's the same for every single value?Thanks again for your thoughts !

Forum Discussion

Order of reponses when using pagination with http client

Recent Discussions

Pagination and nextCursor in header

Javascript to promote top level lists

Google Sheets Subscribe questions

Basic string transformations not working

Can we generate XML file in pretty print format using native snapLogic snaps?