Forum Discussion
Hi @ptaylor,
Appreciate you in depth response.
I agree I should use “Snowflake bulk Load” snap with Kafka Consumer`s Batch Mode option, I usually Validate pipeline and then I select the $variable this ensures correctness, but the pipeline did not work last time, I re-validated and it start to working.
So then I replace snowflake insert to Snowflake bulk insert, but then again it started to timeout
So, I thought Bulk insert might work with “Kafka Consumer`s” Batch acknowledge property but it does not work, it gives Time Out.
I think I am missing something to add, can you please see the properties what I am missing here and please add pipeline as well if you are trying it out.
Attaching all 3 pipeline, I added Extra mappers etc for debug purpose only.
Bulk insert with Batch not Working Pipeline_2022_02_22.slp (17.3 KB)
Bulk insert not Working Pipeline_2022_02_22.slp (12.1 KB)
Working Pipiline_2022_02_22.slp (11.4 KB)
- dmiller3 years agoFormer Employee
I haven’t used HTTP Client much yet, so I’m not sure, but I know there a couple other posts on pagination in this Snap:
- ddellsperger3 years agoAdmin
With HTTP Client, we provide maybe a different way to do this, but it’s still possible though likely a bit more manual. The Http Client has a pagination section (which you can expand - see the screenshot below)
The best way to explain these options is that the “Has next” input should evaluate to a boolean to determine if there are more pages to get or not (so in your case, if$entities.length < 10
would likely be what you want to enter here, that would break the pagination if you have fewer than 10 entries in your output document. The “Total pages to fetch” would be a secondary block for stopping pagination, so if you wanted to get all pages, you’d never set the “Total pages to fetch” if you want to limit the number of pages to however many are available up to 10, you’d set that value to 10. Override URI is the uri you’d have to request to get the next page, in this case (for offset paging) you’d have to do some work to get this working, if your original request url is something likehttps://example.com/api/output
with query parameters ofcount
10
, then you could either change to “reuse request parameters” and define the new ones there, or change the full url completely to something like"https://example.com/api/output?count=10&offset=" + (snap.out.totalCount * 10)
. This would allow your offset to increase by 10 for every output document produced, etc. You could do this either in the “Override URI” or in the “Override parameters” to support this capability. You’ll somewhat note that these are both only visible when the checkbox is checked, so that will determine which process it takes. Typically, “Override URI” is useful for cursor-based pagination and “Override parameters” are useful for count-based pagination assuming you externalize all of your query parameters and the triggers are able to be set via the query parameters rather than path parameters.
- chmurray883 years agoNew Contributor II
How does the above work if there is no offset param? I'm following your post, but I'm not sure how to build the Next URI link when there is no offset option for me. I've been told the API I'm pulling from supports "cursor-style" pagination. No offset, limit, or start....
Any help on this would be greatly appreciated!
Chris
- ddellsperger3 years agoAdmin
Cursor pagination is actually typically much more straightforward, you should typically have a "next url" or "next page cursor" and if there's no next page, that should be null or empty, in that case your has next would be something like
$.entity.paging.nextCursor != null
and you would likely use an override query parameter with something likecursor
being equal to$.entity.paging.nextCursor
. Typically, cursor pagination uses the cursor as a query parameter after the initial page, you'd have to see how the documentation claims to pass in the cursor, though.