How to Chunk data for REST PUT
Hello,
I am trying to source data from Oracle table to send the documents to REST PUT Snap. Typically the Oracle Fetch would return 1.7 million rows and by the time it sends all the 1.7 million for...
Forum Discussion
You can do this in multiple ways.
In Parent Pipeline you can do Shard Offsets and in Child Pipeline to can do fetch offset to query data and send it to Rest Put.
Also using pipeline Execute reuse and pool size is a good way to achieve parallelism and improves performance.
Using Pipeline Execute Snap with Re-use executions and Pool size needs to be set appropriately.
Useful ideas, but where do you set up a “Shard Offset” (in 33 years I’ve never heard the term), and where does the fetch offset go? And… Is the SQL query in the parent and the docstream passed to the child, or the other way around?
