I want to call an external service in a pipeline execute that can only run a single call at a time, but each document I could conceivably call against it can actually be aggregated and passed in a batch call.
Is there any way to queue documents up and collapse them until the pipeline execute is finished running, then submit those documents to the pipeline execute?
Currently the only way I can think to do this is to simply group documents by a predefined number at a time, but the pipeline execute that’s sending data to the endpoint could conceivably support much more than the batch size at a time to hand off.
