cancel
Showing results for 
Search instead for 
Did you mean: 

Reliable, High-Throughput Batching with the Kafka Consumer Snap

ptaylor
Employee
Employee

In this article just published on Medium, we take a closer look at the (Confluent) Kafka Consumer Snap’s new Output Mode setting, and how it can be used to achieve reliable, high-throughput performance for some common use cases where it’s important to process records in batches.

OutputMode-OnePerBatch

Here are the release notes for the 423patches7900 version where this feature was introduced.

4 REPLIES 4

rvadde
New Contributor

Hi Patrick,

Can we get the SLP of this pipeline.

Thanks,
Ranjith

Yes, here you go:

Kafka Consumer Output Mode_2022_02_16.slp (10.1 KB)

rvadde
New Contributor

Please share below pipeline with child pipeline

image

Yes, here’s the more complicated example:

child:
ProcessBatch_2022_02_16.slp (12.5 KB)

parent:
KafkaBatchesMain_2022_02_16.slp (8.2 KB)

Change the value of the Topic and OutputPath parameters in the parent pipeline. Make sure the directory referenced by OutputPath exists.