Reliable, High-Throughput Batching with the Kafka Consumer Snap

[ptaylor]

In this article just published on Medium, we take a closer look at the (Confluent) Kafka Consumer Snap’s new Output Mode setting, and how it can be used to achieve reliable, high-throughput performance for some common use cases where it’s important to process records in batches.

Here are the release notes for the 423patches7900 version where this feature was introduced.

[rvadde]

Hi Patrick,

Can we get the SLP of this pipeline.

Thanks,
Ranjith

[ptaylor]

Yes, here you go:

Kafka Consumer Output Mode_2022_02_16.slp (10.1 KB)

[rvadde]

Please share below pipeline with child pipeline

[ptaylor]

Yes, here’s the more complicated example:

child:
ProcessBatch_2022_02_16.slp (12.5 KB)

parent:
KafkaBatchesMain_2022_02_16.slp (8.2 KB)

Change the value of the Topic and OutputPath parameters in the parent pipeline. Make sure the directory referenced by OutputPath exists.