Forum Discussion

Employee

5 years ago

Reliable, High-Throughput Batching with the Kafka Consumer Snap

In this article just published on Medium, we take a closer look at the (Confluent) Kafka Consumer Snap’s new Output Mode setting, and how it can be used to achieve reliable, high-throughput performan...

confluent_kafka

kafka

rvadde

New Contributor

4 years ago

Hi Patrick,

Can we get the SLP of this pipeline.

Thanks,
Ranjith

ptaylor

Employee

4 years ago

Yes, here you go:

Kafka Consumer Output Mode_2022_02_16.slp (10.1 KB)

Kafka Consumer Output Mode_2022_02_16.slp10 KB

rvadde
New Contributor
4 years ago
Please share below pipeline with child pipeline
- ptaylor
  Employee
  4 years ago
  Yes, here’s the more complicated example:
  
  child:
  ProcessBatch_2022_02_16.slp (12.5 KB)
  
  parent:
  KafkaBatchesMain_2022_02_16.slp (8.2 KB)
  
  Change the value of the Topic and OutputPath parameters in the parent pipeline. Make sure the directory referenced by OutputPath exists.
  ProcessBatch_2022_02_16.slp12 KB
  Kafka Batches Main_2022_02_16 (1).slp8 KB

Forum Discussion

Reliable, High-Throughput Batching with the Kafka Consumer Snap

Recent Discussions

Ingesting Data into Veeva Vault CRM via SnapLogic – Alternatives to SFDC Snaps

SFTP: Worked ok then settings changed not not working

SQS Consumer snap is ready all message at pipeline startup

Salesforce Update REST API Throughput

Using Windows Authentication with SQL Server Accounts