Move data from Database to Kafka and then to Amazon S3 Storage

Contributed by @pkona

---

There are two pipelines in this pattern. The first pipeline extracts data from a Database and publishes to a Kafka Topic. The second pipeline consumes from the Kafka Topic and ingests into the Amazon S3 Storage by calling a third pipeline.

Publish to Kafka

Source: Oracle
Target: Kafka
Snaps used: Oracle Select, Group By N, Confluent Kafka Producer

Consume from Kafka to S3

Source: Kafka
Target: Pipeline Execute
Snaps used: Confluent Kafka Consumer, Mapper, Pipeline Execute (calling Write file to S3 pipeline)

Write file to S3

Source: Kafka
Target: Amazon S3 Storage
Snaps used: Mapper, JSON Formatter, File Writer

Downloads

Publish to Kafka.slp (5.4 KB)
Consume from Kafka to S3.slp (5.4 KB)
Write file to S3.slp (4.5 KB)