Contributed by @pkona
There are two pipelines in this pattern. The first pipeline extracts data from a Database and publishes to a Kafka Topic. The second pipeline consumes from the Kafka Topic and ingests into the Amazon S3 Storage by calling a third pipeline.
Publish to Kafka
Source: Oracle
Target: Kafka
Snaps used: Oracle Select, Group By N, Confluent Kafka Producer
Consume from Kafka to S3
Source: Kafka
Target: Pipeline Execute
Snaps used: Confluent Kafka Consumer, Mapper, Pipeline Execute (calling Write file to S3 pipeline)
Write file to S3
Source: Kafka
Target: Amazon S3 Storage
Snaps used: Mapper, JSON Formatter, File Writer
Downloads
Publish to Kafka.slp (5.4 KB)
Consume from Kafka to S3.slp (5.4 KB)
Write file to S3.slp (4.5 KB)
