Forum Discussion

aanchalrajpal's avatar
aanchalrajpal
New Contributor
8 years ago

Kafka Consumer Snap

Hi I wanted to read all the records from the kafka queue in first run and then latest records from the queue from kafka consumer snap.I start reading for the first time from the queue,what setting ...
kzhou's avatar
kzhou
Former Employee
8 years ago

Hi,
Based on your scenario, when first run the consumer snap, you may use a new group ID in the Consumer, and set ‘auto offset reset’ to earliest. This would let the Consumer to read from earliest at the first running (when initialize the new group). For ‘seek type’, you may set it to End, so that it would continue reading when resume. Hope it helps 🙂

GBekkanti's avatar
GBekkanti
New Contributor III
7 years ago

Hi @kzhou,

Very thanks for your above statements. Those will be work irrespective of partition number right?

Please confirm the same. Thanks in advance!

  • kzhou's avatar
    kzhou
    Former Employee
    7 years ago

    @GBekkanti, sorry for the late response. The offset in Kafka is a per-partition based, and the configuration ‘auto offset reset’ and ‘seek type’ would be in the partition where the Consumer is assigned to.