More detailed explanations are given in the KafkaConsumer API and constants are defined in ConsumerConfig API. Is there anything which drive us to a specific solution or it's something which is upon us. This method will be called when the commit request sent to the server has been acknowledged. But on restarting the consumer, it is not reprocessing the above . Consumer with many threads. auto_commit_interval_ms . To commit offsets asynchronously we can use following method of KafkaConsumer: This method commits offsets returned on the last poll (Duration) for all the subscribed list of topics and partition. For example, you set the interval to 10 seconds and consumer . cgf :. . The property auto.commit.interval.ms specifies the frequency in milliseconds that the consumer offsets are auto-committed to Kafka. Once you've enabled this, the Kafka consumer will commit the offset of the last message received in response to its poll () call. Automatic Commit The easiest way to commit offsets is to allow the consumer to do it for you. Those auto commits are done within poll () ( which is typically called in a loop ). Introduction to Apache Kafka: Kafka Producer & Consumer Ask Question Asked 4 years, 11 months ago. Note Configuration parameters are described in more detail at https://kafka.apache.org/documentation/#consumerconfigs It also interacts with the assigned kafka Group Coordinator node to allow multiple consumers to load balance consumption of topics (requires kafka >= 0.9.0.0). The connector uses this strategy by default if you explicitly enabled Kafka's auto-commit (with the enable.auto.commit attribute set to true ). For a simple data transformation service, "processed" means, simply, that a message has come in and been transformed and then produced back to Kafka. Consumer Offset no commit! · Issue #693 · segmentio/kafka-go - GitHub You can rate examples to help us improve the quality of examples. KafkaConsumer (kafka 2.2.0 API) 2.2.0 To Reproduce Steps to reproduce the behavior. Note that the first offset provided to the consumer during a partition assignment will not contain metadata. How does kafka consumer auto commit work? The consumer is not thread safe and should not be shared across threads. . KafkaConsumer#committed () method public Map<TopicPartition, OffsetAndMetadata> committed(final Set<TopicPartition> partitions) Gets the last committed offsets for the given partitions (whether the commit happened by this process or another).