Basic Kafka 续

Basic Kafka 上次关注这个居然已经是近一年前了。

业务上需要处理：如果确保消费的消息没有处理完成的话，依然可以在下次继续使用？（sonar扫描任务记录到kafka，多个consumer从kafka读取出来进行处理。但有可能处理失败）

Basic concepts of Kafka

10分钟视频介绍，官网入门介绍、核心概念 读完这几篇就有个基本的概念

Kafka in a Nutshell——这篇还没读完。

Producer：写入消息到topic；
Consumer：读取topic的消息；读取的位置有offset维护（对应的哪个topci在哪个partions上的哪个位置）
Cluster：保存topic的消息。包含多个broker，每个broker将topic维护到多个partitions，每个partition是个完整的副本）

相当有一个server端，两个client端（P&C）

同一组内的consumer（目前业务的状态）会由Kafka维护以确保每个consumer不会读到重复的event/message

Consumer Group

• Consumers can join a group by using the samegroup.id.
• The maximum parallelism of a group is that the number of consumers in the group ← no of partitions.
• Kafka assigns the partitions of a topic to the consumer in a group, so that each partition is consumed by exactly one consumer in the group.
• Kafka guarantees that a message is only ever read by a single consumer in the group.
• Consumers can see the message in the order they were stored in the log.

Kafka - Understanding Consumer Group with examples

• If we have three partitions for a topic and we start three consumers for the same topic then each consumer is assigned one partition automatically.

• If we have three partitions for a topic and we start two consumers for the same topic then one consumer is assigned one partition and other is assigned two partitions.

• If we have three partitions for a topic and we start one consumer for the same topic then one consumer is assigned all three partitions.

• If we have three partitions for a topic and we start four consumers for the same topic then three of four consumers are assigned one partition each, and one consumer will not receive any messages.