Kafka

kafka是一个分布式的，可分区的，可备份的日志提交服务，它使用独特的设计实现了一个消息系统的功能。

相关术语

  1. Broker(代理): Kafka集群包含一个或多个服务器，这种服务器被称为broker。
  2. Topic(类别/主题) : 每条发布到 Kafka 集群的消息都有一个类别，这个类别被称为 Topic。（物理上不同 Topic 的消息分开存储，逻辑上一个 Topic 的消息虽然保存于一个或多个 broker 上，但用户只需指定消息的 Topic 即可生产或消费数据而不必关心数据存于何处）。
  3. Partition(分区)：Partition 是物理上的概念，每个Topic包含一个或多个 Partition。每个Partition都是一个排序的和不可变的消息队列，新消息都会append到结尾，每个message都会分配一个sequential id, 叫做offset，作为message的唯一的id

  4. Yahoo's Pulsar: a competitor

  5. Offset Commit : the action of updating the current postion in the partition a commit. How does a consumer commit an offset? it produces a message to kafka, to a special _consumer\_offsets topic, with the committed offset for each partition._

  6. As everything else in the consumer, the automatic commits are driven by the poll loop. Whenever you poll, the consumer checks if its time to commit, and if it is, it will commit the offsets it returned in the last poll.
  7. The consumer API allows you to run your own code when partitions are added or removed from the consumer. You do this by passing a ConsumerRebalanceListener when calling the subscribe() method.

References

  1. Kafka深度解析
  2. Active-Active for Multi-Regional Resiliency
  3. Multi Datacenter Replication with Apache Kafka
  4. Kafka mirroring (MirrorMaker)
  5. Kafka文件存储机制那些事
  6. Kafka protocol guide
  7. Tips for Improving Performance of Kafka Producer
  8. How to choose the number of topics/partitions in a Kafka Cluster?
  9. Kafka 0.9 Configuration Best Practices
  10. 万亿级数据洪峰下的分布式消息引擎

  11. Should you put several event types in the same Kafka topic (Klepmann)

  12. Putting Apache Kafka To Use: A Practical Guide to Building a Streaming Platform (Part 1)

  13. Putting Apache Kafka To Use: A Practical Guide to Building a Streaming Platform (Part 2)

  14. 17 个方面，综合对比 Kafka、RabbitMQ、RocketMQ、ActiveMQ

  15. RabbitMQ vs. Kafka

  16. 爱奇艺在日志实时数据监控的探索与实践

results matching ""

    No results matching ""