The publish-subscribe model is also called the observer model (there are many people on the Internet saying that the two models are different, but I personally think there is not much difference). In the publish-subscribe model, there are mainly two parts.

It is publishing and subscribing, so the connection point between publishing and subscribing is topic. For example, when ordering takeout for lunch, Sister Yan (broker) posted two options in the takeout group.

restaurants, Ducheng and Huiji (this can be called the theme), Xiao Ming (Consumer) ordered Huiji’s, Brother Wen (Consumer) ordered Ducheng’s (this can be called a subscription), Ducheng Restaurant (producer) and Huiji

When the restaurant (producer) prepares the food, it will be delivered to the delivery boy (message agreement). After the food arrives at Sister Yan's place, Xiao Ming and Brother Wen can go to Sister Yan's place to pick it up (pull), or Sister Yan can deliver it to them.

(push).

This is the most common publish-subscribe model in our lives.

As can be seen from the above, the takeout group is a carrier (MQ) that carries the storage and transmission of messages. From here, the concept of message queue can be derived. Next, continue to talk about message queue.

MQ (Message Queue) is also called message queue. We all know the queue, but what is a message?

Messages refer to data transmitted between processes on the same machine, or between different machines.

To put it simply, the data carried by our Rpc request is a message.

This is the traditional communication model.

But this model has many flaws. For example, when the network is not good, this call may be lost.

Queues provide a one-step communication protocol, which means that the sender and receiver of a message do not need to be in contact with the message at the same time. The sender's message will be stored in the queue until the receiver gets it.

Generally, we call the sender of a message a producer and the receiver of a message a consumer.

Since producers and consumers are opaque, they are connected through the middle link - the queue. So in the queue, is the consumer or the producer the active one? In fact, it can be divided into different ways according to different ways of obtaining messages.

There are two types: pull or push.

Literally understood, pull means that consumers need to control themselves to pull messages from the queue, while push means that producers take the initiative and push the generated messages to consumers. This kind of push can be point-to-point or one-stop.

One-to-many, and this one-to-many mode is what we often call the broadcast mode. In distributed systems, message middleware is a very important component, which mainly solves problems such as application coupling, asynchronous messages, and traffic peak cutting.

Commonly used message queue middleware include activeMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, RocketMQ (refer to /s/ad7jibTb5nTzh3nDQYKFeg? I think this article is very well written and very detailed) This time I mainly write about kafka, the message middleware

, Kafka uses the pull mode to consume information. The producer puts the message into the queue, and the consumer can obtain the message through the epull method for consumption. Let’s first talk about a few key concepts of kafka. Kafka was originally developed by

Developed by Linkedin, it is a distributed, partitioned, multi-copy, multi-subscriber distributed log system based on zookeeper coordination (can also be used as an MQ system). It can be commonly used for web/nginx logs, access logs, and message services.

Wait, Linkedin was contributed to the Apache Foundation in 2010 and became a top open source project. The main application scenarios are: log collection system and messaging system.

The main design goals of Kafka are as follows: Support both offline data processing and real-time data processing.

A typical Kafka cluster contains several producers, several brokers, several consumers, and a Zookeeper cluster.

Kafka uses Zookeeper to manage cluster configuration, elect leaders, and rebalance when consumer groups change.

The producer uses push mode to publish messages to the broker, and the consumer uses pull mode to subscribe and consume messages from the broker.

Topic & Partition A topic can be considered as a first-class message. Each topic will be divided into multiple partitions. Each partition is an append log file at the storage level.