1.RESP协议 (Redis遵循)
2.AMQP协议 (RabbitMQ遵循)
3.Kafka通信协议(自定义)

1.RESP协议(Redis Serialization Protocol)

client-server 模式
https://redis.io/topics/protocol (官网)

// RESP 概述  (REdis Serialization Protocol)
redis-client 与 redis-server 之间的通信称之为 RESP.
client 发送特定的数据类型 request, server 产生特定数据类型的 response.
RESP 是二进制安全的, 因为其采用特定前缀来传输数据.
注意: 这里指的是 client-server 模式下, cluster 使用不同的二进制协议进行节点之间的信息交换.

// 请求-响应模型 (Request-Response model)
除了下述两种情况外, Redis protocol 都是简单的 request-response protocol:
1) redis 支持 pipelining. 所以其支持一次发送多条命令, 然后等待响应.
2) 当 redis-client 订阅 pub/sub 通道时, redis-client 不再需要发送命令, redis-server 也会自动推送消息给 redis-client.

// RESP 详述
RESP 是一种序列化协议, 支持的数据类型有: 
Simple Strings, Errors, Integers, Bulk Strings and Arrays.
redis 中如何使用 RESP 作为 request-response protocol 的呢?
1) redis-client 发送 a RESP Array of Bulk Strings 到 redis-server.
2) redis-server 根据命令实现响应 RESP types, 
其响应通常以 "\r\n" (CRLF, 即回车换行符) 结尾, 
其响应的第一个字符随着数据类型的不同而不同:
>> For Simple Strings the first byte of the reply is "+"
>> For Errors the first byte of the reply is "-"
>> For Integers the first byte of the reply is ":"
>> For Bulk Strings the first byte of the reply is "$"
>> For Arrays the first byte of the reply is "*"
>> For null value is Nil

2.AMQP协议 (Advanced Message Queuing Protocol)

http://www.amqp.org/specification/0-10/amqp-org-download (官网)
client-server(broker) 模式

AMQP1-0.10版本目录.png

2.1 概述

概述

AMQP协议用于 clients 和 消息中间件 servers(brokers) 之间进行信息交互.
是一种二进制协议, 具有多通道, 异步, 安全, 有效等特征.

AMQP 协议分为三层

>> AMQP model layer 
定义一组代表应用程序执行有用工作的命令（分组为逻辑功能类）。 
该层目的:
1. To guarantee interoperability between conforming implementations.
2. To provide explicit control over the quality of service.
3. To support any middleware domain: messaging, file transfer, streaming, RPC, etc.
4. To accommodate existing open messaging API standards (for example, Sun's JMS).
5. To be consistent and explicit in naming.
6. To allow complete configuration of server wiring via the protocol.
7. To use a command notation that maps easily into application-level API's.
8. To be clear, so each operation does exactly one thing.

>> AMQP session layer 
通过响应、同步和错误处理提供从应用程序到服务器的可靠传输的命令。

>> AMQP transport layer 
提供帧、信道复用、故障检测和数据呈现功能。 该层目的: 
1. To be compact, using a binary encoding that packs and unpacks rapidly.
2. To handle messages of any size without significant limit.
3. To permit zero-copy data transfer (e.g. remote DMA).
4. To carry multiple sessions across a single connection.
5. To allow sessions to survive network failure, server failover, and application recovery.
6. To be long-lived, with no significant in-built limitations.
7. To be asynchronous.
8. To be easily extended to handle new and changed needs.
9. To be forward compatible with future versions.
10. To be repairable, using a strong assertion model.
11. To be neutral with respect to programming languages.
12. To fit a code generation process.

AMQP三层.png

2.2 AMQP Model layer

AMQP Model diagram.png

2.2.1 server

可以把消息中间件 server 看做是 data server, 主要做2件事:
>> 根据任意标准将message路由到不同的consumers.
>> 当consumers不能及时的消费message时, 将message放到内存或磁盘里.

2.2.2 Virtual Hosts

Virtual Hosts 包括其命名空间, Exchanges, Message Queues, 
以及相关联的对象, 每一个 client 发起的 connection 必须指定唯一一个 Virtual Hosts.
client 在 authentication 之后, 将会选择一个 virtual host, 
只能在一个 virtual host 中建立 channel 并进行通信, 不能与其他 virtual host 建立通信连接.

2.2.3 Exchanges

Exchange 从生产者处接收 messages, 并根据预先设定的标准(bindings),
将messages路由/转发到对应的message queue中.
Exchanges不存储messages.
Exchanges有多种类型, 每一种类型实现了不同的路由算法.

2.2.3.1 Types of Exchange

1.The Direct Exchange Type
2.The Fanout Exchange Type
3.The Topic Exchange Type
4.The Headers Exchange Type
5.The System Exchange Type
6.Implementation-defined Exchange Types 

其中:
1,2,3,4 是常用的 Exchange. 
5为留给 AMQP 服务器使用的 Exchange.
6不是标准的 Exchange.

1.The Direct Exchange Type

是默认的 Exchange 类型. 其工作原理为(完全匹配才发消息):
1. A message queue is bound to the exchange using a binding key, K.
2. A publisher sends the exchange a message with the routing key R.
3. The message is passed to all message queues bound to the exchange with key K where K = R.

2.The Fanout Exchange Type

其工作原理为(发送消息到所有message queue中):
1. A message queue is bound to the exchange with no arguments.
2. A publisher sends the exchange a message.
3. The message is passed to the all message queues bound to the exchange unconditionally.

3.The Topic Exchange Type

其工作原理为 (正则匹配后才发消息):
1. A message queue is bound to the exchange using a binding key, K.
2. A publisher sends the exchange a message with the routing key R.
3. The message is passed to the all message queues where K matches R

4.The Headers Exchange Type

其工作原理为 (headers匹配后才发消息):
1. A message queue is bound to the exchange with a table of arguments containing the headers to be matched for that
binding and optionally the values they should hold.
2. A publisher sends a message to the exchange where the 'headers' property contains a table of names and values.
3. The message is passed to the queue if the headers property matches the arguments with which the queue was bound.

5.The System Exchange Type

其工作原理为:
1. A publisher sends the exchange a message with the routing key S.
2. The system exchange passes this to a system service S.
以“amq”开头的系统服务保留给AMQP使用。
brokers 可以自由使用所有其他名称的 exchanges。

6.Implementation-defined Exchange Types

所有的非标准的 exchanges 必须以 "x-" 开头.

2.2.3.2 Exchange Life-cycle

#内置 Exchanges
每一个 AMQP server (即broker)启动的时候, 都会创建一些以 "amq." 开头的 无法被删除的Exchanges.

#自定义的 Exchanges
当然 AMQP 应用 (如 producer, consumer) 也可以创建 Exchanges, 注意: 
这里用的是 declare 命令(如果不存在, 则创建; 否则, 继续), 不是 create 命令.

2.2.4 Message Queues

message queue 是一个命名缓冲区，代表一组 consumer 应用 保存消息。 
应用可以在自身权限范围内, 自由的 create, share, use, and destroy message queue.

message queue 提供 FIFO (先进先出) 的顺序保证, 即来自于同一个 producer 的
同等优先级的 message, 将会被先后传送给 consumer.

2.2.4.1 Message Queue Properties

当一个 client 创建一个 message queue 时, 可选一些属性:
1. name
Generally, when applications share a message queue they agree on a message queue name beforehand.
2. durable 
If specified, the message queue remains present and active when the server restarts. 
It may lose nonpersistent messages if the server restarts.
3. auto-delete
If specified, the server will delete the message queue when all clients have finished using it, or shortly thereafter.

2.2.4.2 Queue Life-cycles

#三种情况(前两种较多, 第三种是变种):

#Durable message queues
当有许多consumers时, 不同consumers之间对消息的持有是独立的.
不管有没有consumer订阅消息, messages queue 将会持续保持并接收消息.

#Temporary message queues
只能绑定到一个consumer, 当这个consumer取消订阅时, 这个message queue将会被删除.

#shared message queues
当然, 也存在一些变种, 当最后一个consumer取消订阅的时候, message queues 将会被删除.

Temporary message queue的创建和删除过程.png

2.2.5 Bindings

Bindings 是 Exchange 与 message queues 之间联系的桥梁.
bindings 定义了一些路由参数, 从而决定了来自 exchanges 的哪些 messages 将会被 message queues 获取.

Bindings 的生命周期依赖于 Exchange 与 message queues, 
当 exchange 或者 message queues 被销毁的时候, bindings 将会被销毁.

// 表述一个 binding 的伪代码是:
Exchange.Bind <exchange> TO <queue> WHERE <condition>

2.2.6 Messages

message 是路由和队列的原子单位. 
message 由 header(包含一系列属性)和body(包含二进制数据)组成.
messages 可以通过持久化来保证数据安全.

messages 也可以有不同的优先级, 在同一个 message queues 中:
当需要发送messages时, 高优先级的messages将会先被发送;
当消息需要被丢弃时, 低优先级的messages将会先被丢弃.

2.2.6.1 Flow Control

流控制可用于匹配消息发送到receiver上可用资源的速率。 
receiver可以是从producer接收message的AMQP服务器, 
也可以是从AMQP服务器接收message的consumer.
简单说, 就是 flow control 使用 credit 的概念来判断在某个时间点, 
不耗尽receiver资源的情况下, 每次发送多少消息.
一旦messages发送完成, credit将会被删除.
Pre-fetch buffers may be used at the receiver to reduce latency.

2.2.6.2 Transfer of Responsibility

message发送(producer-->AMQP server, AMQP server --> consumer)后, 
AMQP对于message的确认机制有两种:
>> receiver 接收message后, 必须给sender反馈
>> receiver 不必反馈, sender发完message即任务receiver已接收.

2.2.7 Subscriptions

消息订阅者, 即consumer.

2.2.8 Transactions (事务)

AMQP defines two separate transaction models, 
a one-phase commit transaction model (known as tx) and 
a two-phase commit model for distributed transactions (known as dtx).

2.2.9 Distributed Transactions (分布式事务)

The following diagram illustrates a messaging scenario where 
an application "Application" transactionally consumes a message from a queue Q1 
(using transaction T1 achieved through the transaction manger TM). 

Based on the consumed message, 
the application updates a database table Tb using DBMS and 
produces a message on queue Q2 on behalf of transaction T1.

Distributed Transactions.png

2.3 Sessions

3.Kafka通信协议

https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol (官方wiki)
https://www.jianshu.com/p/e921628bf461 (中文译文)