Mutual information —— 读A visual

互信息，其实对应地是联合概率分布中的交集，用韦恩图来表示：

从概率论可以得知P（A∩B）= P（A）+ P（B） - P（A∪B）

所以对每一个事件，可以得到point-wise mutual information：

正好对应概率公式。此外：

相当于是，得知了y，未知的就只剩下x|y，或者说y带来的信息就是两者的交集部分P（x∩y）。而且可以看出来，这里x和y是对偶的。

扩充一下到所有的x,y情况，得到：

作者举了一个形状和颜色的例子：

数字角度：

When I(X; Y) = 0 bits, observing the object’s shape does not allow us to eliminate any possibilities of the object’s color. A mutual information of 0 between two random variables is equivalent to those two random variables being independent. Alternatively, when I(X; Y) = 1 bit, after observing the shape only, we can eliminate two of the four possible colors. Finally, when I(X; Y) = 2, we can eliminate 3 of the four possible colors and know exactly what color it is. In this case, we have gained all 2 bits of information that were present in X.

引申开来：

Since we can at most gain 2 bits from observing 1 of 4 possible shapes (i.e. Hmax(Y) = 2), if there were more than 2 bits of information in X, we wouldn’t be able to determine color exactly. For example, if there were 8 possible colors that were all equally probable, we could not uniquely identify all possibilities based on 4 possible shapes.