Shannon information

The Shannon information or surprisal of a discrete random variable $𝑋 : 𝜉 \to 𝑀$ is the random function info

𝐼 𝑋 : 𝑀 \to ℝ 𝑥 \mapsto - l o g 𝑏 𝑝 𝑋 (𝑥) = - l o g 𝑏 ℙ (𝑋 = 𝑥)

where $𝑏 = 2$ corresponds to the unit $S h$ , $𝑏 = 𝑒$ corresponds to the unit $n a t$ , and $𝑏 = 10$ corresponds to the unit $H a r t$ . Up to choice of $𝑏$ , the Shannon information function is the unique function satisfying the following properties:

If $𝑝 𝑋 (𝑥) = 1$ then $𝐼 𝑋 (𝑥) = 0$ , i.e. a certain event is perfectly unsurprising.
If $𝑝 𝑋 (𝑥 1) < 𝑝 𝑋 (𝑥 2)$ then $𝐼 𝑋 (𝑥 1) > 𝐼 𝑋 (𝑥 2)$ , i.e. the more unlikely an event the more surprising.
If two independent events are measured, the total Shannon information gained is the sum of the Shannon information of the individual events.

Proof

proof

develop | en | SemBr

Shannon information

Graph View

Backlinks