Shannon entropy

The Shannon entropy or information-theoretic entropy of a discrete random variable $𝑋 : 𝜉 \to 𝑀$ is the Expectation of its Shannon information $𝐼 𝑋$ , info

𝐻 [𝑋] = 𝔼 [𝐼 𝑋 (𝑋)] = - \sum 𝑥 \in 𝑀 𝑝 𝑋 (𝑥) l o g 𝑏 𝑝 𝑋 (𝑥)

where $𝑏 = 2$ corresponds to the unit $S h$ , $𝑏 = 𝑒$ corresponds to the unit $n a t$ , and $𝑏 = 10$ corresponds to the unit $H a r t$ . Shannon entropy is closely related to thermodynamic entropy.

develop | en | SemBr

Shannon entropy

Graph View

Backlinks