An introduction to Shannon information and entropy, with word- and token-level measurements from a Wikipedia article.
An intuitive introduction to multivariate and conditional entropy, showing how mutual information measures the overlap between two random variables.
How cross entropy measures the cost of encoding one distribution with another’s codebook, and how KL divergence isolates the extra cost from a model’s inaccuracy.