KL(P ‖ Q)
—
KL(Q ‖ P)
—
Units
nats
bits
P
Q — in P’s rank order
(P shown as line)
same axes on both — token at rank
i
sits at the same x in each pane ·
log y-axis
KL(P ‖ Q) contribution per rank: pᵢ·log₂(pᵢ/qᵢ) — sums to KL(P ‖ Q)
Elements N
40
Rank corr. ρ
ρ =
0.90
reshuffle
Distribution P
Entropy
2.2
nats
/3.7
=
Distribution Q
Entropy
2.2
nats
/3.7
Identical
Temperature
Reordered only
Sharp vs flat
Two models
Scrambled