Kullback–Leibler divergence

Definition

In mathematical statistics, the Kullback–Leibler (KL) divergence, denoted , is a type of statistical distance: a measure of how much an approximating probability distribution Q is different from a true probability distribution P. Mathematically, it is defined as

Related concepts

Absolute continuity Absolute value Absolutely continuous measure Additive map Advances in Mathematics Affine connection Akaike information criterion Almost everywhere Almost surely Alphabet (formal languages)Annals of Mathematical Statistics Approximation Asymmetry Base of a logarithm Bayes' theorem Bayesian experimental design Bayesian inference Bayesian information criterion Bayesian statistics Bhattacharyya distance Binomial distribution Bioinformatics Bit Bregman divergence Bretagnolle–Huber inequality Chain rule Change of variables Chapman & Hall Chi-squared test Coding theory Conditional entropy Conference on Neural Information Processing Systems Continuous random variable Convex function Counting measure Covariance matrix Cross-entropy Cross entropy D-optimal design Data compression Data differencing Density matrix Deviance information criterion Differential entropy Dimensional analysis Discrete probability distribution Discrete random variable Discrete uniform distribution Divergence (statistics)Dover Publications Dually flat manifold E.T. Jaynes E (mathematical constant)Earth mover's distance Einstein summation convention Entropic value at risk Entropy Entropy (information theory)Entropy coding Entropy encoding Entropy in thermodynamics and information theory Entropy maximization Entropy power inequality Event (probability theory)Evidence lower bound Exergy Expectation (statistics)Expectation–maximization algorithm Expected value Exponential families Exponential family F-divergence Feature selection Fisher information metric Fluid mechanics Gaussian measure Gibbs' inequality Gibbs free energy Gibbs inequality Haar measure Harold Jeffreys Hellinger distance Hellinger metric Helmholtz free energy Hessian matrix Hilbert space Huffman coding I. J. Good If and only if Independent random variables Inference Infinitesimal generator (stochastic processes)Information content Information entropy Information gain (decision tree)Information gain in decision trees Information gain ratio Information geometry Information projection Information theory and measure theory International Journal of Computer Vision Jacobian Jensen–Shannon divergence John Wiley & Sons Joint probability distribution Josiah Willard Gibbs Kolmogorov–Smirnov test Kraft–McMillan inequality Kronecker delta Laplace Large deviations Lebesgue measure Leonidas J. Guibas Lie group Limiting density of discrete points Logarithm Logit Loss function Lossless compression MAUVE (metric)Machine learning Marginal probability distribution Matching distance Mathematical proof Mathematical statistics Maximum likelihood Maximum likelihood estimation Maximum spacing estimation Mean squared deviation Measurable space Measure (mathematics)Metric (mathematics)Metric tensor Model selection Multivariate normal distribution Mutual information Nat (unit)Natural logarithm Neuroscience Neyman–Pearson lemma Non-negative Normal distribution Numerical Recipes Parameter space Partition function (mathematics)Partition of a set Patch (computing)Pinsker's inequality Poisson distribution Positive-definite matrix Posterior distribution Prefix-free code Principle of Insufficient Reason Principle of Maximum Entropy Prior distribution Prior probability distribution Probability Probability density function Probability distribution Probability measure Probability space Probability spaces Pythagorean theorem Quantum entanglement Quantum information science Quantum relative entropy Radon–Nikodym derivative Random variable Random variate Rate function Real number Richard Leibler Riemann hypothesis Riemannian metric Rényi divergence Sample space Self-information Sergio Verdú Shannon's entropy Shannon entropy Shun'ichi Amari Solomon Kullback Sphere Squared Euclidean distance Standard temperature and pressure Statistical Science Statistical classification Statistical distance Statistical divergence Statistical manifold Statistical model Sufficient statistic Surprisal Taylor series The American Statistician Time series Topology Total-variation distance of probability measures Total variation Triangle inequality Units of information Utility function Variation of information Variational inference Vincenzo Bonnici Work (thermodynamics)

14 concepts already in your glossary