Discrete-time Markov chain

Definition

In probability, a discrete-time Markov chain (DTMC) is a sequence of random variables, known as a stochastic process, in which the value of the next variable depends only on the value of the current variable, and not any variables in the past. For instance, a machine may have two states, A and E. When it is in state A, there is a 40% chance of it moving to state E and a 60% chance of it remaining in state A. When it is in state E, there is a 70% chance of it moving to A and a 30% chance of it staying in E. The sequence of states of the machine is a Markov chain. If we denote the chain by then is the state which the machine starts in and is the random variable describing its state after 10 transitions. The process continues forever, indexed by the natural numbers.

Related concepts

Absorbing Markov chain Balance equation Chapman–Kolmogorov equation Conditional probability Continuous-time Markov chain Countable set Detailed balance Directed acyclic graph Directed graph Equivalence class Equivalence relation Ergodic theory Eugene Seneta Expected value Exponent Exponential distribution Finite-state machine Geoffrey Grimmett Greatest common divisor Greg Lawler Hitting time If and only if Indexed family Inner product space Iverson bracket J. L. Doob J. Laurie Snell James R. Norris John G. Kemeny Kolmogorov's criterion Marginal distribution Markov Chains and Mixing Times Markov chain Markov chain Monte Carlo Markov property Matrix norm Maximal set Natural numbers Optimization problem Periodic function Phase-type distribution Probability Probability distribution Probability vector Random variable Society for Industrial and Applied Mathematics Stochastic matrix Stochastic process

13 concepts already in your glossary