Home > Gibbs sampling
In mathematics and physics,Gibbs sampling is an algorithm to generate a sequence of samples from the joint distribution of two or more variables.
The purpose of such a sequence is to
approximate the joint distribution (as with a histogram),
or to compute an integral (such as an expected value).
Gibbs sampling is a variation on the Metropolis-Hastings algorithm,
and thus it is an example of a Markov chain Monte Carlo algorithm.
The algorithm is named after the physicist J.W. Gibbs, in reference to an analogy between the sampling algorithm and statistical physics. The algorithm was devised by Geman and Geman (citation below), some decades after the passing of Gibbs, and is also called the Gibbs sampler.
Gibbs sampling is applicable when the joint distribution is not known explicitly,
but the conditional distribution of each variable is known.
The Gibbs sampling algorithm is to generate an instance from the distribution of each variable in turn,
conditional on the current values of the other variables.
It can be shown (see, for example, Gelman et al.) that the sequence of samples comprises a Markov chain,
and the stationary distribution of that Markov chain is just the sought-after joint distribution.
Gibbs sampling is particularly well-adapted to sampling the posterior distribution of a Bayesian network,
since Bayesian networks are typically specified as a collection of conditional distributions.
BUGS (link below) is a program for carrying out Gibbs sampling on Bayesian networks.
1 References
- George Casella and Edward I. George. "Explaining the Gibbs sampler". The American Statistician, 46:167-174, 1992. (Basic summary and many references.)
- A.E. Gelfand and A.F.M. Smith. "Sampling-Based Approaches to Calculating Marginal Densities". J. American Statistical Association, 85:398-409, 1990.
- Andrew Gelman, John B. Carlin, Hal S. Stern, and Donald B. Rubin. Bayesian Data Analysis. London: Chapman and Hall. First edition, 1995. (See Chapter 11.)
- S. Geman and D. Geman. "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images". IEEE Transactions on Pattern Analysis and Machine Intelligence, 6:721-741, 1984.
- C.P. Robert and G. Casella. "Monte Carlo Statistical Methods" (second edition). New York: Springer-Verlag, 2004.
2 External links
Probability and statisticsSee the separate articles on probability or the article on statistics. Statistical analysis depends on the characteristics of particular probability distributions, and the two topics are normally studied together. See also List of probability topics List Numerical analysisNumerical analysis is that branch of applied mathematics which studies the methods and algorithms to find (approximate) numerical solutions to various mathematical problems, using a finite sequence of arithmetic and logical operations. Most solutions of n
