digplanet beta 1: Athena
Share digplanet:

Agriculture

Applied sciences

Arts

Belief

Chronology

Culture

Education

Environment

Geography

Health

History

Humanities

Language

Law

Life

Mathematics

Nature

People

Politics

Science

Society

Technology

In statistics, Bayesian linear regression is an approach to linear regression in which the statistical analysis is undertaken within the context of Bayesian inference. When the regression model has errors that have a normal distribution, and if a particular form of prior distribution is assumed, explicit results are available for the posterior probability distributions of the model's parameters.

## Model setup

Consider a standard linear regression problem, in which for $i=1,...,n$ we specify the conditional distribution of $y_i$ given a $k \times 1$ predictor vector $\mathbf{x}_i$:

$y_{i} = \mathbf{x}_{i}^{\rm T} \boldsymbol\beta + \epsilon_{i},$

where $\boldsymbol\beta$ is a $k \times 1$ vector, and the $\epsilon_i$ is independent and identical normally distributed random variables:

$\epsilon_{i} \sim N(0, \sigma^2).$

This corresponds to the following likelihood function:

$\rho(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma^{2}) \propto (\sigma^{2})^{-n/2} \exp\left(-\frac{1}{2{\sigma}^{2}}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)\right).$

The ordinary least squares solution is to estimate the coefficient vector using the Moore-Penrose pseudoinverse:

$\hat{\boldsymbol\beta} = (\mathbf{X}^{\rm T}\mathbf{X})^{-1}\mathbf{X}^{\rm T}\mathbf{y}$

where $\mathbf{X}$ is the $n \times k$ design matrix, each row of which is a predictor vector $\mathbf{x}_{i}^{\rm T}$; and $\mathbf{y}$ is the column $n$-vector $[y_1 \; \cdots \; y_n]^{\rm T}$.

This is a frequentist approach, and it assumes that there are enough measurements to say something meaningful about $\boldsymbol\beta$. In the Bayesian approach, the data are supplemented with additional information in the form of a prior probability distribution. The prior belief about the parameters is combined with the data's likelihood function according to Bayes theorem to yield the posterior belief about the parameters $\boldsymbol\beta$ and $\sigma$. The prior can take different functional forms depending on the domain and the information that is available a priori.

## With conjugate priors

### Conjugate prior distribution

For an arbitrary prior distribution, there may be no analytical solution for the posterior distribution. In this section, we will consider a so called conjugate prior for which the posterior distribution can be derived analytically.

A prior $\rho(\boldsymbol\beta,\sigma^{2})$ is conjugate to this likelihood function if it has the same functional form with respect to $\boldsymbol\beta$ and $\sigma$. Since the log-likelihood is quadratic in $\boldsymbol\beta$, the log-likelihood is re-written such that the likelihood becomes normal in $(\boldsymbol\beta-\hat{\boldsymbol\beta})$. Write

\begin{align} (\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta) &= (\mathbf{y}- \mathbf{X} \hat{\boldsymbol\beta})^{\rm T}(\mathbf{y}- \mathbf{X} \hat{\boldsymbol\beta}) \\ &+ (\boldsymbol\beta - \hat{\boldsymbol\beta})^{\rm T}(\mathbf{X}^{\rm T}\mathbf{X})(\boldsymbol\beta - \hat{\boldsymbol\beta}). \end{align}

The likelihood is now re-written as

\begin{align} \rho(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma^{2}) &\propto (\sigma^2)^{-v/2} \exp\left(-\frac{vs^{2}}{2{\sigma}^{2}}\right)(\sigma^2)^{-(n-v)/2} \\ &\times \exp\left(-\frac{1}{2{\sigma}^{2}}(\boldsymbol\beta - \hat{\boldsymbol\beta})^{\rm T}(\mathbf{X}^{\rm T}\mathbf{X})(\boldsymbol\beta - \hat{\boldsymbol\beta})\right), \end{align}

where

$vs^{2} =(\mathbf{y}- \mathbf{X} \hat{\boldsymbol\beta})^{\rm T}(\mathbf{y}- \mathbf{X} \hat{\boldsymbol\beta}), \text{and}\; v = n-k,$

where $k$ is the number of regression coefficients.

This suggests a form for the prior:

$\rho(\boldsymbol\beta,\sigma^{2}) = \rho(\sigma^{2})\rho(\boldsymbol\beta|\sigma^{2}),$

where $\rho(\sigma^{2})$ is an inverse-gamma distribution

$\rho(\sigma^{2}) \propto (\sigma^2)^{-(v_{0}/2+1)} \exp\left(-\frac{v_{0}s_{0}^{2}}{2{\sigma}^{2}}\right).$

In the notation introduced in the inverse-gamma distribution article, this is the density of an $\text{Inv-Gamma}( a_0,b_0)$ distribution with $a_0=v_0/2$ and $b_0=\frac{1}{2}v_0s_0^2$ with $v_{0}$ and $s_{0}^{2}$ as the prior values of $v$ and $s^{2}$, respectively. Equivalently, it can also be described as a scaled inverse chi-squared distribution, $\mbox{Scale-inv-}\chi^2(v_0, s_0^2).$.

Further the conditional prior density $\rho(\boldsymbol\beta|\sigma^{2})$ is a normal distribution,

$\rho(\boldsymbol\beta|\sigma^{2}) \propto (\sigma^2)^{-k/2} \exp\left(-\frac{1}{2{\sigma}^{2}}(\boldsymbol\beta - \boldsymbol\mu_0)^{\rm T} \mathbf{\Lambda}_0 (\boldsymbol\beta - \boldsymbol\mu_0)\right) ).$

In the notation of the normal distribution, the conditional prior distribution is $\mathcal{N}\left(\boldsymbol\mu_0, \sigma^2\mathbf{\Lambda}_0^{-1}\right).$

### Posterior distribution

With the prior now specified, the posterior distribution can be expressed as

$\rho(\boldsymbol\beta,\sigma^{2}|\mathbf{y},\mathbf{X}) \propto \rho(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma^{2})\rho(\boldsymbol\beta|\sigma^{2})\rho(\sigma^{2})$
$\propto (\sigma^{2})^{-n/2} \exp\left(-\frac{1}{2{\sigma}^{2}}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta)\right)$
$\times (\sigma^{2})^{-k/2} \exp\left(-\frac{1}{2{\sigma}^{2}}(\boldsymbol\beta -\boldsymbol\mu_0)^{\rm T} \boldsymbol\Lambda_0 (\boldsymbol\beta - \boldsymbol\mu_0)\right)$
$\times (\sigma^2)^{-(a_0+1)} \exp\left(-\frac{b_0}{{\sigma}^{2}}\right).$

With some re-arrangement, the posterior can be re-written so that the posterior mean $\boldsymbol\mu_n$ of the parameter vector $\boldsymbol\beta$ can be expressed in terms of the least squares estimator $\hat{\boldsymbol\beta}$ and the prior mean $\boldsymbol\mu_0$, with the strength of the prior indicated by the prior precision matrix $\boldsymbol\Lambda_0$

$\boldsymbol\mu_n = (\mathbf{X}^{\rm T}\mathbf{X}+\boldsymbol\Lambda_0)^{-1}(\mathbf{X}^{\rm T}\mathbf{X}\hat{\boldsymbol\beta}+\boldsymbol\Lambda_0\boldsymbol\mu_0) .$

To justify that $\boldsymbol\mu_n$ is indeed the posterior mean, the quadratic terms in the exponential can be re-arranged as a quadratic form in $\boldsymbol\beta-\boldsymbol\mu_n$.[1]

$(\mathbf{y}- \mathbf{X} \boldsymbol\beta)^{\rm T}(\mathbf{y}- \mathbf{X} \boldsymbol\beta) + (\boldsymbol\beta - \boldsymbol\mu_0)^{\rm T}\boldsymbol\Lambda_0(\boldsymbol\beta - \boldsymbol\mu_0) = (\boldsymbol\beta-\boldsymbol\mu_n)^{\rm T}(\mathbf{X}^{\rm T}\mathbf{X}+\boldsymbol\Lambda_0)(\boldsymbol\beta-\boldsymbol\mu_n)+\mathbf{y}^{\rm T}\mathbf{y}-\boldsymbol\mu_n^{\rm T}(\mathbf{X}^{\rm T}\mathbf{X}+\boldsymbol\Lambda_0)\boldsymbol\mu_n+\boldsymbol\mu_0^{\rm T}\boldsymbol\Lambda_0\boldsymbol\mu_0 .$

Now the posterior can be expressed as a normal distribution times an inverse-gamma distribution:

$\rho(\boldsymbol\beta,\sigma^{2}|\mathbf{y},\mathbf{X}) \propto (\sigma^{2})^{-k/2} \exp\left(-\frac{1}{2{\sigma}^{2}}(\boldsymbol\beta - \boldsymbol\mu_n)^{\rm T}(\mathbf{X}^{\rm T}\mathbf{X}+\mathbf{\Lambda}_0)(\boldsymbol\beta - \boldsymbol\mu_n)\right)$
$\times (\sigma^2)^{-(n+v_{0})/2-1} \exp\left(-\frac{b_0+\mathbf{y}^{\rm T}\mathbf{y}-\boldsymbol\mu_n^{\rm T}(\mathbf{X}^{\rm T}\mathbf{X}+\boldsymbol\Lambda_0)\boldsymbol\mu_n+\boldsymbol\mu_0^{\rm T}\boldsymbol\Lambda_0\boldsymbol\mu_0)}{2{\sigma}^{2}}\right) .$

Therefore the posterior distribution can be parametrized as follows.

$\rho(\boldsymbol\beta,\sigma^{2}|\mathbf{y},\mathbf{X}) \propto \rho(\boldsymbol\beta |\sigma^{2},\mathbf{y},\mathbf{X}) \rho(\sigma^{2}|\mathbf{y},\mathbf{X}),$

where the two factors correspond to the densities of $\mathcal{N}\left( \boldsymbol\mu_n, \sigma^2\boldsymbol\Lambda_n^{-1} \right)\,$ and $\text{Inv-Gamma}\left(a_n,b_n \right)$ distributions, with the parameters of these given by

$\boldsymbol\Lambda_n=(\mathbf{X}^{\rm T}\mathbf{X}+\mathbf{\Lambda}_0), \quad \boldsymbol\mu_n = (\boldsymbol\Lambda_n)^{-1}(\mathbf{X}^{\rm T}\mathbf{X}\hat{\boldsymbol\beta}+\boldsymbol\Lambda_0\boldsymbol\mu_0) ,$
$a_n=\frac{1}{2}(n+v_0), \qquad b_n=b_0+\frac{1}{2}(\mathbf{y}^{\rm T}\mathbf{y}+\boldsymbol\mu_0^{\rm T}\boldsymbol\Lambda_0\boldsymbol\mu_0-\boldsymbol\mu_n^{\rm T}\boldsymbol\Lambda_n\boldsymbol\mu_n) .$

This can be interpreted as Bayesian learning where the parameters are updated according to the following equations.

$\boldsymbol\mu_n=(\mathbf{X}^{\rm T}\mathbf{X}+\boldsymbol\Lambda_0)^{-1} (\boldsymbol\Lambda_0\boldsymbol\mu_0+\mathbf{X}^{\rm T}\mathbf{X}\hat{\boldsymbol\beta})=(\mathbf{X}^{\rm T}\mathbf{X}+\boldsymbol\Lambda_0)^{-1} (\boldsymbol\Lambda_0\boldsymbol\mu_0+\mathbf{X}^{\rm T}\mathbf{y}) ,$
$\boldsymbol\Lambda_n=(\mathbf{X}^{\rm T}\mathbf{X}+\boldsymbol\Lambda_0) ,$
$a_n=a_0+\frac{n}{2} ,$
$b_n=b_0+\frac{1}{2}(\mathbf{y}^{\rm T}\mathbf{y}+\boldsymbol\mu_0^{\rm T}\boldsymbol\Lambda_0\boldsymbol\mu_0-\boldsymbol\mu_n^{\rm T}\boldsymbol\Lambda_n\boldsymbol\mu_n) .$

### Model evidence

The model evidence $p(\mathbf{y}|m)$ is the probability of the data given the model $m$. It is also known as the marginal likelihood, and as the prior predictive density. Here, the model is defined by the likelihood function $p(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma)$ and the prior distribution on the parameters, i.e. $p(\boldsymbol\beta,\sigma)$. The model evidence captures in a single number how well such a model explains the observations. The model evidence of the Bayesian linear regression model presented in this section can be used to compare competing linear models by Bayesian model comparison. These models may differ in the number and values of the predictor variables as well as in their priors on the model parameters. Model complexity is already taken into account by the model evidence, because it marginalizes out the parameters by integrating $p(\mathbf{y},\boldsymbol\beta,\sigma|\mathbf{X})$ over all possible values of $\boldsymbol\beta$ and $\sigma$.

$p(\mathbf{y}|m)=\int p(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma)\, p(\boldsymbol\beta,\sigma)\, d\boldsymbol\beta\, d\sigma$

This integral can be computed analytically and the solution is given in the following equation.[2]

$p(\mathbf{y}|m)=\frac{1}{(2\pi)^{n/2}}\sqrt{\frac{\det(\boldsymbol\Lambda_0)}{\det(\boldsymbol\Lambda_n)}} \cdot \frac{b_0^{a_0}}{b_n^{a_n}} \cdot \frac{\Gamma(a_n)}{\Gamma(a_0)}$

Here $\Gamma$ denotes the gamma function. Because we have chosen a conjugate prior, the marginal likelihood can also be easily computed by evaluating the following equality for arbitrary values of $\boldsymbol\beta$ and $\sigma$.

$p(\mathbf{y}|m)=\frac{p(\boldsymbol\beta,\sigma|m)\, p(\mathbf{y}|\mathbf{X},\boldsymbol\beta,\sigma,m)}{p(\boldsymbol\beta,\sigma|\mathbf{y},\mathbf{X},m)}$

Note that this equation is nothing but a re-arrangement of Bayes theorem. Inserting the formulas for the prior, the likelihood, and the posterior and simplifying the resulting expression leads to the analytic expression given above.

## Other cases

In general, it may be impossible or impractical to derive the posterior distribution analytically. However, it is possible to approximate the posterior by an approximate Bayesian inference method such as Monte Carlo sampling[3] or variational Bayes.

The special case $\boldsymbol\mu_0=0, \mathbf{\Lambda}_0 = c\mathbf{I}$ is called ridge regression.

A similar analysis can be performed for the general case of the multivariate regression and part of this provides for Bayesian estimation of covariance matrices: see Bayesian multivariate linear regression.

## Notes

1. ^ The intermediate steps are in Fahrmeir et al. (2009) on page 188.
2. ^ The intermediate steps of this computation can be found in O'Hagan (1994) on page 257.
3. ^ Carlin and Louis(2008) and Gelman, et al. (2003) explain how to use sampling methods for Bayesian linear regression.

## References

 6032 videos foundNext >
 (ML 10.1) Bayesian Linear RegressionIntroduction to the Bayesian approach to linear regression. We address the question of why a Bayesian approach is preferable to using the MLE or MAP estimate. Bayesian linear regressionBayesian linear regression using the standard noninformative prior. Although the marginal posteriors for the regression coefficients and the variance are ava... (ML 10.2) Posterior for linear regression (part 1)How to compute the posterior distribution for the weight vector w under a Bayesian model for linear regression. (ML 13.6) Graphical model for Bayesian linear regressionAs an example, we write down the graphical model for Bayesian linear regression. We introduce the "plate notation", and the convention of shading random vari... MLAI Lecture 14: Univariate Bayesian Linear Regression Machine learning - Bayesian learningBayesian learning for linear models Slides available at: http://www.cs.ubc.ca/~nando/540-2013/lectures.html Course taught in 2013 at UBC by Nando de Freitas. Estimating posterior distributions in Bayesian Linear RegressionThis video visualizes how posterior distributions for the slope and intercept of a linear regression model are estimated using Markov Chain Monte Carlo. Afte... (ML 10.3) Posterior for linear regression (part 2)How to compute the posterior distribution for the weight vector w under a Bayesian model for linear regression. Machine learning - Bayesian learning part 2Continuation of Bayesian learning for linear regression. Slides available at: http://www.cs.ubc.ca/~nando/540-2013/lectures.html Course taught in 2013 at UBC... (ML 10.7) Predictive distribution for linear regression (part 4)How to compute the (posterior) predictive distribution for a new point, under a Bayesian model for linear regression.
 6032 videos foundNext >

We're sorry, but there's no news about "Bayesian linear regression" right now.

 Limit to books that you can completely read online Include partial books (book previews) .gsc-branding { display:block; }

Oops, we seem to be having trouble contacting Twitter

# Talk About Bayesian linear regression

You can talk about Bayesian linear regression with people all over the world in our discussions.