In Cooperation with:

American Society for Quality Statistics Division

American Statistical Association

Bernoulli Society for Mathematical Statistics and Probability

Institute of Mathematical Statistics

International Biometric Society

International Chinese Statistical Association

International Society for Bayesian Analysis

International Statistical Institute

Royal Statistical Society

Statistical Society of Canada / Société statistique du Canada

Marginal Probability. Its use in Bayesian Statistics as the Evidence of Models and Bayes Factors

MarginalProbabilityItsUseInBayesianStatisticsAsTheEvidenceOfModelsAndBayesFactors3

Marginal Probability. Its use in Bayesian Statistics as the Evidence of Models and Bayes Factors

Luis Raúl Pericchi, Department of Mathematics and Biostatistics and Bioinformatics Center,
University of Puerto Rico, Rio Piedras, San Juan, Puerto Rico.¹

Keywords:Bayes Factors, Evidence of Models, Intrinsic Bayes Factors, Intrinsic Priors, Posterior Model Probabiities

Definition

Suppose that we have vectors of random variables $[\textbf{v,w}]=[v_1,v_2,\ldots,v_I,w_1,\ldots,w_J]$ in $\Re^{(I+J)}$ . Denote as the joint density function: $f_{\textbf{v,w}}$ , which obeys: $f_{\textbf{v,w}}(v,w) \ge 0$ and
$\int^{\infty}_{-\infty}\ldots\int^{\infty}_{-\infty} f_{\textbf{v,w}}(v,w) dv_1\ldots dv_I dw_1\ldots dw_I=1$ . Then the probability of the set

is given by

$\displaystyle P(A_v,B_w)=\int \ldots \int_{A_v,B_w} f_{\textbf{v,w}}(v,w) \textbf{dv} \textbf{dw}.$

The the marginal density $f_{\textbf{v}}$ is obtained as

$\displaystyle f_{\textbf{v}}(v)=\int^{\infty}_{-\infty}\ldots \int^{\infty}_{-\infty}f_{\textbf{v,w}}(v,w) dw_1\ldots dw_I.$

The the marginal probability of the set

is then obtained as,

$\displaystyle P(A_v)=\int \ldots \int_{A_v} f_{\textbf{v}}(v) dv.$

We have assumed that the random variables are continuous. When they are discrete, integrals are substituted by sums.
We proceed to present an important application of marginal densities to construct the Evidence of the Model and marginal probabilities for measuring the Bayesian Probability of a Model.

Measuring the Evidence in Favor of a Model

In Statistics, a parametric model, is denoted as $f(x_1,\ldots,x_n\vert\theta_1,\ldots,\theta_k)$ , where $\textbf{x}=(x_1,\ldots, x_n)$ is the vector of

observations and $\bf {\theta}=(\theta_1,\ldots,\theta_k)$ is the vector of

parameters. For instance we may have

observations normally distributed and the vector of parameters is $(\theta_1,\theta_2)$ the location and scale respectively, denoted by $f_{\textit{Normal}}(\textbf{x}\vert\bf {\theta})=\prod_{i=1}^n \frac{1}{\sqrt{2 \pi} \theta_2} \exp(-\frac{1}{2 \theta^2_2} (x_i-\theta_1)^2)$ .
Assume now that there is reason to suspect that the location is zero. As a second example, it may be suspected that the sampling model which usually has been assumed Normally distributed, is instead a Cauchy, $f_{\textit{Cauchy}}(\textbf{x}\vert\bf {\theta})=\prod_{i=1}^n \frac{1}{\pi \theta_2}\frac{1}{(1+(\frac{x_i-\theta_1}{\theta_2})^{2})}$ . The first problem is a hypothesis test denoted by

$\displaystyle H_0: \theta_1=0$ VS $\displaystyle H_1: \theta_1 \neq 0,$

and the second problem is a model selection problem:

$\displaystyle M_0: f_{\textit{Normal}}$ VS $\displaystyle M_1: f_{\textit{Cauchy}}.$

How to measure the evidence in favor of

? Instead of maximizing likelihoods as it is done in traditional significance testing, in Bayesian statistics the central concept is the evidence or marginal probability density

$\displaystyle m_j({\bf {x}})=\int f_j({\bf {x}}\vert\bf {\theta}_j) \pi(\bf {\theta}_j) d\bf {\theta}_j,$

where

denotes either model or hypothesis

and $\pi(\bf {\theta}_j)$ denotes the prior for the parameters under model or hypothesis

.
Marginal probabilities embodies the likelihood of a model or hypothesis in great generality and can be claimed it is the natural probabilistic quantity to compare models.

Marginal Probability of a Model

Once the marginal densities of the model j, for $j=1,\ldots,J$ models have been calculated and assuming the prior model probabilities $P(M_j), j=1,\ldots, J$ with $\sum_{j=1}^J P(M_j)=1$ then, using Bayes Theorem, the marginal probability of a model $P(M_j\vert\bf {x})$ can be calculated as,

$\displaystyle P(M_j\vert\bf {x})=\frac{m_j({\bf {x}}) \cdot P(M_j)}{\sum_{i=1}^n m_i({\bf {x}}) \cdot P(M_i)}.$

We have then the following formula for any two models or hypotheses:

$\displaystyle \frac{P(M_j\vert\bf {x})}{P(M_i\vert\bf {x})}= \frac{P(M_j)}{P(M_i)} \times \frac{m_j({\bf {x}})}{m_i({\bf {x}})},$

or in words: Posterior Odds equals Prior Odds times Bayes Factor, where the Bayes Factor of

over

$\displaystyle B_{j,i}=\frac{m_j({\bf {x}})}{m_i({\bf {x}})},$

Jeffreys (1961).
In contrast to p-values, which have interpretations heavily dependent on the sample size

, and its definition is not the same as the scientific question, the posterior probabilities and Bayes Factors address the scientific question: "how probable is model or hypothesis j as compared with model or hypothesis i?", and the interpretation is the same for any sample size, Berger and Pericchi (1996a, 2001). Bayes Factors and Marginal Posterior Model Probabilities have several advantages, like for example large sample consistency, that is as the sample size grows the Posterior Model Probability of the sampling model tends to one. Furthermore, if the goal is to predict future observations

it is not necessary to select one model as the predicting model since we may predict by the so called Bayesian Model Averaging, which if quadratic loss is assumed, the optimal predictor takes the form,

$\displaystyle E[Y_f\vert\bf {x}]= \sum_{j=1}^J E[Y_f\vert\bf {x}, M_j] \times P(M_j\vert\bf {x}),$

where $E[Y_f\vert\bf {x},M_j]$ is the expected value of a future observation under the model or hypothesis

Intrinsic Priors for Model Selection and Hypothesis Testing

Having said some of the advantages of the marginal probabilities of models, the question arises: how to assign the conditional priors $\pi(\theta_j)$ ? In the two examples above which priors are sensible to use? The problem is not a simple one since it is not possible to use the usual Uniform priors since then the Bayes Factors are undetermined. To solve this problem with some generality, Berger and Pericchi (1996a,b) introduced the concepts of Intrinsic Bayes Factors and Intrinsic Priors. Start by splitting the sample in two sub-samples $\bf {x}=[\bf {x}(l),\bf {x}(-l)]$ where the training sample $\bf {x}(l)$ is as small as possible such that for $j=1,\ldots,J: 0<m_j(\bf {x}(l))<\infty$ . Thus starting with an improper prior $\pi^N(\theta_j)$ , which does not integrate to one (for example the Uniform), by using the minimal training sample $\bf {x}(l)$ , all the conditional prior densities $\pi(\theta_j\vert\bf {x}(l))$ become proper. So we may form the Bayes Factor using the training sample $\bf {x}(l)$ as

$\displaystyle B_{ji}(\bf {x}(l))=\frac{m_j(\bf {x}(-l)\vert\bf {x}(l))}{m_i(\bf {x}(-l)\vert\bf {x}(l))}.$

This however depends on the particular training sample $\bf {x}(l)$ . So some sort of average of Bayes Factor is necessary. In Berger and Pericchi (1996) it is shown that the average should be the arithmetic average. It is also found a theoretical prior that is an approximation to the procedure just described as the sample size grows. This is called an Intrinsic Prior. In the examples above: Example 1: in the normal case, assuming first that the variance is known $\theta^2_2=\theta^2_{2,0}$ then it turns out that the Intrinsic Prior is Normal centered at the null hypothesis $\theta_1=0$ and with variance $2 \cdot \theta^2_{2,0}$ . More generally when the variance is unknown

$\displaystyle \pi^I(\theta_1\vert\theta_2)=\frac{1-\exp(-\theta_1^2/\theta_2^2)}{2 \sqrt{\pi}\cdot (\theta_1^2/\theta_2)},$ and $\displaystyle \pi^I(\theta_2)=\frac{1}{\theta_2}.$

It turns out that $\pi^I(\theta_1\vert\theta_2)$ is a proper density, Berger and Pericchi (1996ab), Pericchi(2005).
Example 2: in the Normal vs Cauchy example, it turns out that the improper prior $\pi^I(\theta_1,\theta_2)=1/\theta_2$ is the appropriate prior for comparing the models, Pericchi (2005). For other examples of Intrinsic Priors see for instance, Berger and Pericchi (1996a, 1996b, 2001), Moreno, Bertolino and Racugno (1998), Pericchi (2005) and Casella and Moreno (2009), among others.
This article is based on an article from Lovric, Miodrag (2011), International Encyclopedia of Statistical Science. Heidelberg: Springer Science +Business Media, LLC

References

Berger J.O. and Pericchi L.R. (1996a). The Intrinsic Bayes Factor for Model Selection and Prediction. Jour. Amer. Stat. Assoc.,91, p. 109-122.
Berger J.O. and Pericchi L.R. (1996b). The Intrinsic Bayes Factors for Linear Models. In Bayesian Statistics 5, Bernardo J.M. et. al, editors, p. 23-42, Oxford University Press.
Berger J.O. and Pericchi L.R. (2001) Objective Bayesian Methods for Model Selection: Introduction and Comparison. IMS LectureNotes-Monograph Series, 38, p. 135-207.
Casella, G. and Moreno, E. (2009) Assessing robustness of intrinsic tests of independence in two-way contingency tables. Journal of the American Statistical Association, 104, 1261-1271.
Jeffreys, H. (1961) Theory of Probability. 3rd Ed. Oxford University Press.
Moreno E., Bertolino F. and Racugno W. (1998) An Intrinsic Limiting Procedure for Model Selection and Hypothesis Testing. Jour. of the Amer Statist Assoc, 93, 444, pp. 1451-1460.
Lovric, Miodrag (2011), International Encyclopedia of Statistical Science. Heidelberg: Springer Science +Business Media, LLC
Pericchi, L.R. (2005)Model Selection and Hypothesis Testing based on Objective Probabilities and Bayes Factors. Handbook of Statistics, Vol. 25: Bayesian Thinking: Modeling and Computation. Dey D.K. and Rao C.R. Editors. Elsevier, North-Holland.

Footnotes

... Rico.¹: email address: luarpr@uprrp.edu,This work sponsored in part by NIH Grant: P20-RR016470

Marginal Probability. Its use in Bayesian Statistics as the Evidence of Models and Bayes Factors is owned by Luis Raul Pericchi.

Note

NOTE ON THE PROPER CITATION OF THIS ARTICLE: The proper citation for this article is Luis Raul Pericchi, "Marginal Probability: Its Use in Bayesian Statistics as the Evidence of Models and Bayes Factors." In: Lovric, Miodrag (Editor), (2010). International Encyclopedia of Statistical Science. Heidelberg: Springer Science +Business Media, LLC, reprinted and freely available at StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies.

How to Cite This Entry

Luis Raul Pericchi. "Marginal Probability. Its use in Bayesian Statistics as the Evidence of Models and Bayes Factors" (version 2). StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies. Freely available at http://statprob.com/encyclopedia/MarginalProbabilityItsUseInBayesianStatisticsAsTheEvidenceOfModelsAndBayesFactors3.html

Classification

AMS MSC:	62F03 (Statistics :: Parametric inference :: Hypothesis testing)
	62F15 (Statistics :: Parametric inference :: Bayesian inference)

Pending Errata and Addenda

None.

Discussion

No messages.

Username:
Password:

Register
I've forgotten my login details

Login

In Cooperation with:

Marginal Probability. Its use in Bayesian Statistics as the Evidence of Models and Bayes Factors

Marginal Probability. Its use in Bayesian Statistics as the Evidence of Models and Bayes Factors

Definition

Measuring the Evidence in Favor of a Model

Marginal Probability of a Model

Intrinsic Priors for Model Selection and Hypothesis Testing

References

Footnotes

Note

How to Cite This Entry

Classification

Pending Errata and Addenda

Discussion