Posterior Estimator

Posterior Estimator#

With an assumption of the likelihood function (i.e., the distribution of the data), we can determine the distribution of the parameters which is the posterior distribution. Using Bayes rule we find that,

P (θ ∣ X) = \frac{P (X ∣ θ) P (θ)}{P (X)}

$P (X ∣ θ)$ : the likelihood function
$P (X)$ : the normalization distribution called the evidence.

Expectation of Posterior#

If there must be one value instead of a distribution, the expectation is one natural choice

Maximum a Posterior#

If there must be one value, then the mode or maximum a posteriori (MAP) is a another natural choice.

Bayesian Update#

Motivating Example#

Consider a coin where we do not know the chance of heads $p$ . Let’s take the prior that $p$ is uniformally any value between 0 and 1. That is,

p \sim Uniform (0, 1)

The chance that the coin lands head depends on the value of $p$ which is random. We can marginalize over all possible values of $p$ ,

P (H) = \int_{0}^{1} P (H, p) d p = \int_{0}^{1} P (H ∣ p) f (p) d p

In fact this looks like an expected value as a function of $p$ ,

P (H) = E (P (H ∣ p))

The chance of head given $p$ is of course $p$ itself so,

P (H) = E (p) = \frac{1}{2}

This is not so surprising since $p$ is uniform, a large sample of $p$ would neither be advantageous to getting heads or tails.

However, what’s the chance of getting two heads $P (H H)$ out of two tosses?

We know that $P (H H ∣ p) = p^{2}$ therefore,

P (H H) = E (p^{2}) = \int_{0}^{1} p^{2} d p = \frac{1}{3}

The answer is not $1 / 4$ as we should expect. That must mean the two tosses of coins are not independent when $p$ is random. To see why we need to look at the conditional probability that the second coin lands head if we know the first one lands heads for a random $p$

P (H_{2} ∣ H_{1}) = \frac{P (H_{1}, H_{2})}{P (H_{1})} = \frac{2}{3}

Where did the $2 / 3$ come from? Let’s take a look at what happens to the chance of $p$ given we know the first coin lands head. We would need to apply Bayes rule,

f (p ∣ H_{1}) = \frac{P (H_{1} ∣ p) f (p)}{P (H_{1})} \propto p

Normalizing we find that,

f (p ∣ H_{1}) = 2 p

Thus,

P (H_{2} ∣ H_{1}) = \int_{0}^{1} P (H_{2} ∣ p) \cdot f (p ∣ H_{1}) d p = \frac{2}{3}

Beta and Binomial Update#

We are interested in $n$ IID Bernoulli trials where we define the sum as $S = \sum I_{k}$ .

The Bayesian update rule is,

(p ∣ S = k) \sim Beta (r + k, s + n - k)

Let the prior for the parameter $p$ be distributed as,

\begin{array}{r} \begin{array}{c} p \sim Beta (r, s) \\ f (p) = C (r, s) \cdot p^{r - 1} (1 - p)^{s - 1} \end{array} \end{array}

the likelihood for $S$ is then the binomial distribution,

P (S = k ∣ p) = (\binom{n}{s}) p^{k} (1 - p)^{n - k}

Hence, the posterior distribution is,

f (p ∣ S = k) = C (r + k, s + n - k) p^{r + k - 1} (1 - p)^{n + s - k - 1}

Thus, the Bayesian update adds the number of successes $k$ and the number of failures $n - k$ .

(p ∣ S = k) \sim Beta (r + k, s + n - k)

Expectation : $ $E (p ∣ S = k) = \frac{r + k}{r + s + n}$ $

MAP : $ $mode (p ∣ S = k) = \frac{r + k - 1}{r + k + n - 2}$ $

Transition Rule : $ $P (S_{n + 1} = k + 1 ∣ S_{n} = k) = E (p ∣ S_{n} = k)$ $

Evidence : The chance of $k$ heads after $n$ tosses using the beta prior is the beta-binomial distribution.

$$
P(S_n = k) = {n \choose k}\frac{C(r,s)}{C(r+k, s+n-k)}
$$