Expected Value

Expected Value#

The expected value (aka expectation) is defined as,

\[ E[X] \equiv \sum_{x \in \Omega_X}xP(X=x) \]

Existence : The expected value exists only if,

$$
E[\abs{X}] = \sum_{x \in \Omega_X}\abs{x} P(X=x) \lt \infty
$$

Linearity of Expectation : The linear transformation of \(X\) has the expectation,

$$
E(aX + b) = aE(X) + b
$$

Method of Indicators#

If a random variable \(X\) can be expressed as a sum of indicators then its expected value also follows by linearity.

\[ E[X] = \sum_{k=1}^N E[I_k] \]

Function Rule#

If the random variable at interest is a function of another random variable \(X = f(Y)\) then the function rule applies as following to the range of \(Y\)

\[ E[X] = \sum_{y \in \Omega_Y} f(y) P(Y=y) \]

Moments of \(X\) (Corollary) : All powers of \(X\) has the expected value called the \(k\)-th moment of \(X\) is given by,

$$
E[X^k] = \sum_{x \in \Omega_X}X^k P(X=x)
$$

Tail Sum Formula#

If the random variables \(X\) is non-negative, then the expected value can be expressed as the sum of the right tail of the distribution,

\[ E[X] = \sum_{x \in \Omega_X} P(X > x) \]

Expectation by Conditioning#

The expected value can be expressed in conditional probabilities. Consider the random variable \(X\) and \(Y\), the expected value can be expressed using both random variable using the marginal probability

\[ E[X] = \sum_{x \in \Omega_X} x \left[\sum_{y \in \Omega_Y} P(X=x \mid Y=y)P(Y=y)\right] \]

Motivated by this we define the conditional expectation as,

\[ \boxed{E[X \mid Y=y] = \sum_{x \in \Omega_X} x P(X=x \mid Y=y)} \]

The conditional expectation is a random variable because the conditional expectaiton is a function of the random variable \(Y\).

Most useful is the fact that the conditional expectation has the expected value,

[ \begin{align} E \Big[E[X \mid Y] \Big] &= \sum_{y \in \Omega_Y} E[X \mid Y=y] P(Y=y)\ &= \sum_{x \in \Omega_X}x \left[\sum_{x \in \Omega_Y} P(Y=y \mid X=x)P(Y=y)\right]\ \end{align} ]

[ \boxed{E[X] = E \Big[E[X \mid Y] \Big]} ]

Expectation with Known Conditional Expectation#

The expected value of \(X\) conditioned on \(Y\) can be determined if we know exactly the conditional expectation as some function of the range of \(g(y)\),

\[\begin{split} \begin{gather*} E[X \mid Y=y] = g(y)\\ \big\Downarrow\\ E[X \mid Y] = g(Y) \end{gather*} \end{split}\]

Then the expectation of \(X\) is,

\[ E[X] = E\big[g(Y)\big] \]

This makes more sense if we do a few examples:

Conditional Expectation of Binomial : Take for instance the conditional expectation is found to be the expectation of binomial,

$$
E[X \mid Y=y] = (n-y)p
$$

That is to say the events $\set{X \mid Y = y}$ are distributed as binomial of $n' = n-y$ trials with $p'$ chance of trial succeed.

Then, it's simple to determine $E[Y]$, after replacing $y$ with $Y$

$$
\begin{align*}
	E[E(X \mid Y)] &= E \big[(n-Y)p\big]\\
	&= (n-E[Y])p
\end{align*}
$$

Conditional Expectation of Sums : Let the sum of some event be \(S = X_1 + X_2 + \ldots X_N\) for iid \(X_k\) with mean \(\mu_X\). \(N\) is the random variable independent from \(X\) representing the count of sampling with \(\mu_N\).

The expected value of the sum is more easily written first as the conditional expectation

$$
\begin{gather*}
E(S \mid N=n) = n\mu_X\\
E(S \mid N) = N\mu_X\\
\end{gather*}
$$

The expectation of $S$ is then,

$$
E(S) = E[E(S \mid N)] = \mu_N \mu_X
$$

Expectation of Random Vectors#

For a random vector \(X\) the expectation is applied element-wise,

\[ \mu = E(X) \equiv [E(X_1), E(X_2),\ldots, E(X_n)]^\top \]

Linear Transformation : For a random vector \(Y = AX + b\) where \(A\) is a \(m \times n\) matrix and \(b\) is \(m \times 1\) vector, $\( E(Y) = A\mu_X + b \)$

Contents