Quadratic Discriminant Analysis

Contents

Quadratic Discriminant Analysis#

Gaussian#

Let the multivariate gaussian be,

P (x ∣ μ, σ) = {(2 π σ^{2})}^{d / 2} \exp (- \frac{| x - μ |^{2}}{2 σ^{2}})

Let the $N$ classes be represented as $c \in {1, 2, \dots, N} $ such that each class has the following statistics:

$μ_{c}$ : Mean of the class
$σ_{c}$ : Standard deviation of the class
$π_{c}$ : Prior of the class

The optimization problem can be simplified using logarithm to preserve maximization of the Bayes decision,

\begin{array}{r} \begin{aligned} \hat{y} (x) & = \underset{c}{\arg max} \log ((2 π)^{d / 2} P (x ∣ μ, σ) π_{c}) \\ = \underset{c}{\arg max} \frac{| x - μ_{c} |^{2}}{2 σ_{c}^{2}} - d \log σ_{c} + \log π_{c} \end{aligned} \end{array}

General Quadratic Function#

For some quadratic function $Q (x)$ , the Bayes decision rule is given by,

\hat{y} (x) = \underset{c}{\arg max} Q_{c} (x)

The Bayes decision rule for the quadratic discriminant always product a posterior distribution that is a logistic function. We can easily see this by considering only two classes $(A, B)$ .

\begin{array}{r} \hat{y} (x) = {\begin{cases} A & Q_{A} (x) - Q_{B} (x) > 0 \\ B & otherwise \end{cases} \end{array}

\begin{array}{r} \begin{aligned} P (Y ∣ X) & = \frac{e^{Q_{A} (x)}}{e^{Q_{A} (x)} + e^{Q_{B} (x)}} \\ = \frac{1}{1 + e^{Q_{A} (x) - Q_{B} (x)}} \end{aligned} \end{array}