Bayesian decision theory considers how to choose the optimal class label based on the relevant probabilities and misclassification losses

Suppose there are N possible class labels, that is $\mathbf{y}=\\{c_1,c_2,...,c_N\\}$, and $\lambda_{ij}$ is the loss incurred by misclassifying a sample whose true label is $c_j$ as $c_i$. Based on the posterior probability $P(c_i\vert\mathbf{x})$, we can obtain the expected loss incurred by classifying sample $\mathbf{x}$ as $c_i$, that is, the “conditional risk” on sample $\mathbf{x}$:

$$R(c_i\vert\mathbf{x})= \ \sum_{j=1}^N \lambda_{ij}P(c_j\vert\mathbf{x})$$

Our goal is to find a decision rule $h: \ \mathbf{x} \mapsto \mathbf{y}$ that minimizes the overall risk:

$$R(h)=E_{\mathbf{x}}[R(h(\mathbf{x})\vert\mathbf{x})]$$

Since for every sample $\mathbf{x}$, if h can minimize the conditional risk $R(h(\mathbf{x})\vert\mathbf{x})$, the overall risk $R(h)$ will also be minimized, so we have:

Bayesian decision rule:

To minimize the overall risk, one only needs to choose, for each sample, the class label that minimizes the conditional risk $R(c\vert\mathbf{x})$, that is:

$$h^*(\mathbf{x})=argmin_{c\in\mathbf{y}} \ R(c\vert\mathbf{x})$$

Here $h^*$ is called the Bayes optimal classifier, and the corresponding overall risk $R(h^*)$ is called the Bayes risk; $1-R(h^*)$ reflects the best performance a classifier can achieve, that is, the theoretical upper bound of the model accuracy obtainable through machine learning

Naive Bayes

Naive Bayes learns the joint probability distribution $P(X,Y)$ from the training dataset, specifically learning the following prior probability distribution and conditional probability distributions:

Prior probability distribution: $\ P(Y=c_k), \ k=1,2,...,N$

Conditional probability distribution: $\ P(X=x\vert Y=c_k)=P(X^{(1)}=x^{(1)},...,X^{(n)}=x^{(n)}\vert Y=c_k), \ k=1,2,...N$

So the joint probability distribution is: $P(X,Y)=P(Y=c_k)P(X=x\vert Y=c_k)$

Derivation

Naive Bayes has an important assumption, the conditional independence assumption, namely:

$$ \begin{aligned} P(X=x\vert Y=c_k) & =P(X^{(1)}=x^{(1)},...,X^{(n)}=x^{(n)}\vert Y=c_k) \\ & =\prod_{j=1}^n P(X^{(j)}=x^{(j)}\vert Y=c_k) \end{aligned} \tag{1} $$

By Bayes’ theorem the posterior probability is:

$$P(Y=c_k\vert X=x)=\frac{P(X=x\vert Y=c_k)P(Y=c_k)}{\sum_k P(X=x\vert Y=c_k)P(Y=c_k)}$$

Substituting equation (1) into the above gives:

$$ \begin{aligned} & P(Y=c_k\vert X=x)=\frac{P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}\vert Y=c_k)}{\sum_k P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}\vert Y=c_k)} \\ & \text{where} \ k=1,2,...,k \end{aligned} \tag{2} $$

Then the naive Bayes classifier can be expressed as:

$$ y=f(x)=argmax_{c_k} \ \frac{P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}\vert Y=c_k)}{\sum_k P(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}\vert Y=c_k)} \tag{3} $$

Note that in equation (3) the denominator is the same for all $c_k$, so we have:

$$ y=argmax_{c_k} \ p(Y=c_k)\prod_j P(X^{(j)}=x^{(j)}\vert Y=c_k) $$

Use in Scikit-learn

See the official site for the concrete API parameters:

sklearn.naive_bayes

Reference

Zhou Zhihua’s Watermelon Book and Li Hang’s Statistical Learning Methods