Naive Bayes Classifier

The Problem

The Naive Bayes classifier is a famous classification technique that takes a probabilistic approach.

Consider a dataset where, for any input feature $x$, the output $y$ is either $1$ or $-1$. This is binary classification, where we have two categories. Naive Bayes can be extended to multiclass classification, but we'll start with the binary case.

For example, suppose we are given a person's glucose level, and each person is labeled as normal or high risk. Risk increases with glucose level. We'll treat normal as $1$ and high risk as $-1$.

Consider any input $x$ and output $y$ as coming from two random variables, $X$ and $Y$. For new data, we need to calculate the probability of the output given the input:

For simplicity, we'll write this as $P(y \mid x)$.

We can't read this directly from the training data, but we can calculate two related probabilities:

• $P(x \mid y)$, the probability of the input given the output class
• $P(y)$, the probability of the output class

For any output class, we know the inputs that belong to it, so we can calculate both of these from the dataset. What we need is a way to get from them to $P(y \mid x)$.

Bayes' Theorem

Think of a Venn diagram with two overlapping circles, $X$ and $Y$. The overlap is $X \cap Y$.

• $P(y \mid x)$ asks: given that we are in $X$, what is the probability of $Y$?
• $P(x \mid y)$ asks: given that we are in $Y$, what is the probability of ?