Machine learning
Filter

- by Priyankur Sarkar
- 23rd Sep, 2019
- Last updated on 11th Mar, 2021
- 12 mins read

Naive Bayes is a simple but surprisingly powerful probabilistic machine learning algorithm used for predictive modeling and classification tasks. Some typical applications of Naive Bayes are spam filtering, sentiment prediction, classification of documents, etc. It is a popular algorithm mainly because it can be easily written in code and predictions can be made real quick which in turn increases the scalability of the solution. The Naive Bayes algorithm is traditionally considered the algorithm of choice for practical-based applications mostly in cases where instantaneous responses are required for user’s requests.

It is based on the works of the Rev. Thomas Bayes and hence the name. Before starting off with Naive Bayes, it is important to learn about Bayesian learning, what is ‘Conditional Probability’ and ‘Bayes Rule’.

Bayesian learning is a supervised learning technique where the goal is to build a model of the distribution of class labels that have a concrete definition of the target attribute. Naïve Bayes is based on applying Bayes' theorem with the *naïve* assumption of independence between each and every pair of features.

Let us start with the primitives by understanding Conditional Probability with some examples.

**Example 1**

Consider you have a coin and fair dice. When you flip a coin, there is an equal chance of getting either a head or a tail. So you can say that the probability of getting heads or the probability of getting tails is 50%.

Now if you roll the fair dice, the probability of getting 1 out of the 6 numbers would be 1/6 = 0.166. The probability will also be the same for other numbers on the dice.

**Example 2**

Consider another example of playing cards. You are asked to pick a card from the deck. Can you guess the probability of getting a king given the card is a heart?

The given condition here is that the card is a heart, so the denominator has to be 13 (there are 13 hearts in a deck of cards) and not 52. Since there is only one king in hearts, so the probability that the card is a king given it is a heart is **1/13 = 0.077**.

So when you say the conditional probability of A given B, it refers to the probability of the occurrence of A given that B has already occurred. This is a typical example of conditional probability.

Mathematically, the conditional probability of A given B can be defined as **P(A AND B) / P(B).**

**Example 3**

Let us see another slightly complicated example to understand conditional probability better.

Consider a school with a total population of 100 people. These 100 people can be classified as either **‘Students’** and **‘Teachers’** or as a population of **‘Males’** and **‘Females’**.

With the table below of the 100 people tabulated in some form, what will be the conditional probability that a certain person of the school is a **‘Student’** given that she is a **‘Female’**?

Female | Male | Total | |
---|---|---|---|

Teacher | 10 | 10 | 20 |

Student | 30 | 50 | 80 |

Total | 40 | 60 | 100 |

To compute this, you can filter the sub-population of 40 females and focus only on the 30 female students. So the required probability stands as **P(Student | Female) = 30/40 = 0.75 .**

P(Student | Female) =[P(Student ∩ Female)]/[P(Female)]= 30/40 = 0.75

This is defined as the intersection**(∩)** of Student(A) and Female(B) divided by Female(B). Similarly, the conditional probability of B given A can also be calculated using the same mathematical expression.

Bayes' Theorem helps you examine the probability of an event based on the prior knowledge of any event that has correspondence to the former event. Its uses are mainly found in probability theory and statistics. The term naive is used in the sense that the features given to the model are not dependent on each other. In simple terms, if you change the value of one feature in the algorithm, it will not directly influence or change the value of the other features.

Consider for example the probability that the price of a house is high can be calculated better if we have some prior information like the facilities around it compared to another assessment made without the knowledge of the location of the house.

P(A|B) =[P(B|A)P(A)]/[P(B)]

The equation above shows the basic representation of the Bayes' theorem where A and B are two events and:

**P(A|B)**: The conditional probability that event A occurs, given that B has occurred. This is termed as the posterior probability.

**P(A) and P(B)**: The probability of A and B without any correspondence with each other.

**P(B|A)**: The conditional probability of the occurrence of event B, given that A has occurred.

Now the question is how you can use Bayes' Theorem in your machine learning models. To understand it clearly, let us take an example.

Consider a simple problem where you need to learn a machine learning model from a given set of attributes. Then you will have to describe a hypothesis or a relation to a response variable and then using this relation, you will have to predict a response, given the set of attributes you have.

You can create a learner using Bayes' Theorem that can predict the probability of the response variable that will belong to the same class, given a new set of attributes.

Consider the previous question again and then assume that A is the response variable and B is the given attribute. So according to the equation of Bayes' Theorem, we have:

**P(A|B)**: The conditional probability of the response variable that belongs to a particular value, given the input attributes, also known as the posterior probability.

**P(A)**: The prior probability of the response variable.

**P(B)**: The probability of training data(input attributes) or the evidence.

**P(B|A)**: This is termed as the likelihood of the training data.

The Bayes' Theorem can be reformulated in correspondence with the machine learning algorithm as:

posterior = (prior x likelihood) / (evidence)

Let’s look into another problem. Consider a situation where the number of attributes is n and the response is a Boolean value. i.e. Either **True** or **False**. The attributes are categorical (2 categories in this case). You need to train the classifier for all the values in the instance and the response space.

This example is practically not possible in most machine learning algorithms since you need to compute **2∗(2^n-1)** parameters for learning this model. This means for 30 boolean attributes, you will need to learn more than 3 billion parameters which is unrealistic.

A classifier is a machine learning model which is used to classify different objects based on certain behavior. Naive Bayes classifiers in machine learning are a family of simple probabilistic machine learning models that are based on Bayes' Theorem. In simple words, it is a classification technique with an assumption of independence among predictors.

The Naive Bayes classifier reduces the complexity of the Bayesian classifier by making an assumption of conditional dependence over the training dataset.

Consider you are given variables X, Y, and Z. X will be conditionally independent of Y given Z if and only if the probability distribution of X is independent of the value of Y given Z. This is the assumption of conditional dependence.

In other words, you can also say that X and Y are conditionally independent given Z if and only if, the knowledge of the occurrence of X provides no information on the likelihood of the occurrence of Y and vice versa, given that Z occurs. This assumption is the reason behind the term **naive** in Naive Bayes.

The likelihood can be written considering n different attributes as:

n P(X₁...Xₙ|Y) = π P(Xᵢ|Y) i=1

In the mathematical expression, X represents the attributes, Y represents the response variable. So, **P(X|Y)** becomes equal to the product of the probability distribution of each attribute given Y.

**Maximizing a Posteriori**

If you want to find the posterior probability of **P(Y|X)** for multiple values of Y, you need to calculate the expression for all the different values of Y.

Let us assume a new instance variable **X_NEW**. You need to calculate the probability that Y will take any value given the observed attributes of **X_NEW** and given the distributions P(Y) and **P(X|Y)** which are estimated from the training dataset.

In order to predict the response variable depending on the different values obtained for **P(Y|X)**, you need to consider a probable value or the maximum of the values. Hence, this method is known as **maximizing a posteriori**.

**Maximizing Likelihood**

You can simplify the Naive Bayes algorithm if you assume that the response variable is uniformly distributed which means that it is equally likely to get any response. The advantage of this assumption is that the a priori or the P(Y) becomes a constant value.

Since the a priori and the evidence become independent from the response variable, they can be removed from the equation. So, maximizing the posteriori becomes maximizing the likelihood problem.

Consider a situation where you have 1000 fruits which are either ‘banana’ or ‘apple’ or ‘other’. These will be the possible classes of the variable Y.

The data for the following X variables all of which are in binary (0 and 1):

- Long
- Sweet
- Yellow

The training dataset will look like this:

Fruit | Long (x1) | Sweet (x2) | Yellow (x3) |
---|---|---|---|

Apple | 0 | 0 | 1 |

Banana | 1 | 0 | 1 |

Apple | 0 | 1 | 0 |

Other | 1 | 1 | 1 |

.. | .. | .. | .. |

Now let us sum up the training dataset to form a count table as below:

Type | Long | Not Long | Sweet | Not sweet | Yellow | Not Yellow | Total |
---|---|---|---|---|---|---|---|

Banana | 400 | 100 | 350 | 150 | 450 | 50 | 500 |

Apple | 0 | 300 | 150 | 150 | 300 | 0 | 300 |

Other | 100 | 100 | 150 | 50 | 50 | 150 | 200 |

Total | 500 | 500 | 650 | 350 | 800 | 200 | 1000 |

The main agenda of the classifier is to predict if a given fruit is a ‘Banana’ or an ‘Apple’ or ‘Other’ when the three attributes(long, sweet and yellow) are known.

Consider a case where you’re given that a fruit is long, sweet and yellow and you need to predict what type of fruit it is. This case is similar to the case where you need to predict Y only when the X attributes in the training dataset are known. You can easily solve this problem by using Naive Bayes.

The thing you need to do is to compute the 3 probabilities,i.e. the probability of being a banana or an apple or other. The one with the highest probability will be your answer.

**Step 1:**

First of all, you need to compute the proportion of each fruit class out of all the fruits from the population which is the prior probability of each fruit class.

The Prior probability can be calculated from the training dataset:

P(Y=Banana) = 500 / 1000 = 0.50

P(Y=Apple) = 300 / 1000 = 0.30

P(Y=Other) = 200 / 1000 = 0.20

The training dataset contains 1000 records. Out of which, you have 500 bananas, 300 apples and 200 others. So the priors are 0.5, 0.3 and 0.2 respectively.

**Step 2:**

Secondly, you need to calculate the probability of evidence that goes into the denominator. It is simply the product of P of X’s for all X:

P(x_{1}=Long) = 500 / 1000 = 0.50

P(x_{2}=Sweet) = 650 / 1000 = 0.65

P(x_{3}=Yellow) = 800 / 1000 = 0.80

**Step 3:**

The third step is to compute the probability of likelihood of evidence which is nothing but the product of conditional probabilities of the 3 attributes.

The Probability of Likelihood for Banana:

P(x_{1}=Long | Y=Banana) = 400 / 500 = 0.80

P(x_{2}=Sweet | Y=Banana) = 350 / 500 = 0.70

P(x_{3}=Yellow | Y=Banana) = 450 / 500 = 0.90

Therefore, the overall probability of likelihood for banana will be the product of the above three,i.e. 0.8 * 0.7 * 0.9 = 0.504.

**Step 4:**

The last step is to substitute all the 3 equations into the mathematical expression of Naive Bayes to get the probability.

P(Banana|Long,Sweet and Yellow) = [P(Long|Banana)∗P(Sweet|Banana)∗P(Yellow|Banana) x P(Banana)] **/ [**P(Long)∗P(Sweet)∗P(Yellow)**]**

= 0.8∗0.7∗0.9∗0.5/**[**P(Evidence)**] **= 0.252/**[**P(Evidence)**]**

P(Apple|Long,Sweet and Yellow) = 0, because P(Long|Apple) = 0

P(Other|Long,Sweet and Yellow) = 0.01875/P(Evidence)

In a similar way, you can also compute the probabilities for ‘Apple’ and ‘Other’. The denominator is the same for all cases.

Banana gets the highest probability, so that will be considered as the predicted class.

The main types of Naive Bayes classifier are mentioned below:

**Multinomial Naive Bayes**— These types of classifiers are usually used for the problems of document classification. It checks whether the document belongs to a particular category like sports or technology or political etc and then classifies them accordingly. The predictors used for classification in this technique are the frequency of words present in the document.**Complement Naive Bayes**— This is basically an adaptation of the multinomial naive bayes that is particularly suited for imbalanced datasets.**Bernoulli Naive Bayes**— This classifier is also analogous to multinomial naive bayes but instead of words, the predictors are Boolean values. The parameters used to predict the class variable accepts only yes or no values, for example, if a word occurs in the text or not.**Out-of-Core Naive Bayes**— This classifier is used to handle cases of large scale classification problems for which the complete training dataset might not fit in the memory.**Gaussian Naive Bayes**— In a Gaussian Naive Bayes, the predictors take a continuous value assuming that it has been sampled from a Gaussian Distribution. It is also called a Normal Distribution.

Since the likelihood of the features is assumed to be Gaussian, the conditional probability will change in the following manner:

P(xᵢ|y) = 1/(√2пσ²ᵧ) exp[ –(xᵢ - μᵧ)²/2σ²ᵧ]

The naive Bayes algorithm has both its pros and its cons.

Pros of Naive Bayes —

- It is easy and fast to predict the class of the training data set.
- It performs well in multiclass prediction.
- It performs better as compared to other models like logistic regression while assuming the independent variables.
- It requires less training data.
- It performs better in the case of categorical input variables as compared to numerical variables.

Cons of Naive Bayes —

- The model is not able to make a prediction in situations where the categorical variable has a category that was not observed in the training data set and assigns a 0 (zero) probability to it. This is known as the ‘Zero Frequency’. You can solve this using the Laplace estimation.
- Since Naive Bayes is considered to be a bad estimator, the probability outputs are not taken seriously.
- Naive Bayes works on the principle of assumption of independent predictors, but it is practically impossible to get a set of predictors that are completely independent.

When you have a model with a lot of attributes, it is possible that the entire probability might become zero because one of the feature’s values is zero. To overcome this situation, you can increase the count of the variable with zero to a small value like in the numerator so that the overall probability doesn’t come as zero.

This type of correction is called the Laplace Correction. Usually, all naive Bayes models use this implementation as a parameter.

There are a lot of real-life applications of the Naive Bayes classifier, some of which are mentioned below:

**Real-time prediction**— It is a fast and eager machine learning classifier, so it is used for making predictions in real-time.**Multi-class prediction**— It can predict the probability of multiple classes of the target variable.**Text Classification/ Spam Filtering / Sentiment Analysis**— They are mostly used in text classification problems because of its multi-class problems and the independence rule. They are used for identifying spam emails and also to identify negative and positive customer sentiments on social platforms.**Recommendation Systems**— A Recommendation system is built by combining Naive Bayes classifier and Collaborating Filtering. It filters unseen information and predicts whether the user would like a given resource or not using machine learning and data mining techniques.

In Python, the Naive Bayes classifier is implemented in the scikit-learn library. Let us look into an example by importing the standard **iris** dataset to predict the **Species** of flowers:

# Import packages from sklearn.naive_bayes import GaussianNB from sklearn.model_selection import train_test_split from sklearn.metrics import confusion_matrix import numpy as np import pandas as pd import matplotlib.pyplot as plt import seaborn as sns; sns.set() # Import data training = pd.read_csv('/content/iris_training.csv') test = pd.read_csv('/content/iris_test.csv') # Create the X, Y, Training and Test X_Train = training.drop('Species', axis=1) Y_Train = training.loc[:, 'Species'] X_Test = test.drop('Species', axis=1) Y_Test = test.loc[:, 'Species'] # Init the Gaussian Classifier model = GaussianNB() # Train the model model.fit(X_Train, Y_Train) # Predict Output pred = model.predict(X_Test) # Plot Confusion Matrix mat = confusion_matrix(pred, Y_Test) names = np.unique(pred) sns.heatmap(mat, square=True, annot=True, fmt='d', cbar=False, xticklabels=names, yticklabels=names) plt.xlabel('Truth') plt.ylabel('Predicted')

The output will be as follows:

Text(89.18, 0.5, 'Predicted')

You can improve the power of a Naive Bayes model by following these tips:

- Transform variables using transformations like BoxCox and YeoJohnson to make continuous features to normal distribution.
- Use Laplace Correction for handling zero values in X variables and to predict the class of test data set for zero frequency issues.
- Check for correlated features and remove the highly correlated ones because they are voted twice in the model and might lead to over inflation.
- Combine different features to make a new product which makes some intuitive sense.
- Provide more realistic prior probabilities to the algorithm based on knowledge from business. Use ensemble methods like bagging and boosting to reduce the variance.

**Summary**

Let us see what we have learned so far —

- Naive Bayes and its types
- Pros and Cons of Naive Bayes
- Applications of Naive Bayes
- How a Naive Bayes model makes the prediction
- Creating a Naive Bayes classifier
- Improving a Naive Bayes model

Naive Bayes is mostly used in real-world applications like sentiment analysis, spam filtering, recommendation systems, etc. They are extremely fast and easy to implement as compared to other machine learning models. However, the biggest drawback of Naive Bayes is the requirement of predictors to be independent. In most real-life cases, the predictors are dependent in nature which hinders the performance of the classifier.

We have covered most of the topics related to algorithms in our series of machine learning blogs, click here. If you are inspired by the opportunities provided by machine learning, enrol in our Data Science and Machine Learning Courses for more lucrative career options in this landscape.

5737

- by Gaurav Kr. Roy
- 03 Feb 2021
- 11 mins read

Data has become the new game changer for busines... Read More

5876

- by Abhresh S
- 27 May 2021
- 11 mins read

Statistics is a science concerned with collection,... Read More

9261

- by Abhresh S
- 27 May 2021

What is Dispersion in StatisticsDispersion in stat... Read More

Subscribe to our newsletter.

## Join the Discussion

Your email address will not be published. Required fields are marked *