Conditions Apply

Search

Machine learning Filter

Boosting and AdaBoost in Machine Learning

Ensemble learning is a strategy in which a group of models are used to find a solution to a challenging problem, by using a strategy and combining diverse machine learning models into one single predictive model.In general, ensemble methods are mainly used for improving the overall performance accuracy of a model and combine several different models, also known as the base learners, to predict the results, instead of using a single model.In one of the articles related to ensemble learning, we have already discussed about the popular ensemble method, Bootstrap Aggregation. Bagging tries to implement similar learners on small sample populations and then takes a mean of all the predictions. It combines Bootstrapping and Aggregation to form one ensemble model. It basically reduces the variance error and helps to avoid overfitting. In this article we will look into the limitations of bagging and how a boosting algorithm can be used to overcome those limitations. We will also learn about various types of boosting algorithms and implement one of them in Python. Let’s get started.What are the limitations of Bagging?Let us recall the concept of bagging and consider a binary classification problem. We are either classifying an observation as 0 or as 1.In bagging, T bootstrap samples are selected, a classifier is fitted on each of these samples, and the models are trained in parallel. In a Random Forest, decision trees are trained in parallel. Then the results of all classifiers are averaged into a bagging classifier:Formula for a Bagging ClassifierLet us consider 3 classifiers and the result for the classification can either be right or wrong. If we plot the results of the 3 classifiers, there are regions in which the classifiers will be wrong. These regions are represented in red in the figure below.Example case in which Bagging works wellThe above example works pretty well as when one classifier is wrong, the two others are correct. By voting classifier, you can achieve a better accuracy. However, there are cases where Bagging does not work properly, when all classifiers are mistaken to be in the same region.Due to this reason, the intuition behind the discovery of Boosting was the following :instead of training parallel models, one should train models sequentiallyeach model should focus on where the performance of the previous classifier was poorWith this intuition, Boosting algorithm was introduced. Let us understand what Boosting is all about.What is Boosting?Boosting is an ensemble modeling technique which attempts to build a strong classifier from the number of weak classifiers. It is done by building a model using weak models in series. First, a model is built from the training data. Then the second model is built which tries to correct the errors present in the first model. This procedure is continued and models are added until either the complete training data set is predicted correctly or the maximum number of models are added.Boosting being a sequential process, each subsequent model attempts to correct the errors of the previous model. It is focused on reducing the bias unlike bagging. It makes the boosting algorithms prone to overfitting. To avoid overfitting, parameter tuning plays an important role in boosting algorithms, which will be discussed in the later part of this article. Some examples of boosting are XGBoost, GBM, ADABOOST etc..How can boosting identify weak learners?To find weak learners, we apply base learning (ML) algorithms with a different distribution. As each time base learning algorithm is applied, it generates a new weak prediction rule. This is an iterative process. After many iterations, the boosting algorithm combines these weak rules into a single strong prediction rule.How do we choose a different distribution for each round?Step 1: The base learner takes all the distributions and assigns equal weight or attention to each observation.Step 2: If there is any prediction error caused by first base learning algorithm, then we pay higher attention to observations having prediction error. Then, we apply the next base learning algorithm.Step 3: Iterate Step 2 till the limit of base learning algorithm is reached or higher accuracy is achieved.Finally, it combines the outputs from weak learner and creates a strong learner which eventually improves the prediction power of the model. Boosting gives higher focus to examples which are mis-classified or have higher errors by preceding weak rules.How would you classify an email as SPAM or not?Our initial approach would be to identify ‘SPAM’ and ‘NOT SPAM’ emails using the following criteria. If: Email has only one image file (promotional image), It’s a SPAM.Email has only link(s), It’s a SPAM.Email body consists of sentences like “You won a prize money of $ xxxxxx”, It’s a SPAM.Email from our official domain “www.knowledgehut.com” , Not a SPAM.Email from known source, Not a SPAM.Individually, these rules are not powerful enough to classify an email into ‘SPAM’ or ‘NOT SPAM’. Therefore, these rules are called as weak learner.To convert weak learner to strong learner, we’ll combine the prediction of each weak learner using methods like:Using average/ weighted averageConsidering prediction has higher voteExample: Above, we have defined 5 weak learners. Out of these 5, 3 are voted as ‘SPAM’ and 2 are voted as ‘Not a SPAM’. In this case, by default, we’ll consider an email as SPAM because we have higher(3) vote for ‘SPAM’Boosting helps in training a series of low performing algorithms, called weak learners, simply by adjusting the error metric over time. Weak learners are considered to be those algorithms whose error rate is slightly under 50% as illustrated below:Weighted errorsLet us consider data points on a 2D plot. Some of the data points will be well classified, others won’t. The weight attributed to each error when computing the error rate is 1/n where n is the number of data points to classify.Now if we apply some weight to the errors :You might now notice that we give more weight to the data points that are not well classified. An illustration of the weighting process is mentioned below:Example of weighting processIn the end, we want to build a strong classifier that may look like the figure mentioned below:Strong ClassifierTree stumpsThere might be a question in your mind about how many classifiers should one implement in order to ensure it works well. And how is each classifier chosen at each step?Well, Tree stumps defines a 1-level decision tree. At each step, we need to find the best stump, i.e the best data split, which will minimize the overall error. You can see a stump as a test, in which the assumption is that everything that lies on one side belongs to class 1, and everything that lies on the other side belongs to class 0.Many such combinations are possible for a tree stump. Let us look into an example to understand how many combinations we face.3 data points to splitWell there are 12 possible combinations. Let us check how.12 StumpsThere are 12 possible “tests” we could make. The “2” on the side of each separating line simply represents the fact that all points on one side could be points that belong to class 0, or to class 1. Therefore, there are 2 tests embedded in it.At each iteration t, we will choose ht the weak classifier that splits best the data, by reducing the overall error rate the most. Recall that the error rate is a modified error rate version that takes into account what has been introduced before.Finding the best splitThe best split is found by identifying at each iteration t, the best weak classifier ht, generally a decision tree with 1 node and 2 leaves (a stump). Let us consider an example of credit defaulter, i.e whether a person who borrowed money will return or not.Identifying the best splitIn this case, the best split at time t is to stump on the Payment history, since the weighted error resulting from this split is minimum.Simply note that decision tree classifiers like these ones can in practice be deeper than a simple stump. This will be considered as a hyper-parameter.Combining classifiersIn the next step we combine the classifiers into a Sign classifier, and depending on which side of the frontier a point will stand, it is classified as 0 or 1. It can be achieved by:Combining classifiersYou can improve the classifier by adding weights on each classifier, to avoid giving the same importance to the different classifiers.AdaBoostPseudo-codePseudo-codeThe key elements to keep in mind are:Z is a constant whose role is to normalize the weights so that they add up to 1αt is a weight that we apply to each classifierThis algorithm is called AdaBoost or Adaptive Boosting. This is one of the most important algorithms among all boosting methods.ComputationBoosting algorithms are generally fast to train, although we consider every stump possible and compute exponentials recursively.Well, if we choose αt and Z properly, the weights that are supposed to change at each step simplify to:Weights after choice of α and ZTypes of Boosting AlgorithmsUnderlying engine used for boosting algorithms can be anything.  It can be decision stamp, margin-maximizing classification algorithm etc. There are many boosting algorithms which use other types of engines such as: AdaBoost (Adaptive Boosting)Gradient Tree BoostingXGBoostIn this article, we will focus on AdaBoost and Gradient Boosting followed by their respective Python codes and a little bit about XGBoost.Where are Boosted algorithms required?Boosted algorithms are mainly used when there is plenty of data to make a prediction and high predictive power is expected. It is used to reduce bias and variance in supervised learning. It combines multiple weak predictors to build strong predictor.The underlying engine used for boosting algorithms can be anything. For instance, AdaBoost is a boosting done on Decision stump. There are many other boosting algorithms which use other types of engine such as:GentleBoostGradient BoostingLPBoostBrownBoostAdaptive BoostingAdaptive Boosting, or most commonly known AdaBoost, is a Boosting algorithm. This algorithm uses the method to correct its predecessor. It pays more attention to under fitted training instances by the previous model. Thus, at every new predictor the focus is more on the complicated cases more than the others.It fits a sequence of weak learners on different weighted training data. It starts by predicting the original data set and gives equal weight to each observation. If prediction is incorrect using the first learner, then it gives higher weight to observation which have been predicted incorrectly. Being an iterative process, it continues to add learner(s) until a limit is reached in the number of models or accuracy.Mostly, AdaBoost uses decision stamps. But, we can use any machine learning algorithm as base learner if it accepts weight on training data set. We can use AdaBoost algorithms for both classification and regression problems.Let us consider the example of the image mentioned above. In order to build an AdaBoost classifier, consider that as a first base classifier a Decision Tree algorithm is trained to make predictions on our training data. Applying the following methodology of AdaBoost, the weight of the misclassified training instances is increased. Then the second classifier is trained and the updated weights are acknowledged. It repeats the procedure over and over again.At the end of every model prediction we end up boosting the weights of the misclassified instances so that the next model does a better job on them, and so on.This sequential learning technique might sound similar to Gradient Descent, except that instead of tweaking a single predictor’s parameter to minimize the cost function, AdaBoost adds predictors to the ensemble, gradually making it better.One disadvantage of this algorithm is that the model cannot be parallelized since each predictor can only be trained after the previous one has been trained and evaluated.Below are the steps for performing the AdaBoost algorithm:Initially, all observations are given equal weights.A model is built on a subset of data.Using this model, predictions are made on the whole dataset.Errors are calculated by comparing the predictions and actual values.While creating the next model, higher weights are given to the data points which were predicted incorrectly.Weights can be determined using the error value. For instance,the higher the error the more is the weight assigned to the observation.This process is repeated until the error function does not change, or the maximum limit of the number of estimators is reached.Hyperparametersbase_estimators: specify the base type estimator, i.e. the algorithm to be used as base learner.n_estimators: It defines the number of base estimators, where the default is 10 but you can increase it in order to obtain a better performance.learning_rate: same impact as in gradient descent algorithmmax_depth: Maximum depth of the individual estimatorn_jobs: indicates to the system how many processors it is allowed to use. Value of ‘-1’ means there is no limit;random_state: makes the model’s output replicable. It will always produce the same results when you give it a fixed value as well as the same parameters and training data.Now, let us take a quick look at how to use AdaBoost in Python using a simple example on handwritten digit recognition.import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.ensemble import AdaBoostClassifier from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import accuracy_score from sklearn.model_selection import cross_val_score from sklearn.model_selection import cross_val_predict from sklearn.model_selection import train_test_split from sklearn.model_selection import learning_curve from sklearn.datasets import load_digitsLet us load the data :dataset = load_digits() X = dataset['data'] y = dataset['target']X contains arrays of length 64 which are simply flattened 8x8 images. The aim of this dataset is to recognize handwritten digits. Let’s take a look at a given handwritten digit:plt.imshow(X[4].reshape(8,8))If we stick to a Decision Tree Classifier of depth 1 (a stump), here’s how to implement AdaBoost classifier:reg_ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1)) scores_ada = cross_val_score(reg_ada, X, y, cv=6) scores_ada.mean()0.2636257855582272And it should head a result of around 26%, which can largely be improved. One of the key parameters is the depth of the sequential decision tree classifiers. How does accuracy improve with depth of the decision trees?score = [] for depth in [1,2,10] : reg_ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=depth)) scores_ada = cross_val_score(reg_ada, X, y, cv=6) score.append(scores_ada.mean()) score[0.2636257855582272, 0.5902852679072207, 0.9527524912410157]And the maximal score is reached for a depth of 10 in this simple example, with an accuracy of 95.3%.Gradient BoostingThis is another very popular Boosting algorithm which works pretty similar to what we’ve seen for AdaBoost. Gradient Boosting works by sequentially adding the previous predictors underfitted predictions to the ensemble, ensuring the errors made previously are corrected.The difference lies in what it does with the underfitted values of its predecessor. Contrary to AdaBoost, which tweaks the instance weights at every interaction, this method tries to fit the new predictor to the residual errors made by the previous predictor.So that you can understand Gradient Boosting it is important to understand Gradient Descent first.Below are the steps for performing the Gradient Boosting algorithm:A model is built on a subset of data.Using this model, predictions are made on the whole dataset.Errors are calculated by comparing the predictions and actual values.A new model is created using the errors calculated as target variable. Our objective is to find the best split to minimize the error.The predictions made by this new model are combined with the predictions of the previous.New errors are calculated using this predicted value and actual value.This process is repeated until the error function does not change, or the maximum limit of the number of estimators is reached.Hyperparametersn_estimators: It controls the number of weak learners.Learning_rate: Controls the contribution of weak learners in the final combination. There is a trade-off between learning_rate and n_estimators.min_samples_split: Minimum number of observation which is required in a node to be considered for splitting. It is used to control overfitting.min_samples_leaf: Minimum samples required in a terminal or leaf node. Lower values should be chosen for imbalanced class problems since the regions in which the minority class will be in the majority will be very small.min_weight_fraction_leaf: similar to the previous but defines a fraction of the total number of observations instead of an integer.max_depth : maximum depth of a tree. Used to control overfitting.max_lead_nodes : maximum number of terminal leaves in a tree. If this is defined max_depth is ignored.max_features : number of features it should consider while searching for the best split.You can tune loss function for better performance.Implementation in PythonYou can find Gradient Boosting function in Scikit-Learn’s library.# for regression from sklearn.ensemble import GradientBoostingRegressor model = GradientBoostingRegressor(n_estimators=3,learning_rate=1) model.fit(X,Y) # for classification from sklearn.ensemble import GradientBoostingClassifier model = GradientBoostingClassifier() model.fit(X,Y)XGBoostXG Boost or Extreme Gradient Boosting is an advanced implementation of the Gradient Boosting. This algorithm has high predictive power and is ten times faster than any other gradient boosting techniques. Moreover, it includes a variety of regularization which reduces overfitting and improves overall performance.AdvantagesIt implements regularization which helps in reducing overfit (Gradient Boosting does not have);It implements parallel processing which is much faster than Gradient Boosting;Allows users to define custom optimization objectives and evaluation criteria adding a whole new dimension to the model;XGBoost has an in-built routine to handle missing values;XGBoost makes splits up to the max_depth specified and then starts pruning the tree backwards and removes splits beyond which there is no positive gain;XGBoost allows a user to run a cross-validation at each iteration of the boosting process and thus it is easy to get the exact optimum number of boosting iterations in a single run.Boosting algorithms represent a different machine learning perspective which is turning a weak model to a stronger one to fix its weaknesses. I hope this article helped you understand how boosting works.We have covered most of the topics related to algorithms in our series of machine learning blogs, click here. If you are inspired by the opportunities provided by machine learning, enroll in our  Data Science and Machine Learning Courses for more lucrative career options in this landscape.
Rated 4.5/5 based on 12 customer reviews

Boosting and AdaBoost in Machine Learning

8068
Boosting and AdaBoost in Machine Learning

Ensemble learning is a strategy in which a group of models are used to find a solution to a challenging problem, by using a strategy and combining diverse machine learning models into one single predictive model.

In general, ensemble methods are mainly used for improving the overall performance accuracy of a model and combine several different models, also known as the base learners, to predict the results, instead of using a single model.

In one of the articles related to ensemble learning, we have already discussed about the popular ensemble method, Bootstrap Aggregation. Bagging tries to implement similar learners on small sample populations and then takes a mean of all the predictions. It combines Bootstrapping and Aggregation to form one ensemble model. It basically reduces the variance error and helps to avoid overfitting. In this article we will look into the limitations of bagging and how a boosting algorithm can be used to overcome those limitations. We will also learn about various types of boosting algorithms and implement one of them in Python. Let’s get started.

What are the limitations of Bagging?

Let us recall the concept of bagging and consider a binary classification problem. We are either classifying an observation as 0 or as 1.

In bagging, T bootstrap samples are selected, a classifier is fitted on each of these samples, and the models are trained in parallel. In a Random Forest, decision trees are trained in parallel. Then the results of all classifiers are averaged into a bagging classifier:

What are the limitations of BaggingFormula for a Bagging ClassifierLet us consider 3 classifiers and the result for the classification can either be right or wrong. If we plot the results of the 3 classifiers, there are regions in which the classifiers will be wrong. These regions are represented in red in the figure below.
Example case in which Bagging works wellExample case in which Bagging works wellThe above example works pretty well as when one classifier is wrong, the two others are correct. By voting classifier, you can achieve a better accuracy. However, there are cases where Bagging does not work properly, when all classifiers are mistaken to be in the same region.

Bagging limitationsDue to this reason, the intuition behind the discovery of Boosting was the following :

  • instead of training parallel models, one should train models sequentially
  • each model should focus on where the performance of the previous classifier was poor

With this intuition, Boosting algorithm was introduced. Let us understand what Boosting is all about.

What is Boosting?

Boosting is an ensemble modeling technique which attempts to build a strong classifier from the number of weak classifiers. It is done by building a model using weak models in series. First, a model is built from the training data. Then the second model is built which tries to correct the errors present in the first model. This procedure is continued and models are added until either the complete training data set is predicted correctly or the maximum number of models are added.

Boosting being a sequential process, each subsequent model attempts to correct the errors of the previous model. It is focused on reducing the bias unlike bagging. It makes the boosting algorithms prone to overfitting. To avoid overfitting, parameter tuning plays an important role in boosting algorithms, which will be discussed in the later part of this article. Some examples of boosting are XGBoost, GBM, ADABOOST etc..

How can boosting identify weak learners?

To find weak learners, we apply base learning (ML) algorithms with a different distribution. As each time base learning algorithm is applied, it generates a new weak prediction rule. This is an iterative process. After many iterations, the boosting algorithm combines these weak rules into a single strong prediction rule.

How do we choose a different distribution for each round?

Step 1: The base learner takes all the distributions and assigns equal weight or attention to each observation.
Step 2: If there is any prediction error caused by first base learning algorithm, then we pay higher attention to observations having prediction error. Then, we apply the next base learning algorithm.
Step 3: Iterate Step 2 till the limit of base learning algorithm is reached or higher accuracy is achieved.

Finally, it combines the outputs from weak learner and creates a strong learner which eventually improves the prediction power of the model. Boosting gives higher focus to examples which are mis-classified or have higher errors by preceding weak rules.

How would you classify an email as SPAM or not?

Our initial approach would be to identify ‘SPAM’ and ‘NOT SPAM’ emails using the following criteria. If: 

  1. Email has only one image file (promotional image), It’s a SPAM.
  2. Email has only link(s), It’s a SPAM.
  3. Email body consists of sentences like “You won a prize money of $ xxxxxx”, It’s a SPAM.
  4. Email from our official domain “www.knowledgehut.com” , Not a SPAM.
  5. Email from known source, Not a SPAM.

Individually, these rules are not powerful enough to classify an email into ‘SPAM’ or ‘NOT SPAM’. Therefore, these rules are called as weak learner.

To convert weak learner to strong learner, we’ll combine the prediction of each weak learner using methods like:

  • Using average/ weighted average
  • Considering prediction has higher vote

Example: Above, we have defined 5 weak learners. Out of these 5, 3 are voted as ‘SPAM’ and 2 are voted as ‘Not a SPAM’. In this case, by default, we’ll consider an email as SPAM because we have higher(3) vote for ‘SPAM’

Boosting helps in training a series of low performing algorithms, called weak learners, simply by adjusting the error metric over time. Weak learners are considered to be those algorithms whose error rate is slightly under 50% as illustrated below:

Classifier error rate

Weighted errors

Let us consider data points on a 2D plot. Some of the data points will be well classified, others won’t. The weight attributed to each error when computing the error rate is 1/n where n is the number of data points to classify.

Now if we apply some weight to the errors :

Weighted errors

You might now notice that we give more weight to the data points that are not well classified. An illustration of the weighting process is mentioned below:

Example of weighting process Example of weighting processIn the end, we want to build a strong classifier that may look like the figure mentioned below:

Strong ClassifierStrong ClassifierTree stumps

There might be a question in your mind about how many classifiers should one implement in order to ensure it works well. And how is each classifier chosen at each step?

Well, Tree stumps defines a 1-level decision tree. At each step, we need to find the best stump, i.e the best data split, which will minimize the overall error. You can see a stump as a test, in which the assumption is that everything that lies on one side belongs to class 1, and everything that lies on the other side belongs to class 0.

Many such combinations are possible for a tree stump. Let us look into an example to understand how many combinations we face.

Tree stumps3 data points to splitWell there are 12 possible combinations. Let us check how.

Tree Stumps12 StumpsThere are 12 possible “tests” we could make. The “2” on the side of each separating line simply represents the fact that all points on one side could be points that belong to class 0, or to class 1. Therefore, there are 2 tests embedded in it.

At each iteration t, we will choose ht the weak classifier that splits best the data, by reducing the overall error rate the most. Recall that the error rate is a modified error rate version that takes into account what has been introduced before.

Finding the best split

The best split is found by identifying at each iteration t, the best weak classifier ht, generally a decision tree with 1 node and 2 leaves (a stump). Let us consider an example of credit defaulter, i.e whether a person who borrowed money will return or not.

Finding the best splitIdentifying the best splitIn this case, the best split at time t is to stump on the Payment history, since the weighted error resulting from this split is minimum.

Simply note that decision tree classifiers like these ones can in practice be deeper than a simple stump. This will be considered as a hyper-parameter.

Combining classifiers

In the next step we combine the classifiers into a Sign classifier, and depending on which side of the frontier a point will stand, it is classified as 0 or 1. It can be achieved by:

Combining classifiersCombining classifiersYou can improve the classifier by adding weights on each classifier, to avoid giving the same importance to the different classifiers.

AdaBoost AdaBoostPseudo-codePseudo-codePseudo-codeThe key elements to keep in mind are:

  • Z is a constant whose role is to normalize the weights so that they add up to 1
  • αt is a weight that we apply to each classifier

This algorithm is called AdaBoost or Adaptive Boosting. This is one of the most important algorithms among all boosting methods.

Computation

Boosting algorithms are generally fast to train, although we consider every stump possible and compute exponentials recursively.

Well, if we choose αt and Z properly, the weights that are supposed to change at each step simplify to:

Weights after choice of α and ZWeights after choice of α and Z

Types of Boosting Algorithms

Underlying engine used for boosting algorithms can be anything.  It can be decision stamp, margin-maximizing classification algorithm etc. There are many boosting algorithms which use other types of engines such as: 

  1. AdaBoost (Adaptive Boosting)
  2. Gradient Tree Boosting
  3. XGBoost

In this article, we will focus on AdaBoost and Gradient Boosting followed by their respective Python codes and a little bit about XGBoost.

Where are Boosted algorithms required?

Boosted algorithms are mainly used when there is plenty of data to make a prediction and high predictive power is expected. It is used to reduce bias and variance in supervised learning. It combines multiple weak predictors to build strong predictor.

The underlying engine used for boosting algorithms can be anything. For instance, AdaBoost is a boosting done on Decision stump. There are many other boosting algorithms which use other types of engine such as:

  1. GentleBoost
  2. Gradient Boosting
  3. LPBoost
  4. BrownBoost

Adaptive Boosting

Adaptive Boosting, or most commonly known AdaBoost, is a Boosting algorithm. This algorithm uses the method to correct its predecessor. It pays more attention to under fitted training instances by the previous model. Thus, at every new predictor the focus is more on the complicated cases more than the others.

It fits a sequence of weak learners on different weighted training data. It starts by predicting the original data set and gives equal weight to each observation. If prediction is incorrect using the first learner, then it gives higher weight to observation which have been predicted incorrectly. Being an iterative process, it continues to add learner(s) until a limit is reached in the number of models or accuracy.

Mostly, AdaBoost uses decision stamps. But, we can use any machine learning algorithm as base learner if it accepts weight on training data set. We can use AdaBoost algorithms for both classification and regression problems.

Let us consider the example of the image mentioned above. In order to build an AdaBoost classifier, consider that as a first base classifier a Decision Tree algorithm is trained to make predictions on our training data. Applying the following methodology of AdaBoost, the weight of the misclassified training instances is increased. Then the second classifier is trained and the updated weights are acknowledged. It repeats the procedure over and over again.

At the end of every model prediction we end up boosting the weights of the misclassified instances so that the next model does a better job on them, and so on.

This sequential learning technique might sound similar to Gradient Descent, except that instead of tweaking a single predictor’s parameter to minimize the cost function, AdaBoost adds predictors to the ensemble, gradually making it better.

One disadvantage of this algorithm is that the model cannot be parallelized since each predictor can only be trained after the previous one has been trained and evaluated.

Below are the steps for performing the AdaBoost algorithm:

  1. Initially, all observations are given equal weights.
  2. A model is built on a subset of data.
  3. Using this model, predictions are made on the whole dataset.
  4. Errors are calculated by comparing the predictions and actual values.
  5. While creating the next model, higher weights are given to the data points which were predicted incorrectly.
  6. Weights can be determined using the error value. For instance,the higher the error the more is the weight assigned to the observation.
  7. This process is repeated until the error function does not change, or the maximum limit of the number of estimators is reached.

Hyperparameters

base_estimators: specify the base type estimator, i.e. the algorithm to be used as base learner.

n_estimators: It defines the number of base estimators, where the default is 10 but you can increase it in order to obtain a better performance.

learning_rate: same impact as in gradient descent algorithm

max_depth: Maximum depth of the individual estimator

n_jobsindicates to the system how many processors it is allowed to use. Value of ‘-1’ means there is no limit;

random_state: makes the model’s output replicable. It will always produce the same results when you give it a fixed value as well as the same parameters and training data.

Now, let us take a quick look at how to use AdaBoost in Python using a simple example on handwritten digit recognition.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.ensemble import AdaBoostClassifier
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import accuracy_score
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import cross_val_predict
from sklearn.model_selection import train_test_split
from sklearn.model_selection import learning_curve
from sklearn.datasets import load_digits

Let us load the data :

dataset = load_digits()
X = dataset['data']
y = dataset['target']

X contains arrays of length 64 which are simply flattened 8x8 images. The aim of this dataset is to recognize handwritten digits. Let’s take a look at a given handwritten digit:

plt.imshow(X[4].reshape(8,8))

Hyperparameters

If we stick to a Decision Tree Classifier of depth 1 (a stump), here’s how to implement AdaBoost classifier:

reg_ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1))
scores_ada = cross_val_score(reg_ada, X, y, cv=6)
scores_ada.mean()
0.2636257855582272

And it should head a result of around 26%, which can largely be improved. One of the key parameters is the depth of the sequential decision tree classifiers. How does accuracy improve with depth of the decision trees?

score = []
for depth in [1,2,10] :
reg_ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=depth))
scores_ada = cross_val_score(reg_ada, X, y, cv=6)
score.append(scores_ada.mean())
score
[0.2636257855582272, 0.5902852679072207, 0.9527524912410157]

And the maximal score is reached for a depth of 10 in this simple example, with an accuracy of 95.3%.

Gradient Boosting

This is another very popular Boosting algorithm which works pretty similar to what we’ve seen for AdaBoost. Gradient Boosting works by sequentially adding the previous predictors underfitted predictions to the ensemble, ensuring the errors made previously are corrected.

The difference lies in what it does with the underfitted values of its predecessor. Contrary to AdaBoost, which tweaks the instance weights at every interaction, this method tries to fit the new predictor to the residual errors made by the previous predictor.

So that you can understand Gradient Boosting it is important to understand Gradient Descent first.

Below are the steps for performing the Gradient Boosting algorithm:

  1. A model is built on a subset of data.
  2. Using this model, predictions are made on the whole dataset.
  3. Errors are calculated by comparing the predictions and actual values.
  4. A new model is created using the errors calculated as target variable. Our objective is to find the best split to minimize the error.
  5. The predictions made by this new model are combined with the predictions of the previous.
  6. New errors are calculated using this predicted value and actual value.
  7. This process is repeated until the error function does not change, or the maximum limit of the number of estimators is reached.

Hyperparameters

n_estimators: It controls the number of weak learners.
Learning_rate: Controls the contribution of weak learners in the final combination. There is a trade-off between learning_rate and n_estimators.
min_samples_split: Minimum number of observation which is required in a node to be considered for splitting. It is used to control overfitting.
min_samples_leaf: Minimum samples required in a terminal or leaf node. Lower values should be chosen for imbalanced class problems since the regions in which the minority class will be in the majority will be very small.
min_weight_fraction_leaf: similar to the previous but defines a fraction of the total number of observations instead of an integer.
max_depth : maximum depth of a tree. Used to control overfitting.
max_lead_nodes : maximum number of terminal leaves in a tree. If this is defined max_depth is ignored.
max_features : number of features it should consider while searching for the best split.

You can tune loss function for better performance.

Implementation in Python

You can find Gradient Boosting function in Scikit-Learn’s library.

# for regression
from sklearn.ensemble import GradientBoostingRegressor
model = GradientBoostingRegressor(n_estimators=3,learning_rate=1)
model.fit(X,Y)
# for classification
from sklearn.ensemble import GradientBoostingClassifier
model = GradientBoostingClassifier()
model.fit(X,Y)

XGBoost

XG Boost or Extreme Gradient Boosting is an advanced implementation of the Gradient Boosting. This algorithm has high predictive power and is ten times faster than any other gradient boosting techniques. Moreover, it includes a variety of regularization which reduces overfitting and improves overall performance.

Advantages

  • It implements regularization which helps in reducing overfit (Gradient Boosting does not have);
  • It implements parallel processing which is much faster than Gradient Boosting;
  • Allows users to define custom optimization objectives and evaluation criteria adding a whole new dimension to the model;
  • XGBoost has an in-built routine to handle missing values;
  • XGBoost makes splits up to the max_depth specified and then starts pruning the tree backwards and removes splits beyond which there is no positive gain;
  • XGBoost allows a user to run a cross-validation at each iteration of the boosting process and thus it is easy to get the exact optimum number of boosting iterations in a single run.

Boosting algorithms represent a different machine learning perspective which is turning a weak model to a stronger one to fix its weaknesses. I hope this article helped you understand how boosting works.

We have covered most of the topics related to algorithms in our series of machine learning blogs, click here. If you are inspired by the opportunities provided by machine learning, enroll in our  Data Science and Machine Learning Courses for more lucrative career options in this landscape.

Priyankur

Priyankur Sarkar

Data Science Enthusiast

Priyankur Sarkar loves to play with data and get insightful results out of it, then turn those data insights and results in business growth. He is an electronics engineer with a versatile experience as an individual contributor and leading teams, and has actively worked towards building Machine Learning capabilities for organizations.

Join the Discussion

Your email address will not be published. Required fields are marked *

Suggested Blogs

What are Decision Trees in Machine Learning (Classification And Regression)

Introduction to Machine Learning and its typesMachine Learning is an interdisciplinary field of study and is a sub-domain of Artificial Intelligence. It gives computers the ability to learn and infer from a huge amount of homogeneous data, without having to be programmed explicitly.Types of Machine Learning: Machine Learning can broadly be classified into three types:Supervised Learning: If the available dataset has predefined features and labels, on which the machine learning models are trained, then the type of learning is known as Supervised Machine Learning. Supervised Machine Learning Models can broadly be classified into two sub-parts: Classification and Regression. These have been discussed further in detail.Unsupervised Learning: If the available dataset has predefined features but lacks labels, then the Machine Learning algorithms perform operations on this data to assign labels to it or to reduce the dimensionality of the data. There are several types of Unsupervised Learning Models, the most common of them being: Principal Component Analysis (PCA) and Clustering.Reinforcement Learning: Reinforcement Learning is a more advanced type of learning, where, the model learns from “Experience”. Here, features and labels are not clearly defined. The model is just given a “Situation” and is rewarded or penalized based on the “Outcome”. The model thus learns to optimize the “Situation” to maximize the Rewards and hence improves the “Outcome” with “Experience”.ClassificationClassification is the process of determination/prediction of the category to which a data-point may belong to. It is the process by which a Supervised Learning Algorithm learns to draw inference from the features of a given dataset and predict which class or group or category does the particular data point belongs to.Example of Classification: Let’s assume that we are given a few images of handwritten digits (0-9). The problem statement is to “teach” the machine to classify correctly which image corresponds to which digit. A small sample of the dataset is given below:The machine has to be thus trained, such that, when given an input of any such hand-written digit, it has to correctly classify the digits and mention which digit the image represents. This is classed classification of hand-written digits.Looking at another example which is not image-based, we have 2D data (x1 and x2) which is plotted in the form of a graph shown below.The red and green dots represent two different classes or categories of data. The main goal of the classifier is that given one such “dot” of unknown class, based on its “features”, the algorithm should be able to correctly classify if that dot belongs to the red or green class. This is also shown by the line going through the middle, which correctly classifies the majority of the dots.Applications of Classification: Listed below are some of the real-world applications of classification Algorithms.Face Recognition: Face recognition finds its applications in our smartphones and any other place with Biometric security. Face Recognition is nothing but face detection followed by classification. The classification algorithm determines if the face in the image matches with the registered user or not.Medical Image Classification: Given the data of patients, a model that is well trained is often used to classify if the patient has a malignant tumor (cancer), heart ailments, fractures, etc.RegressionRegression is also a type of supervised learning. Unlike classification, it does not predict the class of the given data. Instead, it predicts the corresponding values of a given dataset based on the “features” it encounters.Example of Regression: For this, we will look at a dataset consisting of California Housing Prices. The contents of this dataset are shown below.Here, there are several columns. Each of the columns shows the “features” based on which the machine learning algorithm predicts the housing price (shown by yellow highlight). The primary goal of the regression algorithm is that, given the features of a given house, it should be able to correctly estimate the price of the house. This is called a regression problem. It is similar to curve fitting and is often confused with the same.Applications of Regression: Listed below are some of the real-world applications of regression Algorithms.Stock Market Prediction: Regression algorithms are used to predict the future price of stocks based on certain past features like time of the day or festival time, etc. Stock Market Prediction also falls under a subdomain of study called Time Series Analysis.Object Detection Algorithms: Object Detection is the process of detection of the location of a given object in an image or video. This process returns the coordinates of the pixel values stating the location of the object in the image. These coordinates are determined by using regression algorithms alongside classification.ClassificationRegressionAssign specific classes to the data based on its features.Predict values based on the features of the dataset.Prediction is discrete or categorical in nature.Prediction is continuous in nature.Introduction to the building blocks of Decision TreesIn order to get started with Decision Trees, it is important to understand the basic building blocks of decision trees. Hence, we start building the concepts slowly with some basic theory.1. EntropyDefinition: It is a commonly used concept in Information Theory and is a measure of “purity” of an arbitrary collection of information.Mathematical Equation:Here, given a collection S, containing positive and negative examples, the Entropy of S is given by the above equation, where, p represents the probability of occurrence of that example in the given data.In a more generalized form, Entropy is given by the following equation:Example: As an example, a sample S is taken, which contains 14 data samples and includes 9 positive and 5 negative samples. The same is denoted by the mathematical notion: [9+, 5­­–].Thus, Entropy of the given sample can be calculated as follows:2. Information GainDefinition: With the knowledge of Entropy, the amount of relevant information that is gained form a given random sample size can be calculated and is known as Information Gain.Mathematical Equation:Here, the Gain (S, A) is the Information Gain of an attribute A relative to a sample S. The Values(A) is a set of all possible values for attribute A.Example: As an example, let’s assume S is a collection of 14 training-examples. Here, in this example, we will consider the Attribute to be Wind and the values of that corresponding attribute will be Weak and Strong. In addition to the previous example information, we will assume that out of the previously mentioned 9 positives and 5 negative samples, 6 positive and 2 negative samples have the value of the attribute Wind=Weak, and the remaining have Wind=Strong. Thus, under such a circumstance, the information gained by the attribute Wind is shown below.Decision TreeIntroduction: Since we have the basic building blocks out of the way, let’s try to understand what exactly is a Decision Tree. As the name suggests, it is a Tree which is developed based on certain decisions taken by the algorithm in accordance with the given data that it has been trained on.In simple words, a Decision Tree uses the features in the given data to perform Supervised Learning and develop a tree-like structure (data structure) whose branches are developed in such a way that given the feature-set, the decision tree can predict the expected output relatively accurately.Example:  Let us look at the structure of a decision tree. For this, we will take up an example dataset called the “PlayTennis” dataset. A sample of the dataset is shown below.In summary, the target of the model is to predict if the weather conditions are suitable to play tennis or not, as guided by the dataset shown above.As it can be seen in the dataset, it contains certain information (features) for each day. In this, we have the feature-attributes: Outlook, Temperature, Humidity and Wind and the target-attribute PlayTennis. Each of these attributes can take up certain values, for example, the attribute Outlook has the values Sunny, Rain and Overcast.With a clear idea of the dataset, jumping a bit forward, let us look at the structure of the learned Decision Tree as developed from the above dataset.As shown above, it can clearly be seen that, given certain values for each of the attributes, the learned decision tree is capable of giving a clear answer as to whether the weather is suitable for Tennis or not.Algorithm: With the overall intuition of decision trees, let us look at the formal Algorithm:ID3(Samples, Target_attribute, Attributes):Create a root node for the TreeIf all the Samples are positive, Return a single-node tree Root, with label = +If all the Samples are negative, Return a single-node tree Root, with label = –If Attribute is empty, Return the single-node tree Root with the label = Most common value of the Target_attribute among the Samples.Otherwise:A ← the attribute from Attributes that best classifies the SamplesThe decision attribute for Root ← AFor each possible value of A:Add a new tree branch below Root, corresponding to the test A = viLet the Samplesvi be the subset of Samples that have value vi for AIf Samplesvi is empty:Then below the new branch add a leaf node with the label = most common value of Target_attribute in the samplesElse below the new branch add the subtree:ID3(Samplesvi, Target_attribute, Attributes – {A})EndReturn RootConnecting the dots: Since the overall idea of decision trees have been explained, let’s try to figure out how Entropy and Information Gain fits into this entire process.Entropy (E) is used to calculate Information Gain, which is used to identify which attribute of a given dataset provides the highest amount of information. The attribute which provides the highest amount of information for the given dataset is considered to have more contribution towards the outcome of the classifier and hence is given the higher priority in the tree.For Example, considering the PlayTennis Example, if we calculate the Information Gain for two corresponding Attributes: Humidity and Wind, we would find that Humidity plays a more important role in deciding whether to play tennis or not. Hence, in this case, Humidity is considered as a better classifier. The detailed calculation is shown in the figure below:Applications of Decision TreeWith the basic idea out of the way, let’s look at where decision trees can be used:Select a flight to travel: Decision trees are very good at classification and hence can be used to select which flight would yield the best “bang-for-the-buck”. There are a lot of parameters to consider, such as if the flight is connecting or non-stop, or how reliable is the service record of the given airliner, etc.Selecting alternative products: Often in companies, it is important to determine which product will be more profitable at launch. Given the sales attributes such as market conditions, competition, price, availability of raw materials, demand, etc. a Decision Tree classifier can be used to accurately determine which of the products would maximize the profits.Sentiment Analysis: Sentiment Analysis is the determination of the overall opinion of a given piece of text and is especially used to determine if the writer’s comment towards a given product/service is positive, neutral or negative. Decision trees are very versatile classifiers and are used for sentiment analysis in many Natural Language Processing (NLP) applications.Energy Consumption: It is very important for electricity supply boards to correctly predict the amount of energy consumption in the near future for a particular region. This is to make sure that un-used power can be diverted towards an area with a higher demand to keep a regular and uninterrupted supply of power throughout the grid. Decision Trees are often used to determine which region is expected to require more or less power in the up-coming time-frame.Fault Diagnosis: In the Engineering domain, one of the widely used applications of decision trees is the determination of faults. In the case of load-bearing rotatory machines, it is important to determine which of the component(s) have failed and which ones can directly or indirectly be affected by the failure. This is determined by a set of measurements that are taken. Unfortunately, there are numerous measurements to take and among them, there are some measurements which are not relevant to the detection of the fault. A Decision Tree classifier can be used to quickly determine which of these measurements are relevant in the determination of the fault.Advantages of Decision TreeListed below are some of the advantages of Decision Trees:Comprehensive: Another significant advantage of a decision tree is that it forces the algorithm to take into consideration all the possible outcomes of a decision and traces each path to a conclusion.Specific: The output of decision trees is very specific and reduces uncertainty in the prediction. Hence, they are considered as really good classifiers.Easy to use: Decision Trees are one of the simplest, yet most versatile algorithms in Machine Learning. It is based on simple math and no complex formulas. They are easy to visualize, understand and explain.Versatile: A lot of business problems can be solved using Decision Trees. They find their applications in the field of Engineering, Management, Medicine, etc. basically, any situation where data is available and a decision needs to be taken in uncertain conditions.Resistant to data abnormalities: Data is never perfect and there are always many abnormalities in the dataset. Some of the most common abnormalities are outliers, missing data and noise. While most Machine Learning algorithms fail with even a minor set of abnormalities, Decision Trees are very resilient and is able to handle a fair percentage of such abnormalities quite well without altering the results.Visualization of the decision taken: Often in Machine Learning models, data scientists struggle to reason as to why a certain model is giving a certain set of outputs. Unfortunately, for most of the algorithms, it is not possible to clearly determine and visualize the actual process of classification that leads to the final outcome. However, decision trees are very easy to visualize. Once the tree is trained, it can be visualized and the programmer can see exactly how and why the conclusion was reached. It is also easy to explain the outcome to a non-technical team with the “tree” type visualization. This is why many organizations prefer to use decision trees over other Machine Learning Algorithms.Limitations of Decision TreeListed below are some of the limitations of Decision Trees:Sensitivity to hyperparameter tuning: Decision Trees are very sensitive to hyperparameter tuning. Hyperparameters are those parameters which are in control of the programmer and can be tuned to get better performance out of a given model. Unfortunately, the output of a decision tree can vary drastically if the hyperparameters are inaccurately tuned.Overfitting: Decision trees are prone to overfitting. Overfitting is a concept where the model learns the data too well and hence performs well on training dataset but fails to perform on testing dataset. Decision trees are prone to overfitting if the breadth and depth of the tree is set to very high for a simpler dataset.Underfitting: Similar to overfitting, decision trees are also prone to underfitting. Underfitting is a concept where the model is too simple for it to learn the dataset effectively. Decision tree suffers from underfitting if the breadth and depth of the model or the number of nodes are set too low. This does not allow the model to fit the data properly and hence fails to learn.Code ExamplesWith the theory out of the way, let’s look at the practical implementation of decision tree classifiers and regressors.1. ClassificationIn order to conduct classification, a diabetes dataset from Kaggle has been used. It can be downloaded.The initial step for any data science application is data visualization. Hence, the dataset is shown below:The highlighted column is the target value that the model is expected to predict, given the parameters.Load the Libraries. We will be using pandas to load and manipulate data. Sklearn is used for applying Machine Learning models on the data.# Load libraries import pandas as pd from sklearn.tree import DecisionTreeClassifier # Import Decision Tree Classifier from sklearn.model_selection import train_test_split # Import train_test_split function from sklearn import metrics #Import scikit-learn metrics module for accuracy calculation Load the data. Pandas is used to read the data from the CSV.col_names = ['pregnant', 'glucose', 'bp', 'skin', 'insulin', 'bmi', 'pedigree', 'age', 'label'] # load dataset pima = pd.read_csv("pima-indians-diabetes.csv", header=None, names=col_names)Feature Selection: The relevant features are selected for the classification.#split dataset in features and target variable feature_cols = ['pregnant', 'insulin', 'bmi', 'age','glucose','bp','pedigree'] X = pima[feature_cols] # Features y = pima.label # Target variablesplitting the data: The dataset needs to be split into training and testing data. The training data is used to train the model, while the testing data is used to test the model’s performance on unseen data.# Split dataset into training set and test set X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=1) # 70% training and 30% testBuilding the decision tree. These few lines initialize, train and predict on the dataset.# Create Decision Tree classifier object clf = DecisionTreeClassifier() # Train Decision Tree Classifier clf = clf.fit(X_train,y_train) #Predict the response for test dataset y_pred = clf.predict(X_test)The model’s accuracy is evaluated by using Sklearn’s metrics library. # Model Accuracy, how often is the classifier correct? print("Accuracy:",metrics.accuracy_score(y_test, y_pred)) Output: Accuracy: 0.6753246753246753This will generate the decision tree that is shown in the following image2. RegressionIn order to conduct classification, a diabetes dataset from Kaggle has been used.For this example, we will generate a Numpy Array which will simulate a scatter plot resembling a sine wave with a few randomly added noise elements. # Import the necessary modules and libraries import numpy as np from sklearn.tree import DecisionTreeRegressor import matplotlib.pyplot as plt # Create a random dataset rng = np.random.RandomState(1) X = np.sort(5 * rng.rand(80, 1), axis=0) y = np.sin(X).ravel() y[::5] += 3 * (0.5 - rng.rand(16))This time we create two regression models to experiment and see how overfitting looks like for a decision tree. Hence, we initialize the two Decision Tree Regression objects and train them on the given data.# Fit regression model regr_1 = DecisionTreeRegressor(max_depth=2) regr_2 = DecisionTreeRegressor(max_depth=5) regr_1.fit(X, y) regr_2.fit(X, y)After fitting the model, we predict on a custom test dataset and plot the results to see how it performed.# Predict X_test = np.arange(0.0, 5.0, 0.01)[:, np.newaxis] y_1 = regr_1.predict(X_test) y_2 = regr_2.predict(X_test) # Plot the results plt.figure() plt.scatter(X, y, s=20, edgecolor="black",             c="darkorange", label="data") plt.plot(X_test, y_1, color="cornflowerblue",         label="max_depth=2", linewidth=2) plt.plot(X_test, y_2, color="yellowgreen", label="max_depth=5", linewidth=2) plt.xlabel("data") plt.ylabel("target") plt.title("Decision Tree Regression") plt.legend() plt.show() The graph that is thus generated is shown below. Here we can clearly see that for a simple dataset  when we used max_depth=5 (green), the model started to overfit and learned the patterns of the noise along with the sine wave. Such kinds of models do not perform well. Meanwhile, for max_depth=3 (blue), it has fitted the dataset in a better way when compared to the other one.ConclusionIn this article, we tried to build an intuition, by starting from the basics of the theory behind the working of a decision tree classifier. However, covering every aspect of detail is beyond the scope of this article. Hence, it is suggested to go through this book to dive deeper into the specifics. Further, moving on, the code snippets introduces the “Hello World” of how to use both, real-world data and artificially generated data to train a Decision Tree model and predict using the same. This will allow any novice to get an overall balanced theoretical and practical idea about the workings of Classification and Regression Trees and their implementation.
Rated 4.5/5 based on 12 customer reviews
8113
What are Decision Trees in Machine Learning (Class...

Introduction to Machine Learning and its typesMach... Read More

Top Data Science Trends in 2020

Industry experts are of the view that 2020 will be a huge year for data science and AI. The expected growth rate for AI market will be at 118.6 billion by 2025. The focus areas in the overall AI market will include everything from natural language processing to robotic process automation. Since the beginning of digital era, data has been growing at the speed of light! There will only be a surge in this growth. New data will not only generate more innovative use cases but also spearhead a revolution of innovation.About 77% of the devices today have AI incorporated into them. Smart devices, Netflix recommendations, Amazon’s Alexa Google Home have transformed the way we live in the digital age.  The renowned AI-powered virtual nurses “Molly” and “Angel”, have taken healthcare to new heights and robots have already been performing various surgical procedures.Dynamic technologies like data science and AI have some intriguing data science trends to watch out for, in 2020. Check out the top 6 data science trends in 2020 any data science enthusiast should know:1. Advent of Deep LearningSimply put, deep learning is a machine learning technique that trains computers to think and act like humans i.e., by example. Ever since, deep learning models have proven their efficacy by exceeding human limitations and performance. Deep learning models are usually trained using a large set of labelled data and multi-layered neural network architectures.What’s new for Deep Learning in 2020?In 2020, deep learning will be quite significant. Its capacity to foresee and understand human behaviour and how enterprises can utilize this knowledge to stay ahead of their competitors will come in handy.2. Spotlight on Augmented AnalyticsAlso hailed as the future of Business Intelligence, Augmented analytics employs machine learning/artificial intelligence (ML/AI) techniques to automate data preparation, insight discovery and sharing, data science and ML model development, management and deployment. This can be greatly beneficial for companies to improve their offerings and customer experience. The global augmented analytics market size is projected to reach $29,856 million by 2025. Its growth rate is expected to be at a CAGR of 28.4% from 2018 to 2025.What’s New for Augmented Analytics in 2020?This year, augmented analytics platforms will help enterprises leverage social component. The use of interactive dashboards and visualizations in augmented analytics will help stakeholders share important insights and create a crystal-clear narrative that echoes the company’s mission.3. Impact of IoT, ML and AI2020 will see the rise of AI/ML, 5G, cybersecurity and IoT. The rise in automation will create opportunities for new skills to be explored. Upskilling in emerging new technologies will make professionals competent in the dynamic tech space today. As per a survey by IDC, over 75% of organizations will invest in reskilling programs or their workforce to bridge the rising skill gap by 2025.What’s new for IoT, ML and AI in 2020?It has been estimated that over 24 billion devices will be connected to the Internet of Things this year. This means industries can create a world of difference by developing smart devices that make a difference to the way we live.4. Better Mobile Analytics StrategiesMobile analytics deals with measuring and analysing data created across various mobile platform sites and applications alone. It helps businesses keep track of the behaviour of their users on mobile sites and apps. This technology will aid in boosting the cross-channel marketing initiatives of an enterprise, while optimizing the mobile experience and growing user engagement and retention.What’s new for Mobile Analytics in 2020?With the ever-increasing number of mobile phone users globally, there will be a heightened focus on mobile app marketing and app analytics. Currently, mobile advertising ranks first in digital advertising worldwide. This had made mobile analytics quintessential, as businesses today can track in-app traffic, potential security threats, as well as the levels of customer satisfaction.5. Enhanced Levels of CustomizationAccess to real-time data and customer behaviour has made it possible to cater to each customer’s specific needs. As customer expectations soar, companies would have to buckle up to deliver more personalized, relevant and superior customer experience. The use of data and AI will make it possible.What’s new for User Experience in 2020?Interpreting user experience can be done easily using data derived from conversions, pageviews, and other user actions. These insights will help user experience professionals make better decisions while providing the user exactly what he/she needs.6. Better Cybersecurity2019 brought to light the grim reality of data privacy and security breach. With over 24 billion devices estimated to be connected to the internet this year, stringent measures will be deployed by enterprises to protect data privacy and prevent security breach. Industry experts are of the view that a combination of cybersecurity and AI-enabled technology results will lead to effective attack surface coverage that is a 20x more effective attack than traditional methods.What’s new for Cybersecurity in 2020?Since data shows no signs of stopping in its growth, the number of threats will also keep looming on the horizon. In 2020, cybersecurity professionals would have to gear themselves to conjure new and improved ways to secure data. Therefore, AI-supported cybersecurity measures will be deployed to prevent malicious attacks from malware and ensure better cybersecurity.The Way Ahead in 2020Data Science will be one of the fastest-growing technologies in 2020. Its wide range of applications in various industries will pave way for more innovative trends like the ones above.
Rated 4.5/5 based on 0 customer reviews
5899
Top Data Science Trends in 2020

Industry experts are of the view that 2020 will be... Read More

Top 30 Machine Learning Skills required to get a Machine Learning Job

Machine learning has been making a silent revolution in our lives since the past decade. From capturing selfies with a blurry background and focused face capture to getting our queries answered by virtual assistants such as Siri and Alexa, we are increasingly depending on products and applications that implement machine learning at their core.In more basic terms, machine learning is one of the steps involved in artificial intelligence. Machines learn through machine learning. How exactly? Just like how humans learn – through training, experience, and feedback.Once machines learn through machine learning, they implement the knowledge so acquired for many purposes including, but not limited to, sorting, diagnosis, robotics, analysis and predictions in many fields.It is these implementations and applications that have made machine learning an in-demand skill in the field of programming and technology.Look at the stats that show a positive trend for machine learning projects and careers.Gartner’s report on artificial intelligence showed that as many as 2.3 million jobs in machine learning would be available across the globe by 2020.Another study from Indeed, the online job portal giant, revealed that machine learning engineers, data scientists and software engineers with these skills are topping the list of most in-demand professionals.High profile companies such as Univa, Microsoft, Apple, Google, and Amazon have invested millions of dollars on machine learning research and designing and are developing their future projects on it.With so much happening around machine learning, it is no surprise that any enthusiast who is keen on shaping their career in software programming and technology would prefer machine learning as a foundation to their career. This post is specifically aimed at guiding such enthusiasts and gives comprehensive information on skills that are needed to become a machine learning engineer, who is ready to dive into the real-time challenges.Machine Learning SkillsOrganizations are showing massive interest in using machine learning in their products, which would in turn bring plenty of opportunities for machine learning enthusiasts.When you ask machine learning engineers the question – “What do you do as a machine learning engineer?”, chances are high that individual answers would differ from one professional to another. This may sound a little puzzling, but yes, this is true!Hence, a beginner to machine learning needs to have a clear understanding that there are different roles that they can perform with machine learning skills. And accordingly the skill set that they should possess, would differ. This section will give clarity on machine learning skills that are needed to perform various machine learning roles.Broadly, three main roles come into the picture when you talk about machine learning skills:Data EngineerMachine Learning EngineerMachine Learning ScientistOne must understand that data science, machine learning and artificial intelligence are interlinked. The following quote explains this better:Data science produces insights. Machine learning produces predictions. Artificial intelligence produces actions.A machine learning engineer is someone who deals with huge volumes of data to train a machine and impart it with knowledge that it uses to perform a specified task. However, in practice, there may be a little more to add to this:Machine Learning RoleSkills RequiredRoles and ResponsibilitiesData EngineerPython, R, and DatabasesParallel and distributed Knowledge on quality and reliabilityVirtual machines and cloud environmentMapReduce and HadoopCleaning, manipulating and extracting the required data   Developing code for data analysis and manipulationPlays a major role in statistical analysis of dataMachine Learning EngineerConcepts of computer science and software engineeringData analysis and feature engineeringMetrics involved in MLML algorithm selection, and cross validationMath and StatisticsAnalyses and checks the suitability of an algorithm if it caters the needs of the current taskPlays main role in deciding and selecting machine learning libraries for given task.Machine Learning ScientistExpert knowledge in:Robotics and Machine LearningCognitive ScienceEngineeringMathematics and mathematical modelsDesigns new models and algorithms of machine learningResearches intensively on machine learning and publishes their research papers.Thus, gaining machine learning skills should be a task associated with clarity on the job role and of course the passion to learn them. As it is widely known, becoming a machine learning engineer is not a straightforward task like becoming a web developer or a tester.Irrespective of the role, a learner is expected to have solid knowledge on data science. Besides, many other subjects are intricately intertwined in learning machine learning and for a learner it requires a lot of patience and zeal to learn skills and build them up as they move ahead in their career.The following diagram shows the machine learning skills that are in demand year after year:AI - Artificial IntelligenceTensorFlowApache KafkaData ScienceAWS - Amazon Web Services                                                                                                                                                                                                                                                                                                                                Image SourceIn the coming sections, we would be discussing each of these skills in detail and how proficient you are expected to be in them.Technical skills required to become ML EngineerBecoming a machine learning engineer means preparing oneself to handle interesting and challenging tasks that would change the way humanity is experiencing things right now. It demands both technical and non-technical expertise. Firstly, let’s talk about the technical skills needed for a machine learning engineer. Here is a list of technical skills a machine learning engineer is expected to possess:Applied MathematicsNeural Network ArchitecturesPhysicsData Modeling and EvaluationAdvances Signal Processing TechniquesNatural Language ProcessingAudio and video ProcessingReinforcement LearningLet us delve into each skill in detail now:1.Applied MathematicsMathematics plays an important role in machine learning, and hence it is the first one on the list. If you wish to see yourself as a proven machine learning engineer, you ought to love math and be an expert in the following specializations of math.But first let us understand why a machine learning engineer would need math at all? There are many scenarios where a machine learning engineer should depend on math. For example:Choosing the right algorithm that suits the final needsUnderstanding and working with parameters and their settings.Deciding on validation strategiesApproximating the confidence intervals.How much proficiency in Math does a machine learning engineer need to have?It depends on the level at which a machine learning engineer works. The following diagram gives an idea about how important various concepts of math are for a machine learning enthusiast.A) Linear algebra: 15%B) Probability Theory and Statistics: 25%C) Multivariate Calculus: 15%D) Algorithms and Optimization: 15%F) Other concepts: 10%Data SourceA) Linear AlgebraThis concept plays a main role in machine learning. One has to be skilled in the following sub-topics of linear algebra:Principal Component Analysis (PCA), Singular Value Decomposition (SVD)Eigen decomposition of a matrixLU DecompositionQR Decomposition/FactorizationSymmetric MatricesOrthogonalization & OrthonormalizationMatrix OperationsProjectionsEigenvalues & EigenvectorsVector Spaces and NormsB) Probability Theory and StatisticsThe core aim of machine learning is to reduce the probability of error in the final output and decision making of the machine. Thus, it is no wonder that probability and statistics play a major role.The following topics are important in these subjects:CombinatoricsProbability Rules & AxiomsBayes’ TheoremRandom VariablesVariance and ExpectationConditional and Joint DistributionsStandard Distributions (Bernoulli, Binomial, Multinomial, Uniform and Gaussian)Moment Generating Functions, Maximum Likelihood Estimation (MLE)Prior and PosteriorMaximum a Posteriori Estimation (MAP)Sampling Methods.C) CalculusIn calculus, the following concepts have notable importance in machine learning:Integral CalculusPartial Derivatives,Vector-Values FunctionsDirectional GradientHessian, Jacobian, Laplacian and Lagrangian Distributions.D) Algorithms and OptimizationThe scalability and the efficiency of computation of a machine learning algorithm depends on the chosen algorithm and optimization technique adopted. The following areas are important from this perspective:Data structures (Binary Trees, Hashing, Heap, Stack etc)Dynamic ProgrammingRandomized & Sublinear AlgorithmGraphsGradient/Stochastic DescentsPrimal-Dual methodsE) Other ConceptsBesides, the ones mentioned above, other concepts of mathematics are also important for a learner of machine learning. They are given below:Real and Complex Analysis (Sets and Sequences, Topology, Metric Spaces, Single-Valued and Continuous Functions Limits, Cauchy Kernel, Fourier Transforms)Information Theory (Entropy, Information Gain)Function Spaces and Manifolds2.Neural Network ArchitecturesNeural networks are the predefined set of algorithms for implementing machine learning tasks. They offer a class of models and play a key role in machine learning.The following are the key reasons why a machine learning enthusiast needs to be skilled in neural networks:Neural networks let one understand how the human brain works and help to model and simulate an artificial one.Neural networks give a deeper insight of parallel computations and sequential computationsThe following are the areas of neural networks that are important for machine learning:Perceptrons Convolutional Neural Networks Recurrent Neural NetworkLong/Short Term Memory Network (LSTM)Hopfield Networks Boltzmann Machine NetworkDeep Belief NetworkDeep Auto-encoders3.PhysicsHaving an idea of physics definitely helps a machine learning engineer. It makes a difference in designing complex systems and is a skill that is a definite bonus for a machine learning enthusiast.4.Data Modeling and EvaluationA machine learning has to work with huge amounts of data and leverage them into predictive analytics. Data modeling and evaluation is important in working with such bulky volumes of data and estimating how good the final model is.For this purpose, the following concepts are worth learnable for a machine learning engineer:Classification AccuracyLogarithmic LossConfusion MatrixArea under CurveF1 ScoreMean Absolute ErrorMean Squared Error5.Advanced Signal Processing TechniquesThe crux of signal processing is to minimize noise and extract the best features of a given signal.For this purpose, it uses certain concepts such as:convex/greedy optimization theory and algorithmsspectral time-frequency analysis of signalsAlgorithms such as wavelets, shearlets, curvelets, contourlets, bandlets, etc.All these concepts find their application in machine learning as well.6. Natural language processingThe importance of natural language processing in artificial intelligence and machine learning is not to be forgotten. Various libraries and techniques of natural language processing used in machine learning are listed here:Gensim and NLTKWord2vecSentiment analysisSummarization7. Audio and Video ProcessingThis differs from natural language processing in the sense that we can apply audio and video processing on audio signals only. For achieving this, the following concepts are essential for a machine learning engineer:Fourier transformsMusic theoryTensorFlow8. Reinforcement LearningThough reinforcement learning plays a major role in learning and understanding deep learning and artificial intelligence, it is good for a beginner of machine learning to know the basic concepts of reinforcement learning.Programming skills required to become ML EngineerMachine learning, ultimately, is coding and feeding the code to the machines and getting them to do the tasks we intend them to do. As such, a machine learning engineer should have hands-on expertise in software programming and related concepts. Here is a list of programming skills a machine learning engineer is expected to have knowledge on:Computer Science Fundamentals and ProgrammingSoftware Engineering and System DesignMachine Learning Algorithms and LibrariesDistributed computingUnixLet us look into each of these programming skills in detail now:1.Computer Science Fundamentals and ProgrammingIt is important that a machine learning engineer apply the concepts of computer science and programming correctly as the situation demands. The following concepts play an important role in machine learning and are a must on the list of the skillsets a machine learning engineer needs to have:Data structures (stacks, queues, multi-dimensional arrays, trees, graphs)Algorithms (searching, sorting, optimization, dynamic programming)Computability and complexity (P vs. NP, NP-complete problems, big-O notation, approximate algorithms, etc.)Computer architecture (memory, cache, bandwidth, deadlocks, distributed processing, etc.)2.Software Engineering and System DesignWhatever a machine learning engineer does, ultimately it is a piece of software code – a beautiful conglomerate of many essential concepts and the one that is entirely different from coding in other software languages.Hence, it is quintessential that a machine learning engineer have solid knowledge of the following areas of software programming and system design:Scaling algorithms with the size of dataBasic best practices of software coding and design, such as requirement analysis, version control, and testing.Communicating with different modules and components of work using library calls, REST APIs and querying through databases.Best measures to avoid bottlenecks and designing the final product such that it is user-friendly.3. Machine Learning Algorithms and LibrariesA machine learning engineer may need to work with multiple packages, libraries, algorithms as a part of day-to-day tasks. It is important that a machine learning engineer is well-versed with the following aspects of machine learning algorithms and libraries:A thorough idea of various learning procedures including linear regression, gradient descent, genetic algorithms, bagging, boosting, and other model-specific methods.Sound knowledge in packages and APIs such as scikit-learn, Theano, Spark MLlib, H2O, TensorFlow, etc.Expertise in models such as decision trees, nearest neighbor, neural net, support vector machine and a knack to deciding which one fits the best.Deciding and choosing hyperparameters that affect learning model and the outcome.Comfortable to work with concepts such as gradient descent, convex optimization, quadratic programming, partial differential equations.Select an algorithm which yields the best performance from random forests, support vector machines (SVMs), and Naive Bayes Classifiers, etc.4   Distributed Computing Working as a machine learning engineer means working with huge sets of data, not just focused on one isolated system, but spread among a cluster of systems. For this purpose, it is important that a machine learning engineer knows the concepts of distributed computing.5. UnixMost clusters and servers that machine learning engineers need to work are variants of Linux(Unix). Though randomly they work on Windows and Mac, more than half of the time, they need to work on Unix systems only. Hence having sound knowledge on Unix and Linux is a key skill to become a machine learning engineer.Programming Languages for Machine LearningMachine learning engineers need to code to train machines. Several programming languages can be used to do this. The list of programming languages that a machine learning expert should essentially know are as under:C, C++ and JavaSpark and HadoopR ProgrammingApache KafkaPythonWeka PlatformMATLAB/OctaveIn this section, let us know in detail why each of these programming languages is important for a machine learning engineer:1.C, C++ and JavaThese languages give essentials of programming and teach many concepts in a simple manner that form a foundation stone for working on complex programming patterns of machine learning. Knowledge of C++ helps to improve the speed of the program, while Java is needed to work with Hadoop and Hive, and other tools that are essential for a machine learning engineer.2.Spark and HadoopHadoop skills are needed for working in a distributed computing environment. Spark, a recent variant of Hadoop is gaining popularity among the machine learning tribe. It is a framework to implement machine learning on a large scale.3.R ProgrammingR is a programming language built by statisticians specifically to work with programming that involves statistics. Many mathematical computations of machine learning are based on statistics; hence it is no wonder that a machine learning engineer needs to have sound knowledge in R programming.4.Apache KafkaApache Kafka concepts such as Kafka Streams and KSQL play a major role in pre-processing of data in machine learning. Also, a sound knowledge of Apache Kafka lets a machine learning engineer to design solutions that are both multi-cloud based or hybrid cloud-based.  Other concepts such as business information such as latency and model accuracy are also from Kafka and find use in Machine learning.5.PythonOf late, Python has become the unanimous programming language for machine learning. In fact, experts quote that humans communicate with machines through Python language.Why Python is preferred for Machine Learning?Python Programming Language has several key features and benefits that make it the monarch of programming languages for machine learning:It is an all-in-one purpose programming language that can do a lot more than dealing with statistics.It is beginner friendly and easy to learn.It boasts of rich libraries and APIs that solve various needs of machine learning pretty easily.Its productivity is higher than its other counterparts.It offers ease of integration and gets the workflow smoothly from the designing stage to the production stage.Python EcoSystemThere are various components of Python that make it preferred language for machine learning. Such components are discussed below:Jupyter NotebookNumpyPandasScikit-LearnTensorFlow1.Jupyter NotebookJupyter offers excellent computational environment for Python based data science applications. With the help of Jupyter notebook, a machine learning engineer can illustrate the flow of the process step-by-step very clearly.2.NumPyNumPy or Numerical Python is one of the components of Python that allows the following operations of machine learning in a smooth way:Fourier transformationLinear algebraic operationsLogical and numerical operations on arrays.Of late, NumPy is gaining attention because it makes an excellent substitute to MATLAB, as it coordinates with Matplotlib and SciPy very smoothly.3.PandasPandas is a Python library that offers various features for loading, manipulating, analysing, modeling and preparing data. It is entirely dedicated for data analysis and manipulation.4.Scikit-learnBuilt on NumPy, SciPy, and Matplotlib, it is an open-source library of Python. It offers excellent features and functionalities for major aspects of machine learning such as clustering, dimensionality reduction, model reduction, regression and classification.5.TensorFlowTensorFlow is another framework of Python. It finds its usage in deep learning and having a knowledge of its libraries such as Keras, helps a machine learning engineer to move ahead confidently in their career.6.Weka PlatformIt is widely known that machine learning is a non-linear process that involves many iterations. Weka or Waikato Environment for Knowledge Analysis is a recent platform that is designed specifically designed for applied machine learning. This tool is also slowing gaining its popularity and thus is a must-include on the list of skills for a machine learning engineer.7.MATLAB/OctaveThis is a basic programming language that was used for simulation of various engineering models. Though not popularly used in machine learning, having sound knowledge in MATLAB lets one learns the other mentioned libraries of Python easily.Soft skills or behavioural skills required to become ML engineerTechnical skills are relevant only when they are paired with good soft skills. And the machine learning profession is no exception to this rule. Here is a list of soft skills that a machine learning engineer should have:Domain knowledgeCommunication SkillsProblem-solving skillsRapid prototypingTime managementLove towards constant learningLet us move ahead and discuss how each of these skills make a difference to a machine learning engineer.1.Domain knowledgeMachine learning is such a subject that needs the best of its application in real-time. Choosing the best algorithm while solving a machine learning problem in your academia is far different from what you do in practice. Various aspects of business come into picture when you are a real-time machine learning engineer. Hence, a solid understanding of the business and domain of machine learning is of utmost importance to succeed as a good machine learning engineer.2.Communication SkillsAs a machine learning engineer, you need to communicate with offshore teams, clients and other business teams. Excellent communication skills are a must to boost your reputation and confidence and to bring up your work in front of peers.3.Problem-solving skillsMachine learning is all about solving real time challenges. One must have good problem-solving skills and be able to weigh the pros and cons of the given problem and apply the best possible methods to solve it.4.Rapid PrototypingChoosing the correct learning method or the algorithm are signs of a machine learning engineer’s good prototyping skills. These skills would be a great saviour in real time as they would show a huge impact on budget and time taken for successfully completing a machine learning project.5.Time managementTraining a machine is not a cake-walk. It takes huge time and patience to train a machine. But it’s not always that machine learning engineers are allotted ample time for completing tasks. Hence, time management is an essential skill a machine learning professional should have to effectively deal with bottlenecks and deadlines.6.Love towards constant learningSince its inception, machine learning has witnessed massive change – both in the way it is implemented and in its final form. As we have seen in the previous section, technical and programming skills that are needed for machine learning are constantly evolving. Hence, to prove oneself a successful machine learning expert, it is very crucial that they have a zeal to update themselves – constantly!ConclusionThe skills that one requires to begin their journey in machine learning are exactly what we have discussed in this post. The future for machine learning is undoubtedly bright with companies ready to offer millions of dollars as remuneration, irrespective of the country and the location.Machine learning and deep learning will create a new set of hot jobs in the next five years. – Dave WatersAll it takes to have an amazing career in machine learning is a strong will to hone one’s skills and gain a solid knowledge of them. All the best for an amazing career in machine learning!
Rated 4.5/5 based on 43 customer reviews
14020
Top 30 Machine Learning Skills required to get a M...

Machine learning has been making a silent revoluti... Read More