Search

Series List Filter

What is Machine Learning and Why It Matters: Everything You Need to Know

If you are a machine learning enthusiast and stay in touch with the latest developments, you would have definitely come across the news “Machine learning identifies links between the world's oceans”. Wait, we all know how complex it would be to analyse a concept such as oceans and their behaviour which would undoubtedly involve billions of data points associated with many critical parameters such as wind velocities, temperatures, earth’s rotation and many such. Doesn’t this piece of information gives you a glimpse of the wondrous possibilities of machine learning and its potential uses? And this is just a drop in the ocean!As you move across this post, you would get a comprehensive idea of various aspects that you ought to know about machine learning.What is Machine Learning and Why It Matters?Machine learning is a segment of artificial intelligence. It is designed to make computers learn by themselves and perform operations without human intervention, when they are exposed to new data. It means a computer or a system designed with machine learning will identify, analyse and change accordingly and give the expected output when it comes across a new pattern of data, without any need of humans.The power behind machine learning’s self-identification and analysis of new patterns, lies in the complex and powerful ‘pattern recognition’ algorithms that guide them in where to look for what. Thus, the demand for machine learning programmers who have extensive knowledge on working with complex mathematical calculations and applying them to big data and AI is growing year after year.Machine learning, though a buzz word only since recent times, has conceptually been in existence since World War II when Alan Turing’s Bombe, an enigma deciphering machine was introduced to the world. However, it's only in the past decade or so that there has been such great progress made in context to machine learning and its uses, driven mainly by our quest for making this world more futuristic  with lesser human intervention and more precision. Pharma, education technology, industries, science and space, digital inventions, maps and navigation, robotics – you name the domain and you will have instances of machine learning innovations made in it.The Timeline of Machine Learning and the Evolution of MachinesVoice activated home appliances, self-driven cars and online marketing campaigns are some of the applications of machine learning that we experience and enjoy the benefit of in our day to day life. However, the development of such amazing inventions date back to decades. Many great mathematicians and futuristic thinkers were involved in the foundation and development of machine learning.A glimpse of the timeline of machine learning reveals many hidden facts and the efforts of great mathematicians and scientists to whom we should attribute all the fruits that we are enjoying today.1812- 1913: The century that laid the foundation of machine learningThis age laid the mathematical foundation for the development of machine learning. Bayes’ theorem and Markovs Chains took birth during this period.Late 1940s: First computers Computers were recognised as machines that can ‘store data’. The famous Manchester Small-Scale Experimental Machine (nicknamed 'The Manchester Baby') belongs to this era.1950: The official Birth of Machine LearningDespite many researches and theoretical studies done prior to this year, it was the year 1950 that is always remembered as the foundation of the machine learning that we are witnessing today. Alan Turing, researcher, mathematician, computer genius and thinker, submitted a paper where he mentioned something called ‘imitation game’ and astonished the world by questioning “Can Machines Think?”. His research grabbed the attention of the BBC which took an exclusive interview with Alan.1951: The First neural networkThe first artificial neural network was built by Marvin Minsky and Dean Edmonds this year. Today, we all know that artificial neural networks play a key role in the thinking process of computers and machines. This should be attributed to the invention made by these two scientists.1974: Coining of the term ‘Machine Learning’Though there were no specific terms till then for the things that machines did by thinking on their own, it was in 1974 that the term ‘machine learning’ was termed. Other words such as artificial intelligence, informatics and computational intelligence were also proposed the same year.1996: Machine beats man in a game of chessIBM developed its own computer called Deep Blue, that can think. This machine beat the world famous champion in chess, Garry Kasparov. It was then proved to the world that machines can really think like humans.2006-2017: Backpropagation, external memory access and AlphaGoBack propagation is an important technique that machines use for image recognition. This technique was developed in this period of time.Besides in 2014, a neural network developed by DeepMind, a British based company, developed a neural network that can access external memory and get things done.In 2016, AlphaGo was designed by DeepMind researchers. It beat the world famous Go players Lee Sedol and Ke Jie and proved that machines have come a long way.What’s next?Scientists are talking about ‘singularity’ –a phenomenon that would occur if humans develop a humanoid robot that could think better than humans and will recreate itself. So far, we have been witnessing how AI is entering our personal lives too in the form of voice activated devices, smart systems and many more. The results of this singularity – we shall have to wait and watch!Basics of Machine LearningTo put it simply, machine learning involves learning by machines. It means computers learn and there are many concepts, methods, algorithms and processes involved in making this happen. Let us try to understand some of the more important machine learning terms.Three concepts – artificial intelligence, machine learning and deep learning – are often thought to be synonymous. Though they belong to the same family, conceptually they are different.Machine LearningIt implies that machines can ‘learn on their own’ and give the output without any need of programming explicitly.Artificial IntelligenceThis term means machines can ‘think on their own’ just like humans and take decisions.Deep LearningThis involves creation of artificial neural networks which can think and act based on algorithms.How do machines learn?Quite simply, machines learn just like humans do. Humans learn from their training, experiences and through teachers. Sometimes they use knowledge that is fed into their brains, or sometimes take decisions by analysing the current situation using their past experiences.Similarly, machines learn from the inputs given to them which tell them which is right and which is wrong. Then they are given data that they would have to analyse based on the training they have received so far. In some other cases, they do not have any idea of which is right or wrong, but just take the decision based on their own experiences. We will analyse the various concepts of learning and the methods involved.How Machine Learning Works?The process of machine learning occurs in five steps as shown in the following diagram.The steps are explained in simple words below:Gathering the data includes data collection from varied, rich and dense content of various formats and types. In real time, this includes feeding the data from different sources such as text files, word documents or excel sheets.Data preparation involves extracting the actual data out of the entire content fed. Only the data that really makes sense to the machine is used for processing. This step also involves checking for missing data, unwanted data and treatment of outliers.Training involves using an appropriate algorithm and modelling the data. The data filtered in the second step is split into two parts and a part of it is used as training data and the second part is used as reference data. The training data is used to create the model.Evaluating the model includes testing its accuracy. To verify its accuracy to the fullest, the model so developed is tested on such data which is not present in the data during the second step.Finally, the performance of the machine is improved by choosing a different model that suits the different type of data that is present altogether. This is the step where the machine thinks and rethinks in selecting the model best suited for various types of data.Examples of Machine LearningThe below examples will help you understand where machine learning is used in real time:Speech RecognitionVoice based searching and call rerouting are the best examples for speech recognition using machine learning. The principle lies in translating  spoken words into text and then segmenting them on the basis of their frequencies.Image RecognitionWe all use this in day to day life in sorting our pictures on our Google drive or Photos. The main technique that is used here is classifying the pictures based on the intensity (in case of black and white pictures) and measurement of intensities of red, blue and green for coloured images.HealthcareVarious diagnoses are increasingly made using machine learning these days. Here, various clinical parameters are input to the machine which makes a prognosis  and then predicts the disease status and other health parameters of the person under study.Financial ServicesMachine learning helps in predicting chances of financial fraud, customer’s credit habits, spending patterns etc. The financial and banking sector is also doing market analysis using machine learning techniques.Machine Learning – MethodsMachine learning is all about machines learning through the inputs provided. This learning is carried out in the following ways:Supervised LearningAs the name says, the machine learns under supervision. Let’s see how this is done:The entire process of learning takes place in the presence or supervision of a teacher.This mode of learning contains basic steps as follows:First, the machine is trained using a predefined data also called ‘labeled’ data.Then, the correct answer is fed into the computer which allows it to understand what the right and wrong answers should be.Lastly, the system is given a new set of data or unlabelled data, which it would now analyse using techniques such as classification and regression to predict the correct outcome for the current unlabelled data.Example:Consider a shape sorting game that kids play. A bunch of different shapes of wooden pieces are given to kids, say of square shape, triangular shape, circular shape and star shape. Assume that all blocks of a similar shape are of a unique colour. First, you teach the kids which shape is what  and then you ask them to do the sorting on their own.Similarly, in machine learning, you teach the machine through labelled data. Then, the machine is given some unknown data, which it analyses based on the previous labelled data and gives the correct outcome.In this case, if you observe, two techniques have been used.Classification: Based on colors.Regression: Based on shapes.As a further explanation,Classification: A classification problem is when the output variable is a category, such as “Red” or “blue” or “disease” and “no disease”.Regression: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.Unsupervised LearningIn this type of learning, there is no previous knowledge, no previous training, nor a teacher to supervise. This learning is all instantaneous based on the data that is available at that given time.Example:Consider a kid playing with a mix of tomatoes and capsicums. They would sort them involuntarily based on their shape or color. This is an instantaneous reaction without any predefined set of attributes or training.A machine working on unsupervised learning would produce the results based on a similar mechanism. For this purpose, it uses two algorithms as explained below:Clustering: This involves grouping a cluster of data. For example, this is used in analysing the online customer’s purchase patterns and shopping habits.Association: This involves associating the given items based on the portion of their sizes. For example, analysing that people who bought large number of a given item would also prefer other similar items. Semi-supervised LearningThe name itself says the pattern of this algorithm.It is a hybrid mix of both supervised and unsupervised learning and uses both labelled data and unlabelled data to predict the results.In most occurrences, unlabelled data is given more in quantity than labelled data, because of cost considerations.For example, in a folder of thousands of photographs, the machine sorts pictures based on the maximum number of common features (unsupervised) and already defined names of persons in the pictures, if any(supervised)Reinforcement LearningIn reinforcement learning, there is no correct answer known to the system. The system learns from its own experience through a reinforcement agent. Since the answer is not known, the reinforcement agent decides what to do with the given task and for this it uses its experience from the current situation only.Example: In a robotic game that involves earning the hidden treasure, the algorithm focuses on bringing out the best outcome through trial and error method. Mainly three components are observed in this type of learning: the user, the environment and the action the user is performing. The algorithm adjusts itself accordingly to guide the user towards the best result that can be achieved.The diagram shown below summarizes the four types of learning we have learnt so far:Machine Learning – AlgorithmsMachine learning is rich in algorithms that allow programmers to pick one that best suits the context. Some of the machine learning algorithms are:Neural networksDecision treesRandom forestsSupport vector machinesNearest-neighbor mappingk-means clusteringSelf-organizing mapsExpectation maximizationBayesian networksKernel density estimationPrincipal component analysisSingular value decompositionMachine Learning Tools and LibrariesTo start the journey with machine learning, a learner should have knowledge of tools and libraries that are quintessential to designing machine learning code. Here is a list of such tools and libraries:ToolsProgramming LanguageMachine learning can be coded either using R programming language or Python. Of late, Python has become more popular due to its rich libraries, ease of learning and coding friendliness.IDEMachine learning is widely coded in Jupyter Notebook. It simplifies writing of Python code and embedding plots and charts. Google Colab is another free tool that you can choose for the same purpose.LibrariesScikit-LearnA very popular and beginner friendly library.Supports most of the standard algorithms from supervised and unsupervised learning.Offers models for data pre-processing and result analysis.Limited support for deep learning.TensorFlowSupports Neural networks and deep learning.Bulky compared to scikit learnOffers best computational efficiencySupports many classical algorithms of machine learning.PandasThe data gathering and preparation part of machine learning that we have seen in the stages involved in machine learning is taken care of by Pandas. This library:Gathers and prepares data that other libraries of machine learning can use at a later point in time.Gathers data from any type of data source such as text, SQL DB, MS Excel or JSON files.Contains many statistical functionalities that can be used to work on the data that’s gathered.NumPy and SciPyNumPy supports all array based and linear algebraic functions needed while working on data, while SciPy offers many scientific calculations. NumPy is more widely used in many real time applications of machine learning as compared to SciPy.MatplotlibThis is a machine learning library that has an extensive collection of plots and charts. This library is a collection of many other packages. Of them, Seaborn is the most popular and is widely used to work on data structures.PyTorch and KerasThese are known for their usage in Deep learning.PyTorch library is extensively used for Deep Learning. It is known for its amazingly speedy calculations and is very popular among deep learning programmers.Keras uses other libraries such as Tensor flow and is apt for developing neural networks.Machine Learning – ProcessesBesides algorithms, machine learning offers many tools and processes to pair best with big data. Various such processes and tools that are at hand for developers are:Data quality and managementGUIs that ease models and process flowsData exploration in an interactive modeVisualized outputs for modelsChoosing the best learning model by comparisonModel evaluation done automatically that identifies the best performersUser friendly model deployment and data-to-decision processMachine Learning Use CasesHere is a list of five use cases that are based on machine learning:PayPal: The online money transfers giant uses machine learning for detecting any suspicious activities related to financial transactions.Amazon: The company’s Alexa, the digital assistant, is the best example of speech processing application of machine learning. The online retailing giant is also using machine learning to display recommendation to its customers.Facebook: The social media company is using machine learning extensively to filter out spam posts and forwards, and to shred out poor quality content.IBM: The company’s self-driven vehicle uses machine learning in taking a decision whether to give the driving control to a human or computer.Kaspersky: The anti-virus manufacturing company is using machine learning to detect security breaches, or unknown malware threats and also for high quality endpoint security for businesses.Which Industries Use Machine Learning?As we have seen just now, machine learning is being adopted in many industries for the potential advantages it offers. Machine learning can be applied to any industry that deals with huge volumes of data, and which has many challenges to be answered. For instance, machine learning has been found to be extremely useful to organizations in the following domains which are  making the best use of the technology:PharmaceuticalsPharma industry spends billions of dollars on drug design and testing every year across the globe. Machine learning helps in cutting down such costs and to obtain results with accuracy just by entering the entire data of the drugs and their chemical compounds and comparing with various other parameters.Banks and Financial ServicesThis industry has two major needs to be addressed: attracting investor attention and increasing investments, and staying alert and preventing financial frauds and cyber threats. Machine learning does these two major tasks with ease and accuracy.Health Care and TreatmentsBy predicting the possible  diseases that could affect a patient, based on the medical, genetic and lifestyle data, machine learning helps patients stay alert to probable health threats that they may encounter. Wearable smart devices are an example of the machine learning applications in health care.Online SalesCompanies study the patterns that online shoppers are adopting through machine learning and use the results to display related ads, offers and discounts. Personalisation of internet shopping experience, merchandise supply panning and marketing campaigns are all based on the outcomes of machine learning results themselves.Mining, Oil and GasMachine learning helps in predicting accurately the best location of availability of minerals, gas, oil and other such natural resources, which would otherwise need huge investments, manpower and time.Government SchemesMany governments are taking the help of machine learning to study the interests and needs of their people. They are accordingly using the results in plans and schemes, both for the betterment of people and optimum usage of financial resources.Space Exploration and Science StudiesMachine learning greatly helps in studying stars, planets and finding out the secrets of other celestial bodies with far lesser investments and manpower. Scientists are also maximising the use of machine learning to discover various fascinating facts about the earth and its components.Future of Machine LearningCurrently, machine learning is entering our lives with baby steps. By the next decade, radical changes can be expected in machine learning and the way it impacts our lives. Customers have already started trusting the power and comfort of machine learning, and would definitely welcome more such innovations in the near future.Gartner says:Artificial Intelligence and Machine Learning have reached a critical tipping point and will increasingly augment and extend virtually every technology enabled service, thing, or application.So, it would not be surprising if in the future, machine learning would:Make its entry in almost every aspect of human  lifeBe omnipresent in business and industries, irrespective of their sizeEnter  cloud based servicesBring drastic changes in CPU design keeping in mind the need for computational efficiencyAltogether change the shape of data, its processing and usageChange the way connected systems work and look  owing to the ever increasing data on the internet.ConclusionMachine learning is quite different in its own way. While many experts are raising concerns over the ever increasing dependence and presence of machine learning in our everyday lives, on the positive side, machine learning can work wonders. And the world is already witnessing its magic – in health care, finance industry, automotive industry, image processing and voice recognition and many other fields.While many of us worry that machines may take over the world, it is totally up to us, how we design effective, yet safe and controllable machines. There is no doubt that machine learning would change the way we do many things including education, business and health services making the world a safer and better place.
Rated 4.5/5 based on 3 customer reviews

What is Machine Learning and Why It Matters: Everything You Need to Know

9218
  • by Animikh Aich
  • 26th Apr, 2019
  • Last updated on 12th Sep, 2019
  • 15 mins read
What is Machine Learning and Why It Matters: Everything You Need to Know

If you are a machine learning enthusiast and stay in touch with the latest developments, you would have definitely come across the news “Machine learning identifies links between the world's oceans”. Wait, we all know how complex it would be to analyse a concept such as oceans and their behaviour which would undoubtedly involve billions of data points associated with many critical parameters such as wind velocities, temperatures, earth’s rotation and many such. Doesn’t this piece of information gives you a glimpse of the wondrous possibilities of machine learning and its potential uses? And this is just a drop in the ocean!

As you move across this post, you would get a comprehensive idea of various aspects that you ought to know about machine learning.

What is Machine Learning and Why It Matters?

Machine learning is a segment of artificial intelligence. It is designed to make computers learn by themselves and perform operations without human intervention, when they are exposed to new data. It means a computer or a system designed with machine learning will identify, analyse and change accordingly and give the expected output when it comes across a new pattern of data, without any need of humans.

The power behind machine learning’s self-identification and analysis of new patterns, lies in the complex and powerful ‘pattern recognition’ algorithms that guide them in where to look for what. Thus, the demand for machine learning programmers who have extensive knowledge on working with complex mathematical calculations and applying them to big data and AI is growing year after year.

What is ML and Why It Matters

Machine learning, though a buzz word only since recent times, has conceptually been in existence since World War II when Alan Turing’s Bombe, an enigma deciphering machine was introduced to the world. However, it's only in the past decade or so that there has been such great progress made in context to machine learning and its uses, driven mainly by our quest for making this world more futuristic  with lesser human intervention and more precision. Pharma, education technology, industries, science and space, digital inventions, maps and navigation, robotics – you name the domain and you will have instances of machine learning innovations made in it.

The Timeline of Machine Learning and the Evolution of Machines

Voice activated home appliances, self-driven cars and online marketing campaigns are some of the applications of machine learning that we experience and enjoy the benefit of in our day to day life. However, the development of such amazing inventions date back to decades. Many great mathematicians and futuristic thinkers were involved in the foundation and development of machine learning.

A glimpse of the timeline of machine learning reveals many hidden facts and the efforts of great mathematicians and scientists to whom we should attribute all the fruits that we are enjoying today.

Timeline of Machine Learning and Evolution of Machines

  • 1812- 1913: The century that laid the foundation of machine learning

This age laid the mathematical foundation for the development of machine learning. Bayes’ theorem and Markovs Chains took birth during this period.

  • Late 1940s: First computers 

Computers were recognised as machines that can ‘store data’. The famous Manchester Small-Scale Experimental Machine (nicknamed 'The Manchester Baby') belongs to this era.

  • 1950: The official Birth of Machine Learning

Despite many researches and theoretical studies done prior to this year, it was the year 1950 that is always remembered as the foundation of the machine learning that we are witnessing today. Alan Turing, researcher, mathematician, computer genius and thinker, submitted a paper where he mentioned something called ‘imitation game’ and astonished the world by questioning “Can Machines Think?”. His research grabbed the attention of the BBC which took an exclusive interview with Alan.

  • 1951: The First neural network

The first artificial neural network was built by Marvin Minsky and Dean Edmonds this year. Today, we all know that artificial neural networks play a key role in the thinking process of computers and machines. This should be attributed to the invention made by these two scientists.

  • 1974: Coining of the term ‘Machine Learning’

Though there were no specific terms till then for the things that machines did by thinking on their own, it was in 1974 that the term ‘machine learning’ was termed. Other words such as artificial intelligence, informatics and computational intelligence were also proposed the same year.

  • 1996: Machine beats man in a game of chess

IBM developed its own computer called Deep Blue, that can think. This machine beat the world famous champion in chess, Garry Kasparov. It was then proved to the world that machines can really think like humans.

  • 2006-2017: Backpropagation, external memory access and AlphaGo

Back propagation is an important technique that machines use for image recognition. This technique was developed in this period of time.

Besides in 2014, a neural network developed by DeepMind, a British based company, developed a neural network that can access external memory and get things done.

In 2016, AlphaGo was designed by DeepMind researchers. It beat the world famous Go players Lee Sedol and Ke Jie and proved that machines have come a long way.

  • What’s next?

Scientists are talking about ‘singularity’ –a phenomenon that would occur if humans develop a humanoid robot that could think better than humans and will recreate itself. So far, we have been witnessing how AI is entering our personal lives too in the form of voice activated devices, smart systems and many more. The results of this singularity – we shall have to wait and watch!

Basics of Machine Learning

To put it simply, machine learning involves learning by machines. It means computers learn and there are many concepts, methods, algorithms and processes involved in making this happen. Let us try to understand some of the more important machine learning terms.

Three concepts – artificial intelligence, machine learning and deep learning – are often thought to be synonymous. Though they belong to the same family, conceptually they are different.

Basics of Machine Learning

Machine Learning

It implies that machines can ‘learn on their own’ and give the output without any need of programming explicitly.

Artificial Intelligence

This term means machines can ‘think on their own’ just like humans and take decisions.

Deep Learning

This involves creation of artificial neural networks which can think and act based on algorithms.

How do machines learn?

Quite simply, machines learn just like humans do. Humans learn from their training, experiences and through teachers. Sometimes they use knowledge that is fed into their brains, or sometimes take decisions by analysing the current situation using their past experiences.

Similarly, machines learn from the inputs given to them which tell them which is right and which is wrong. Then they are given data that they would have to analyse based on the training they have received so far. In some other cases, they do not have any idea of which is right or wrong, but just take the decision based on their own experiences. We will analyse the various concepts of learning and the methods involved.

How Machine Learning Works?

The process of machine learning occurs in five steps as shown in the following diagram.

How Machine Learning Works

The steps are explained in simple words below:

  • Gathering the data includes data collection from varied, rich and dense content of various formats and types. In real time, this includes feeding the data from different sources such as text files, word documents or excel sheets.
  • Data preparation involves extracting the actual data out of the entire content fed. Only the data that really makes sense to the machine is used for processing. This step also involves checking for missing data, unwanted data and treatment of outliers.
  • Training involves using an appropriate algorithm and modelling the data. The data filtered in the second step is split into two parts and a part of it is used as training data and the second part is used as reference data. The training data is used to create the model.
  • Evaluating the model includes testing its accuracy. To verify its accuracy to the fullest, the model so developed is tested on such data which is not present in the data during the second step.
  • Finally, the performance of the machine is improved by choosing a different model that suits the different type of data that is present altogether. This is the step where the machine thinks and rethinks in selecting the model best suited for various types of data.

Examples of Machine Learning

The below examples will help you understand where machine learning is used in real time:

Machine Learning Examples

Speech Recognition

Voice based searching and call rerouting are the best examples for speech recognition using machine learning. The principle lies in translating  spoken words into text and then segmenting them on the basis of their frequencies.

Image Recognition

We all use this in day to day life in sorting our pictures on our Google drive or Photos. The main technique that is used here is classifying the pictures based on the intensity (in case of black and white pictures) and measurement of intensities of red, blue and green for coloured images.

Healthcare

Various diagnoses are increasingly made using machine learning these days. Here, various clinical parameters are input to the machine which makes a prognosis  and then predicts the disease status and other health parameters of the person under study.

Financial Services

Machine learning helps in predicting chances of financial fraud, customer’s credit habits, spending patterns etc. The financial and banking sector is also doing market analysis using machine learning techniques.

Machine Learning – Methods

Machine learning is all about machines learning through the inputs provided. This learning is carried out in the following ways:

Supervised Learning

As the name says, the machine learns under supervision. Let’s see how this is done:

  • The entire process of learning takes place in the presence or supervision of a teacher.
  • This mode of learning contains basic steps as follows:
    • First, the machine is trained using a predefined data also called ‘labeled’ data.
    • Then, the correct answer is fed into the computer which allows it to understand what the right and wrong answers should be.
  • Lastly, the system is given a new set of data or unlabelled data, which it would now analyse using techniques such as classification and regression to predict the correct outcome for the current unlabelled data.

Example:

Consider a shape sorting game that kids play. A bunch of different shapes of wooden pieces are given to kids, say of square shape, triangular shape, circular shape and star shape. Assume that all blocks of a similar shape are of a unique colour. First, you teach the kids which shape is what  and then you ask them to do the sorting on their own.

Similarly, in machine learning, you teach the machine through labelled data. Then, the machine is given some unknown data, which it analyses based on the previous labelled data and gives the correct outcome.

In this case, if you observe, two techniques have been used.

  • Classification: Based on colors.
  • Regression: Based on shapes.

As a further explanation,

  • Classification: A classification problem is when the output variable is a category, such as “Red” or “blue” or “disease” and “no disease”.
  • Regression: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.

Unsupervised Learning

  • In this type of learning, there is no previous knowledge, no previous training, nor a teacher to supervise. This learning is all instantaneous based on the data that is available at that given time.

Example:

Consider a kid playing with a mix of tomatoes and capsicums. They would sort them involuntarily based on their shape or color. This is an instantaneous reaction without any predefined set of attributes or training.

A machine working on unsupervised learning would produce the results based on a similar mechanism. For this purpose, it uses two algorithms as explained below:

  • Clustering: This involves grouping a cluster of data. For example, this is used in analysing the online customer’s purchase patterns and shopping habits.
  • Association: This involves associating the given items based on the portion of their sizes. For example, analysing that people who bought large number of a given item would also prefer other similar items. 

Semi-supervised Learning

The name itself says the pattern of this algorithm.

  • It is a hybrid mix of both supervised and unsupervised learning and uses both labelled data and unlabelled data to predict the results.
  • In most occurrences, unlabelled data is given more in quantity than labelled data, because of cost considerations.
  • For example, in a folder of thousands of photographs, the machine sorts pictures based on the maximum number of common features (unsupervised) and already defined names of persons in the pictures, if any(supervised)

Reinforcement Learning

In reinforcement learning, there is no correct answer known to the system. The system learns from its own experience through a reinforcement agent. Since the answer is not known, the reinforcement agent decides what to do with the given task and for this it uses its experience from the current situation only.

Example: In a robotic game that involves earning the hidden treasure, the algorithm focuses on bringing out the best outcome through trial and error method. Mainly three components are observed in this type of learning: the user, the environment and the action the user is performing. The algorithm adjusts itself accordingly to guide the user towards the best result that can be achieved.

The diagram shown below summarizes the four types of learning we have learnt so far:

Types of Machine Learning:- Supervised, Unsupervised, Semi-supervised and Reinforced Learning.

Machine Learning – Algorithms

Machine learning is rich in algorithms that allow programmers to pick one that best suits the context. Some of the machine learning algorithms are:

  • Neural networks
  • Decision trees
  • Random forests
  • Support vector machines
  • Nearest-neighbor mapping
  • k-means clustering
  • Self-organizing maps
  • Expectation maximization
  • Bayesian networks
  • Kernel density estimation
  • Principal component analysis
  • Singular value decomposition

Machine Learning Tools and Libraries

To start the journey with machine learning, a learner should have knowledge of tools and libraries that are quintessential to designing machine learning code. Here is a list of such tools and libraries:

Tools

Programming Language

Machine learning can be coded either using R programming language or Python. Of late, Python has become more popular due to its rich libraries, ease of learning and coding friendliness.

IDE

Machine learning is widely coded in Jupyter Notebook. It simplifies writing of Python code and embedding plots and charts. Google Colab is another free tool that you can choose for the same purpose.

Libraries

Scikit-Learn

  • A very popular and beginner friendly library.
  • Supports most of the standard algorithms from supervised and unsupervised learning.
  • Offers models for data pre-processing and result analysis.
  • Limited support for deep learning.

TensorFlow

  • Supports Neural networks and deep learning.
  • Bulky compared to scikit learn
  • Offers best computational efficiency
  • Supports many classical algorithms of machine learning.

Pandas

The data gathering and preparation part of machine learning that we have seen in the stages involved in machine learning is taken care of by Pandas. This library:

  • Gathers and prepares data that other libraries of machine learning can use at a later point in time.
  • Gathers data from any type of data source such as text, SQL DB, MS Excel or JSON files.
  • Contains many statistical functionalities that can be used to work on the data that’s gathered.

NumPy and SciPy

NumPy supports all array based and linear algebraic functions needed while working on data, while SciPy offers many scientific calculations. NumPy is more widely used in many real time applications of machine learning as compared to SciPy.

Matplotlib

This is a machine learning library that has an extensive collection of plots and charts. This library is a collection of many other packages. Of them, Seaborn is the most popular and is widely used to work on data structures.

PyTorch and Keras

These are known for their usage in Deep learning.

  • PyTorch library is extensively used for Deep Learning. It is known for its amazingly speedy calculations and is very popular among deep learning programmers.
  • Keras uses other libraries such as Tensor flow and is apt for developing neural networks.

Tools and Libraries of Machine Learning

Machine Learning – Processes

Besides algorithms, machine learning offers many tools and processes to pair best with big data. Various such processes and tools that are at hand for developers are:

  • Data quality and management
  • GUIs that ease models and process flows
  • Data exploration in an interactive mode
  • Visualized outputs for models
  • Choosing the best learning model by comparison
  • Model evaluation done automatically that identifies the best performers
  • User friendly model deployment and data-to-decision process

Machine Learning Use Cases

Here is a list of five use cases that are based on machine learning:

  • PayPal: The online money transfers giant uses machine learning for detecting any suspicious activities related to financial transactions.
  • Amazon: The company’s Alexa, the digital assistant, is the best example of speech processing application of machine learning. The online retailing giant is also using machine learning to display recommendation to its customers.
  • Facebook: The social media company is using machine learning extensively to filter out spam posts and forwards, and to shred out poor quality content.
  • IBM: The company’s self-driven vehicle uses machine learning in taking a decision whether to give the driving control to a human or computer.
  • Kaspersky: The anti-virus manufacturing company is using machine learning to detect security breaches, or unknown malware threats and also for high quality endpoint security for businesses.

Which Industries Use Machine Learning?

As we have seen just now, machine learning is being adopted in many industries for the potential advantages it offers. Machine learning can be applied to any industry that deals with huge volumes of data, and which has many challenges to be answered. For instance, machine learning has been found to be extremely useful to organizations in the following domains which are  making the best use of the technology:

Pharmaceuticals

Pharma industry spends billions of dollars on drug design and testing every year across the globe. Machine learning helps in cutting down such costs and to obtain results with accuracy just by entering the entire data of the drugs and their chemical compounds and comparing with various other parameters.

Banks and Financial Services

This industry has two major needs to be addressed: attracting investor attention and increasing investments, and staying alert and preventing financial frauds and cyber threats. Machine learning does these two major tasks with ease and accuracy.

Health Care and Treatments

By predicting the possible  diseases that could affect a patient, based on the medical, genetic and lifestyle data, machine learning helps patients stay alert to probable health threats that they may encounter. Wearable smart devices are an example of the machine learning applications in health care.

Online Sales

Companies study the patterns that online shoppers are adopting through machine learning and use the results to display related ads, offers and discounts. Personalisation of internet shopping experience, merchandise supply panning and marketing campaigns are all based on the outcomes of machine learning results themselves.

Mining, Oil and Gas

Machine learning helps in predicting accurately the best location of availability of minerals, gas, oil and other such natural resources, which would otherwise need huge investments, manpower and time.

Government Schemes

Many governments are taking the help of machine learning to study the interests and needs of their people. They are accordingly using the results in plans and schemes, both for the betterment of people and optimum usage of financial resources.

Space Exploration and Science Studies

Machine learning greatly helps in studying stars, planets and finding out the secrets of other celestial bodies with far lesser investments and manpower. Scientists are also maximising the use of machine learning to discover various fascinating facts about the earth and its components.

Future of Machine Learning

Future of Machine Learning

Currently, machine learning is entering our lives with baby steps. By the next decade, radical changes can be expected in machine learning and the way it impacts our lives. Customers have already started trusting the power and comfort of machine learning, and would definitely welcome more such innovations in the near future.

Gartner says:

Artificial Intelligence and Machine Learning have reached a critical tipping point and will increasingly augment and extend virtually every technology enabled service, thing, or application.

So, it would not be surprising if in the future, machine learning would:

  • Make its entry in almost every aspect of human  life
  • Be omnipresent in business and industries, irrespective of their size
  • Enter  cloud based services
  • Bring drastic changes in CPU design keeping in mind the need for computational efficiency
  • Altogether change the shape of data, its processing and usage
  • Change the way connected systems work and look  owing to the ever increasing data on the internet.

Conclusion

Machine Learning can work wonders

Machine learning is quite different in its own way. While many experts are raising concerns over the ever increasing dependence and presence of machine learning in our everyday lives, on the positive side, machine learning can work wonders. And the world is already witnessing its magic – in health care, finance industry, automotive industry, image processing and voice recognition and many other fields.

While many of us worry that machines may take over the world, it is totally up to us, how we design effective, yet safe and controllable machines. There is no doubt that machine learning would change the way we do many things including education, business and health services making the world a safer and better place.

Animikh

Animikh Aich

Computer Vision Engineer

Animikh Aich is a Deep Learning enthusiast, currently working as a Computer Vision Engineer. His work includes three International Conference publications and several projects based on Computer Vision and Machine Learning.

Join the Discussion

Your email address will not be published. Required fields are marked *

3 comments

vintage House restaurant 09 May 2019

Greetings! Very helpful advice within this article! It's the little changes that produce the greatest changes. Thanks a lot for sharing!

Aditya 21 Jun 2019

Excellent web site difficult to find high quality writing like yours nowadays,I honestly appreciate people like you! Take care

amith singh 06 Aug 2019

Hi, I read the complete blog and got full details of machine learning. It has been presented in such a way that anyone from a development background can understand easily. Thank you for the wonderful blog. Thank you Knowledgehut!

Suggested Blogs

Boosting and AdaBoost in Machine Learning

Ensemble learning is a strategy in which a group of models are used to find a solution to a challenging problem, by using a strategy and combining diverse machine learning models into one single predictive model.In general, ensemble methods are mainly used for improving the overall performance accuracy of a model and combine several different models, also known as the base learners, to predict the results, instead of using a single model.In one of the articles related to ensemble learning, we have already discussed about the popular ensemble method, Bootstrap Aggregation. Bagging tries to implement similar learners on small sample populations and then takes a mean of all the predictions. It combines Bootstrapping and Aggregation to form one ensemble model. It basically reduces the variance error and helps to avoid overfitting. In this article we will look into the limitations of bagging and how a boosting algorithm can be used to overcome those limitations. We will also learn about various types of boosting algorithms and implement one of them in Python. Let’s get started.What are the limitations of Bagging?Let us recall the concept of bagging and consider a binary classification problem. We are either classifying an observation as 0 or as 1.In bagging, T bootstrap samples are selected, a classifier is fitted on each of these samples, and the models are trained in parallel. In a Random Forest, decision trees are trained in parallel. Then the results of all classifiers are averaged into a bagging classifier:Formula for a Bagging ClassifierLet us consider 3 classifiers and the result for the classification can either be right or wrong. If we plot the results of the 3 classifiers, there are regions in which the classifiers will be wrong. These regions are represented in red in the figure below.Example case in which Bagging works wellThe above example works pretty well as when one classifier is wrong, the two others are correct. By voting classifier, you can achieve a better accuracy. However, there are cases where Bagging does not work properly, when all classifiers are mistaken to be in the same region.Due to this reason, the intuition behind the discovery of Boosting was the following :instead of training parallel models, one should train models sequentiallyeach model should focus on where the performance of the previous classifier was poorWith this intuition, Boosting algorithm was introduced. Let us understand what Boosting is all about.What is Boosting?Boosting is an ensemble modeling technique which attempts to build a strong classifier from the number of weak classifiers. It is done by building a model using weak models in series. First, a model is built from the training data. Then the second model is built which tries to correct the errors present in the first model. This procedure is continued and models are added until either the complete training data set is predicted correctly or the maximum number of models are added.Boosting being a sequential process, each subsequent model attempts to correct the errors of the previous model. It is focused on reducing the bias unlike bagging. It makes the boosting algorithms prone to overfitting. To avoid overfitting, parameter tuning plays an important role in boosting algorithms, which will be discussed in the later part of this article. Some examples of boosting are XGBoost, GBM, ADABOOST etc..How can boosting identify weak learners?To find weak learners, we apply base learning (ML) algorithms with a different distribution. As each time base learning algorithm is applied, it generates a new weak prediction rule. This is an iterative process. After many iterations, the boosting algorithm combines these weak rules into a single strong prediction rule.How do we choose a different distribution for each round?Step 1: The base learner takes all the distributions and assigns equal weight or attention to each observation.Step 2: If there is any prediction error caused by first base learning algorithm, then we pay higher attention to observations having prediction error. Then, we apply the next base learning algorithm.Step 3: Iterate Step 2 till the limit of base learning algorithm is reached or higher accuracy is achieved.Finally, it combines the outputs from weak learner and creates a strong learner which eventually improves the prediction power of the model. Boosting gives higher focus to examples which are mis-classified or have higher errors by preceding weak rules.How would you classify an email as SPAM or not?Our initial approach would be to identify ‘SPAM’ and ‘NOT SPAM’ emails using the following criteria. If: Email has only one image file (promotional image), It’s a SPAM.Email has only link(s), It’s a SPAM.Email body consists of sentences like “You won a prize money of $ xxxxxx”, It’s a SPAM.Email from our official domain “www.knowledgehut.com” , Not a SPAM.Email from known source, Not a SPAM.Individually, these rules are not powerful enough to classify an email into ‘SPAM’ or ‘NOT SPAM’. Therefore, these rules are called as weak learner.To convert weak learner to strong learner, we’ll combine the prediction of each weak learner using methods like:Using average/ weighted averageConsidering prediction has higher voteExample: Above, we have defined 5 weak learners. Out of these 5, 3 are voted as ‘SPAM’ and 2 are voted as ‘Not a SPAM’. In this case, by default, we’ll consider an email as SPAM because we have higher(3) vote for ‘SPAM’Boosting helps in training a series of low performing algorithms, called weak learners, simply by adjusting the error metric over time. Weak learners are considered to be those algorithms whose error rate is slightly under 50% as illustrated below:Weighted errorsLet us consider data points on a 2D plot. Some of the data points will be well classified, others won’t. The weight attributed to each error when computing the error rate is 1/n where n is the number of data points to classify.Now if we apply some weight to the errors :You might now notice that we give more weight to the data points that are not well classified. An illustration of the weighting process is mentioned below:Example of weighting processIn the end, we want to build a strong classifier that may look like the figure mentioned below:Strong ClassifierTree stumpsThere might be a question in your mind about how many classifiers should one implement in order to ensure it works well. And how is each classifier chosen at each step?Well, Tree stumps defines a 1-level decision tree. At each step, we need to find the best stump, i.e the best data split, which will minimize the overall error. You can see a stump as a test, in which the assumption is that everything that lies on one side belongs to class 1, and everything that lies on the other side belongs to class 0.Many such combinations are possible for a tree stump. Let us look into an example to understand how many combinations we face.3 data points to splitWell there are 12 possible combinations. Let us check how.12 StumpsThere are 12 possible “tests” we could make. The “2” on the side of each separating line simply represents the fact that all points on one side could be points that belong to class 0, or to class 1. Therefore, there are 2 tests embedded in it.At each iteration t, we will choose ht the weak classifier that splits best the data, by reducing the overall error rate the most. Recall that the error rate is a modified error rate version that takes into account what has been introduced before.Finding the best splitThe best split is found by identifying at each iteration t, the best weak classifier ht, generally a decision tree with 1 node and 2 leaves (a stump). Let us consider an example of credit defaulter, i.e whether a person who borrowed money will return or not.Identifying the best splitIn this case, the best split at time t is to stump on the Payment history, since the weighted error resulting from this split is minimum.Simply note that decision tree classifiers like these ones can in practice be deeper than a simple stump. This will be considered as a hyper-parameter.Combining classifiersIn the next step we combine the classifiers into a Sign classifier, and depending on which side of the frontier a point will stand, it is classified as 0 or 1. It can be achieved by:Combining classifiersYou can improve the classifier by adding weights on each classifier, to avoid giving the same importance to the different classifiers.AdaBoostPseudo-codePseudo-codeThe key elements to keep in mind are:Z is a constant whose role is to normalize the weights so that they add up to 1αt is a weight that we apply to each classifierThis algorithm is called AdaBoost or Adaptive Boosting. This is one of the most important algorithms among all boosting methods.ComputationBoosting algorithms are generally fast to train, although we consider every stump possible and compute exponentials recursively.Well, if we choose αt and Z properly, the weights that are supposed to change at each step simplify to:Weights after choice of α and ZTypes of Boosting AlgorithmsUnderlying engine used for boosting algorithms can be anything.  It can be decision stamp, margin-maximizing classification algorithm etc. There are many boosting algorithms which use other types of engines such as: AdaBoost (Adaptive Boosting)Gradient Tree BoostingXGBoostIn this article, we will focus on AdaBoost and Gradient Boosting followed by their respective Python codes and a little bit about XGBoost.Where are Boosted algorithms required?Boosted algorithms are mainly used when there is plenty of data to make a prediction and high predictive power is expected. It is used to reduce bias and variance in supervised learning. It combines multiple weak predictors to build strong predictor.The underlying engine used for boosting algorithms can be anything. For instance, AdaBoost is a boosting done on Decision stump. There are many other boosting algorithms which use other types of engine such as:GentleBoostGradient BoostingLPBoostBrownBoostAdaptive BoostingAdaptive Boosting, or most commonly known AdaBoost, is a Boosting algorithm. This algorithm uses the method to correct its predecessor. It pays more attention to under fitted training instances by the previous model. Thus, at every new predictor the focus is more on the complicated cases more than the others.It fits a sequence of weak learners on different weighted training data. It starts by predicting the original data set and gives equal weight to each observation. If prediction is incorrect using the first learner, then it gives higher weight to observation which have been predicted incorrectly. Being an iterative process, it continues to add learner(s) until a limit is reached in the number of models or accuracy.Mostly, AdaBoost uses decision stamps. But, we can use any machine learning algorithm as base learner if it accepts weight on training data set. We can use AdaBoost algorithms for both classification and regression problems.Let us consider the example of the image mentioned above. In order to build an AdaBoost classifier, consider that as a first base classifier a Decision Tree algorithm is trained to make predictions on our training data. Applying the following methodology of AdaBoost, the weight of the misclassified training instances is increased. Then the second classifier is trained and the updated weights are acknowledged. It repeats the procedure over and over again.At the end of every model prediction we end up boosting the weights of the misclassified instances so that the next model does a better job on them, and so on.This sequential learning technique might sound similar to Gradient Descent, except that instead of tweaking a single predictor’s parameter to minimize the cost function, AdaBoost adds predictors to the ensemble, gradually making it better.One disadvantage of this algorithm is that the model cannot be parallelized since each predictor can only be trained after the previous one has been trained and evaluated.Below are the steps for performing the AdaBoost algorithm:Initially, all observations are given equal weights.A model is built on a subset of data.Using this model, predictions are made on the whole dataset.Errors are calculated by comparing the predictions and actual values.While creating the next model, higher weights are given to the data points which were predicted incorrectly.Weights can be determined using the error value. For instance,the higher the error the more is the weight assigned to the observation.This process is repeated until the error function does not change, or the maximum limit of the number of estimators is reached.Hyperparametersbase_estimators: specify the base type estimator, i.e. the algorithm to be used as base learner.n_estimators: It defines the number of base estimators, where the default is 10 but you can increase it in order to obtain a better performance.learning_rate: same impact as in gradient descent algorithmmax_depth: Maximum depth of the individual estimatorn_jobs: indicates to the system how many processors it is allowed to use. Value of ‘-1’ means there is no limit;random_state: makes the model’s output replicable. It will always produce the same results when you give it a fixed value as well as the same parameters and training data.Now, let us take a quick look at how to use AdaBoost in Python using a simple example on handwritten digit recognition.import pandas as pd import numpy as np import matplotlib.pyplot as plt from sklearn.ensemble import AdaBoostClassifier from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import accuracy_score from sklearn.model_selection import cross_val_score from sklearn.model_selection import cross_val_predict from sklearn.model_selection import train_test_split from sklearn.model_selection import learning_curve from sklearn.datasets import load_digitsLet us load the data :dataset = load_digits() X = dataset['data'] y = dataset['target']X contains arrays of length 64 which are simply flattened 8x8 images. The aim of this dataset is to recognize handwritten digits. Let’s take a look at a given handwritten digit:plt.imshow(X[4].reshape(8,8))If we stick to a Decision Tree Classifier of depth 1 (a stump), here’s how to implement AdaBoost classifier:reg_ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=1)) scores_ada = cross_val_score(reg_ada, X, y, cv=6) scores_ada.mean()0.2636257855582272And it should head a result of around 26%, which can largely be improved. One of the key parameters is the depth of the sequential decision tree classifiers. How does accuracy improve with depth of the decision trees?score = [] for depth in [1,2,10] : reg_ada = AdaBoostClassifier(DecisionTreeClassifier(max_depth=depth)) scores_ada = cross_val_score(reg_ada, X, y, cv=6) score.append(scores_ada.mean()) score[0.2636257855582272, 0.5902852679072207, 0.9527524912410157]And the maximal score is reached for a depth of 10 in this simple example, with an accuracy of 95.3%.Gradient BoostingThis is another very popular Boosting algorithm which works pretty similar to what we’ve seen for AdaBoost. Gradient Boosting works by sequentially adding the previous predictors underfitted predictions to the ensemble, ensuring the errors made previously are corrected.The difference lies in what it does with the underfitted values of its predecessor. Contrary to AdaBoost, which tweaks the instance weights at every interaction, this method tries to fit the new predictor to the residual errors made by the previous predictor.So that you can understand Gradient Boosting it is important to understand Gradient Descent first.Below are the steps for performing the Gradient Boosting algorithm:A model is built on a subset of data.Using this model, predictions are made on the whole dataset.Errors are calculated by comparing the predictions and actual values.A new model is created using the errors calculated as target variable. Our objective is to find the best split to minimize the error.The predictions made by this new model are combined with the predictions of the previous.New errors are calculated using this predicted value and actual value.This process is repeated until the error function does not change, or the maximum limit of the number of estimators is reached.Hyperparametersn_estimators: It controls the number of weak learners.Learning_rate: Controls the contribution of weak learners in the final combination. There is a trade-off between learning_rate and n_estimators.min_samples_split: Minimum number of observation which is required in a node to be considered for splitting. It is used to control overfitting.min_samples_leaf: Minimum samples required in a terminal or leaf node. Lower values should be chosen for imbalanced class problems since the regions in which the minority class will be in the majority will be very small.min_weight_fraction_leaf: similar to the previous but defines a fraction of the total number of observations instead of an integer.max_depth : maximum depth of a tree. Used to control overfitting.max_lead_nodes : maximum number of terminal leaves in a tree. If this is defined max_depth is ignored.max_features : number of features it should consider while searching for the best split.You can tune loss function for better performance.Implementation in PythonYou can find Gradient Boosting function in Scikit-Learn’s library.# for regression from sklearn.ensemble import GradientBoostingRegressor model = GradientBoostingRegressor(n_estimators=3,learning_rate=1) model.fit(X,Y) # for classification from sklearn.ensemble import GradientBoostingClassifier model = GradientBoostingClassifier() model.fit(X,Y)XGBoostXG Boost or Extreme Gradient Boosting is an advanced implementation of the Gradient Boosting. This algorithm has high predictive power and is ten times faster than any other gradient boosting techniques. Moreover, it includes a variety of regularization which reduces overfitting and improves overall performance.AdvantagesIt implements regularization which helps in reducing overfit (Gradient Boosting does not have);It implements parallel processing which is much faster than Gradient Boosting;Allows users to define custom optimization objectives and evaluation criteria adding a whole new dimension to the model;XGBoost has an in-built routine to handle missing values;XGBoost makes splits up to the max_depth specified and then starts pruning the tree backwards and removes splits beyond which there is no positive gain;XGBoost allows a user to run a cross-validation at each iteration of the boosting process and thus it is easy to get the exact optimum number of boosting iterations in a single run.Boosting algorithms represent a different machine learning perspective which is turning a weak model to a stronger one to fix its weaknesses. I hope this article helped you understand how boosting works.We have covered most of the topics related to algorithms in our series of machine learning blogs, click here. If you are inspired by the opportunities provided by machine learning, enroll in our  Data Science and Machine Learning Courses for more lucrative career options in this landscape.
Rated 4.5/5 based on 12 customer reviews
7942
Boosting and AdaBoost in Machine Learning

Ensemble learning is a strategy in which a group o... Read More

Bagging and Random Forest in Machine Learning

In today’s world, innovations happen on a daily basis, rendering all the previous versions of that product, service or skill-set outdated and obsolete. In such a dynamic and chaotic space, how can we make an informed decision without getting carried away by plain hype? To make the right decisions, we must follow a set of processes; investigate the current scenario, chart down your expectations, collect reviews from others, explore your options, select the best solution after weighing the pros and cons, make a decision and take the requisite action. For example, if you are looking to purchase a computer, will you simply walk up to the store and pick any laptop or notebook? It’s highly unlikely that you would do so. You would probably search on Amazon, browse a few web portals where people have posted their reviews and compare different models, checking for their features, specifications and prices. You will also probably ask your friends and colleagues for their opinion. In short, you would not directly jump to a conclusion, but will instead make a decision considering the opinions and reviews of other people as well. Ensemble models in machine learning also operate on a similar manner. They combine the decisions from multiple models to improve the overall performance. The objective of this article is to introduce the concept of ensemble learning and understand algorithms like bagging and random forest which use a similar technique. What is Ensemble Learning? Ensemble methods aim at improving the predictive performance of a given statistical learning or model fitting technique. The general principle of ensemble methods is to construct a linear combination of some model fitting method, instead of using a single fit of the method. An ensemble is itself a supervised learning algorithm, because it can be trained and then used to make predictions. Ensemble methods combine several decision trees classifiers to produce better predictive performance than a single decision tree classifier. The main principle behind the ensemble model is that a group of weak learners come together to form a strong learner, thus increasing the accuracy of the model.When we try to predict the target variable using any machine learning technique, the main causes of difference in actual and predicted values are noise, variance, and bias. Ensemble helps to reduce these factors (except noise, which is irreducible error). The noise-related error is mainly due to noise in the training data and can't be removed. However, the errors due to bias and variance can be reduced.The total error can be expressed as follows: Total Error = Bias + Variance + Irreducible Error A measure such as mean square error (MSE) captures all of these errors for a continuous target variable and can be represented as follows: Where, E stands for the expected mean, Y represents the actual target values and fˆ(x) is the predicted values for the target variable. It can be broken down into its components such as bias, variance and noise as shown in the following formula: Using techniques like Bagging and Boosting helps to decrease the variance and increase the robustness of the model. Combinations of multiple classifiers decrease variance, especially in the case of unstable classifiers, and may produce a more reliable classification than a single classifier. Ensemble Algorithm The goal of ensemble algorithms is to combine the predictions of several base estimators built with a given learning algorithm in order to improve generalizability / robustness over a single estimator. There are two families of ensemble methods which are usually distinguished: Averaging methods. The driving principle is to build several estimators independently and then to average their predictions. On average, the combined estimator is usually better than any of the single base estimator because its variance is reduced.|Examples: Bagging methods, Forests of randomized trees. Boosting methods. Base estimators are built sequentially and one tries to reduce the bias of the combined estimator. The motivation is to combine several weak models to produce a powerful ensemble.Examples: AdaBoost, Gradient Tree Boosting.Advantages of Ensemble Algorithm Ensemble is a proven method for improving the accuracy of the model and works in most of the cases. Ensemble makes the model more robust and stable thus ensuring decent performance on the test cases in most scenarios. You can use ensemble to capture linear and simple as well nonlinear complex relationships in the data. This can be done by using two different models and forming an ensemble of two. Disadvantages of Ensemble Algorithm Ensemble reduces the model interpret-ability and makes it very difficult to draw any crucial business insights at the end It is time-consuming and thus might not be the best idea for real-time applications The selection of models for creating an ensemble is an art which is really hard to master Basic Ensemble Techniques Max Voting: Max-voting is one of the simplest ways of combining predictions from multiple machine learning algorithms. Each base model makes a prediction and votes for each sample. The sample class with the highest votes is considered in the final predictive class. It is mainly used for classification problems.  Averaging: Averaging can be used while estimating the probabilities in classification tasks. But it is usually used for regression problems. Predictions are extracted from multiple models and an average of the predictions are used to make the final prediction. Weighted Average: Like averaging, weighted averaging is also used for regression tasks. Alternatively, it can be used while estimating probabilities in classification problems. Base learners are assigned different weights, which represent the importance of each model in the prediction. Ensemble Methods Ensemble methods became popular as a relatively simple device to improve the predictive performance of a base procedure. There are different reasons for this: the bagging procedure turns out to be a variance reduction scheme, at least for some base procedures. On the other hand, boosting methods are primarily reducing the (model) bias of the base procedure. This already indicates that bagging and boosting are very different ensemble methods. From the perspective of prediction, random forests is about as good as boosting, and often better than bagging.  Bootstrap Aggregation or Bagging tries to implement similar learners on small sample populations and then takes a mean of all the predictions. It combines Bootstrapping and Aggregation to form one ensemble model Reduces the variance error and helps to avoid overfitting Bagging algorithms include: Bagging meta-estimator Random forest Boosting refers to a family of algorithms which converts weak learner to strong learners. Boosting is a sequential process, where each subsequent model attempts to correct the errors of the previous model. Boosting is focused on reducing the bias. It makes the boosting algorithms prone to overfitting. To avoid overfitting, parameter tuning plays an important role in boosting algorithms. Some examples of boosting are mentioned below: AdaBoost GBM XGBM Light GBM CatBoost Why use ensemble models? Ensemble models help in improving algorithm accuracy as well as the robustness of a model. Both Bagging and Boosting should be known by data scientists and machine learning engineers and especially people who are planning to attend data science/machine learning interviews. Ensemble learning uses hundreds to thousands of models of the same algorithm and then work hand in hand to find the correct classification. You may also consider the fable of the blind men and the elephant to understand ensemble learning, where each blind man found a feature of the elephant and they all thought it was something different. However, if they would work together and discussed among themselves, they might have figured out what it is. Using techniques like bagging and boosting leads to increased robustness of statistical models and decreased variance. Now the question becomes, between these different “B” words. Which is better? Which is better, Bagging or Boosting? There is no perfectly correct answer to that. It depends on the data, the simulation and the circumstances. Bagging and Boosting decrease the variance of your single estimate as they combine several estimates from different models. So the result may be a model with higher stability. If the problem is that the single model gets a very low performance, Bagging will rarely get a better bias. However, Boosting could generate a combined model with lower errors as it optimizes the advantages and reduces pitfalls of the single model. By contrast, if the difficulty of the single model is overfitting, then Bagging is the best option. Boosting for its part doesn’t help to avoid over-fitting; in fact, this technique is faced with this problem itself. For this reason, Bagging is effective more often than Boosting. In this article we will discuss about Bagging, we will cover Boosting in the next post. But first, let us look into the very important concept of bootstrapping. Bootstrap Sampling Sampling is the process of selecting a subset of observations from the population with the purpose of estimating some parameters about the whole population. Resampling methods, on the other hand, are used to improve the estimates of the population parameters. In machine learning, the bootstrap method refers to random sampling with replacement. This sample is referred to as a resample. This allows the model or algorithm to get a better understanding of the various biases, variances and features that exist in the resample. Taking a sample of the data allows the resample to contain different characteristics then it might have contained as a whole. This is demonstrated in figure 1 where each sample population has different pieces, and none are identical. This would then affect the overall mean, standard deviation and other descriptive metrics of a data set. In turn, it can develop more robust models. Bootstrapping is also great for small size data sets that can have a tendency to overfit. In fact, we recommended this to one company who was concerned because their data sets were far from “Big Data”. Bootstrapping can be a solution in this case because algorithms that utilize bootstrapping can be more robust and handle new data sets depending on the methodology chosen(boosting or bagging). The reason behind using the bootstrap method is because it can test the stability of a solution. By using multiple sample data sets and then testing multiple models, it can increase robustness. Perhaps one sample data set has a larger mean than another, or a different standard deviation. This might break a model that was overfit, and not tested using data sets with different variations. One of the many reasons bootstrapping has become very common is because of the increase in computing power. This allows for many times more permutations to be done with different resamples than previously. Bootstrapping is used in both Bagging and Boosting Let us assume we have a sample of ‘n’ values (x) and we’d like to get an estimate of the mean of the sample. mean(x) = 1/n * sum(x) Consider a sample of 100 values (x) and we’d like to get an estimate of the mean of the sample. We can calculate the mean directly from the sample as: We know that our sample is small and that the mean has an error in it. We can improve the estimate of our mean using the bootstrap procedure: Create many (e.g. 1000) random sub-samples of the data set with replacement (meaning we can select the same value multiple times). Calculate the mean of each sub-sample Calculate the average of all of our collected means and use that as our estimated mean for the data Example: Suppose we used 3 re-samples and got the mean values 2.3, 4.5 and 3.3. Taking the average of these we could take the estimated mean of the data to be 3.367. This process can be used to estimate other quantities like the standard deviation and even quantities used in machine learning algorithms, like learned coefficients. While using Python, we do not have to implement the bootstrap method manually. The scikit-learn library provides an implementation that creates a single bootstrap sample of a dataset. The resample() scikit-learn function can be used for sampling. It takes as arguments the data array, whether or not to sample with replacement, the size of the sample, and the seed for the pseudorandom number generator used prior to the sampling. For example, let us create a bootstrap that creates a sample with replacement with 4 observations and uses a value of 1 for the pseudorandom number generator. boot = resample(data, replace=True, n_samples=4, random_state=1)As the bootstrap API does not allow to easily gather the out-of-bag observations that could be used as a test set to evaluate a fit model, in the univariate case we can gather the out-of-bag observations using a simple Python list comprehension. # out of bag observations  oob = [x for x in data if x not in boot]Let us look at a small example and execute it.# scikit-learn bootstrap  from sklearn.utils import resample  # data sample  data = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6]  # prepare bootstrap sample  boot = resample(data, replace=True, n_samples=4, random_state=1)  print('Bootstrap Sample: %s' % boot)  # out of bag observations  oob = [x for x in data if x not in boot]  print('OOB Sample: %s' % oob) The output will include the observations in the bootstrap sample and those observations in the out-of-bag sample.Bootstrap Sample: [0.6, 0.4, 0.5, 0.1]  OOB Sample: [0.2, 0.3]Bagging Bootstrap Aggregation, also known as Bagging, is a powerful ensemble method that was proposed by Leo Breiman in 1994 to prevent overfitting. The concept behind bagging is to combine the predictions of several base learners to create a more accurate output. Bagging is the application of the Bootstrap procedure to a high-variance machine learning algorithm, typically decision trees. Suppose there are N observations and M features. A sample from observation is selected randomly with replacement (Bootstrapping). A subset of features are selected to create a model with sample of observations and subset of features. Feature from the subset is selected which gives the best split on the training data. This is repeated to create many models and every model is trained in parallel Prediction is given based on the aggregation of predictions from all the models. This approach can be used with machine learning algorithms that have a high variance, such as decision trees. A separate model is trained on each bootstrap sample of data and the average output of those models used to make predictions. This technique is called bootstrap aggregation or bagging for short. Variance means that an algorithm’s performance is sensitive to the training data, with high variance suggesting that the more the training data is changed, the more the performance of the algorithm will vary. The performance of high variance machine learning algorithms like unpruned decision trees can be improved by training many trees and taking the average of their predictions. Results are often better than a single decision tree. What Bagging does is help reduce variance from models that are might be very accurate, but only on the data they were trained on. This is also known as overfitting. Overfitting is when a function fits the data too well. Typically this is because the actual equation is much too complicated to take into account each data point and outlier. Bagging gets around this by creating its own variance amongst the data by sampling and replacing data while it tests multiple hypothesis(models). In turn, this reduces the noise by utilizing multiple samples that would most likely be made up of data with various attributes(median, average, etc). Once each model has developed a hypothesis. The models use voting for classification or averaging for regression. This is where the “Aggregating” in “Bootstrap Aggregating” comes into play. Each hypothesis has the same weight as all the others. When we later discuss boosting, this is one of the places the two methodologies differ. Essentially, all these models run at the same time, and vote on which hypothesis is the most accurate. This helps to decrease variance i.e. reduce the overfit. Advantages Bagging takes advantage of ensemble learning wherein multiple weak learners outperform a single strong learner.  It helps reduce variance and thus helps us avoid overfitting. Disadvantages There is loss of interpretability of the model. There can possibly be a problem of high bias if not modeled properly. While bagging gives us more accuracy, it is computationally expensive and may not be desirable depending on the use case. There are many bagging algorithms of which perhaps the most prominent would be Random Forest.  Decision Trees Decision trees are simple but intuitive models. Using a top-down approach, a root node creates binary splits unless a particular criteria is fulfilled. This binary splitting of nodes results in a predicted value on the basis of the interior nodes which lead to the terminal or the final nodes. For a classification problem, a decision tree will output a predicted target class for each terminal node produced. We have covered decision tree algorithm  in detail for both classification and regression in another article. Limitations to Decision Trees Decision trees tend to have high variance when they utilize different training and test sets of the same data, since they tend to overfit on training data. This leads to poor performance when new and unseen data is added. This limits the usage of decision trees in predictive modeling. However, using ensemble methods, models that utilize decision trees can be created as a foundation for producing powerful results. Bootstrap Aggregating Trees We have already discussed about bootstrap aggregating (or bagging), we can create an ensemble (forest) of trees where multiple training sets are generated with replacement, meaning data instances. Once the training sets are created, a CART model can be trained on each subsample. Features of Bagged Trees Reduces variance by averaging the ensemble's results. The resulting model uses the entire feature space when considering node splits. Bagging trees allow the trees to grow without pruning, reducing the tree-depth sizes and resulting in high variance but lower bias, which can help improve predictive power. Limitations to Bagging Trees The main limitation of bagging trees is that it uses the entire feature space when creating splits in the trees. Suppose some variables within the feature space are indicating certain predictions, there is a risk of having a forest of correlated trees, which actually  increases bias and reduces variance. Why a Forest is better than One Tree?The main objective of a machine learning model is to generalize properly to new and unseen data. When we have a flexible model, overfitting takes place. A flexible model is said to have high variance because the learned parameters (such as the structure of the decision tree) will vary with the training data. On the other hand, an inflexible model is said to have high bias as it makes assumptions about the training data. An inflexible model may not have the capacity to fit even the training data and in both cases — high variance and high bias — the model is not able to generalize new and unseen data properly. You can through the article on one of the foundational concepts in machine learning, bias-variance tradeoff which will help you understand that the balance between creating a model that is so flexible memorizes the training data and an inflexible model cannot learn the training data.  The main reason why decision tree is prone to overfitting when we do not limit the maximum depth is because it has unlimited flexibility, which means it keeps growing unless there is one leaf node for every single observation. Instead of limiting the depth of the tree which results in reduced variance and increase in bias, we can combine many decision trees into a single ensemble model known as the random forest. What is Random Forest algorithm? Random forest is like bootstrapping algorithm with Decision tree (CART) model. Suppose we have 1000 observations in the complete population with 10 variables. Random forest will try to build multiple CART along with different samples and different initial variables. It will take a random sample of 100 observations and then chose 5 initial variables randomly to build a CART model. It will go on repeating the process say about 10 times and then make a final prediction on each of the observations. Final prediction is a function of each prediction. This final prediction can simply be the mean of each prediction. The random forest is a model made up of many decision trees. Rather than just simply averaging the prediction of trees (which we could call a “forest”), this model uses two key concepts that gives it the name random: Random sampling of training data points when building trees Random subsets of features considered when splitting nodes How the Random Forest Algorithm Works The basic steps involved in performing the random forest algorithm are mentioned below: Pick N random records from the dataset. Build a decision tree based on these N records. Choose the number of trees you want in your algorithm and repeat steps 1 and 2. In case of a regression problem, for a new record, each tree in the forest predicts a value for Y (output). The final value can be calculated by taking the average of all the values predicted by all the trees in the forest. Or, in the case of a classification problem, each tree in the forest predicts the category to which the new record belongs. Finally, the new record is assigned to the category that wins the majority vote. Using Random Forest for Regression Here we have a problem where we have to predict the gas consumption (in millions of gallons) in 48 US states based on petrol tax (in cents), per capita income (dollars), paved highways (in miles) and the proportion of population with the driving license. We will use the random forest algorithm via the Scikit-Learn Python library to solve this regression problem. First we import the necessary libraries and our dataset. import pandas as pd  import numpy as np  dataset = pd.read_csv('/content/petrol_consumption.csv')  dataset.head() Petrol_taxAverage_incomepaved_HighwaysPopulation_Driver_licence(%)Petrol_Consumption09.0357119760.52554119.0409212500.57252429.0386515860.58056137.5487023510.52941448.043994310.544410You will notice that the values in our dataset are not very well scaled. Let us scale them down before training the algorithm. Preparing Data For Training We will perform two tasks in order to prepare the data. Firstly we will divide the data into ‘attributes’ and ‘label’ sets. The resultant will then be divided into training and test sets. X = dataset.iloc[:, 0:4].values  y = dataset.iloc[:, 4].valuesNow let us divide the data into training and testing sets:from sklearn.model_selection import train_test_split  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)Feature Scaling The dataset is not yet a scaled value as you will see that the Average_Income field has values in the range of thousands while Petrol_tax has values in the range of tens. It will be better if we scale our data. We will use Scikit-Learn's StandardScaler class to do the same. # Feature Scaling  from sklearn.preprocessing import StandardScaler  sc = StandardScaler()  X_train = sc.fit_transform(X_train)  X_test = sc.transform(X_test)Training the Algorithm Now that we have scaled our dataset, let us train the random forest algorithm to solve this regression problem. from sklearn.ensemble import Random Forest Regressor  regressor = Random Forest Regressor(n_estimators=20,random_state=0)  regressor.fit(X_train, y_train)  y_pred = regressor.predict(X_test)The RandomForestRegressor is used to solve regression problems via random forest. The most important parameter of the RandomForestRegressor class is the n_estimators parameter. This parameter defines the number of trees in the random forest. Here we started with n_estimator=20 and check the performance of the algorithm. You can find details for all of the parameters of RandomForestRegressor here. Evaluating the Algorithm Let us evaluate the performance of the algorithm. For regression problems the metrics used to evaluate an algorithm are mean absolute error, mean squared error, and root mean squared error.  from sklearn import metrics  print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))  print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))  print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred))) Mean Absolute Error: 51.76500000000001 Mean Squared Error: 4216.166749999999 Root Mean Squared Error: 64.93201637097064 With 20 trees, the root mean squared error is 64.93 which is greater than 10 percent of the average petrol consumption i.e. 576.77. This may indicate, among other things, that we have not used enough estimators (trees). Let us now change the number of estimators to 200, the results are as follows: Mean Absolute Error: 48.33899999999999 Mean Squared Error: 3494.2330150000003  Root Mean Squared Error: 59.112037818028234 The graph below shows the decrease in the value of the root mean squared error (RMSE) with respect to number of estimators.  You will notice that the error values decrease with the increase in the number of estimators. You may consider 200 a good number for n_estimators as the rate of decrease in error diminishes. You may try playing around with other parameters to figure out a better result. Using Random Forest for ClassificationNow let us consider a classification problem to predict whether a bank currency note is authentic or not based on four attributes i.e. variance of the image wavelet transformed image, skewness, entropy, andkurtosis of the image. We will use Random Forest Classifier to solve this binary classification problem. Let’s get started. import pandas as pd  import numpy as np  dataset = pd.read_csv('/content/bill_authentication.csv')  dataset.head()VarianceSkewnessKurtosisEntropyClass03.621608.6661-2.8073-0.44699014.545908.1674-2.4586-1.46210023.86600-2.63831.92420.10645033.456609.5228-4.0112-3.59440040.32924-4.45524.5718-0.988800Similar to the data we used previously for the regression problem, this data is not scaled. Let us prepare the data for training. Preparing Data For Training The following code divides data into attributes and labels: X = dataset.iloc[:, 0:4].values  y = dataset.iloc[:, 4].values The following code divides data into training and testing sets:from sklearn.model_selection import train_test_split  X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0) Feature Scaling We will do the same thing as we did for the previous problem. # Feature Scaling  from sklearn.preprocessing import StandardScaler  sc = StandardScaler()  X_train = sc.fit_transform(X_train)  X_test = sc.transform(X_test)Training the Algorithm Now that we have scaled our dataset, let us train the random forest algorithm to solve this classification problem. from sklearn.ensemble import Random Forest Classifier  classifier = RandomForestClassifier(n_estimators=20, random_state=0)  classifier.fit(X_train, y_train)  y_pred = classifier.predict(X_test)For classification, we have used RandomForestClassifier class of the sklearn.ensemble library. It takes n_estimators as a parameter. This parameter defines the number of trees in out random forest. Similar to the regression problem, we have started with 20 trees here. You can find details for all of the parameters of Random Forest Classifier here. Evaluating the Algorithm For evaluating classification problems,  the metrics used are accuracy, confusion matrix, precision recall, and F1 valuesfrom sklearn.metrics import classification_report, confusion_matrix, accuracy_score  print(confusion_matrix(y_test,y_pred))  print(classification_report(y_test,y_pred))  print(accuracy_score(y_test, y_pred)) The output will look something like this: Output:[ [ 155   2] [     1  117] ]Precisionrecallf1-scoresupport00.990.990.9915710.980.990.99118accuracy0.99275macro avg0.990.990.992750.98909090909090910.990.990.99275The accuracy achieved by our random forest classifier with 20 trees is 98.90%. Let us change the number of trees to 200.from sklearn.ensemble import Random Forest Classifier  classifier = Random Forest Classifier(n_estimators=200, random_state=0)  classifier.fit(X_train, y_train)  y_pred = classifier.predict(X_test) Output:[ [ 155   2] [     1  117] ]Precisionrecallf1-scoresupport00.990.990.9915710.980.990.99118accuracy0.99275macro avg0.990.990.992750.98909090909090910.990.990.99275Unlike the regression problem, changing the number of estimators for this problem did not make any difference in the results.An accuracy of 98.9% is pretty good. In this case, we have seen that there is not much improvement if the number of trees are increased. You may try playing around with other parameters of the RandomForestClassifier class and see if you can improve on our results. Advantages and Disadvantages of using Random Forest As with any algorithm, there are advantages and disadvantages to using it. Let us look into the pros and cons of using Random Forest for classification and regression. Advantages Random forest algorithm is unbiased as there are multiple trees and each tree is trained on a subset of data.  Random Forest algorithm is very stable. Introducing a new data in the dataset does not affect much as the new data impacts one tree and is pretty hard to impact all the trees. The random forest algorithm works well when you have both categorical and numerical features. With missing values in the dataset, the random forest algorithm performs very well. Disadvantages A major disadvantage of random forests lies in their complexity. More computational resources are required and also results in the large number of decision trees joined together. Due to their complexity, training time is more compared to other algorithms. Summary In this article we have covered what is ensemble learning and discussed about basic ensemble techniques. We also looked into bootstrap sampling involves iteratively resampling of a dataset with replacement which allows the model or algorithm to get a better understanding various features. Then we moved on to bagging followed by random forest. We also implemented random forest in Python for both regression and classification and came to a conclusion that increasing number of trees or estimators does not always make a difference in a classification problem. However, in regression there is an impact.  We have covered most of the topics related to algorithms in our series of machine learning blogs,click here. If you are inspired by the opportunities provided by machine learning, enrol in our  Data Science and Machine Learning Courses for more lucrative career options in this landscape. 0.99
Rated 4.5/5 based on 12 customer reviews
16648
Bagging and Random Forest in Machine Learning

In today’s world, innovations happen on a daily ... Read More

Support Vector Machines in Machine Learning

While many classifiers exist that can classify linearly separable data such as logistic regression, Support Vector Machines can handle highly non-linear problems using a kernel trick which implicitly maps the input vectors to higher-dimensional feature spaces. The transformation rearranges the dataset in such a way that it is then linearly solvable. In this article we are going to look at how SVM works, learn about kernel functions, hyperparameters and pros and cons of SVM along with some of the real life applications of SVM. Support Vector Machines (SVMs), also known as support vector networks, are a family of extremely powerful models which use method based learning and can be used in classification and regression problems. They aim at finding decision boundaries that separate observations with differing class memberships. In other words, SVM is a discriminative classifier formally defined by a separating hyperplane.Method Based Learning There are several learning models namely:Association rules basedEnsemble method basedDeep Learning basedClustering method basedRegression Analysis basedBayesian method basedDimensionality reduction based Instance basedKernel method basedLet us understand what Kernel method based learning is all about.In simple terms, a kernel is a similarity function which is fed into a machine learning algorithm. It accepts two inputs and suggests the similarity. For example, suppose we want to classify images, the input data is a key-value pair (image, label). The image data is taken into consideration, features are computed, and a vector of features are fed into the Machine learning algorithm. But in the case of similarity functions, a kernel function can be defined which internally computes the similarity between images, and then feeds into the learning algorithm along with the images and label data. The outcome of this is a classifier. Perceptron frameworks or Support vector machines work with kernels and use vectors only. Here, the machine learning algorithms are expressed as dot products so that kernel functions can be used.Feature vectors generally prefer kernels. Its ease of computing makes it one of the key reasons, also, feature vectors need more storage space in comparison to dot products. You can writeMachine learning algorithms to use dot products and later map them to use kernels. This completely avoids the usage of feature vectors. This allows us to work with highly complex, efficient-to-compute, and yet high performing kernels effortlessly, without really developing multi-dimensional vectors.Kernel functionsLet us understand what kernel functions are: The figure shown below represents a 1D function using a simple 1-Dimensional example. Assume that given points are as follows, it will depict a vertical line and no other vertical lines will separate the dataset.Now, if we consider a 2-Dimensional representation, as shown in the figure below, there is a hyperplane (an arbitrary line in 2-Dimensions) which separates red and blue points, which can be separated using Support Vector Machines.As we keep increasing dimensional space, the need to be able to separate data will eventually decrease. This mapping, x -> (x, x2), is called the kernel function. In case of growing dimensional space, the computations become more complex and kernel trick needs to be applied to address these computations cheaply. What is Support Vector Machine? Support Vector Machine (SVM) is a supervised machine learning algorithm which can be used for both classification or regression challenges. However,  it is mostly used in classification problems. In this algorithm, each data is plotted in n-dimensional space (where n is the number of features you have) with the value of each feature being the value of a particular coordinate. After that, we perform classification by locating the hyperplane which differentiates both the classes.Let us create a dataset to understand support vector classification:# importing scikit learn with make_blobs from sklearn.datasets.samples_generator import make_blobs# creating datasets X containing n_samples # Y containing two classes X, Y = make_blobs(n_samples=500, centers=2,        random_state=0, cluster_std=0.40)# plotting scatters plt.scatter(X[:, 0], X[:, 1], c=Y, s=50, cmap='spring'); plt.show()Support vector machine is based on the concept of decision planes that define decision boundaries. A decision plane is one that separates between a set of objects with different class memberships. For example, in the figure mentioned below, there are objects which belong to either class Green or Red. The separating line defines a boundary on the right side of which all objects are Green and to the left of which all objects are Red. Any new object (white circle) falling to the right is labeled, i.e., classified, as Green (or classified as Red should it fall to the left of the separating line).Support vector machines not only draw a line between two classes, but consider a region about the line of some given width. Here’s an example of what it can look like:# creating line space between -1 to 3.5 xfit = np.linspace(-1, 3.5) # plotting scatter plt.scatter(X[:, 0], X[:, 1], c=Y, s=50, cmap='spring') # plot a line between the different sets of data for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:     yfit = m * xfit + b     plt.plot(xfit, yfit, '-k')     plt.fill_between(xfit, yfit - d, yfit + d, edgecolor='none',     color='#AAAAAA', alpha=0.4)plt.xlim(-1, 3.5); plt.show()Another scenario, where it is clear that a full separation of the Green and Red objects would require a curve (which is more complex than a line). Classification tasks based on drawing separating lines to distinguish between objects of different class memberships are known as hyperplane classifiers. Support Vector Machines are particularly suited to handle such tasks.The figure below shows the basic idea behind Support Vector Machines. Here you will see that the original objects (left side of the schematic) mapped, are rearranged using a set of mathematical functions called kernels. This process of rearranging objects is known as mapping or transformation. You will notice that the right side of the schematic is linearly separable. All we can do is find an optimal line that will separate red and green objects.What is a hyperplane?The goal of Support Vector Machine is to find the hyperplane which separates these two objects or classes. Let us consider another figure which shows some of the possible hyperplanes which can help in separating or dividing the dataset. It is the choice of the best hyperplane which is also the goal. The best hyperplane is defined by the extent to which a maximum margin is left for both classes. The margin is the distance between the hyperplane and the closest point in the classification.Let us consider two hyperplanes among all and then check the margins represented by M1 and M2. You will notice that margin M1 > M2, so the choice of the hyperplane which separates the best one is the new plane between the green and blue planes.How do we find the right hyperplane?Now, let us represent the new plane by a linear equation as: f(x) = ax + bLet us consider that this equation delivers all values ≥ 1 from the green triangle class and ≤ -1 for the gold star class. The distance of this plane from the closest points in both the classes is at least one; the modulus is one. f(x) ≥ 1 for triangles and f(x) ≤ 1 or |f(x)| = 1 for starThe distance between the hyperplane and the point can be computed using the following equation. M1 = |f(x)| / ||a|| = 1 / ||a||The total margin is 1 / ||a|| + 1 / ||a|| = 2 / ||a|. In order to maximize the separability, we will have to maximize the ||a|| value. This particular value is known as a weight vector. We can minimize the weight value which is a non-linear optimization task. One of the methods is to use the Karush-Kuhn-Tucker (KKT) condition, using the Lagrange multiplier λi.What is a support vector in SVM?Let's take an example of two points between the two attributes X and Y. We need to find a point between these two points that has a maximum distance between these points. This requirement is represented in the graph depicted next. The optimal point is depicted using the red circle.The maximum margin weight vector is parallel to the line from (1, 1) to (2, 3). The weight vector is at (1,2), and this becomes a decision boundary that is halfway between and in perpendicular, that passes through (1.5, 2). So, y = x1 +2x2 − 5.5 and the geometric margin is computed as √5. Following are the steps to compute SVMs: With w = (a, 2a) for the functions of the points (1,1) and (2,3) can be represented as shown here: a + 2a + ω0 = -1 for the point (1,1) 2a + 6a + ω0 = 1 for the point (2,3) The weights can be computed as follows:These are the support vectors:Lastly, the final equation is as follows:Large Margin IntuitionIn logistic regression, the output of linear function is taken and the value is squashed within the range of [0,1] using the sigmoid function. If the value is greater than a threshold value, say 0.5, label 1 is assigned else label 0.  In case of support vector machines, the linear function is taken and if the output is greater than 1 and we identify it with one class and if the output is -1, it is identified with another class. Since the threshold values are changed to 1 and -1 in SVM, we obtain this reinforcement range of values([-1,1]) which acts as margin. Cost Function and Gradient UpdatesIn the SVM algorithm, we maximize the margin between the data points and the hyperplane. The loss function that helps maximize the margin is called the hinge loss.Hinge loss function (function on the left can be represented as a function on the right)   If the predicted value and the actual value are of the same sign, the cost is 0 . If not, we calculate the loss value. We also add a regularization parameter the cost function. The objective of the regularization parameter is to balance the margin maximization and loss. After adding the regularization parameter, the cost functions looks as below.Loss function for SVM  Now that we have the loss function, we take partial derivatives with respect to the weights to find the gradients. Using gradients, we can update our weights.Gradients  When there is no misclassification, i.e our model correctly predicts the class of our data point, we only have to update the gradient from the regularization parameter.Gradient Update — No misclassification  When there is a misclassification, i.e our model makes a mistake on the prediction of the class of our data point, we include the loss along with the regularization parameter to perform gradient update.Gradient Update — Misclassification  Let us start with a code and import the necessary libraries:import pandas as pd  import numpy as np  from sklearn.model_selection import train_test_split  from sklearn.model_selection import cross_val_score, GridSearchCV  from sklearn import metrics  from sklearn.preprocessing import MinMaxScaler  pd.set_option('display.max_columns', None)Read the Wisconsin Breast Cancer dataset using pandas.read_csv function into an object 'data' from the current directorydata = pd.read_csv('wisconsin.csv')After reading the data, we have prepared the data as per requirement. Feature scaling is a method used to standardize the range of independent variables or features of data. The min-max scaling (or min-max normalization) shrinks the range of feature such that the range is in between 0 and 1 (or -1 to 1 if there are negative values).sclr = MinMaxScaler() predictor_sc = sclr.fit_transform(predictor)predictor_sc.shapeSplit the scaled data into train-test split:x_train_sc,x_test_sc, y_train, y_test = train_test_split(predictor_sc, target, test_size = 0.30, random_state=101) print("Scaled train and test split") print("x_train ",x_train_sc.shape) print("x_test ",x_test_sc.shape) print("y_train ",y_train.shape) print("y_test ",y_test.shape)Scaled train and test split x_train  (398, 30) x_test  (171, 30) y_train  (398,) y_test  (171,)But what happens when there is no clear hyperplane? Support Vector Machines can probably help you to find a separating hyperplane but only if it exists. There are certain cases when it is not possible to define a hyperplane, this happens due to noise in the data. Another possible reason can be a non-linear boundary. The first graph below depicts noise and the next one shows a non-linear boundary.There might be cases where there is no possibility to define a hyperplane, which can happen due to noise in the data. In fact, another reason can be a non-linear boundary as well. The following first graph depicts noise and the second one shows a non-linear boundary.For such problems which arise due to noise in the data, the best way is to reduce the margin itself and introduce slack.The non-linear boundary problem can be solved if we introduce a kernel. Some of the kernel functions that can be introduced are mentioned below:A radial basis function is a real-valued function whose value is dependent on the distance between the input and some fixed point. In machine learning, the radial basis function kernel, or RBF kernel, is a popular kernel function used in various kernelized learning algorithms.The RBF kernel on two samples x and x', represented as feature vectors in some input space, is defined as:Applying SVM with default hyperparametersLet us get back to the example and apply SVM after data pre-processsing with default hyperparameters. Linear Kernelfrom sklearn import svm svm2 = svm.SVC(kernel='linear') svm2 SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False) model2 = svm2.fit(x_train_sc, y_train) y_pred2 = svm2.predict(x_test_sc) print('Accuracy Score’) print(metrics.accuracy_score(y_test,y_pred2))Accuracy Score:0.9707602339181286Gaussian Kernelsvm3 = svm.SVC(kernel='rbf') svm3 SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False) model3 = svm3.fit(x_train_sc, y_train) y_pred3 = svm3.predict(x_test_sc) print('Accuracy Score’) print(metrics.accuracy_score(y_test, y_pred3))Accuracy Score:0.935672514619883Polynomial Kernelsvm4 = svm.SVC(kernel='poly') svm4SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, decision_function_shape='ovr', degree=3, gamma='auto', kernel='poly', max_iter=-1, probability=False, random_state=None, shrinking=True, tol=0.001, verbose=False)model4 = svm4.fit(x_train_sc, y_train) y_pred4 = svm4.predict(x_test_sc) print('Accuracy Score’) print(metrics.accuracy_score(y_test,y_pred4)) Accuracy Score:0.6198830409356725How to tune Parameters of SVM? Kernel: Kernel in support vector machine is responsible for the transformation of the input data into the required format. Some of the kernels used in support vector machines are linear, polynomial and radial basis function (RBF). In order to create a non-linear hyperplane, we use RBF and Polynomial function, and for complex applications, you should use more advanced kernels to separate classes that are nonlinear in nature. With this transformation, you can obtain accurate classifiers. Regularization: Using the Scikit-learn’s C parameters and adjusting we can maintain regularization. C denotes a penalty parameter representing an error or any form of misclassification. This misclassification allows you to understand how much of the error is actually bearable. This helps you nullify the compensation between the misclassified term and the decision boundary. With a smaller C value, you obtain hyperplane of small margin and with a larger C value, hyperplane of larger value is obtained. Gamma: Lower value of Gamma creates a loose fit of the training dataset. On the other hand, a high value of gamma allows the model to get fit more appropriately. A low value of gamma will only provide consideration to the nearby points for the calculation of a separate plane. However, the high value of gamma will consider all the data-points to calculate the final separation line. Do we need to tune parameters always?? You do not need to tune parameter in all cases. There are inbuilt functions in sklearn tool kit which can be used. Tuning HyperparametersThe 'C' and 'gamma' hyperparameterC is the parameter for the soft margin cost function, which controls the influence of each individual support vector. This process involves trading error penalty for stability. Small C tends to emphasize the margin while ignoring the outliers in the training data(Soft Margin), while large C may tend to overfit the training data(Hard Margin). Thus for a very large values we can cause overfitting of the model and for a very small value of C we can cause underfitting.Thus the value of C must be chosen in such a manner that it generalises the unseen data well. The gamma parameter is the inverse of the standard deviation of the RBF kernel (Gaussian function), which is used as a similarity measure between two points. A small gamma value define a Gaussian function with a large variance. In this case, two points can be considered similar even if are far from each other. On the other hand, a large gamma value define a Gaussian function with a small variance and in this case, two points are considered similar just if they are close to each other. Taking kernel as linear and tuning C hyperparameterC_range=list(range(1,26)) acc_score=[] for c in C_range: svc = svm.SVC(kernel='linear', C=c) scores = cross_val_score(svc, predictor_sc, target, cv=10, scoring='accuracy') acc_score.append(scores.mean()) print(acc_score) [0.9772210699161695, 0.9772210699161695, 0.9806995938121164, 0.9824539797770286, 0.9789754558810818, 0.9789452078472042, 0.9806995938121164, 0.9789452078472041, 0.9789452078472041, 0.9789452078472041, 0.9806995938121164, 0.9789452078472041, 0.9789452078472041, 0.9772210699161695, 0.9772210699161695, 0.9772210699161695, 0.9772210699161695, 0.9754666839512574, 0.9754666839512574, 0.9754666839512574, 0.9754666839512574, 0.9754666839512574, 0.9754666839512574, 0.9754666839512574, 0.9754666839512574]Let us visualize the above points:import matplotlib.pyplot as plt %matplotlib inline C_Val_list = list(range(1,26)) plt.plot(C_Val_list,acc_score) plt.xticks(np.arange(0,27,2)) plt.xlabel('Value of C for SVC') plt.ylabel('Cross-Validated Accuracy')From the plot we can see that accuracy has been close to 98% somewhere in between C=4 and C=5 and then it drops.#Taking a close look at the cross-validation accuracy in the range C(4,5) C_range=list(np.arange(4,5,0.2)) acc_score=[] for c in C_range: svc = svm.SVC(kernel='linear', C=c) scores = cross_val_score(svc, predictor_sc, target, cv=10, scoring='accuracy') acc_score.append(scores.mean()) print(acc_score) [0.9824539797770286, 0.9806995938121164, 0.9789754558810818, 0.9789754558810818, 0.9789754558810818] Accuracy score is highest for C=4Taking kernel as gaussian and tuning gamma hyperparametergamma_range=[0.0001,0.001,0.01,0.1,1,10,100] acc_score=[] for g in gamma_range: svc = svm.SVC(kernel='rbf', gamma=g) scores = cross_val_score(svc, predictor_sc, target, cv=10, scoring='accuracy') acc_score.append(scores.mean()) print(acc_score) [0.6274274047186933, 0.6274274047186933, 0.9195035001296346, 0.9561651974764496, 0.9806995938121164, 0.9420026359000951, 0.6274274047186933] Let us visualize the above points: gamma_range=[0.0001,0.001,0.01,0.1,1,10,100]# plotting the value of gamma for SVM versus the cross-validated accuracy plt.plot(gamma_range,acc_score) plt.xlabel('Value of gamma for SVC ') plt.xticks(np.arange(0.0001,100,5)) plt.ylabel('Cross-Validated Accuracy')Text(0,0.5,'Cross-Validated Accuracy')For gamma between 5 and 100 the kernel performs very poorly.Let us take a closer look at the cross-validated accuracy for gamma value in between 0 and 5.gamma_range=list(np.arange(0.1,5,0.1))  acc_score=[] for g in gamma_range:  svc = svm.SVC(kernel='rbf', gamma=g)  scores = cross_val_score(svc, predictor_sc, target, cv=10, scoring='accuracy') acc_score.append(scores.mean())  print(acc_score)[0.9561651974764496, 0.9718952553798289, 0.9754051075965776, 0.9737122979863452, 0.9806995938121164, 0.9806995938121164, 0.9806995938121164, 0.9806995938121164, 0.9806995938121164, 0.9806995938121164, 0.9789754558810818, 0.9754969319851352, 0.9754969319851352, 0.9754969319851352, 0.9754969319851352, 0.9737727940541007, 0.9737727940541007, 0.9737727940541007, 0.9737727940541007, 0.9720184080891883, 0.9720184080891883, 0.9720184080891883, 0.9720184080891883, 0.9720184080891883, 0.9720184080891883, 0.9702326938034741, 0.9702326938034741, 0.9702326938034741, 0.9702326938034741, 0.9702326938034741, 0.9702326938034741, 0.9702326938034741, 0.9702326938034741, 0.9666925935528475, 0.9666925935528475, 0.9684167314838821, 0.9684167314838821, 0.9684167314838821, 0.9701711174487941, 0.9701711174487941, 0.96838540316308, 0.9649068792671333, 0.9649068792671333, 0.9649068792671333, 0.9649068792671333, 0.9649068792671333, 0.9649068792671333, 0.963152493302221, 0.963152493302221] gamma_range=list(np.arange(0.1,5,0.1)) plt.plot(gamma_range,acc_score) plt.xlabel('Value of gamma for SVC ') #plt.xticks(np.arange(0.0001,5,5)) plt.ylabel('Cross-Validated Accuracy') Text(0,0.5,'Cross-Validated Accuracy')The highest cross-validated accuracy for rbf kernel remains constant in between gamma=0.5 and gamma=1Taking polynomial kernel and tuning degree hyperparameterdegree=[2,3,4,5,6] acc_score=[] for d in degree: svc = svm.SVC(kernel='poly', degree=d) scores = cross_val_score(svc, predictor_sc, target, cv=10, scoring='accuracy') acc_score.append(scores.mean()) print(acc_score) [0.8350974418805635, 0.6450652493302222, 0.6274274047186933, 0.6274274047186933, 0.6274274047186933] plt.plot(degree,acc_score) plt.xlabel('degrees for SVC ') plt.ylabel('Cross-Validated Accuracy') Text(0,0.5,'Cross-Validated Accuracy')Score is high for second degree polynomial. There is drop in the accuracy score as degree of polynomial increases.Thus increase in polynomial degree results in high complexity of the model. Advantages and Disadvantages of Support Vector MachineAdvantages of SVMSVM Classifiers offer good accuracy and perform faster prediction compared to Naïve Bayes algorithm. SVM guarantees optimality due to the nature of Convex Optimization, the solution will always be global minimum not a local minimum. SVMcan be access it conveniently, be it from Python or Matlab. SVM can be used for both linearly separable as well as non-linearly separable data. Linearly separable data is the hard margin however, non-linearly separable data poses a soft margin. SVM provides compliance to the semi-supervised learning models as well. It can be implemented in both labelled and unlabelled data. The only thing it requires is a condition to the minimization problem which is known as the Transductive SVM. Feature Mapping used to be complex with respect to computation of the overall training performance of the model. With the help of Kernel Trick, SVM can carry out the feature mapping using simple dot product. SVM works well with a clear margin of separation and with high dimensional space.  Disadvantages of SVM SVM is not at all capable of handling text structures. It leads to bad performance as it results in the loss of sequential information. SVM is not suitable for large datasets because of its high training time and it also takes more time in training compared to Naïve Bayes. SVM works poorly with overlapping classes and is also sensitive to the type of kernel used. In cases where the number of features for each data point exceeds the number of training data samples , the SVM under performs. Applications of SVM in Real WorldSupport vector machines depend on supervised learning algorithms. The main goal of using SVM is to classify unseen data correctly. SVMs can be used to solve various real-world problems: Face detection – SVM can be used to classify parts of the image as a face and non-face and create a square boundary around the face. Text and hypertext categorization – SVM allows text and hypertext categorization for both inductive and transductive models. It uses training data for classification of documents into different categories. It categorizes based on the score generated and then compares with the threshold value. Classification of images – SVMs enhances search accuracy for image classification. In comparison to the traditional query-based searching techniques, SVM provides better accuracy. Bioinformatics – It includes classification of proteins and classification of cancer. SVM is used for identifying the classification of genes, patients on the basis of genes and other biological problems. Protein fold and remote homology detection – SVM algorithms are applied for protein remote homology detection. Handwriting recognition –  SVMs are used widely to recognize handwritten characters.  Generalized predictive control(GPC) – You can use SVM based GPC in order to control chaotic dynamics with useful parameters. Summary In this article, we looked at the machine learning algorithm, Support Vector Machine in detail. We have discussed the concept behind support vector machines, how it works, the process of implementation in Python.  We also looked into how to tune its parameters and make efficient models. Lastly, we came across the advantages and disadvantages of SVM along with various real world applications of support vector machines.We have covered most of the topics related to algorithms in our series of machine learning blogs,click here. If you are inspired by the opportunities provided by machine learning, enrol in our  Data Science and Machine Learning Courses for more lucrative career options in this landscape.
Rated 4.0/5 based on 67 customer reviews
27729
Support Vector Machines in Machine Learning

While many classifiers exist that can classify lin... Read More

20% Discount