Linear Regression in Machine Learning [with Examples]

Read it in 15 Mins

Last updated on
04th Nov, 2022
Published
20th Sep, 2019
Views
12,369
Linear Regression in Machine Learning [with Examples]

Statistical techniques have been used for Data Analysis and Interpretation for a long time. Linear Regression in Machine Learning analysis is important for evaluating data and establishing a definite relationship between two or more variables. Regression quantifies how the dependent variable changes as the independent variable itself take different values. Regression is referred to as simple or multiple regression depending on the number of independent variables, like single or multiple variables respectively. 

Machine Learning is the solution when data is large, and relation becomes difficult to quantify manually. Here, the model is trained on available data of a number of independent variables with the statistical tool of Linear Regression to determine how the relationship can be obtained with great accuracy. This article has a practical example of Regression in Machine Learning for beginners. These days a comprehensive data science online course  can help build the necessary foundation to the essential concepts of Regression in Machine Learning. 

What is Linear Regression in Machine Learning?

Linear Regression is an algorithm that belongs to supervised Machine Learning. It tries to apply relations that will predict the outcome of an event based on the independent variable data points. The relation is usually a straight line that best fits the different data points as close as possible. The output is of a continuous form, i.e., numerical value. For example, the output could be revenue or sales in currency, the number of products sold, etc. In the above example, the independent variable can be single or multiple. 

1. Linear Regression Equation

Linear regression can be expressed mathematically as: 

y= β0β 1x+ ε 

Here, 

  • Y= Dependent Variable  
  • X= Independent Variable  
  • β 0= intercept of the line  
  • β1 = Linear regression coefficient (slope of the line) 
  • ε = random error 

The last parameter, random error ε, is required as the best fit line also doesn't include the data points perfectly. 

2. Linear Regression Model 

Since the Linear Regression algorithm represents a linear relationship between a dependent (y) and one or more independent (y) variables, it is known as Linear Regression. This means it finds how the value of the dependent variable changes according to the change in the value of the independent variable. The relation between independent and dependent variables is a straight line with a slope. 

Types of Linear Regression

Linear Regression can be broadly classified into two types of algorithms: 

1. Simple Linear Regression

A simple straight-line equation involving slope (dy/dx) and intercept (an integer/continuous value) is utilized in simple Linear Regression. Here a simple form is: 

y=mx+c where y denotes the output, x is the independent variable, and c is the intercept when x=0. With this equation, the algorithm trains the model of machine learning and gives the most accurate output 

2. Multiple Linear Regression

When a number of independent variables are more than one, the governing linear equation applicable to regression takes a different form like: 

y= c+m1x1+m2x2… mnxn where represents the coefficient responsible for impact of different independent variables x1, x2 etc. This machine learning algorithm, when applied, finds the values of coefficients m1, m2, etc., and gives the best fitting line. 

3. Non-Linear Regression

When the best fitting line is not a straight line but a curve, it is referred to as Non-Linear Regression.   

Linear Regression Terminologies

1. Cost Function

The output which is obtained or predicted by an algorithm is referred to as yˆ (pronounced as yhat). The difference between the actual and predicted values is the error, i.e., y -  yˆy^. Different values of y- yˆy^ (loss function) are obtained when the model repeatedly tries to find the best relation. The average summation of all loss function values is called the cost function. The machine learning algorithm tries to obtain the minimum value of the cost function. In other words, it tries to reach the global minimum. 

where J = cost function, n= number of observations (i = 1 to n), ∑ = summation, predi = predicted output and yi = actual value.  

As shown above, the error difference is squared for each value and then the average of the sum of squares of error gives us the cost function. It is also referred to as Mean Square Error (MSE). 

2. Gradient Descent 

Another important concept in Linear Regression is Gradient Descent. It is a popular optimization approach employed in training machine learning models by reducing errors between actual and predicted outcomes. Optimization in machine learning is the task of minimizing the cost function parameterized by the model's parameters. The primary goal of gradient descent is to minimize the convex function by parameter iteration.

A slower learning rate helps to reach the global minimum but takes an unusually long time and computationally proves expensive. The faster learning rate may make the model wander and lead to an undesired position, making it difficult to come back on the correct track to reach the global minimum. Hence the learning rate should be neither too slow nor too fast if the global minimum is to be reached efficiently.  

Interested in learning Data Science and its importance? Check out data science online course. 

How Does Linear Regression Work?

After understanding the concept of Linear Regression and its adoption to solve many engineering and business problems, we now will consider the process of applying Linear Regression in a Machine Learning project. Let us import the necessary libraries: 

import pandas as pd 
import matplotlib.pyplot as plt 
import seaborn as sns 
from sklearn.model_selection import train_test_split 
from sklearn.linear_model import LinearRegression 
from sklearn import metrics 

We will load the dataset using the following command: 

# Loading the data  
car_data = pd.read_csv('/content/car_data.csv') 

Let us check the first few rows of the dataset: 

car_data.head() 

We can describe the dataset using .info() command 

# Getting some information about the dataset 
car_data.info() 

It is 301 rows and 9 columns dataset, and there are no null values in it. The output ‘Selling_Price’ is the target and there are multiple independent variables which affect this value. This is a type of supervised Machine Learning problem where the output variables are labelled and the model is first trained on split data. The model is then verified for its accuracy on validation/test data. 

Let us convert categorical variables i.e., "Fuel_Type", "Seller_Type" and "Transmission" (dtype=object), into numerical variables before applying a regression algorithm.  

# encoding Columns 
car_data.replace({'Fuel_Type':{'Petrol':0,'Diesel':1,'CNG':2}},inplace=True) 
car_data.replace({'Seller_Type':{'Dealer':0,'Individual':1}},inplace=True) 
car_data.replace({'Transmission':{'Manual':0,'Automatic':1}},inplace=True) 

To understand the relationship between different attributes in the dataset, we will plot a correlation matrix using the following code: 

corrMatrix = car_data.corr() 
sns.heatmap(corrMatrix, annot=True, cmap=”viridis”) 
plt.show() 

The above correlation matrix shows that more than one independent variable affects the output (higher negative value or positive values indicate a strong correlation), i.e., Selling_Price. For example, a correlation value of -0.8 or lower will indicate a strong negative relationship whereas a value of 0.8 or higher will indicate a strong positive relationship between the input and output variables. 

Splitting the Dataset 

We will use an 80:20 split for training and testing the model 

X = car_data.drop(['Car_Name','Selling_Price'],axis=1) 
Y = car_data['Selling_Price'] 
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size = 0.2, random_state=42) 

We can call the Linear Regression module from the sklearn library using the following commands: 

# loading the linear regression model 
lin_reg_model = LinearRegression() 
Now we can fit the model to our dataset 
lin_reg_model.fit(X_train,Y_train) 

Once the training is completed, we can make predictions and print the r-squared error for regression-  

# prediction on Training data 

training_data_prediction = lin_reg_model.predict(X_train) 

# R squared Error 

train_error_score = metrics.r2_score(Y_train, training_data_prediction) 
print("R squared Error - Training : ", train_error_score) 

# prediction on Training data 

Y_pred = lin_reg_model.predict(X_test) 

# R squared Error 

test_error_score = metrics.r2_score(Y_test, Y_pred) 
print("R squared Error - Test: ", test_error_score) 

To plot the best fit line, we will use the following code: 

# create scatterplot with regression line 
sns.regplot(Y_test, Y_pred, scatter_kws={"color": "green"}, line_kws={"color": "blue"}) 

Assumptions of Linear Regression

Normally most statistical tests and results rely upon some specific assumptions regarding the variables involved. Naturally, if these assumptions are not considered, the results will not be reliable. Linear Regression also comes under same consideration. There are some common assumptions to be considered while using Linear Regression: 

  1. Linearity: The models of Linear Regression models must be linear in the sense that the output must have a linear association with the input values, and it only suits data that has a linear relationship between the two entities. 
  2. Homoscedasticity: Homoscedasticity means the standard deviation and the variance of the residuals (difference of (y-
  3. yˆ)y^)
  4.  2 ) must be the same for any value of x. Multiple Linear Regression assumes that the amount of error in the residuals is similar at each point of the linear model. We can check the Homoscedasticity using Scatter plots.  
  5. Non-multicollinearity: The data should not have multicollinearity, which means the independent variables should not be highly correlated with each other. If this occurs, it will be difficult to identify those specific variables which actually contribute to the variance in the dependent variable. We can check the data for this using a correlation matrix. 
  6. No Autocorrelation: When data are obtained across time, we assume that successive values of the disturbance component are momentarily independent in the conventional Linear Regression model. When this assumption is not followed, the situation is referred to be autocorrelation. 
  7. Not applicable to Outliers: The value of the dependent variable cannot be estimated for a value of an independent variable which lies outside the range of values in the sample data. 

All the above assumptions are critical because if they are not followed, they can lead to drawing conclusions that may become invalid and unreliable. You can check data science bootcamp cos provided by KnowledgeHut for a better understanding of the course. 

Advantages and Disadvantages  

Advantages of Linear Regression 

  1. For linear datasets, Linear Regression performs well to find the nature of the relationship among different variables. 
  2. Linear Regression algorithms are easy to train and the Linear Regression models are easy to implement. 
  3. Although, the Linear Regression models are likely to over-fit, but can be avoided using dimensionality reduction techniques such as regularization (L1 and L2) and cross-validation. 

Disadvantages of Linear Regression

  1. An important disadvantage of Linear Regression is that it assumes linearity between the dependent and independent variables, which is rarely represented in real-world data. It assumes a straight-line relationship between the dependent and independent variables, which is unlikely many times. 
  2. It is prone to noise and overfitting. In datasets where the number of observations is lesser than the attributes, Linear Regression might not be a good choice as it can lead to overfitting. This is because the algorithm can start considering the noise while building the model. 
  3. Sensitive to outliers, it is essential to pre-process the dataset and remove the outliers before applying Linear Regression to the data. 
  4. It does not assume multicollinearity. If there is any relationship between the independent variables, i.e., multicollinearity, then it needs to be removed using dimensionality reduction techniques before applying Linear Regression as the algorithm assumes that there is no relationship among independent variables.  

Key Benefits of Linear Regression 

Linear Regression is popular in statistics. It offers several benefits in Data Science as follows: 

1. Easy to Implement 

Linear Regression machine learning model is computationally simple and does not require much engineering overhead. Hence, it is easy to implement and maintain. 

2. Scalability

Since Linear Regression is computationally inexpensive, it can be applied to cases where scaling is needed, such as applications that handle big data. 

3. Interpretability

Linear Regression is easy to interpret and very efficient to train. It is relatively simple, unlike deep learning neural networks which require more data and time to efficiently train.  

4. Applicability in real-time 

As Linear Regression models are easy to train and do not require much computational power, these can be retrained quickly with new data and hence, can be applied to scenarios where real-time predictions are important. 

Use Cases of Linear Regression 

Linear Regression finds applications in several domains such as agriculture, banking and finance, education, marketing, and many more. Linear Regression is applicable in real-world scenarios where machine learning problems can be used to predict the output as a continuous variable. 

In agriculture, Linear Regression can be used to predict the amount of rainfall and crop yield, while in banking, it is implemented to predict probability of loan defaults. For the Finance sector, Linear Regression is used to predict stock prices and assess associated risks. In the healthcare sector, Linear Regression is helpful in modelling the healthcare costs, predicting the length of stay in hospitals for patients, etc.  

In the domain of sports analytics, Linear Regression can be used to predict the performance of players in upcoming games. Similarly, it can be used in education to predict student performances in different courses. 

Businesses also use Linear Regression to forecast product demands, predict product sales, decide on marketing and advertising strategies, and so on. 

Best Practices for Linear Regression

The success of every attempt to apply any machine learning model to a specific problem depends on the best practice of implementation. Best practices mean sticking to the characteristics of the selected algorithms, the right type of data being used, and some specific causes related to the problem at hand. Some of the tips for best practices are listed below: 

1. Follow the Assumptions

The different assumptions in the application of Linear Regression in machine learning have already been discussed above. When applying a regression algorithm, the assumptions must be considered. 

2. Start with a Simple Model First

A simple model is easy to build and execute, takes less time, and can be applied to similar different datasets. 

3. Use Visualizations

Using visualizations to analyze and evaluate the performance of models frequently helps to understand the correctness of the model and can be used to improve the same by removing any shortcomings.  

4. Start with Sample Dataset

When applied to a large dataset, the model takes more time to reach its accuracy and high computational power. If the performance on a large dataset is not satisfactory, the time and power spent are of no use. Hence, starting with a small sample dataset initially is better to try out the model.  

The estimate obtained on this dataset gives a good indication of correct progress or otherwise. The findings of this approach help to take corrective action in the new model if required. 

5. Shifting to Multi-Linear Regression

If the market conditions change in due course of time, the algorithm's controlling parameters also need to be changed. In such situations, it is better to go for a multi-linear model so that new affecting parameters can be included to build up the model. 

6. Applying Linear Regression Model to Real-life Problems

It is always a good practice to apply Linear Regression to real-life problems like stock prediction, probability of promotion chances, growth percentage of crop yield, and so on. The results obtained can be matched to some previous example outcomes for gaining confidence. 

7. Choosing Appropriate Data

The success of any algorithm is as good as the data used. One can choose appropriate data to be given to the model based on the outcome of the project in mind. A lot of open-source websites provide a variety of datasets suitable for applying regression algorithms in machine learning. 

Beginner Projects to Try Out Linear Regression 

While regression analysis is utilized in practically every area, from finance to education and from banking to advertising, there are some beginner machine learning projects for Linear Regression as mentioned below: 

Project 1: Loan Default Prediction

Banks employ Machine learning to predict loan defaults and decide upon loan applications. Lending in the form of loans is a major source of revenue for banks, credit unions, and other financial organizations, which accounts for a sizable portion of the bank's assets. However, when these loans default, the financial institutions suffer severe consequences. You can get the dataset here. 

Project 2: House Price Prediction

Predicting house prices may assist in determining the selling price of a property in a certain location and in determining the best time to buy a home. The dataset is available here. 

Project 3: Stock Market Prediction

Stock market analysis and forecasting are extremely challenging tasks. Stock prices are dynamic and affected by a variety of factors. When forecasting stocks, most stockbrokers use methodological and fundamental analysis and time series analysis. The dataset is available here. 

Project 4: Market Sales Forecasting

Sales forecasting is crucial because it assists companies in discovering which strategies work effectively and where a particular strategy needs to be modified to ensure future success. The dataset is available here. 

Project 5: Advertising

Businesses usually promote their products through websites and social media channels. However, their main challenge lies in finding the correct demographic to target for internet marketing. Since advertising is expensive, targeting advertisements to an audience that is unlikely to purchase the products can be a loss for the company. The dataset is available here.  

Conclusion

In this article, we discussed Linear Regression for Machine Learning, its concepts, terminologies, and the types of Linear Regression. We understood the assumptions as well as the advantages and disadvantages of Linear Regression. Further, we learned how Linear Regression works with an example of Multiple Linear Regression on the car prices dataset. And finally, explored the best practices along with some beginner-level projects to try and gain confidence in Linear Regression problems. 

Check out KnowledgeHut’s data science bootcamp cost, where you are offered a variety of Data Science training that will give you the experience you need to land a top Data scientist role. 

Frequently Asked Questions (FAQs) 

1. What is the output of Linear Regression in machine learning?

The output is a continuous value, integer, or probability percentage based on the selected problems. Thus, it can be sales amount, profit percentage, probability of success or failure in some activities like admission possibility, winning an election, etc. With the best fit line of regression, the output value for any new value of input variable can be easily calculated. 

2. What are the benefits of using Linear Regression?

There are many benefits of Linear Regression, including simplicity of understanding and implementation. It can be applied to obtain relations in linear or multi-linear parameters and thus can be applied to various business problems. 

3. How do you explain a Linear Regression model?

A Linear Regression model will use a mathematical equation to derive a relation between a predicted variable which varies with independent variables. The best fit line will be obtained based on the given data after applying the algorithm, and this line can then be used to give expected predictions. 

4. Which type of dataset is used for Linear Regression?

Many datasets can be used for Linear Regression like stock price prediction, house price prediction, disease prediction probability, medical insurance costs, etc. 

5. Which ML model is best for regression?

Although it is not easy to specify a particular best ML model for regression yet, one can select a regression model that best fits to predict outcomes of numerical nature. A multi-Linear Regression model would probably be a good choice in most cases. 

Profile

Devashree Madhugiri

Author

Devashree holds an M.Eng degree in Information Technology from Germany and a background in Data Science. She likes working with statistics and discovering hidden insights in varied datasets to create stunning dashboards. She enjoys sharing her knowledge in AI by writing technical articles on various technological platforms.
She loves traveling, reading fiction, solving Sudoku puzzles, and participating in coding competitions in her leisure time.