Ashish is a techology consultant with 13+ years of experience and specializes in Data Science, the Python ecosystem and Django, DevOps and automation. He specializes in the design and delivery of key, impactful programs.
HomeBlogData ScienceWhat is Linear Discriminant Analysis for Machine Learning?
Linear Discriminant Analysis or LDA in machine learning is a dimensionality reduction technique. It is used as a pre-processing step in Machine Learning and applications of pattern classification. The goal of LDA is to project the features in higher dimensional space onto a lower-dimensional space in order to avoid the curse of dimensionality and also reduce resources and dimensional costs.
LDA is a powerful supervised classification technique, playing a very integral role in crafting competitive machine learning models. Its versatility spans across various domains, from image recognition to predictive analysis in marketing.
Learn more about machine learning and data science through our detailed Data Science course. Now, let's dive in to understand LDA in machine learning, how it works, its limitations, and more through our article.
Linear Discriminant Analysis for Machine Learning (LDA) is a common statistical method for machine learning and pattern identification. It is used mostly for dimensionality reduction and classification tasks. The primary principle behind LDA is to identify a linear combination of attributes that best distinguishes two or more classes of objects or events.
LDA assumes that separate classes create data with Gaussian distributions and seeks to maximize the ratio of between-class variance to within-class variance in any given dataset, resulting in maximum separability. The combination of attributes produces the greatest "discriminant" or distinguishing information about the classes.
Let's consider a situation where you have plotted the relationship between two variables where each color represents a different class. One is shown with a red color and the other with blue.
If you are willing to reduce the number of dimensions to 1, you can just project everything to the x-axis as shown below:
This approach neglects any helpful information provided by the second feature. However, you can use LDA to plot it. The advantage of LDA is that it uses information from both the features to create a new axis which in turn minimizes the variance and maximizes the class distance of the two variables.
LDA focuses primarily on projecting the features in higher dimension space to lower dimensions. You can achieve this in three steps:
The representation of LDA is pretty straight-forward. The model consists of the statistical properties of your data that has been calculated for each class. The same properties are calculated over the multivariate Gaussian in the case of multiple variables. The multivariates are means and covariate matrix.
Predictions are made by providing the statistical properties into the Linear Discriminant Analysis for Machine Learning equation. The properties are estimated from your data. Finally, the model values are saved to file to create the LDA model.
The assumptions made by an LDA model about your data:
The LDA model is able to estimate the mean and variance from your data for each class with the help of these assumptions.
The mean value of each input for each of the classes can be calculated by dividing the sum of values by the total number of values:
Mean =Sum(x)/Nk
where Mean = mean value of x for class
N = number of
k = number of
Sum(x) = sum of values of each input x.
The variance is computed across all the classes as the average of the square of the difference of each value from the mean:
Σ²=Sum((x - M)²)/(N - k)
where Σ² = Variance across all inputs x.
N = number of instances.
k = number of classes.
Sum((x - M)²) = Sum of values of all (x - M)².
M = mean for input x.
LDA, i.e. linear discriminant analysis in machine learning, models uses Bayes’ Theorem to estimate probabilities. They make predictions based upon the probability that a new input dataset belongs to each class. The class which has the highest probability is considered the output class and then the LDA makes a prediction.
The prediction is made simply by the use of Bayes’ Theorem which estimates the probability of the output class given the input. They also make use of the probability of each class and the probability of the data belonging to each class:
P(Y=x|X=x) = [(Plk * fk(x))] / [(sum(PlI * fl(x))]
Where x = input.
k = output class.
Plk = Nk/n or base probability of each class observed in the training data. It is also called prior probability in Bayes’ Theorem.
fk(x) = estimated probability of x belonging to class k.
The f(x) is plotted using a Gaussian Distribution function and then it is plugged into the equation above and the result we get is the equation as follows:
Dk(x) = x∗(mean/Σ²) – (mean²/(2*Σ²)) + ln(PIk)
The Dk(x) is called the discriminant function for class k given input x, mean, Σ² and Plk are all estimated from the data and the class is calculated as having the largest value, will be considered in the output classification.
Check out KnowledgeHut’s Data Science Courses for more lucrative career options in this landscape and become a certified Data Scientist.
Some suggestions you should keep in mind while preparing your data to build your LDA model:
You can implement a Linear Discriminant Analysis model from scratch using Python. Let’s start by importing the libraries that are required for the model:
from sklearn.datasets import load_wine import pandas as pd import numpy as np np.set_printoptions(precision=4) from matplotlib import pyplot as plt import seaborn as sns sns.set() from sklearn.preprocessing import LabelEncoder from sklearn.tree import DecisionTreeClassifier from sklearn.model_selection import train_test_split from sklearn.metrics import confusion_matrix
Since we will work with the wine dataset, you can obtain it from the UCI machine learning repository. The scikit-learn library in Python provides a wrapper function for downloading it:
wine_info = load_wine() X = pd.DataFrame(wine_info.data, columns=wine_info.feature_names) y = pd.Categorical.from_codes(wine_info.target, wine_info.target_names)
The wine dataset comprises of 178 rows of 13 columns each:
X.shape
(178, 13)
The attributes of the wine dataset comprise of various characteristics such as alcohol content of the wine, magnesium content, color intensity, hue and many more:
X.head()
The wine dataset contains three different kinds of wine:
wine_info.target_names array(['class_0', 'class_1', 'class_2'], dtype='<U7')
Now we create a DataFrame which will contain both the features and the content of the dataset:
df = X.join(pd.Series(y, name='class'))
We can divide the process of Linear Discriminant Analysis into 5 steps as follows:
Step 1 - Computing the within-class and between-class scatter matrices.
Step 2 - Computing the eigenvectors and their corresponding eigenvalues for the scatter matrices.
Step 3 - Sorting the eigenvalues and selecting the top k.
Step 4 - Creating a new matrix that will contain the eigenvectors mapped to the k eigenvalues.
Step 5 - Obtaining new features by taking the dot product of the data and the matrix from Step 4.
To calculate the within-class scatter matrix, you can use the following mathematical expression:
where, c = total number of distinct classes and
where, x = a sample (i.e. a row).
n = total number of samples within a given class.
Now we create a vector with the mean values of each feature:
feature_means1 = pd.DataFrame(columns=wine_info.target_names) for c, rows in df.groupby('class'): feature_means1[c] = rows.mean() feature_means1
The mean vectors (mi ) are now plugged into the above equations to obtain the within-class scatter matrix:
withinclass_scatter_matrix = np.zeros((13,13)) for c, rows in df.groupby('class'): rows = rows.drop(['class'], axis=1) s = np.zeros((13,13)) for index, row in rows.iterrows(): x, mc = row.values.reshape(13,1), feature_means1[c].values.reshape(13,1) s += (x - mc).dot((x - mc).T) withinclass_scatter_matrix += s
We can calculate the between-class scatter matrix using the following mathematical expression:
where,
and
feature_means2 = df.mean() betweenclass_scatter_matrix = np.zeros((13,13)) for c in feature_means1: n = len(df.loc[df['class'] == c].index) mc, m = feature_means1[c].values.reshape(13,1), feature_means2.values.reshape(13,1) betweenclass_scatter_matrix += n * (mc - m).dot((mc - m).T)
Now we will solve the generalized eigenvalue problem to obtain the linear discriminants for:
eigen_values, eigen_vectors = np.linalg.eig(np.linalg.inv(withinclass_scatter_matrix).dot(betweenclass_scatter_matrix))
We will sort the eigenvalues from the highest to the lowest since the eigenvalues with the highest values carry the most information about the distribution of data is done. Next, we will first k eigenvectors. Finally, we will place the eigenvalues in a temporary array to make sure the eigenvalues map to the same eigenvectors after the sorting is done:
eigen_pairs = [(np.abs(eigen_values[i]), eigen_vectors[:,i]) for i in range(len(eigen_values))] eigen_pairs = sorted(eigen_pairs, key=lambda x: x[0], reverse=True) for pair in eigen_pairs: print(pair[0])
237.46123198302251
46.98285938758684
1.4317197551638386e-14
1.2141209883217706e-14
1.2141209883217706e-14
8.279823065850476e-15
7.105427357601002e-15
6.0293733655173466e-15
6.0293733655173466e-15
4.737608877108813e-15
4.737608877108813e-15
2.4737196789039026e-15
9.84629525010022e-16
Now we will transform the values into percentage since it is difficult to understand how much of the variance is explained by each component.
sum_of_eigen_values = sum(eigen_values) print('Explained Variance') for i, pair in enumerate(eigen_pairs): print('Eigenvector {}: {}'.format(i, (pair[0]/sum_of_eigen_values).real))
Explained Variance
Eigenvector 0: 0.8348256799387275
Eigenvector 1: 0.1651743200612724
Eigenvector 2: 5.033396012077518e-17
Eigenvector 3: 4.268399397827047e-17
Eigenvector 4: 4.268399397827047e-17
Eigenvector 5: 2.9108789097898625e-17
Eigenvector 6: 2.498004906118145e-17
Eigenvector 7: 2.119704204950956e-17
Eigenvector 8: 2.119704204950956e-17
Eigenvector 9: 1.665567688286435e-17
Eigenvector 10: 1.665567688286435e-17
Eigenvector 11: 8.696681541121664e-18
Eigenvector 12: 3.4615924706522496e-18
First, we will create a new matrix W using the first two eigenvectors:
W_matrix = np.hstack((eigen_pairs[0][1].reshape(13,1), eigen_pairs[1][1].reshape(13,1))).real
Next, we will save the dot product of X and W into a new matrix Y:
Y = X∗W
where, X = n x d matrix with n sample and d dimensions.
Y = n x k matrix with n sample and k dimensions.
In simple terms, Y is the new matrix or the new feature space.
X_lda = np.array(X.dot(W_matrix))
Our next work is to encode every class a member in order to incorporate the class labels into our plot. This is done because matplotlib cannot handle categorical variables directly.
Finally, we plot the data as a function of the 2 LDA components using different color for each class:
plt.xlabel('LDA1') plt.ylabel('LDA2') plt.scatter( X_lda[:,0], X_lda[:,1], c=y, cmap='rainbow', alpha=0.7, edgecolors='b' )
<matplotlib.collections.PathCollection at 0x7fd08a20e908>
LDA is considered to be a very simple and effective method, especially for classification techniques. Since it is simple and well understood, so it has a lot of extensions and variations:
Below are the differences between LDA and PCA in machine learning:
Let us create and fit an instance of the PCA class:
from sklearn.decomposition import PCA pca_class = PCA(n_components=2) X_pca = pca.fit_transform(X, y)
Again, to view the values in percentage for a better understanding, we will access the explained_variance_ratio_ property:
pca.explained_variance_ratio_
array([0.9981, 0.0017])
Clearly, PCA selected the components which will be able to retain the most information and ignores the ones which maximize the separation between classes.
plt.xlabel('PCA1') plt.ylabel('PCA2') plt.scatter( X_pca[:,0], X_pca[:,1], c=y, cmap='rainbow', alpha=0.7, edgecolors='b
Now to create a classification model using the LDA components as features, we will divide the data into training datasets and testing datasets:
X_train, X_test, y_train, y_test = train_test_split(X_lda, y, random_state=1)
The next thing we will do is create a Decision Tree. Then, we will predict the category of each sample test and create a confusion matrix to evaluate the LDA model’s performance:
data = DecisionTreeClassifier() data.fit(X_train, y_train) y_pred = data.predict(X_test) confusion_matrix(y_test, y_pred)
array([[18, 0, 0], [ 0, 17, 0], [ 0, 0, 10]])
So it is clear that the Decision Tree Classifier has correctly classified everything in the test dataset.
Advantages | Disadvantages |
Dimensionality Reduction: LDA reduces data dimensionality while retaining class-discriminatory information, potentially improving classification algorithm performance. | Sensitivity to outliers: LDA is sensitive to outliers, which can have a major impact on model outputs and performance. |
Performance on Small Datasets: LDA performs effectively on tiny datasets that meet the requirements of normality and equal class covariance. | Poor Performance with Non-Gaussian Data: If the data distribution is non-Gaussian, LDA may perform poorly. |
Adaptability: LDA can be used for both binary and multiclass classification tasks, increasing its adaptability. | Requirement for large sample size: Large sample sizes are required for reliable mean and covariance estimation. |
Simplicity and Computational Efficiency: LDA is easy to implement and computationally efficient, making it appropriate for huge datasets. | Linearity Assumption: LDA assumes a linear relationship between the features, which may not be true for complicated datasets. |
Some of the practical applications of LDA are listed below:
The Linear Discriminant Analysis in Python or LDA in machine learning to be more precise is a very simple and well-understood approach of classification in machine learning. Though there are other dimensionality reduction techniques like Logistic Regression or PCA, but LDA is preferred in many special classification cases.
If you want to be an expert in machine learning, knowledge of Linear Discriminant Analysis would lead you to that position effortlessly and Data Science and Machine Learning Courses are the best means to do so.
Gain the skills to excel in business analysis and drive success. Enroll now with CCBA training and seize new opportunities today.
LDA is most effective when you have a dataset with clearly defined classes and want to reduce dimensionality while keeping as much class discriminatory information as possible.
The basic purpose of LDA is to minimize the dimensionality of a dataset while increasing class separability. It seeks to identify a linear combination of features that best distinguishes between classes, making it easier to categorize fresh data points.
LDA is a supervised learning technique that employs labeled data to identify linear discriminants that maximize class separability. Class labels are required for computing the between-class and within-class variances utilized during analysis.
Name | Date | Fee | Know more |
---|