Search

Machine learning Filter

What is K-Nearest Neighbor in Machine Learning: K-NN Algorithm

If you are thinking of a simple, easy-to-implement supervised machine learning algorithm which can be used to solve both classification as well as regression problems, K-Nearest Neighbor (K-NN) is the perfect choice. Learning K-NN is a great way to introduce yourself to machine learning and classification in general. Also, you will find a lot of intense application of K-NN in data mining, pattern recognition, semantic searching, intrusion detection and anomaly detection.K-Nearest Neighbors is one of the most basic supervised machine learning algorithms, yet very essential. A supervised machine learning algorithm is one of the types of machine learning algorithm which is dependent on labelled input data in order to learn a function which is capable of producing an output whenever a new unlabeled data is given as input.In real life scenarios, K-NN is widely used as it is non-parametric which means it does not make any underlying assumptions about the distributions of data. With the business world entirely revolving around Data Science, it has become one of the most lucrative fields. Hence, the heavy demand for a Data Science Certification.Parametric vs Non-parametric MethodsLet us look into how different is a parametric machine learning algorithm from a nonparametric machine learning algorithm.Machine learning, in other words can be called as learning a function (f) which maps input variables (X) to the output variables (Y).Y=f(X)An algorithm learns about the target mapping function from the training data. As we are unaware of the form of the function, we have to evaluate various machine learning algorithms and figure out which algorithms perform better at providing an approximation of the underlying function.Statistical Methods are classified on the basis of what we know about the population we are studying.Parametric statistics is a branch of statistics which assumes that sample data comes from a population that follows a probability distribution based on a fixed set of parameters.Nonparametric statistics is the branch of statistics that is not based solely on population parameters.Parametric Machine Learning AlgorithmsThis particular algorithm involves two steps:Selecting a form for the functionLearning the coefficients for the function from the training dataLet us consider a line to understand functional form for the mapping function as it is used in linear regression and simplify the learning process.b0 + b1*x1 + b2*x2 = 0Where b0, b1 and b2 are the coefficients of the line which control the intercept and slope, and x1 and x2 are two input variables.All we have to do now is to estimate the coefficients of the line equation to get a predictive model for the problem. Now, the problem is that the actual unknown underlying function may not be a linear function like a line. In that case, the approach will give poor results. Some of the examples of parametric machine learning algorithms are mentioned below:Logistic RegressionLinear Discriminant AnalysisPerceptronNaive BayesSimple Neural NetworksNonparametric Machine Learning AlgorithmsNonparametric methods always try to find the best fit training data while constructing the mapping function which also allows it to fit a large number of functional forms. Some of the examples of nonparametric machine learning algorithms are mentioned below:k-Nearest NeighborsDecision Trees like CART and C4.5Support Vector MachinesThe best example of nonparametric machine learning algorithms would be k-nearest neighbors algorithm which makes predictions based on the k most similar training patterns for a given set of new data instance. This method simply assumes that the patterns which are close are likely to be of similar type.Parametric Machine Learning AlgorithmsNonparametric Machine Learning AlgorithmsBenefitsSimple to understand and interpret resultsSpeed of learning from data in fastLess training data is requiredFlexible enough to fit a large number of functional formsNo assumptions about the underlying functionsProvides high performance for predictionLimitationsChoosing a functional form constrains the method to the specified formIt has limited complexity and more suited to simpler problemsIt is unlikely to match the underlying mapping function and results in poor fitRequires more training data in order to estimate the mapping functionDue to more parameters to train, it is slower comparativelyThere is a risk to overfit the training dataMethod Based LearningThere are several learning models namely:Association rules basedEnsemble method basedDeep Learning basedClustering method basedRegression Analysis basedBayesian method basedDimensionality reduction basedKernel method basedInstance basedLet us understand what Instance Based Learning is all about.Instance Based Learning (IBL)Instance-Based methods are the simplest form of learningInstance -Based learning is lazy learningK-NN model works on identified instanceInstances are retrieved from memory and then this data is used to classify the new query instanceInstance based learning is also called memory-based or case-basedUnder Instance-based Learning we have,Nearest-neighbor classifierUses k “closest” points (nearest neighbors) for performing classification. For example: It’s how people judge by observing our peers. We tend to move with people of similar attributes.Lazy Learning vs Eager LearningLazy LearningEager LearningSimply stores the training data and waits until it is given a test tuple.Munges the training data as soon as it receives it.It's slow as it calculates based on the current data set instead of coming up with an algorithm based on historical data.It's fast as it has pre-calculated algorithm.Localized data so generalization takes time at every iteration.On the basis of training set ,it constructs a classification model before receiving new data to classify.What is K-NN?One of the biggest applications of K-Nearest Neighbor search is Recommender Systems. If you have noticed while you are shopping as a user on Amazon and you like a particular item, you are recommended with similar items.It also recommends similar items bought by other users and other set of items which are often bought together. Basically, the algorithm compares the set of users who like each item and looks for similarity. This not only applies to recommending items or products but also recommending media and even advertisements to display to a user.K nearest neighbors or K-NN Algorithm is a simple algorithm which uses the entire dataset in its training phase. Whenever a prediction is required for an unseen data instance, it searches through the entire training dataset for k-most similar instances and the data with the most similar instance is finally returned as the prediction.This algorithm suggests that if you’re similar to your neighbours, then you are one of them. Let us consider a simple example, if apple looks more similar to peach, pear, and cherry (fruits) than monkey, cat or a rat (animals), then most likely apple is a fruit.Nearest Neighbours algorithm has been in action for the last sixty years. It is mainly used in statistical estimation and pattern recognition, as a non-parametric method, for regression and classification. The main aim of the K-Nearest Neighbor algorithm is to classify a new data point by comparing it to all previously seen data points. The classification of the k most similar previous cases are used for predicting the classification of the current data point. It is a simple algorithm which stores all available cases and classifies new cases based on a similarity measure (e.g., distance functions).When do we use K-NN algorithm?K-NN algorithm can be used for applications which require high accuracy as it makes highly accurate predictions. The quality of predictions is completely dependent on the distance measure. Thus, this algorithm is suitable for applications for which you have sufficient domain knowledge so that it can help you select an appropriate measure.As we have already seen K-NN algorithm is a type of lazy learning, the computation for the generation is postponed until classification which indeed increases the costs of computation compared to other machine learning algorithms. But still K-NN is considered to be the better choice for applications where accuracy is more important and predictions are not requested frequently.K-NN can be used for both regression and classification predictive problems. However, in the industry it is mostly used in classification problems.Generally we mainly look at 3 important aspects in order to evaluate any technique:Ease to interpret outputCalculation timePredictive PowerLet us consider a few examples to place K-NN in the scale :If you notice the chart mentioned above, K-NN algorithm exceeds in most of the parameters. It is most commonly used for ease of interpretation and low calculation time.How does the K-NN algorithm work?K-NN algorithm works on the basis of feature similarity. The classification of a given data point is determined by how closely out-of-sample features resemble our training set.The above figure shows an example of k-NN classification. If you consider the nearest neighbor to the test sample, it is a blue square (Class 1) and k=1. This falls inside the inner circle.Now, if you consider k=3, then you will see 2 red triangles and only 1 blue square falls under the outer circle. Thus, the test sample is classified as a red triangle now (Class 2).Similarly, if you consider k=5, it is assigned to the first class (3 squares vs. 2 triangles outside the outer circle).K-NN in RegressionIn regression problems, K-NN is used for prediction based on the mean or the median of the K-most similar instances.K-NN in ClassificationK-nearest-neighbor classification was actually developed from the need to perform discriminant analysis when reliable parametric estimates of probability densities are unknown or are difficult to determine. When K-NN is used for classification, the output is easily calculated by the class having the highest frequency from the K-most similar instances. The class with maximum vote is taken into consideration for prediction.The probabilities of Classes can be calculated as the normalized frequency of samples that belong to each class in the set of K most similar instances for a new data instance.For example, in a binary classification problem (class is 0 or 1):p(class=0) = count(class=0) / (count(class=0)+count(class=1))If you are using K and you have an even number of classes (e.g. 2) it is a good idea to choose a K value with an odd number to avoid a tie. And the inverse, use an even number for K when you have an odd number of classes.Ties can be broken consistently by expanding K by 1 and looking at the class of the next most similar instance in the training dataset.Making Predictions with K-NNA case can be classified by a majority vote of its neighbors. The case is then assigned to the most common class amongst its K nearest neighbors measured by a distance function. Suppose the value of K is 1, then the case is simply assigned to the class of its nearest neighbor.The three distance measures mentioned above are valid only for continuous variables. For categorical variables, the Hamming distance is used. It also brings up the issue of standardization of the numerical variables between 0 and 1 when there is a mixture of numerical and categorical variables in the dataset.By inspecting the data, you can choose the best optimal value for K. Generally, a large value of K is more accurate as it tends to reduce the overall noise but is not always true. Another way to retrospectively determine a good K value by using an independent dataset to validate the K value is Cross-validation. According to observation, the optimal K for most datasets has been between 3-10 which provides better results than 1NN.For example, let us consider an example where the data mentioned below us concerned with credit default. Age and Loan are two numerical variables (predictors) and Default is the target.By observing the data mentioned above, we can use the training set in order to classify an unknown case (Age=48 and Loan=$142,000) using Euclidean distance. If K=1 then the nearest neighbor is the last case in the training set with Default=Y.AgeLoanDefaultDistance25$40,000N10200035$60,000N8200045$80,000N6200020$20,000N12200035$120,000N22000252$18,000N12400023$95,000Y4700040$62,000Y8000060$100,000Y42000348$220,000Y7800033$150,000Y8000148$142,000?Euclidean DistanceWith K=3, there are two Default=Y and one Default=N out of three closest neighbors. The prediction for the unknown case is again Default=Y.Standardized DistanceOne major drawback in calculating distance measures directly from the training set is in the case where variables have different measurement scales or there is a mixture of numerical and categorical variables. For example, if one variable is based on annual income in dollars, and the other is based on age in years then income will have a much higher influence on the distance calculated. One solution is to standardize the training set as shown below.AgeLoanDefaultDistance0.1250.11N0.76520.3750.21N0.52000.6250.31N0.316000.01N0.92450.3750.50N0.34280.80.00N0.62200.0750.38Y0.66690.50.22Y0.443710.41Y0.36500.71.00Y0.38610.3250.65Y0.37710.70.61?Standardized VariableUsing the standardized distance on the same training set, the unknown case returned a different neighbor which is not a good sign of robustness.Between-sample geometric distanceThe k-nearest-neighbor classifier is commonly based on the Euclidean distance between a test sample and the specified training samples. Let  xi be an input sample with p features, (xi1, xi2, …, xip), n be the total number of input samples (i=1,2,…,n) and p the total number of features (j=1,2,…,p) . The Euclidean distance between sample xi and xl (l=1,2,…,n) is defined as:A graphical representation of the nearest neighbor concept is illustrated in the Voronoi tessellation. The tessellation shows 19 samples marked with a "+", and the Voronoi cell, R, surrounding each sample. A Voronoi cell encapsulates all neighboring points that are nearest to each sample and is defined as:Where Ri is the Voronoi cell for sample xi, and x represents all possible points within Voronoi cell Ri.Voronoi tessellation showing Voronoi cells of 19 samples marked with a "+"The Voronoi tessellation reflects two characteristics of the example 2-dimensional coordinate system: i) all possible points within a sample's Voronoi cell are the nearest neighboring points for that sample, and ii) for any sample, the nearest sample is determined by the closest Voronoi cell edge.According to the latter characteristic, the k-nearest-neighbor classification rule is to assign to a test sample the majority category label of its k nearest training samples. In practice, k is usually chosen to be odd, so as to avoid ties. The k = 1 rule is generally called the nearest-neighbor classification rule.Curse of DimensionalityThe curse of dimensionality refers to various phenomena that are witnessed while analyzing and organizing data in high-dimensional spaces (often with hundreds or thousands of dimensions). Such phenomenon do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience.K-NN algorithm will work absolutely fine when you are dealing with a small number of input variables (p)  but will struggle when there are a large number of inputs.K-NN works well with a small number of input variables (p), but struggles when the number of inputs is very large. Each input variable can be considered a dimension of a p-dimensional input space. For example, suppose you have two input variables x1 and x2, the input space would be 2-dimensional. With the increase in the number of dimensions, the volume of the input space increases at an exponential rate.In case of higher dimensions, the points which are similar may have large distances. All these points will be then away from each other and our intuition about 2 to 3 dimensional spaces will not be applicable. This kind of problem is called the “Curse of Dimensionality“. How is K in K-means different from K in K-NN?K-Means Clustering and k-Nearest Neighbors algorithm, both are commonly used algorithms in Machine Learning. They are often confused with each other, especially when we are talking about the k-factor. The ‘K’ in K-Means Clustering has nothing to do with the ‘K’ in K-NN algorithm. k-Means Clustering is an unsupervised learning algorithm that is used for clustering whereas K-NN is a supervised learning algorithm used for classification.K-Means AlgorithmThe k-means algorithm is an unsupervised clustering algorithm which takes a couple of unlabeled points and then groups them into “k” number of clusters.The “k” in k-means denotes the number of clusters you would like to have in the end. Suppose the value of k is 5, it means you will have 5 clusters on the data set.Let us see how it works.Step 1: First you determine the value of K by Elbow method and then specify the number of clusters KStep 2: Next you have to randomly assign each data point to a clusterStep 3: Determine the cluster centroid coordinatesStep 4: Determine the distances of each data point to the centroids and re-assign each point to the closest cluster centroid based upon minimum distanceStep 5: Calculate cluster centroids againStep 6: Repeat steps 4 and 5 until we reach global optima where further improvements are not possible and there is no provision to switch data points from one cluster to another.Implementation in Python#Finding the optimum number of clusters for k-means clustering Nc = range(1, 10) kmeans = [KMeans(n_clusters=i) for i in Nc] kmeans score = [kmeans[i].fit(x).score(x) for i in range(len(kmeans))] score pl.plot(Nc,score) pl.xlabel('Number of Clusters') pl.ylabel('Score') pl.title('Elbow Curve') pl.show()You can clearly see why it is called 'The elbow method' from the above graph, the optimum clusters is where the elbow occurs.Now that we have the optimum amount of clusters (k=3), we can move on to applying K-means clustering to the Iris dataset.#Implementation of K-Means Clustering model = KMeans(n_clusters = 3) model.fit(x) model.labels_ colormap = np.array(['Red', 'Blue', 'Green']) z = plt.scatter(x.sepal_length, x.sepal_width, x.petal_length, c = colormap[model.labels_])#Accuracy of K-Means Clustering accuracy_score(iris.target,model.labels_) 0.8933333333333333K-NN AlgorithmBy now, we already know that K-NN algorithm is a supervised classification algorithm. It takes into consideration a couple of labelled points and then uses those points to learn how to label other points. To be able to assign label to other points, K-NN algorithm looks for the closest neighbor of the new point and checks for voting. The most number of neighbors around the new point decide the label of the new point.The “k” in K-Nearest Neighbors is the number of neighbors it checks. It is supervised because it is trying to classify a point on the basis of the known classification of other points.Let us see how it works.Step 1: Firstly, you determine the value for K.Step 2: Then you calculate the distances between the new input (test data) and all the training data. The most commonly used metrics for calculating distance are Euclidean, Manhattan and MinkowskiStep 3: Sort the distance and determine k nearest neighbors based on minimum distance valuesStep 4: Analyze the category of those neighbors and assign the category for the test data based on majority voteStep 5: Return the predicted classImplementation using Pythonerror = [] # Calculating error for K values between 1 and 40 for i in range(1, 40): K-NN = KNeighborsClassifier(n_neighbors=i) K-NN.fit(X_train, y_train) pred_i = K-NN.predict(X_test) error.append(np.mean(pred_i != y_test)) plt.figure(figsize=(12, 6)) plt.plot(range(1, 40), error, color='black', linestyle='dashed', marker='o',     markerfacecolor='grey', markersize=10) plt.title('Error Rate K Value') plt.xlabel('K Value') plt.ylabel('Mean Error') Text(0, 0.5, 'Mean Error')Now we know for what values of ‘K’, the error rate will be less. Let’s fix k=5 and implement K-NN algorithm.#Creating training and test splits from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20) #Performing Feature Scaling from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(X_train) X_train = scaler.transform(X_train) X_test = scaler.transform(X_test) #Training K-NN with k=5 from sklearn.neighbors import KNeighborsClassifier classifier = KNeighborsClassifier(n_neighbors=5) classifier.fit(X_train, y_train) KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',                     metric_params=None, n_jobs=None, n_neighbors=5, p=2,                     weights='uniform') y_pred = classifier.predict(X_test) from sklearn.metrics import classification_report, confusion_matrix print(confusion_matrix(y_test, y_pred)) print(classification_report(y_test, y_pred)) [[10  0  0] [ 0  9  2] [ 0  1  8]]                 precision recall   f1-score   support     Iris-setosa        1.00         1.00       1.00       10 Iris-versicolor       0.90       0.82     0.86       11 Iris-virginica     0.80         0.89       0.84       9       accuracy                   0.90         30       macro avg     0.90       0.90     0.90     30   weighted avg    0.90       0.90     0.90       30Practical Applications of K-NNNow that we have we have seen how K-NN works, let us look into some of the practical applications of K-NN.Recommending products to people with similar interests, recommending movies and TV shows as per viewer’s choice and interest, recommending hotels and other accommodation facilities while you are travelling based on your previous bookings.Assigning credit ratings based on financial characteristics, comparing people with similar financial features in a database. By analyzing the nature of a credit rating, people with similar financial details, they would be assigned similar credit ratings.Should the bank give a loan to an individual? Would an individual default on his or her loan? Is that person closer in characteristics to people who defaulted or did not default on their loans?Some advanced examples could include handwriting detection (like OCR), image recognition and even video recognition.Some pros and cons of K-NNProsTraining phase of K-nearest neighbor classification is faster in comparison with other classification algorithms.Training of a model is not required for generalization.Simple algorithm — to explain and understand/interpret.High accuracy (relatively) — it is pretty high but not competitive in comparison to better supervised learning models.K-NN can be useful in case of nonlinear data.Versatile — useful for classification or regression.ConsTesting phase of K-nearest neighbor classification is slower and costlier with respect to time and memory. High memory requirement - Requires large memory for storing the entire training dataset.K-NN requires scaling of data because K-NN uses the Euclidean distance between two data points to find nearest neighbors.Euclidean distance is sensitive to magnitudes. The features with high magnitudes will weigh more than features with low magnitudes.Not suitable for large dimensional data.How to improve the performance of K-NN?Rescaling Data: K-NN performs much better if all of the data has the same scale. Normalizing your data to the range [0, 1] is a good idea. It may also be a good idea to standardize your data if it has a Gaussian distribution.Addressing Missing Data: Missing data will mean that the distance between samples can not be calculated. These samples could be excluded or the missing values could be imputed.Reducing Dimensionality: K-NN is suited for lower dimensional data. You can try it on high dimensional data (hundreds or thousands of input variables) but be aware that it may not perform as good as other techniques. K-NN can benefit from feature selection that reduces the dimensionality of the input feature space.In this article we have learned about the K-Nearest Neighbor algorithm, where we should use it, how it works and so on. Also, we have discussed about parametric and nonparametric machine learning algorithms, instance based learning, eager and lazy learning, advantages and disadvantages of using K-NN, performance improvement suggestions and have implemented K-NN in Python. To learn more about other machine learning algorithms, join our Data Science Certification course and expand your learning skill set and career opportunities.

What is K-Nearest Neighbor in Machine Learning: K-NN Algorithm

9767
What is K-Nearest Neighbor in Machine Learning: K-NN Algorithm

If you are thinking of a simple, easy-to-implement supervised machine learning algorithm which can be used to solve both classification as well as regression problems, K-Nearest Neighbor (K-NN) is the perfect choice. Learning K-NN is a great way to introduce yourself to machine learning and classification in general. Also, you will find a lot of intense application of K-NN in data mining, pattern recognition, semantic searching, intrusion detection and anomaly detection.

K-Nearest Neighbors is one of the most basic supervised machine learning algorithms, yet very essential. A supervised machine learning algorithm is one of the types of machine learning algorithm which is dependent on labelled input data in order to learn a function which is capable of producing an output whenever a new unlabeled data is given as input.

In real life scenarios, K-NN is widely used as it is non-parametric which means it does not make any underlying assumptions about the distributions of data. With the business world entirely revolving around Data Science, it has become one of the most lucrative fields. Hence, the heavy demand for a Data Science Certification.

Parametric vs Non-parametric Methods

Let us look into how different is a parametric machine learning algorithm from a nonparametric machine learning algorithm.

Machine learning, in other words can be called as learning a function (f) which maps input variables (X) to the output variables (Y).

Y=f(X)

An algorithm learns about the target mapping function from the training data. As we are unaware of the form of the function, we have to evaluate various machine learning algorithms and figure out which algorithms perform better at providing an approximation of the underlying function.

Statistical Methods are classified on the basis of what we know about the population we are studying.

  • Parametric statistics is a branch of statistics which assumes that sample data comes from a population that follows a probability distribution based on a fixed set of parameters.
  • Nonparametric statistics is the branch of statistics that is not based solely on population parameters.

Parametric Machine Learning Algorithms

This particular algorithm involves two steps:

  1. Selecting a form for the function
  2. Learning the coefficients for the function from the training data

Let us consider a line to understand functional form for the mapping function as it is used in linear regression and simplify the learning process.

b0 + b1*x1 + b2*x2 = 0

Where b0, b1 and b2 are the coefficients of the line which control the intercept and slope, and x1 and x2 are two input variables.

All we have to do now is to estimate the coefficients of the line equation to get a predictive model for the problem. Now, the problem is that the actual unknown underlying function may not be a linear function like a line. In that case, the approach will give poor results. Some of the examples of parametric machine learning algorithms are mentioned below:

  • Logistic Regression
  • Linear Discriminant Analysis
  • Perceptron
  • Naive Bayes
  • Simple Neural Networks

Nonparametric Machine Learning Algorithms

Nonparametric methods always try to find the best fit training data while constructing the mapping function which also allows it to fit a large number of functional forms. Some of the examples of nonparametric machine learning algorithms are mentioned below:

  • k-Nearest Neighbors
  • Decision Trees like CART and C4.5
  • Support Vector Machines

The best example of nonparametric machine learning algorithms would be k-nearest neighbors algorithm which makes predictions based on the k most similar training patterns for a given set of new data instance. This method simply assumes that the patterns which are close are likely to be of similar type.


Parametric Machine Learning AlgorithmsNonparametric Machine Learning Algorithms
Benefits
  • Simple to understand and interpret results
  • Speed of learning from data in fast
  • Less training data is required

  • Flexible enough to fit a large number of functional forms
  • No assumptions about the underlying functions
  • Provides high performance for prediction

Limitations
  • Choosing a functional form constrains the method to the specified form
  • It has limited complexity and more suited to simpler problems
  • It is unlikely to match the underlying mapping function and results in poor fit

  • Requires more training data in order to estimate the mapping function
  • Due to more parameters to train, it is slower comparatively
  • There is a risk to overfit the training data

Method Based LearningMethod Based Learning in machine learning

There are several learning models namely:

  • Association rules based
  • Ensemble method based
  • Deep Learning based
  • Clustering method based
  • Regression Analysis based
  • Bayesian method based
  • Dimensionality reduction based
  • Kernel method based
  • Instance based

Let us understand what Instance Based Learning is all about.

Instance Based Learning (IBL)Instance Based Learning (IBL) in machine learning

  • Instance-Based methods are the simplest form of learning
  • Instance -Based learning is lazy learning
  • K-NN model works on identified instance
  • Instances are retrieved from memory and then this data is used to classify the new query instance
  • Instance based learning is also called memory-based or case-based

Under Instance-based Learning we have,

Nearest-neighbor classifier

Uses k “closest” points (nearest neighbors) for performing classification. For example: It’s how people judge by observing our peers. We tend to move with people of similar attributes.

Lazy Learning vs Eager Learning

Lazy LearningEager Learning
Simply stores the training data and waits until it is given a test tuple.Munges the training data as soon as it receives it.
It's slow as it calculates based on the current data set instead of coming up with an algorithm based on historical data.It's fast as it has pre-calculated algorithm.
Localized data so generalization takes time at every iteration.On the basis of training set ,it constructs a classification model before receiving new data to classify.

What is K-NN?

One of the biggest applications of K-Nearest Neighbor search is Recommender Systems. If you have noticed while you are shopping as a user on Amazon and you like a particular item, you are recommended with similar items.

What is K-NN? in Machine Leaning

It also recommends similar items bought by other users and other set of items which are often bought together. Basically, the algorithm compares the set of users who like each item and looks for similarity. This not only applies to recommending items or products but also recommending media and even advertisements to display to a user.

K nearest neighbors or K-NN Algorithm is a simple algorithm which uses the entire dataset in its training phase. Whenever a prediction is required for an unseen data instance, it searches through the entire training dataset for k-most similar instances and the data with the most similar instance is finally returned as the prediction.

This algorithm suggests that if you’re similar to your neighbours, then you are one of them. Let us consider a simple example, if apple looks more similar to peach, pear, and cherry (fruits) than monkey, cat or a rat (animals), then most likely apple is a fruit.

Nearest Neighbours algorithm has been in action for the last sixty years. It is mainly used in statistical estimation and pattern recognition, as a non-parametric method, for regression and classification. The main aim of the K-Nearest Neighbor algorithm is to classify a new data point by comparing it to all previously seen data points. The classification of the k most similar previous cases are used for predicting the classification of the current data point. It is a simple algorithm which stores all available cases and classifies new cases based on a similarity measure (e.g., distance functions).

When do we use K-NN algorithm?

K-NN algorithm can be used for applications which require high accuracy as it makes highly accurate predictions. The quality of predictions is completely dependent on the distance measure. Thus, this algorithm is suitable for applications for which you have sufficient domain knowledge so that it can help you select an appropriate measure.

As we have already seen K-NN algorithm is a type of lazy learning, the computation for the generation is postponed until classification which indeed increases the costs of computation compared to other machine learning algorithms. But still K-NN is considered to be the better choice for applications where accuracy is more important and predictions are not requested frequently.

K-NN can be used for both regression and classification predictive problems. However, in the industry it is mostly used in classification problems.

Generally we mainly look at 3 important aspects in order to evaluate any technique:

  1. Ease to interpret output
  2. Calculation time
  3. Predictive Power

Let us consider a few examples to place K-NN in the scale :

K-NN algorithm example scale in Machine Learning

If you notice the chart mentioned above, K-NN algorithm exceeds in most of the parameters. It is most commonly used for ease of interpretation and low calculation time.

How does the K-NN algorithm work?

K-NN algorithm works on the basis of feature similarity. The classification of a given data point is determined by how closely out-of-sample features resemble our training set.

How does the K-NN algorithm work? in Machine Learning

The above figure shows an example of k-NN classification. If you consider the nearest neighbor to the test sample, it is a blue square (Class 1) and k=1. This falls inside the inner circle.

Now, if you consider k=3, then you will see 2 red triangles and only 1 blue square falls under the outer circle. Thus, the test sample is classified as a red triangle now (Class 2).

Similarly, if you consider k=5, it is assigned to the first class (3 squares vs. 2 triangles outside the outer circle).

K-NN in Regression

In regression problems, K-NN is used for prediction based on the mean or the median of the K-most similar instances.

K-NN in Classification

K-nearest-neighbor classification was actually developed from the need to perform discriminant analysis when reliable parametric estimates of probability densities are unknown or are difficult to determine. When K-NN is used for classification, the output is easily calculated by the class having the highest frequency from the K-most similar instances. The class with maximum vote is taken into consideration for prediction.

The probabilities of Classes can be calculated as the normalized frequency of samples that belong to each class in the set of K most similar instances for a new data instance.

For example, in a binary classification problem (class is 0 or 1):

p(class=0) = count(class=0) / (count(class=0)+count(class=1))

If you are using K and you have an even number of classes (e.g. 2) it is a good idea to choose a K value with an odd number to avoid a tie. And the inverse, use an even number for K when you have an odd number of classes.

Ties can be broken consistently by expanding K by 1 and looking at the class of the next most similar instance in the training dataset.

Making Predictions with K-NN

A case can be classified by a majority vote of its neighbors. The case is then assigned to the most common class amongst its K nearest neighbors measured by a distance function. Suppose the value of K is 1, then the case is simply assigned to the class of its nearest neighbor.

Distance functions in Machine Learning

The three distance measures mentioned above are valid only for continuous variables. For categorical variables, the Hamming distance is used. It also brings up the issue of standardization of the numerical variables between 0 and 1 when there is a mixture of numerical and categorical variables in the dataset.

Hamming Distance In Machine Learning

By inspecting the data, you can choose the best optimal value for K. Generally, a large value of K is more accurate as it tends to reduce the overall noise but is not always true. Another way to retrospectively determine a good K value by using an independent dataset to validate the K value is Cross-validation. According to observation, the optimal K for most datasets has been between 3-10 which provides better results than 1NN.

For example, let us consider an example where the data mentioned below us concerned with credit default. Age and Loan are two numerical variables (predictors) and Default is the target.

Making Predictions with K-NN In Machine Learning

By observing the data mentioned above, we can use the training set in order to classify an unknown case (Age=48 and Loan=$142,000) using Euclidean distance. If K=1 then the nearest neighbor is the last case in the training set with Default=Y.

Making Predictions with K-NN

AgeLoanDefaultDistance
25$40,000N102000
35$60,000N82000
45$80,000N62000
20$20,000N122000
35$120,000N220002
52$18,000N124000
23$95,000Y47000
40$62,000Y80000
60$100,000Y420003
48$220,000Y78000
33$150,000Y80001





48$142,000
?

Euclidean DistanceEuclidean Distance

With K=3, there are two Default=Y and one Default=N out of three closest neighbors. The prediction for the unknown case is again Default=Y.

Standardized Distance

One major drawback in calculating distance measures directly from the training set is in the case where variables have different measurement scales or there is a mixture of numerical and categorical variables. For example, if one variable is based on annual income in dollars, and the other is based on age in years then income will have a much higher influence on the distance calculated. One solution is to standardize the training set as shown below.

AgeLoanDefaultDistance
0.1250.11N0.7652
0.3750.21N0.5200
0.6250.31N
0.3160
00.01N0.9245
0.3750.50N0.3428
0.80.00N0.6220
0.0750.38Y0.6669
0.50.22Y0.4437
10.41Y0.3650
0.71.00Y0.3861
0.3250.65Y0.3771




0.7
0.61
?

Standardized VariableStandardized Variable

Using the standardized distance on the same training set, the unknown case returned a different neighbor which is not a good sign of robustness.

Between-sample geometric distance

The k-nearest-neighbor classifier is commonly based on the Euclidean distance between a test sample and the specified training samples. Let  xbe an input sample with features, (xi1, xi2, …, xip), be the total number of input samples (i=1,2,…,n) and the total number of features (j=1,2,…,p) . The Euclidean distance between sample xand x(l=1,2,…,n) is defined as:

Standardized Variable

A graphical representation of the nearest neighbor concept is illustrated in the Voronoi tessellation. The tessellation shows 19 samples marked with a "+", and the Voronoi cell, R, surrounding each sample. A Voronoi cell encapsulates all neighboring points that are nearest to each sample and is defined as:

Standardized Variable

Where Ris the Voronoi cell for sample xi, and represents all possible points within Voronoi cell Ri.

Standardized geometric distance in Machine LearningVoronoi tessellation showing Voronoi cells of 19 samples marked with a "+"

The Voronoi tessellation reflects two characteristics of the example 2-dimensional coordinate system: i) all possible points within a sample's Voronoi cell are the nearest neighboring points for that sample, and ii) for any sample, the nearest sample is determined by the closest Voronoi cell edge.

According to the latter characteristic, the k-nearest-neighbor classification rule is to assign to a test sample the majority category label of its k nearest training samples. In practice, k is usually chosen to be odd, so as to avoid ties. The k = 1 rule is generally called the nearest-neighbor classification rule.

Curse of Dimensionality

The curse of dimensionality refers to various phenomena that are witnessed while analyzing and organizing data in high-dimensional spaces (often with hundreds or thousands of dimensions). Such phenomenon do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience.

K-NN algorithm will work absolutely fine when you are dealing with a small number of input variables (p)  but will struggle when there are a large number of inputs.

K-NN works well with a small number of input variables (p), but struggles when the number of inputs is very large. Each input variable can be considered a dimension of a p-dimensional input space. For example, suppose you have two input variables x1 and x2, the input space would be 2-dimensional. With the increase in the number of dimensions, the volume of the input space increases at an exponential rate.

In case of higher dimensions, the points which are similar may have large distances. All these points will be then away from each other and our intuition about 2 to 3 dimensional spaces will not be applicable. This kind of problem is called the “Curse of Dimensionality“. 

How is K in K-means different from K in K-NN?

K-Means Clustering and k-Nearest Neighbors algorithm, both are commonly used algorithms in Machine Learning. They are often confused with each other, especially when we are talking about the k-factor. The ‘K’ in K-Means Clustering has nothing to do with the ‘K’ in K-NN algorithm. k-Means Clustering is an unsupervised learning algorithm that is used for clustering whereas K-NN is a supervised learning algorithm used for classification.

K-Means Algorithm

The k-means algorithm is an unsupervised clustering algorithm which takes a couple of unlabeled points and then groups them into “k” number of clusters.

The “k” in k-means denotes the number of clusters you would like to have in the end. Suppose the value of k is 5, it means you will have 5 clusters on the data set.

Let us see how it works.

Step 1: First you determine the value of K by Elbow method and then specify the number of clusters K

Step 2: Next you have to randomly assign each data point to a cluster

Step 3: Determine the cluster centroid coordinates

Step 4: Determine the distances of each data point to the centroids and re-assign each point to the closest cluster centroid based upon minimum distance

Step 5: Calculate cluster centroids again

Step 6: Repeat steps 4 and 5 until we reach global optima where further improvements are not possible and there is no provision to switch data points from one cluster to another.

Implementation in Python

#Finding the optimum number of clusters for k-means clustering
Nc = range(1, 10)
kmeans = [KMeans(n_clusters=i) for i in Nc]
kmeans
score = [kmeans[i].fit(x).score(x) for i in range(len(kmeans))]
score
pl.plot(Nc,score)
pl.xlabel('Number of Clusters')
pl.ylabel('Score')
pl.title('Elbow Curve')
pl.show()

Python Implementation in Machine Learning

You can clearly see why it is called 'The elbow method' from the above graph, the optimum clusters is where the elbow occurs.

Now that we have the optimum amount of clusters (k=3), we can move on to applying K-means clustering to the Iris dataset.

#Implementation of K-Means Clustering
model = KMeans(n_clusters = 3)
model.fit(x)
model.labels_
colormap = np.array(['Red', 'Blue', 'Green'])
z = plt.scatter(x.sepal_length, x.sepal_width, x.petal_length, c = colormap[model.labels_])

The elbow method

#Accuracy of K-Means Clustering
accuracy_score(iris.target,model.labels_)
0.8933333333333333

K-NN Algorithm

By now, we already know that K-NN algorithm is a supervised classification algorithm. It takes into consideration a couple of labelled points and then uses those points to learn how to label other points. To be able to assign label to other points, K-NN algorithm looks for the closest neighbor of the new point and checks for voting. The most number of neighbors around the new point decide the label of the new point.

The “k” in K-Nearest Neighbors is the number of neighbors it checks. It is supervised because it is trying to classify a point on the basis of the known classification of other points.

Let us see how it works.

Step 1: Firstly, you determine the value for K.

Step 2: Then you calculate the distances between the new input (test data) and all the training data. The most commonly used metrics for calculating distance are Euclidean, Manhattan and Minkowski

Step 3: Sort the distance and determine k nearest neighbors based on minimum distance values

Step 4: Analyze the category of those neighbors and assign the category for the test data based on majority vote

Step 5: Return the predicted class

Implementation using Python

error = []
# Calculating error for K values between 1 and 40
for i in range(1, 40):
K-NN = KNeighborsClassifier(n_neighbors=i)
K-NN.fit(X_train, y_train)
pred_i = K-NN.predict(X_test)
error.append(np.mean(pred_i != y_test))
plt.figure(figsize=(12, 6))
plt.plot(range(1, 40), error, color='black', linestyle='dashed', marker='o',
    markerfacecolor='grey', markersize=10)
plt.title('Error Rate K Value')
plt.xlabel('K Value')
plt.ylabel('Mean Error')
Text(0, 0.5, 'Mean Error')

Error Rate K value Python Implementation in Machine Learning

Now we know for what values of ‘K’, the error rate will be less. Let’s fix k=5 and implement K-NN algorithm.

#Creating training and test splits
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)
#Performing Feature Scaling
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
scaler.fit(X_train)

X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)

#Training K-NN with k=5
from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors=5)
classifier.fit(X_train, y_train)
KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
                    metric_params=None, n_jobs=None, n_neighbors=5, p=2,
                    weights='uniform')
y_pred = classifier.predict(X_test)
from sklearn.metrics import classification_report, confusion_matrix
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred))
[[10  0  0]
 [ 0  9  2]
 [ 0  1  8]]
                       precision    recall     f1-score     support
    Iris-setosa        1.00         1.00       1.00         10
Iris-versicolor        0.90         0.82       0.86         11
 Iris-virginica        0.80         0.89       0.84         9
       accuracy                                0.90         30
      macro avg        0.90         0.90       0.90         30
   weighted avg        0.90         0.90       0.90         30

Practical Applications of K-NN

Now that we have we have seen how K-NN works, let us look into some of the practical applications of K-NN.

  • Recommending products to people with similar interests, recommending movies and TV shows as per viewer’s choice and interest, recommending hotels and other accommodation facilities while you are travelling based on your previous bookings.
  • Assigning credit ratings based on financial characteristics, comparing people with similar financial features in a database. By analyzing the nature of a credit rating, people with similar financial details, they would be assigned similar credit ratings.
  • Should the bank give a loan to an individual? Would an individual default on his or her loan? Is that person closer in characteristics to people who defaulted or did not default on their loans?
  • Some advanced examples could include handwriting detection (like OCR), image recognition and even video recognition.

Some pros and cons of K-NN

Pros

  • Training phase of K-nearest neighbor classification is faster in comparison with other classification algorithms.
  • Training of a model is not required for generalization.
  • Simple algorithm — to explain and understand/interpret.
  • High accuracy (relatively) — it is pretty high but not competitive in comparison to better supervised learning models.
  • K-NN can be useful in case of nonlinear data.
  • Versatile — useful for classification or regression.

Cons

  • Testing phase of K-nearest neighbor classification is slower and costlier with respect to time and memory. 
  • High memory requirement - Requires large memory for storing the entire training dataset.
  • K-NN requires scaling of data because K-NN uses the Euclidean distance between two data points to find nearest neighbors.
  • Euclidean distance is sensitive to magnitudes. The features with high magnitudes will weigh more than features with low magnitudes.
  • Not suitable for large dimensional data.

How to improve the performance of K-NN?

  • Rescaling Data: K-NN performs much better if all of the data has the same scale. Normalizing your data to the range [0, 1] is a good idea. It may also be a good idea to standardize your data if it has a Gaussian distribution.
  • Addressing Missing Data: Missing data will mean that the distance between samples can not be calculated. These samples could be excluded or the missing values could be imputed.
  • Reducing Dimensionality: K-NN is suited for lower dimensional data. You can try it on high dimensional data (hundreds or thousands of input variables) but be aware that it may not perform as good as other techniques. K-NN can benefit from feature selection that reduces the dimensionality of the input feature space.

In this article we have learned about the K-Nearest Neighbor algorithm, where we should use it, how it works and so on. Also, we have discussed about parametric and nonparametric machine learning algorithms, instance based learning, eager and lazy learning, advantages and disadvantages of using K-NN, performance improvement suggestions and have implemented K-NN in Python. To learn more about other machine learning algorithms, join our Data Science Certification course and expand your learning skill set and career opportunities.

Priyankur

Priyankur Sarkar

Data Science Enthusiast

Priyankur Sarkar loves to play with data and get insightful results out of it, then turn those data insights and results in business growth. He is an electronics engineer with a versatile experience as an individual contributor and leading teams, and has actively worked towards building Machine Learning capabilities for organizations.

Join the Discussion

Your email address will not be published. Required fields are marked *

Suggested Blogs

Role of Unstructured Data in Data Science

Data has become the new game changer for businesses. Typically, data scientists categorize data into three broad divisions - structured, semi-structured, and unstructured data. In this article, you will get to know about unstructured data, sources of unstructured data, unstructured data vs. structured data, the use of structured and unstructured data in machine learning, and the difference between structured and unstructured data. Let us first understand what is unstructured data with examples. What is unstructured data? Unstructured data is a kind of data format where there is no organized form or type of data. Videos, texts, images, document files, audio materials, email contents and more are considered to be unstructured data. It is the most copious form of business data, and cannot be stored in a structured database or relational database. Some examples of unstructured data are the photos we post on social media platforms, the tagging we do, the multimedia files we upload, and the documents we share. Seagate predicts that the global data-sphere will expand to 163 zettabytes by 2025, where most of the data will be in the unstructured format. Characteristics of Unstructured DataUnstructured data cannot be organized in a predefined fashion, and is not a homogenous data model. This makes it difficult to manage. Apart from that, these are the other characteristics of unstructured data. You cannot store unstructured data in the form of rows and columns as we do in a database table. Unstructured data is heterogeneous in structure and does not have any specific data model. The creation of such data does not follow any semantics or habits. Due to the lack of any particular sequence or format, it is difficult to manage. Such data does not have an identifiable structure. Sources of Unstructured Data There are various sources of unstructured data. Some of them are: Content websites Social networking sites Online images Memos Reports and research papers Documents, spreadsheets, and presentations Audio mining, chatbots Surveys Feedback systems Advantages of Unstructured Data Unstructured data has become exceptionally easy to store because of MongoDB, Cassandra, or even using JSON. Modern NoSQL databases and software allows data engineers to collect and extract data from various sources. There are numerous benefits that enterprises and businesses can gain from unstructured data. These are: With the advent of unstructured data, we can store data that lacks a proper format or structure. There is no fixed schema or data structure for storing such data, which gives flexibility in storing data of different genres. Unstructured data is much more portable by nature. Unstructured data is scalable and flexible to store. Database systems like MongoDB, Cassandra, etc., can easily handle the heterogeneous properties of unstructured data. Different applications and platforms produce unstructured data that becomes useful in business intelligence, unstructured data analytics, and various other fields. Unstructured data analysis allows finding comprehensive data stories from data like email contents, website information, social media posts, mobile data, cache files and more. Unstructured data, along with data analytics, helps companies improve customer experience. Detection of the taste of consumers and their choices becomes easy because of unstructured data analysis. Disadvantages of Unstructured data Storing and managing unstructured data is difficult because there is no proper structure or schema. Data indexing is also a substantial challenge and hence becomes unclear due to its disorganized nature. Search results from an unstructured dataset are also not accurate because it does not have predefined attributes. Data security is also a challenge due to the heterogeneous form of data. Problems faced and solutions for storing unstructured data. Until recently, it was challenging to store, evaluate, and manage unstructured data. But with the advent of modern data analysis tools, algorithms, CAS (content addressable storage system), and big data technologies, storage and evaluation became easy. Let us first take a look at the various challenges used for storing unstructured data. Storing unstructured data requires a large amount of space. Indexing of unstructured data is a hectic task. Database operations such as deleting and updating become difficult because of the disorganized nature of the data. Storing and managing video, audio, image file, emails, social media data is also challenging. Unstructured data increases the storage cost. For solving such issues, there are some particular approaches. These are: CAS system helps in storing unstructured data efficiently. We can preserve unstructured data in XML format. Developers can store unstructured data in an RDBMS system supporting BLOB. We can convert unstructured data into flexible formats so that evaluating and storage becomes easy. Let us now understand the differences between unstructured data vs. structured data. Unstructured Data Vs. Structured Data In this section, we will understand the difference between structured and unstructured data with examples. STRUCTUREDUNSTRUCTUREDStructured data resides in an organized format in a typical database.Unstructured data cannot reside in an organized format, and hence we cannot store it in a typical database.We can store structured data in SQL database tables having rows and columns.Storing and managing unstructured data requires specialized databases, along with a variety of business intelligence and analytics applications.It is tough to scale a database schema.It is highly scalable.Structured data gets generated in colleges, universities, banks, companies where people have to deal with names, date of birth, salary, marks and so on.We generate or find unstructured data in social media platforms, emails, analyzed data for business intelligence, call centers, chatbots and so on.Queries in structured data allow complex joining.Unstructured data allows only textual queries.The schema of a structured dataset is less flexible and dependent.An unstructured dataset is flexible but does not have any particular schema.It has various concurrency techniques.It has no concurrency techniques.We can use SQL, MySQL, SQLite, Oracle DB, Teradata to store structured data.We can use NoSQL (Not Only SQL) to store unstructured data.Types of Unstructured Data Do you have any idea just how much of unstructured data we produce and from what sources? Unstructured data includes all those forms of data that we cannot actively manage in an RDBMS system that is a transactional system. We can store structured data in the form of records. But this is not the case with unstructured data. Before the advent of object-based storage, most of the unstructured data was stored in file-based systems. Here are some of the types of unstructured data. Rich media content: Entertainment files, surveillance data, multimedia email attachments, geospatial data, audio files (call center and other recorded audio), weather reports (graphical), etc., comes under this genre. Document data: Invoices, text-file records, email contents, productivity applications, etc., are included under this genre. Internet of Things (IoT) data: Ticker data, sensor data, data from other IoT devices come under this genre. Apart from all these, data from business intelligence and analysis, machine learning datasets, and artificial intelligence data training datasets are also a separate genre of unstructured data. Examples of Unstructured Data There are various sources from where we can obtain unstructured data. The prominent use of this data is in unstructured data analytics. Let us now understand what are some examples of unstructured data and their sources – Healthcare industries generate a massive volume of human as well as machine-generated unstructured data. Human-generated unstructured data could be in the form of patient-doctor or patient-nurse conversations, which are usually recorded in audio or text formats. Unstructured data generated by machines includes emergency video camera footage, surgical robots, data accumulated from medical imaging devices like endoscopes, laparoscopes and more.  Social Media is an intrinsic entity of our daily life. Billions of people come together to join channels, share different thoughts, and exchange information with their loved ones. They create and share such data over social media platforms in the form of images, video clips, audio messages, tagging people (this helps companies to map relations between two or more people), entertainment data, educational data, geolocations, texts, etc. Other spectra of data generated from social media platforms are behavior patterns, perceptions, influencers, trends, news, and events. Business and corporate documents generate a multitude of unstructured data such as emails, presentations, reports containing texts, images, presentation reports, video contents, feedback and much more. These documents help to create knowledge repositories within an organization to make better implicit operations. Live chat, video conferencing, web meeting, chatbot-customer messages, surveillance data are other prominent examples of unstructured data that companies can cultivate to get more insights into the details of a person. Some prominent examples of unstructured data used in enterprises and organizations are: Reports and documents, like Word files or PDF files Multimedia files, such as audio, images, designed texts, themes, and videos System logs Medical images Flat files Scanned documents (which are images that hold numbers and text – for example, OCR) Biometric data Unstructured Data Analytics Tools  You might be wondering what tools can come into use to gather and analyze information that does not have a predefined structure or model. Various tools and programming languages use structured and unstructured data for machine learning and data analysis. These are: Tableau MonkeyLearn Apache Spark SAS Python MS. Excel RapidMiner KNIME QlikView Python programming R programming Many cloud services (like Amazon AWS, Microsoft Azure, IBM Cloud, Google Cloud) also offer unstructured data analysis solutions bundled with their services. How to analyze unstructured data? In the past, the process of storage and analysis of unstructured data was not well defined. Enterprises used to carry out this kind of analysis manually. But with the advent of modern tools and programming languages, most of the unstructured data analysis methods became highly advanced. AI-powered tools use algorithms designed precisely to help to break down unstructured data for analysis. Unstructured data analytics tools, along with Natural language processing (NLP) and machine learning algorithms, help advanced software tools analyze and extract analytical data from the unstructured datasets. Before using these tools for analyzing unstructured data, you must properly go through a few steps and keep these points in mind. Set a clear goal for analyzing the data: It is essential to clear your intention about what insights you want to extract from your unstructured data. Knowing this will help you distinguish what type of data you are planning to accumulate. Collect relevant data: Unstructured data is available everywhere, whether it's a social media platform, online feedback or reviews, or a survey form. Depending on the previous point, that is your goal - you have to be precise about what data you want to collect in real-time. Also, keep in mind whether your collected details are relevant or not. Clean your data: Data cleaning or data cleansing is a significant process to detect corrupt or irrelevant data from the dataset, followed by modifying or deleting the coarse and sloppy data. This phase is also known as the data-preprocessing phase, where you have to reduce the noise, carry out data slicing for meaningful representation, and remove unnecessary data. Use Technology and tools: Once you perform the data cleaning, it is time to utilize unstructured data analysis tools to prepare and cultivate the insights from your data. Technologies used for unstructured data storage (NoSQL) can help in managing your flow of data. Other tools and programming libraries like Tableau, Matplotlib, Pandas, and Google Data Studio allows us to extract and visualize unstructured data. Data can be visualized and presented in the form of compelling graphs, plots, and charts. How to Extract information from Unstructured Data? With the growth in digitization during the information era, repetitious transactions in data cause data flooding. The exponential accretion in the speed of digital data creation has brought a whole new domain of understanding user interaction with the online world. According to Gartner, 80% of the data created by an organization or its application is unstructured. While extracting exact information through appropriate analysis of organized data is not yet possible, even obtaining a decent sense of this unstructured data is quite tough. Until now, there are no perfect tools to analyze unstructured data. But algorithms and tools designed using machine learning, Natural language processing, Deep learning, and Graph Analysis (a mathematical method for estimating graph structures) help us to get the upper hand in extracting information from unstructured data. Other neural network models like modern linguistic models follow unsupervised learning techniques to gain a good 'knowledge' about the unstructured dataset before going into a specific supervised learning step. AI-based algorithms and technologies are capable enough to extract keywords, locations, phone numbers, analyze image meaning (through digital image processing). We can then understand what to evaluate and identify information that is essential to your business. ConclusionUnstructured data is found abundantly from sources like documents, records, emails, social media posts, feedbacks, call-records, log-in session data, video, audio, and images. Manually analyzing unstructured data is very time-consuming and can be very boring at the same time. With the growth of data science and machine learning algorithms and models, it has become easy to gather and analyze insights from unstructured information.  According to some research, data analytics tools like MonkeyLearn Studio, Tableau, RapidMiner help analyze unstructured data 1200x faster than the manual approach. Analyzing such data will help you learn more about your customers as well as competitors. Text analysis software, along with machine learning models, will help you dig deep into such datasets and make you gain an in-depth understanding of the overall scenario with fine-grained analyses.
5744
Role of Unstructured Data in Data Science

Data has become the new game changer for busines... Read More

What Is Statistical Analysis and Its Business Applications?

Statistics is a science concerned with collection, analysis, interpretation, and presentation of data. In Statistics, we generally want to study a population. You may consider a population as a collection of things, persons, or objects under experiment or study. It is usually not possible to gain access to all of the information from the entire population due to logistical reasons. So, when we want to study a population, we generally select a sample. In sampling, we select a portion (or subset) of the larger population and then study the portion (or the sample) to learn about the population. Data is the result of sampling from a population.Major ClassificationThere are two basic branches of Statistics – Descriptive and Inferential statistics. Let us understand the two branches in brief. Descriptive statistics Descriptive statistics involves organizing and summarizing the data for better and easier understanding. Unlike Inferential statistics, Descriptive statistics seeks to describe the data, however, it does not attempt to draw inferences from the sample to the whole population. We simply describe the data in a sample. It is not developed on the basis of probability unlike Inferential statistics. Descriptive statistics is further broken into two categories – Measure of Central Tendency and Measures of Variability. Inferential statisticsInferential statistics is the method of estimating the population parameter based on the sample information. It applies dimensions from sample groups in an experiment to contrast the conduct group and make overviews on the large population sample. Please note that the inferential statistics are effective and valuable only when examining each member of the group is difficult. Let us understand Descriptive and Inferential statistics with the help of an example. Task – Suppose, you need to calculate the score of the players who scored a century in a cricket tournament.  Solution: Using Descriptive statistics you can get the desired results.   Task – Now, you need the overall score of the players who scored a century in the cricket tournament.  Solution: Applying the knowledge of Inferential statistics will help you in getting your desired results.  Top Five Considerations for Statistical Data AnalysisData can be messy. Even a small blunder may cost you a fortune. Therefore, special care when working with statistical data is of utmost importance. Here are a few key takeaways you must consider to minimize errors and improve accuracy. Define the purpose and determine the location where the publication will take place.  Understand the assets to undertake the investigation. Understand the individual capability of appropriately managing and understanding the analysis.  Determine whether there is a need to repeat the process.  Know the expectation of the individuals evaluating reviewing, committee, and supervision. Statistics and ParametersDetermining the sample size requires understanding statistics and parameters. The two being very closely related are often confused and sometimes hard to distinguish.  StatisticsA statistic is merely a portion of a target sample. It refers to the measure of the values calculated from the population.  A parameter is a fixed and unknown numerical value used for describing the entire population. The most commonly used parameters are: Mean Median Mode Mean :  The mean is the average or the most common value in a data sample or a population. It is also referred to as the expected value. Formula: Sum of the total number of observations/the number of observations. Experimental data set: 2, 4, 6, 8, 10, 12, 14, 16, 18, 20  Calculating mean:   (2 + 4 + 6 + 8 + 10 + 12 + 14 + 16 + 18 + 20)/10  = 110/10   = 11 Median:  In statistics, the median is the value separating the higher half from the lower half of a data sample, a population, or a probability distribution. It’s the mid-value obtained by arranging the data in increasing order or descending order. Formula:  Let n be the data set (increasing order) When data set is odd: Median = n+1/2th term Case-I: (n is odd)  Experimental data set = 1, 2, 3, 4, 5  Median (n = 5) = [(5 +1)/2]th term      = 6/2 term       = 3rd term   Therefore, the median is 3 When data set is even: Median = [n/2th + (n/2 + 1)th] /2 Case-II: (n is even)  Experimental data set = 1, 2, 3, 4, 5, 6   Median (n = 6) = [n/2th + (n/2 + 1)th]/2     = ( 6/2th + (6/2 +1)th]/2     = (3rd + 4th)/2      = (3 + 4)/2      = 7/2      = 3.5  Therefore, the median is 3.5 Mode: The mode is the value that appears most often in a set of data or a population. Experimental data set= 1, 2, 2, 2, 3, 3, 3, 3, 3, 4, 4,4,5, 6  Mode = 3 (Since 3 is the most repeated element in the sequence.) Terms Used to Describe DataWhen working with data, you will need to search, inspect, and characterize them. To understand the data in a tech-savvy and straightforward way, we use a few statistical terms to denote them individually or in groups.  The most frequently used terms used to describe data include data point, quantitative variables, indicator, statistic, time-series data, variable, data aggregation, time series, dataset, and database. Let us define each one of them in brief: Data points: These are the numerical files formed and organized for interpretations. Quantitative variables: These variables present the information in digit form.  Indicator: An indicator explains the action of a community's social-economic surroundings.  Time-series data: The time-series defines the sequential data.  Data aggregation: A group of data points and data set. Database: A group of arranged information for examination and recovery.  Time-series: A set of measures of a variable documented over a specified time. Step-by-Step Statistical Analysis ProcessThe statistical analysis process involves five steps followed one after another. Step 1: Design the study and find the population of the study. Step 2: Collect data as samples. Step 3: Describe the data in the sample. Step 4: Make inferences with the help of samples and calculations Step 5: Take action Data distributionData distribution is an entry that displays entire imaginable readings of data. It shows how frequently a value occurs. Distributed data is always in ascending order, charts, and graphs enabling visibility of measurements and frequencies. The distribution function displaying the density of values of reading is known as the probability density function. Percentiles in data distributionA percentile is the reading in a distribution with a specified percentage of clarifications under it.  Let us understand percentiles with the help of an example.  Suppose you have scored 90th percentile on a math test. A basic interpretation is that merely 4-5% of the scores were higher than your scores. Right? The median is 50th percentile because the assumed 50% of the values are higher than the median. Dispersion Dispersion explains the magnitude of distribution readings anticipated for a specific variable and multiple unique statistics like range, variance, and standard deviation. For instance, high values of a data set are widely scattered while small values of data are firmly clustered. Histogram The histogram is a pictorial display that arranges a group of data facts into user detailed ranges. A histogram summarizes a data series into a simple interpreted graphic by obtaining many data facts and combining them into reasonable ranges. It contains a variety of results into columns on the x-axis. The y axis displays percentages of data for each column and is applied to picture data distributions. Bell Curve distribution Bell curve distribution is a pictorial representation of a probability distribution whose fundamental standard deviation obtained from the mean makes the bell, shaped curving. The peak point on the curve symbolizes the maximum likely occasion in a pattern of data. The other possible outcomes are symmetrically dispersed around the mean, making a descending sloping curve on both sides of the peak. The curve breadth is therefore known as the standard deviation. Hypothesis testingHypothesis testing is a process where experts experiment with a theory of a population parameter. It aims to evaluate the credibility of a hypothesis using sample data. The five steps involved in hypothesis testing are:  Identify the no outcome hypothesis.  (A worthless or a no-output hypothesis has no outcome, connection, or dissimilarities amongst many factors.) Identify the alternative hypothesis.  Establish the importance level of the hypothesis.  Estimate the experiment statistic and equivalent P-value. P-value explains the possibility of getting a sample statistic.  Sketch a conclusion to interpret into a report about the alternate hypothesis. Types of variablesA variable is any digit, amount, or feature that is countable or measurable. Simply put, it is a variable characteristic that varies. The six types of variables include the following: Dependent variableA dependent variable has values that vary according to the value of another variable known as the independent variable.  Independent variableAn independent variable on the other side is controllable by experts. Its reports are recorded and equated.  Intervening variableAn intervening variable explicates fundamental relations between variables. Moderator variableA moderator variable upsets the power of the connection between dependent and independent variables.  Control variableA control variable is anything restricted to a research study. The values are constant throughout the experiment. Extraneous variableExtraneous variable refers to the entire variables that are dependent but can upset experimental outcomes. Chi-square testChi-square test records the contrast of a model to actual experimental data. Data is unsystematic, underdone, equally limited, obtained from independent variables, and a sufficient sample. It relates the size of any inconsistencies among the expected outcomes and the actual outcomes, provided with the sample size and the number of variables in the connection. Types of FrequenciesFrequency refers to the number of repetitions of reading in an experiment in a given time. Three types of frequency distribution include the following: Grouped, ungrouped Cumulative, relative Relative cumulative frequency distribution. Features of FrequenciesThe calculation of central tendency and position (median, mean, and mode). The measure of dispersion (range, variance, and standard deviation). Degree of symmetry (skewness). Peakedness (kurtosis). Correlation MatrixThe correlation matrix is a table that shows the correlation coefficients of unique variables. It is a powerful tool that summarises datasets points and picture sequences in the provided data. A correlation matrix includes rows and columns that display variables. Additionally, the correlation matrix exploits in aggregation with other varieties of statistical analysis. Inferential StatisticsInferential statistics use random data samples for demonstration and to create inferences. They are measured when analysis of each individual of a whole group is not likely to happen. Applications of Inferential StatisticsInferential statistics in educational research is not likely to sample the entire population that has summaries. For instance, the aim of an investigation study may be to obtain whether a new method of learning mathematics develops mathematical accomplishment for all students in a class. Marketing organizations: Marketing organizations use inferential statistics to dispute a survey and request inquiries. It is because carrying out surveys for all the individuals about merchandise is not likely. Finance departments: Financial departments apply inferential statistics for expected financial plan and resources expenses, especially when there are several indefinite aspects. However, economists cannot estimate all that use possibility. Economic planning: In economic planning, there are potent methods like index figures, time series investigation, and estimation. Inferential statistics measures national income and its components. It gathers info about revenue, investment, saving, and spending to establish links among them. Key TakeawaysStatistical analysis is the gathering and explanation of data to expose sequences and tendencies.   Two divisions of statistical analysis are statistical and non-statistical analyses.  Descriptive and Inferential statistics are the two main categories of statistical analysis. Descriptive statistics describe data, whereas Inferential statistics equate dissimilarities between the sample groups.  Statistics aims to teach individuals how to use restricted samples to generate intellectual and precise results for a large group.   Mean, median, and mode are the statistical analysis parameters used to measure central tendency.   Conclusion Statistical analysis is the procedure of gathering and examining data to recognize sequences and trends. It uses random samples of data obtained from a population to demonstrate and create inferences on a group. Inferential statistics applies economic planning with potent methods like index figures, time series investigation, and estimation.  Statistical analysis finds its applications in all the major sectors – marketing, finance, economic, operations, and data mining. Statistical analysis aids marketing organizations in disputing a survey and requesting inquiries concerning their merchandise. 
5881
What Is Statistical Analysis and Its Business Appl...

Statistics is a science concerned with collection,... Read More

Measures of Dispersion: All You Need to Know

What is Dispersion in StatisticsDispersion in statistics is a way of describing how spread out a set of data is. Dispersion is the state of data getting dispersed, stretched, or spread out in different categories. It involves finding the size of distribution values that are expected from the set of data for the specific variable. The statistical meaning of dispersion is “numeric data that is likely to vary at any instance of average value assumption”.Dispersion of data in Statistics helps one to easily understand the dataset by classifying them into their own specific dispersion criteria like variance, standard deviation, and ranging.Dispersion is a set of measures that helps one to determine the quality of data in an objectively quantifiable manner.The measure of dispersion contains almost the same unit as the quantity being measured. There are many Measures of Dispersion found which help us to get more insights into the data: Range Variance Standard Deviation Skewness IQR  Image SourceTypes of Measure of DispersionThe Measure of Dispersion is divided into two main categories and offer ways of measuring the diverse nature of data. It is mainly used in biological statistics. We can easily classify them by checking whether they contain units or not. So as per the above, we can divide the data into two categories which are: Absolute Measure of Dispersion Relative Measure of DispersionAbsolute Measure of DispersionAbsolute Measure of Dispersion is one with units; it has the same unit as the initial dataset. Absolute Measure of Dispersion is expressed in terms of the average of the dispersion quantities like Standard or Mean deviation. The Absolute Measure of Dispersion can be expressed  in units such as Rupees, Centimetre, Marks, kilograms, and other quantities that are measured depending on the situation. Types of Absolute Measure of Dispersion: Range: Range is the measure of the difference between the largest and smallest value of the data variability. The range is the simplest form of Measure of Dispersion. Example: 1,2,3,4,5,6,7 Range = Highest value – Lowest value   = ( 7 – 1 ) = 6 Mean (μ): Mean is calculated as the average of the numbers. To calculate the Mean, add all the outcomes and then divide it with the total number of terms. Example: 1,2,3,4,5,6,7,8 Mean = (sum of all the terms / total number of terms)                = (1 + 2 + 3 + 4 + 5 + 6 + 7 + 8) / 8                = 36 / 8                = 4.5 Variance (σ2): In simple terms, the variance can be calculated by obtaining the sum of the squared distance of each term in the distribution from the Mean, and then dividing this by the total number of the terms in the distribution.  It basically shows how far a number, for example, a student’s mark in an exam, is from the Mean of the entire class. Formula: (σ2) = ∑ ( X − μ)2 / N Standard Deviation: Standard Deviation can be represented as the square root of Variance. To find the standard deviation of any data, you need to find the variance first. Formula: Standard Deviation = √σ Quartile: Quartiles divide the list of numbers or data into quarters. Quartile Deviation: Quartile Deviation is the measure of the difference between the upper and lower quartile. This measure of deviation is also known as interquartile range. Formula: Interquartile Range: Q3 – Q1. Mean deviation: Mean Deviation is also known as an average deviation; it can be computed using the Mean or Median of the data. Mean deviation is represented as the arithmetic deviation of a different item that follows the central tendency. Formula: As mentioned, the Mean Deviation can be calculated using Mean and Median. Mean Deviation using Mean: ∑ | X – M | / N Mean Deviation using Median: ∑ | X – X1 | / N Relative Measure of DispersionRelative Measures of dispersion are the values without units. A relative measure of dispersion is used to compare the distribution of two or more datasets.  The definition of the Relative Measure of Dispersion is the same as the Absolute Measure of Dispersion; the only difference is the measuring quantity.  Types of Relative Measure of Dispersion: Relative Measure of Dispersion is the calculation of the co-efficient of Dispersion, where 2 series are compared, which differ widely in their average.  The main use of the co-efficient of Dispersion is when 2 series with different measurement units are compared.  1. Co-efficient of Range: it is calculated as the ratio of the difference between the largest and smallest terms of the distribution, to the sum of the largest and smallest terms of the distribution.  Formula: L – S / L + S  where L = largest value S= smallest value 2. Co-efficient of Variation: The coefficient of variation is used to compare the 2 data with respect to homogeneity or consistency.  Formula: C.V = (σ / X) 100 X = standard deviation  σ = mean 3. Co-efficient of Standard Deviation: The co-efficient of Standard Deviation is the ratio of standard deviation with the mean of the distribution of terms.  Formula: σ = ( √( X – X1)) / (N - 1) Deviation = ( X – X1)  σ = standard deviation  N= total number  4. Co-efficient of Quartile Deviation: The co-efficient of Quartile Deviation is the ratio of the difference between the upper quartile and the lower quartile to the sum of the upper quartile and lower quartile.  Formula: ( Q3 – Q3) / ( Q3 + Q1) Q3 = Upper Quartile  Q1 = Lower Quartile 5. Co-efficient of Mean Deviation: The co-efficient of Mean Deviation can be computed using the mean or median of the data. Mean Deviation using Mean: ∑ | X – M | / N Mean Deviation using Mean: ∑ | X – X1 | / N Why dispersion is important in a statisticThe knowledge of dispersion is vital in the understanding of statistics. It helps to understand concepts like the diversification of the data, how the data is spread, how it is maintained, and maintaining the data over the central value or central tendency. Moreover, dispersion in statistics provides us with a way to get better insights into data distribution. For example,  3 distinct samples can have the same Mean, Median, or Range but completely different levels of variability. How to Calculate DispersionDispersion can be easily calculated using various dispersion measures, which are already mentioned in the types of Measure of Dispersion described above. Before measuring the data, it is important to understand the diversion of the terms and variation. One can use the following method to calculate the dispersion: Mean Standard deviation Variance Quartile deviation For example, let us consider two datasets: Data A:97,98,99,100,101,102,103  Data B: 70,80,90,100,110,120,130 On calculating the mean and median of the two datasets, both have the same value, which is 100. However, the rest of the dispersion measures are totally different as measured by the above methods.  The range of B is 10 times higher, for instance. How to represent Dispersion in Statistics Dispersion in Statistics can be represented in the form of graphs and pie-charts. Some of the different ways used include: Dot Plots Box Plots Stems Leaf Plots Example: What is the variance of the values 3,8,6,10,12,9,11,10,12,7?  Variation of the values can be calculated using the following formula: (σ2) = ∑ ( X − μ)2 / N (σ2) = 7.36 What is an example of dispersion? One of the examples of dispersion outside the world of statistics is the rainbow- where white light is split into 7 different colours separated via wavelengths.  Some statistical ways of measuring it are- Standard deviation Range Mean absolute difference Median absolute deviation Interquartile change Average deviation Conclusion: Dispersion in statistics refers to the measure of variability of data or terms. Such variability may give random measurement errors where some of the instrumental measurements are found to be imprecise. It is a statistical way of describing how the terms are spread out in different data sets. The more sets of values, the more scattered data is found, and it is always directly proportional. This range of values can vary from 5 - 10 values to 1000 - 10,000 values. This spread of data is described by the range of descriptive range of statistics. The dispersion in statistics can be represented using a Dot Plot, Box Plot, and other different ways. 
9264
Measures of Dispersion: All You Need to Know

What is Dispersion in StatisticsDispersion in stat... Read More