Machine Learning Tutorial

Softmax regression is also known as multi nomial logistic regression, which is a generalization of logistic regression. It is used in cases where multiple classes need to be worked with, i.e data points in the dataset need to be classified into more than 2 classes. Softmax function performs the below functions: Converts all the scores into probabilities Ensures that the sum of the probabilities is 1 Feature matrix Assume a dataset to have ‘m’ columns, ‘n’ rows, and ‘k’ classes into which these values need to be classified into. The feature matrix can be represented as: x = (1  11...1) (1  21 ... 2) (1   1 ...)Weight matrix The weight matrix represents the weight of the ith row and jth colum W = (  01...1) (  11  ….   2  ) (   1....)The scores need to be normalized so that it is easy to implement gradient descent algorithm so as to minimize the cost function. Hence, we use a softmax function, which is defined below: P(y|) = S() (vector form) One hot encoded target matrix: The softmax function gives a vector of probabilities for every class label, with respect to a data point. This needs to be converted into the same format so as to calculate the cost function. Hence, every data point has a target vector which has zeroes and ones where a correct label is set to 1. This process is known as one-hot encoding. Let us understand how softmax regression can be implemented using TensorFlow library: Import the required libraries to implement softmax regression, and download the MNIST handwritten digit dataset. The MNIST data is split into a training, testing, and validation dataset. Next a computation graph is created. In the training data, a placeholder is supplied at run time. This technique uses mini batches to train the model using gradient descent, and this is known as stochastic gradient descent. The weight matrix explained prior is initialized using random values with a normal distribution. The bias is initialized to 0. The input data points are multiplied with weight matrix and bias value is added to it. Next the softmax is calculated using TensorFlow. Next, the cost function is minimized using the gradient descent algorithm. Let us look at the code now: import tensorflow as tf  import numpy as np  import matplotlib.pyplot as plt  from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)  print("Shape of the feature matrix:", mnist.train.images.shape)  print("Shape of the target matrix:", mnist.train.labels.shape)  print("One-hot encoding for the first observation is:\n", mnist.train.labels[0])  visualizing data by plotting the images fig,ax = plt.subplots(10,10)  k = 0  for i in range(10): for j in range(10):  ax[i][j].imshow(mnist.train.images[k].reshape(28,28), aspect='auto') k += 1  plt.show()  number of features  num_features = 784  number of target labels num_labels = 10  learning rate (also knwon as alpha) learning_rate = 0.05  batch size  batch_size = 128  number of epochs num_steps = 5001  input dataset  train_dataset = mnist.train.images  train_labels = mnist.train.labels  test_dataset = mnist.test.images  test_labels = mnist.test.labels  valid_dataset = mnist.validation.images  valid_labels = mnist.validation.labels  initializing a tensorflow graph graph = tf.Graph()  with graph.as_default():  # Inputs  tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, num_features))  tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))  tf_valid_dataset = tf.constant(valid_dataset)  tf_test_dataset = tf.constant(test_dataset)  # Variables.  weights = tf.Variable(tf.truncated_normal([num_features, num_labels]))  biases = tf.Variable(tf.zeros([num_labels]))  # Training computation  logits = tf.matmul(tf_train_dataset, weights) + biases  loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( labels=tf_train_labels, logits=logits))  # Optimizer  optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(loss)  Predictions for the training, validation, and test datasets train_prediction = tf.nn.softmax(logits)  valid_prediction = tf.nn.softmax(tf.matmul(tf_valid_dataset, weights) + biases) test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)  utility function that calculates accuracy  def accuracy(predictions, labels):  correctly_predicted = np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))  accu = (100.0 * correctly_predicted) / predictions.shape[0]  return accu  with tf.Session(graph=graph) as session:  initialize the weights and biases tf.global_variables_initializer().run() print("Initialized")  for step in range(num_steps):  # randomized offset is picked  offset = np.random.randint(0, train_labels.shape[0] - batch_size - 1)  # Generating a minibatch.  batch_data = train_dataset[offset:(offset + batch_size), :] ba feed_dict=feed_dict)  if (step % 500 == 0):  print("Minibatch loss at step {0}: {1}".format(step, l))  print("Minibatch accuracy: {:.1f}%".format(  accuracy(predictions, batch_labels)))  print("Validation accuracy: {:.1f}%".format(  accuracy(valid_prediction.eval(), valid_labels)))  print("\nTest accuracy: {:.1f}%".format(  accuracy(test_prediction.eval(), test_labels))) Output: Initialized  Minibatch loss at step 0: 11.68  Minibatch accuracy: 10.2%  Validation accuracy: 14.3%  Minibatch loss at step 500: 2.25  Minibatch accuracy: 46.9%  Validation accuracy: 67.6%  Minibatch loss at step 1000: 1.10  Minibatch accuracy: 78.1%  Validation accuracy: 75.0%  Minibatch loss at step 1500: 0.67  Minibatch accuracy: 78.9%  Validation accuracy: 78.6%  Minibatch loss at step 2000: 0.22  Minibatch accuracy: 91.4%  Validation accuracy: 81.0%  Minibatch loss at step 2500: 0.60  Minibatch accuracy: 84.4%  Validation accuracy: 82.5%  Minibatch loss at step 3000: 0.97  Minibatch accuracy: 85.2%  Validation accuracy: 83.9%  Minibatch loss at step 3500: 0.64  Minibatch accuracy: 85.2%  Validation accuracy: 84.4%  Minibatch loss at step 4000: 0.79  Minibatch accuracy: 82.8%  Validation accuracy: 85.0%  Minibatch loss at step 4500: 0.60  Minibatch accuracy: 80.5%  Validation accuracy: 85.6%  Minibatch loss at step 5000: 0.48  Minibatch accuracy: 89.1%tch_labels = train_labels[offset:(offset + batch_size), :]  feed_dict = {tf_train_dataset : batch_data,  tf_train_labels : batch_labels}  _, l, predictions = session.run([optimizer, loss, train_prediction],  Validation accuracy: 86.2%  Test accuracy: 86.49% Conclusion In this post, we understood the meaning of softmax regression, how it can be used, and its implementation using TensorFlow library.

Softmax Regression using TensorFlow

Softmax regression is also known as multi nomial logistic regression, which is a generalization of logistic regression. It is used in cases where multiple classes need to be worked with, i.e data points in the dataset need to be classified into more than 2 classes.

Softmax function performs the below functions:

• Converts all the scores into probabilities
• Ensures that the sum of the probabilities is 1

Feature matrix

Assume a dataset to have ‘m’ columns, ‘n’ rows, and ‘k’ classes into which these values need to be classified into. The feature matrix can be represented as:

x = (1  11...1)
(1  21 ... 2)
(1   1 ...)

Weight matrix

The weight matrix represents the weight of the ith row and jth colum

W = (  01...1)
(  11  ….   2  )
(   1....)

The scores need to be normalized so that it is easy to implement gradient descent algorithm so as to minimize the cost function. Hence, we use a softmax function, which is defined below:

P(y|) = S() (vector form)

One hot encoded target matrix: The softmax function gives a vector of probabilities for every class label, with respect to a data point. This needs to be converted into the same format so as to calculate the cost function. Hence, every data point has a target vector which has zeroes and ones where a correct label is set to 1. This process is known as one-hot encoding.

Let us understand how softmax regression can be implemented using TensorFlow library:

Import the required libraries to implement softmax regression, and download the MNIST handwritten digit dataset.

The MNIST data is split into a training, testing, and validation dataset. Next a computation graph is created. In the training data, a placeholder is supplied at run time. This technique uses mini batches to train the model using gradient descent, and this is known as stochastic gradient descent.

The weight matrix explained prior is initialized using random values with a normal distribution. The bias is initialized to 0.

The input data points are multiplied with weight matrix and bias value is added to it. Next the softmax is calculated using TensorFlow.

Next, the cost function is minimized using the gradient descent algorithm.

Let us look at the code now:

import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
from tensorflow.examples.tutorials.mnist import input_data mnist = input_data.read_data_sets("MNIST_data/", one_hot=True)
print("Shape of the feature matrix:", mnist.train.images.shape)
print("Shape of the target matrix:", mnist.train.labels.shape)
print("One-hot encoding for the first observation is:\n", mnist.train.labels[0])
visualizing data by plotting the images fig,ax = plt.subplots(10,10)
k = 0
for i in range(10): for j in range(10):
ax[i][j].imshow(mnist.train.images[k].reshape(28,28), aspect='auto') k += 1
plt.show()
number of features
num_features = 784
number of target labels num_labels = 10
learning rate (also knwon as alpha) learning_rate = 0.05
batch size
batch_size = 128
number of epochs num_steps = 5001
input dataset
train_dataset = mnist.train.images
train_labels = mnist.train.labels
test_dataset = mnist.test.images
test_labels = mnist.test.labels
valid_dataset = mnist.validation.images
valid_labels = mnist.validation.labels
initializing a tensorflow graph graph = tf.Graph()
with graph.as_default():
# Inputs
tf_train_dataset = tf.placeholder(tf.float32, shape=(batch_size, num_features))
tf_train_labels = tf.placeholder(tf.float32, shape=(batch_size, num_labels))
tf_valid_dataset = tf.constant(valid_dataset)
tf_test_dataset = tf.constant(test_dataset)
# Variables.
weights = tf.Variable(tf.truncated_normal([num_features, num_labels]))
biases = tf.Variable(tf.zeros([num_labels]))
# Training computation
logits = tf.matmul(tf_train_dataset, weights) + biases
loss = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits( labels=tf_train_labels, logits=logits))
# Optimizer
Predictions for the training, validation, and test datasets train_prediction = tf.nn.softmax(logits)
valid_prediction = tf.nn.softmax(tf.matmul(tf_valid_dataset, weights) + biases) test_prediction = tf.nn.softmax(tf.matmul(tf_test_dataset, weights) + biases)
utility function that calculates accuracy
def accuracy(predictions, labels):
correctly_predicted = np.sum(np.argmax(predictions, 1) == np.argmax(labels, 1))
accu = (100.0 * correctly_predicted) / predictions.shape[0]
return accu
with tf.Session(graph=graph) as session:
initialize the weights and biases tf.global_variables_initializer().run() print("Initialized")
for step in range(num_steps):
# randomized offset is picked
offset = np.random.randint(0, train_labels.shape[0] - batch_size - 1)
# Generating a minibatch.
batch_data = train_dataset[offset:(offset + batch_size), :] ba feed_dict=feed_dict)
if (step % 500 == 0):
print("Minibatch loss at step {0}: {1}".format(step, l))
print("Minibatch accuracy: {:.1f}%".format(
accuracy(predictions, batch_labels)))
print("Validation accuracy: {:.1f}%".format(
accuracy(valid_prediction.eval(), valid_labels)))
print("\nTest accuracy: {:.1f}%".format(
accuracy(test_prediction.eval(), test_labels))) 

Output:

Initialized
Minibatch loss at step 0: 11.68
Minibatch accuracy: 10.2%
Validation accuracy: 14.3%
Minibatch loss at step 500: 2.25
Minibatch accuracy: 46.9%
Validation accuracy: 67.6%
Minibatch loss at step 1000: 1.10
Minibatch accuracy: 78.1%
Validation accuracy: 75.0%
Minibatch loss at step 1500: 0.67
Minibatch accuracy: 78.9%
Validation accuracy: 78.6%
Minibatch loss at step 2000: 0.22
Minibatch accuracy: 91.4%
Validation accuracy: 81.0%
Minibatch loss at step 2500: 0.60
Minibatch accuracy: 84.4%
Validation accuracy: 82.5%
Minibatch loss at step 3000: 0.97
Minibatch accuracy: 85.2%
Validation accuracy: 83.9%
Minibatch loss at step 3500: 0.64
Minibatch accuracy: 85.2%
Validation accuracy: 84.4%
Minibatch loss at step 4000: 0.79
Minibatch accuracy: 82.8%
Validation accuracy: 85.0%
Minibatch loss at step 4500: 0.60
Minibatch accuracy: 80.5%
Validation accuracy: 85.6%
Minibatch loss at step 5000: 0.48
Minibatch accuracy: 89.1%tch_labels = train_labels[offset:(offset + batch_size), :]
feed_dict = {tf_train_dataset : batch_data,
tf_train_labels : batch_labels}
_, l, predictions = session.run([optimizer, loss, train_prediction],
Validation accuracy: 86.2%
Test accuracy: 86.49% 

Conclusion

In this post, we understood the meaning of softmax regression, how it can be used, and its implementation using TensorFlow library.