How to implement a neural network Part 2

This page is part of a 5 (+2) parts tutorial on how to implement a simple neural network model. You can find the links to the rest of the tutorial here:

Logistic regression (classification) ¶

This part will cover:

While the previous tutorial described a very simple one-input-one-output linear regression model, this tutorial will describe a 2-class classification neural network with two input dimensions. This model is known in statistics as the logistic regression model. This network can be represented graphically as:

The notebook starts out with importing the libraries we need:

In [1]:

Define the class distributions ¶

In this example the target classes $t$ will be generated from 2 class distributions: blue ($t=1$) and red ($t=0$). Samples from both classes are sampled from their respective distributions. These samples are plotted in the figure below. Note that $X$ is a $N \times 2$ matrix of individual input samples $\mathbf{x}_i$, and that $\mathbf{t}$ is a corresponding $N \times 1$ vector of target values $t_i$.

In [2]:
# Define and generate the samples
nb_of_samples_per_class = 20  # The number of sample in each class
red_mean = [-1,0]  # The mean of the red class
blue_mean = [1,0]  # The mean of the blue class
std_dev = 1.2  # standard deviation of both classes
# Generate samples from both classes
x_red = np.random.randn(nb_of_samples_per_class, 2) * std_dev + red_mean
x_blue = np.random.randn(nb_of_samples_per_class, 2) * std_dev + blue_mean

# Merge samples in set of input variables x, and corresponding set of output variables t
X = np.vstack((x_red, x_blue))
t = np.vstack((np.zeros((nb_of_samples_per_class,1)), np.ones((nb_of_samples_per_class,1))))

In [3]:

Logistic function and cross-entropy cost function ¶

Logistic function ¶

The goal is to predict the target class $t$ from the input values $x$. The network is defined as having an input $\mathbf{x} = [x_1, x_2]$ which gets transformed by the weights $\mathbf{w} = [w_1, w_2]$ to generate the probability that sample $\mathbf{x}$ belongs to class $t=1$. This probability $P(t=1| \mathbf{x},\mathbf{w})$ is represented by the output $y$ of the network computed as $y = \sigma(\mathbf{x} * \mathbf{w}^T)$. $\sigma$ is the logistic function and is defined as: $$\sigma(z) = \frac{1}{1+e^{-z}}$$

This logistic function and its derivative are explained in detail in intermezzo 1 of this tutorial. The logistic function is implemented below by the  logistic(z)  method.

Cross-entropy cost function ¶

The cost function used to optimize the classification is the cross-entropy error function . And is defined for sample $i$ as:

$$\xi(t_i,y_i) = -t_i log(y_i) - (1-t_i)log(1-y_i)$$

Which will give $\xi(t,y) = - \sum_{i=1}^{n} \left[ t_i log(y_i) + (1-t_i)log(1-y_i) \right]$ if we sum over all $N$ samples.

The explanation and derivative of this cost function are given in detail in intermezzo 1 of this tutorial. The cost function is implemented below by the  cost(y, t)  method, and its output with respect to the parameters $\mathbf{w}$ over all samples $\mathbf{x}$ is plotted in the figure below.

The neural network output is implemented by the  nn(x, w)  method, and the neural network prediction by the  nn_predict(x,w)  method.

In [4]:
# Define the logistic function
def logistic(z):
return 1 / (1 + np.exp(-z))

# Define the neural network function y = 1 / (1 + numpy.exp(-x*w))
def nn(x, w):
return logistic(x.dot(w.T))

# Define the neural network prediction function that only returns
#  1 or 0 depending on the predicted class
def nn_predict(x,w):
return np.around(nn(x,w))

# Define the cost function
def cost(y, t):
return - np.sum(np.multiply(t, np.log(y)) + np.multiply((1-t), np.log(1-y)))

In [5]:

Gradient descent optimization of the cost function ¶

The gradient descent algorithm works by taking the derivative of the cost function $\xi$ with respect to the parameters, and updates the parameters in the direction of the negative gradient .

The parameters $\mathbf{w}$ are updated by taking steps proportional to the negative of the gradient: $\mathbf{w}(k+1) = \mathbf{w}(k) - \Delta \mathbf{w}(k+1)$. $\Delta \mathbf{w}$ is defined as: $\Delta \mathbf{w} = \mu \frac{\partial \xi}{\partial \mathbf{w}}$ with $\mu$ the learning rate.

${\partial \xi_i}/{\partial \mathbf{w}}$, for each sample $i$ is computed as follows:

$$\frac{\partial \xi_i}{\partial \mathbf{w}} = \frac{\partial z_i}{\partial \mathbf{w}} \frac{\partial y_i}{\partial z_i} \frac{\partial \xi_i}{\partial y_i}$$

Where $y_i = \sigma(z_i)$ is the output of the logistic neuron, and $z_i = \mathbf{x}_i * \mathbf{w}^T$ the input to the logistic neuron.

• Intermezzo 1 derived that ${\partial \xi_i}/{\partial y_i}$ can be calculated as:
$$\frac{\partial \xi_i}{\partial y_i} = \frac{y_i - t_i}{y_i (1 - y_i)}$$
• Intermezzo 1 derived that ${\partial y_i}/{\partial z_i}$ can be calculated as:
$$\frac{\partial y_i}{\partial z_i} = y_i (1 - y_i)$$
• ${\partial z_i}/{\partial \mathbf{w}}$ can be calculated as:
$$\frac{\partial z}{\partial \mathbf{w}} = \frac{\partial (\mathbf{x} * \mathbf{w})}{\partial \mathbf{w}} = \mathbf{x}$$

Bringing this together we can write:

$$\frac{\partial \xi_i}{\partial \mathbf{w}} = \frac{\partial z_i}{\partial \mathbf{w}} \frac{\partial y_i}{\partial z_i} \frac{\partial \xi_i}{\partial y_i} = \mathbf{x} * y_i (1 - y_i) * \frac{y_i - t_i}{y_i (1-y_i)} = \mathbf{x} * (y_i-t_i)$$

Notice how this gradient is the same (negating the constant factor) as the gradient of the squared error regression.

So the full update function $\Delta w_j$ for each weight will become

$$\Delta w_j = \mu * \frac{\partial \xi_i}{\partial w_j} = \mu * x_j * (y_i-t_i)$$

In the batch processing, we just add up all the gradients for each sample:

$$\Delta w_j = \mu * \sum_{i=1}^{N} x_{ij} (y_i - t_i)$$

To start out the gradient descent algorithm, you typically start with picking the initial parameters at random and start updating these parameters according to the delta rule with $\Delta w$ until convergence.

The gradient ${\partial \xi}/{\partial \mathbf{w}}$ is implemented by the  gradient(w, x, t)  function. $\Delta \mathbf{w}$ is computed by the  delta_w(w_k, x, t, learning_rate)  .

In [6]:
# define the gradient function.
def gradient(w, x, t):
return (nn(x, w) - t).T * x

# define the update function delta w which returns the
#  delta w for each weight in a vector
def delta_w(w_k, x, t, learning_rate):
return learning_rate * gradient(w_k, x, t)


Gradient descent is run on the example inputs $X$ and targets $\mathbf{t}$ for 10 iterations. The first 3 iterations are shown in the figure below. The blue dots represent the weight parameter values $\mathbf{w}(k)$ at iteration $k$.

In [7]:
# Set the initial weight parameter
w = np.asmatrix([-4, -2])
# Set the learning rate
learning_rate = 0.05

# Start the gradient descent updates and plot the iterations
nb_of_iterations = 10  # Number of gradient descent updates
w_iter = [w]  # List to store the weight values over the iterations
for i in range(nb_of_iterations):
dw = delta_w(w, X, t, learning_rate)  # Get the delta w update
w = w-dw  # Update the weights
w_iter.append(w)  # Store the weights for plotting

In [8]:

Visualization of the trained classifier ¶

The resulting decision boundary of running gradient descent on the example inputs $X$ and targets $\mathbf{t}$ is shown in the figure below. The background color refers to the classification decision of the trained classifier. Note that since this decision plane is linear that not all examples can be classified correctly. Two blue dots will be misclassified as red, and four red spots will be misclassified as blue.

Note that the decision boundary goes through the point $(0,0)$ since we don't have a bias parameter on the logistic output unit.

In [9]:

This post at peterroelants.github.io is generated from an IPython notebook file. Link to the full IPython notebook file