Appunti - Artificial neural network and deep learning

Notes artificial neural network and deep learning

Contents

• 1. Machine Learning vs Deep Learning 4
• 2. From Perceptron to Feed Forward Neural Network 4
• 2.1 Hebbian Learning 4
• 2.2 Feed Forward Neural Network 4
• 2.2.1 Output Layer in Regression and Classification 5
• 2.2.2. Loss Functions 6
• 2.2.3 Optimization and Learning 6
• 3. Neural Network Training 9
• 3.1 Prevent Neural Network Overfitting 9
• 3.2 Tips and Trick in NN 9
• 3.2.1 Better Activation Function. 9
• 3.2.2 Weights initialization 10
• 3.2.3 Batch Normalization 11
• 3.2.4 Gradient Descent trick 11
• 3.2.5 Data Preprocessing (for Image Recognition) 11
• 3.2.6 Pick The correct Learning Rate 12
• 4. Image Classification 13
• 4.1 Local (Spatial) Transformation 13
• 4.2 Linear Classifier 13
• 4.3 Image Classification Problem 14
• 5. Convolutional Neural Network 15
• 5.1 Convolutional Neural Network 15
• 5.2 Convolutional Layer 16
• 5.3 Other Layers 16
• 5.4 Dense Layers 17
• 5.5 Useful Formulas 18
• 6. CNN Parameter and Training 19
• 6.1 Training a CNN 19
• 6.2 Data Scarcity 20
• 6.2.1. Data Augmentation 20
• 6.2.2 Transfer Learning 20
• 7. Famous CNN Architectures 21
• 7.1 AlexNet 21
• 7.2 VGG16 21
• 7.3 Network In Network 21
• 7.4 GoogLeNet - Inception Module 22
• 7.5 ResNet 22
• 7.6 DenseNet 23
• 7.7. CNN Visualization 24
• 8. CNN For Semantic Segmentation 25
• 8.1 Data Pre-Processing and Batch Normalization 25
• 8.2 Visual Recognition problems 25
• 8.3 Fully convolutional Neural Network (FCNN) 25
• 8.3.1 Semantic Segmentation with FCNN 26
• 8.3.2 Training a FCNN 27
• 8.4. U-Net 27
• 9. GAP, Localization and CNN explanation 28
• 9.1 Localization 28
• 9.1.1. Weakly Supervised Localization and CAM 28
• 10. CNN for Object Detection 30
• 10.1 R-CNN Paper 30
• 10.2 Fast R-CNN 30
• 10.3 Faster R-CNN 31
• 10.4 YOLO 31
• 10.5 Instance Segmentation 32
• 10.6 Metric Learning 32
• 10.6.1 Siamese Networks 32
• 11. Auto Encoders and Generative Adversarial network 34
• 11.1 Autoencoders 34
• 11.2 Generative Models 34
• 11.2.1 Generative Adversarial Networks (GAN) 34
• 12. Recurrent Neural Network 36
• 12.1 Models with memory 36
• 12.1.1 Recurrent Neural Network (RNN) 36
• 12.1.2. Long-Short-Term Memories (LSTM) 37
• 12.2. Seq2Seq Model 38
• 13. Word Embedding 39
• 13.1 Some Good Model 39
• 13.2 Loss Function (extra) 40
• 14. Beyond Seq2Seq Architectures 41
• 14.1 Enchanted Seq2Seq 42

1. Machine learning vs deep learning

Deep learning is a subset of machine learning, which is essentially a neural network with three or more layers. These neural networks attempt to simulate the behavior of the human brain-albeit far from matching its ability-allowing it to "learn" from large amounts of data.

Deep learning distinguishes itself from classical machine learning by the type of data that it works with and the methods in which it learns. Deep learning eliminates some of data pre-processing that is typically involved with machine learning. These algorithms can ingest and process unstructured data.

In machine learning, the hierarchy of features is established manually by a human expert, instead Deep Learning can compute it by itself using various levels of cascaded non-linear units to perform feature extraction and transformation tasks.

2. From perceptron to feed forward neural network

Perceptron: a computational model based on the brain. That means: distributed, redundant and Parallel.

Perceptron maps input\output following a function f : f(x) = X(<w,x> + b) where: x = input, w=features and b= bias (like ML).

Different functions can be taken as X(), most common is line function. Perceptron is a linear classifier so to work we have to assume the input space is linearly separable.

2.1 Hebbian learning

“The strength of a synapse increases according to the simultaneous activation of the relative input and the desired target”.

Hebbian learning is a rule for adjusting the strength, or weight, of connections between neurons in an artificial neural network. can be summarized by the following parameter rules: ki k where: η = learning rate, x = input at time k, t = output at time k, w= parameter.

The most famous example of hebbian learning is perceptron (remember ML) where:

• We start from a random (not 0) initialization
• We pick a learning rate according to the kind of problem
• Cycle through the records by fixing those which are not correct (using the sign function as check)

2.2 Feed forward neural network

The Universal Approximation Theorem states that A single hidden layer feedforward neural network with S shaped activation functions can approximate any measurable function to any desired degree of accuracy on a compact set. (But sometimes this could be computational unfeasible).

The activation function of a neuron takes in the output signal from the previous cell and converts it into some form that can be taken as input to the next cell.

A Neural Network is a Non Linear model characterized by the number of neurons, activation functions and the value of the weights. We assume also:

• Activation function must be differentiable to train the network
• Layers are connected through weights W l jl l-1 l
• The output of a network depends only on the previous layer h = {h (h ,W )} (no input memory)

2.2.1 Output layer in regression and classification

• In Regression the output spans the whole R domain: so we use a linear actuator function for the output neuron because we need an infinite space output. Should not be used in hidden-layer because it can be always summed up in a single neuron composing the linear activations
• In Classification with 2 classes we chose according to their coding:
• [-1,1] → Tanh Output Activator (approximation of sign())
• [0,1] → Sigmoid Output Activator

These functions are also used in hidden layer to add non-linearity to the deep learning model, problem of vanishing gradient is present for both.

• When dealing with multiple classes (K) use as many neurons as classes (one hot encoding) with a softmax unit as activation function.

This function normalize all the vector that receive as an input and set one as “soft max” and the sum among all the output elements is 1.

We have 2 major classifiers: SVM and Softmax. Unlike the SVM which treats the outputs f(xi,W) as (uncalibrated and possibly difficult to interpret) scores for each class, the Softmax classifier gives a slightly more intuitive output (normalized class probabilities) and also has a probabilistic interpretation that we will describe shortly.

2.2.2. Loss functions

A loss function, also known as a cost function, is a mathematical function that measures the difference between the predicted output and the actual output of a machine learning model. Different types of loss functions are used depending on the type of problem being solved:

• Mean Squared Error (MSE): MSE is a common loss function used for regression tasks, where the goal is to predict a continuous value. It measures the average squared difference between the predicted value and the true value.
• Categorical Cross-Entropy (CCE): CCE is a common loss function used for multi-class classification tasks, where the goal is to predict one of several possible classes. It measures the difference between the predicted class probability distribution and the true class probability distribution.
• Binary Cross-Entropy (BCE): BCE is similar to CCE, but it is used for binary classification tasks, where the goal is to predict one of two possible classes.

2.2.3 Optimization and learning

Gradient Descent Technique - Backpropagation (GD)

In Parametric model we want to reach an output function y(x | θ) t , for this we have to minimize the ～n n error function (typically sum of square error): 2 () = ∑( − ( |)) Closed-form solution are practically never available so we can use iterative solution.

To avoid Local Minima we can use momentum, that increase the movements of the gradient in the most useful direction and decrease oscillation based on the past values of the gradient:

We have some variation in gradient descent technique:

• Batch Gradient Descent: group input and change the weight after the computation of a group
• Stochastic Gradient Descent: Use one sample at a time, online learning. Stochastic because the sample is chosen randomly
• Mini-Batch gradient descent: use a subset of the sample, good tradeoff between variance and computational cost

BackPropagation and Chain Rule

Backpropagation is a way to perform gradient descent exploiting the Chain Rule and update weights parallel, locally, and require just 2 passes:

• Forward pass: In which we compute in middle layer-neuron: the output value g(x) and the local gradient of its output with respect to its inputs(g’(x)).
• Backward Pass: perform the update using all the intermediate values computed before. Chain rule says that the node should take that gradient and multiply it into every gradient it normally computes for all of its inputs

The key idea is that the error at the output layer can be propagated backwards through the layers of the network to adjust the weights of each layer in such a way that the overall error of the network is minimized.

Keeping in mind that the output of each neuron in the network is a function of the inputs to that neuron and the weights of the connections between the neurons and to calculate the gradient of the error with respect to the weights of a neuron, we need to compute the derivative of the error with respect to the output of that neuron, and then the derivative of the output with respect to the weights.

After this premise, is trivially to understand the backpropagation steps:

• 1. Feed the input data into the network and compute the predicted output.
• 2. Calculate the error between the predicted output and the actual output.
• 3. Propagate the error back through the network by computing the gradient of the error with respect to the weights of the neurons.
• 4. Update the weights of the neurons by subtracting the gradient of the error from the current weights. This is done using an optimization algorithm such as gradient descent.

Maximum Likelihood Estimation (MLE)

If we have samples from an i.i.d distribution we can choose parameters which maximize data probability.

Let θ a vector of parameter, find the MLE for θ is done by:

• Write the Likelihood L=P(data|θ) for the data [and l= Log(L)] MLE
• Find the extreme points of both functions and pick the max of l θ . To find it we can use:
• Analytical Technique (solve the equation)
• Optimization Technique (Lagrange Multiplier)
• Numerical Technique (Gradient Descent)

Note that the distribution used for regressionand classification is different to fit better the features of the model.

MLE is used to find the set of weights that maximizes the likelihood of the observed output given the input data and the model architecture. The likelihood is a function of the model parameters, and it is defined as the probability of observing the data given the parameters.

MLE and GD are related in the sense that the optimization problem solved by MLE and SGD are equivalent, in the sense that the optimal solution found by SGD is the solution that maximizes the likelihood function.

3. Neural network training

Training error is not a good indicator of performance on future data, we need to test on an independent new test set. This Test set must be meaningful regarding the themes: it has to maintain data distribution in regression and class distribution in classification. We define:

• Training Set: data used to learn model parameter
• Test Set: data used to perform final model assessment
• Validation set: data used to perform model selection (hyper parameter tuning)

Cross-Validation is the use of the training database to both train the model (fitting + model selection) and estimate its error on new data, splitting each time in a different way. We have different version:

• When lots of data are available we use a Hold Out set and perform validation (no cross-val)
• When having few data we use Leave-One-Out Cross Validation
• K-Fold Cross Validation is a good trade off between computational cost and effectiveness

3.1 Prevent neural network overfitting

We have different methods to prevent overfitting of a neural network. The most relevant regularization techniques are:

• Hold Out some training Data
• Early Stopping: Stop training when validation error increase
• Cross-Validation and Hyperparameters Tuning at (hyper) parameter layer
• Weight Decay: Limiting Overfitting by weight regularization. We add some constraints of weight and prior distribution reducing model freedom. (i.e. Ridge Regression or Lasso)
• Dropout: Limiting Overfitting by Stochastic Regularization. By turning off randomly some neurons we force neurons to be “balanced”, this because near layer cannot fix overfitting of a neuron when turned off (co-adaptation). This method trains weaker classified that should be averaged to have the “real classifier”

3.2 Tips and trick in NN

3.2.1 Better activation function.

The activation functions shown since now have one problem: the gradient can tend to 0. This leads to have a slower learning phase and trouble with backpropagation (during the backward phase).

Vanishing gradient descent is an issue that arises when training a model with stochastic gradient descent combined with the activation functions that could be taken to 0. When output is near 0 the gradient backpropagation tends to the same value. This entails a stop in the learning process locking the weight update.

When the output is too high instead (unbounded) we have the opposite problem, called exploding gradient.

A famous activation function that fix this problem is the rectified Linear Unit.

Rectified Linear Unit has a peculiar feature: its gradient do not tend to 0 so we have a faster SGD converge and efficient gradient propagation. But has also some potential disadvantages:

• Non Differentiable at 0
• Non zero centered output
• Unbounded (explosive gradient problem)
• Dying Neuron: Some neurons can be pushed in a state with gradient equal to 0 for essentially all inputs. This can be avoided modifying the left part of the function

3.2.2 Weights initialization

Weight initialization can influence the learning process. In deep network:

• If weights is too small, then gradient shrinks as it passes through each layer, slow convergence
• If weights is too large, gradients granularity is too big for a good optimization phase

Neural network weights should be initialized in order not to increase nor decrease (on average) the magnitude of the gradients.

Xavier Initialization propose to initialize based on the fact that variance of output is the variance of the input but scaled by I*var(w ) (i= number of components of the input).i

The idea behind Xavier initialization is to initialize the weights of the neurons such that the variance of the input is the same as the variance of the output. These techniques take into account the amount of neurons in input/output. The idea is that if we have many inputs, then the variance of the value that enters an output neuron will become higher, and vice versa if we have few inputs.

To do so we have to initialize the weight w N(0,1*(2)/n ).∼ in

Glorot Initialization for similar reasoning arrive to w N(0,2/(n +n ))∼ in out.

Note that another good weight initialization can be given by transfer learning that can help to mitigate the vanishing gradient problem, reducing the amount of training required to achieve good performance, and providing a good initialization point that can help to regularize the network.

3.2.3 Batch normalization

Networks converge faster if inputs have been whitened (0 means, unit variance) and are uncorrelated to account for covariate shift. Batch Normalization layer Forces activations to take values on a unit Gaussian at the beginning of the training.

Batch Normalization is a technique that normalizes the activations of the previous layer for each mini-batch. It helps to improve the stability and speed of the training process, and it also helps to prevent overfitting by reducing the internal covariate shift.

Note that meanwhile the train set is normalized using its own properties, the test set should be normalized using the same parameter or it should not be normalized (we cannot see its data to normalize it) so Batch normalization layer has different behavior at Test \ Training time.

A linear function is applied after normalization to spread the data in a specific range and let the model learn the best “range” of data for the problem.

Batch normalization layer has different advantages:

• Improve gradient flow through the network
• Allow using higher learning rates (faster learning)
• Reduce the strong dependence on weights initialization
• Act as a form of regularization slightly reducing the need for dropout

In practice:

• Each unit’s pre-activation is normalized (mean subtraction, stddev division)
• During training, mean and stddev are computed for each minibatch
• Backpropagation takes into account normalization
• At test time, global mean / stddev are used (global statistics are estimated using training data)

Parameters: [ ɣ weights, weights, moving_mean(non-trainable), moving_variance(non-trainable)] o

• All Parameters: N of filter *4o
• Non-Trainable Param: N of filter *2

3.2.4 Gradient descent momentum trick

• Nesterov Accelerated Gradient: make a jump as momentum, then adjust
• Gradient magnitudes vary across layers

3.2.5 Pick the correct learning rate

The learning rate is a hyperparameter that controls the step size at which the optimizer makes updates to the model's weights during training. Using the correct learning rate is fundamental for good exploitation of gradient descent.

• A small learning rate means that the optimizer makes smaller updates to the weights, which can result in slower convergence but can also help to prevent overshooting the optimal solution.
• A larger learning rate means that the optimizer makes larger updates to the weights, which can result in faster convergence but can also make the optimizer more likely to overshoot the optimal solution and can cause sometimes the non-converg