Appunti Lezione Machine Learning

Regularization

For each output t we have

k T 1 Tŵ = ( ) t k

where t is an N -dimensional column vector. Thus the solution decouples between the di↵erent outputs,kand we need only to compute a single pseudo-inverse matrix, which is shared by all of the vectors w .k4 It has variance on the diagonal and covariance on the rest of the matrixP 2(y(x ,w) t )5 n nnMean Squared Error: N3.5.

REGULARIZATION 153.5

Regularization3.5.1 Under-fitting vs Over-fitting The test error is a measure of how well we aredoing in predicting the values of t for a new dataA model with low complexity is usually not capable of fitting the data observation of xand so representing appropriately the true model (under-fitting).A model with high complexity (e.g. high-order polynomials) givesexcellent fit over the training data, but a poor representation of thetrue function (over-fitting).

The goal is to achieve a good generalization and we obtain some quan-titative insight about that by reserving a part of the set for

testing.By calculating the error also for the test set it is possible to evaluate Here thetraining errorthe generalization. The root-mean-square error (RMS) is used: goes to 0r because it⇤2 RSS( ŵ)E = used all theRM S N dof

Figure 3.3: E on trainingRM SIn which the division by N allows to compare sets of di↵erent sizes and test setsand the square root ensures that the error is based on the same scaleas the target variable.

As we can see from Figure 3.3, from a specific value of the model complexity, even if the training errorcontinues decreasing, ato some point the test error starts to increase. That’s because there is over-fittingand the model has huge values for the parameters in order to try to fit all the points of the training set,as showed in Figure 3.4.

Figure 3.4: Table of the coefficients w* for polynomials of various order.

For a given model complexity, the over-fitting problem becomes less severe as the size of the data setincreases. In other words, the larger

The data set, the more complex (and flexible) the model we can afford to fit the data.

3.5.2 Ridge regression

Regularization involves adding a penalty term to the error function to discourage the coefficients from reaching large values and so, to make the curve smoother.

L(w) = L (w) + L (w)D W

Where L (w) is the empirical loss on data (e.g. RSS) and L (w) is the measure of the size of the parameters (the model complexity).

12 12T 22||w||w w = we obtain the ridge regression (or weight decay)

By taking L (w) = W NX1 T 2 22||w||

L(w) = (t w (x )) +i i2 2i=1

Note that if there are a lot of samples, the regularization is useless and it may even worsen the performance of the model.

The loss function is still quadratic in w: This means that we are able to find a unique solution T 1 Tŵ = ( I + ) t

As lambda increases, var decreases, bias increases ridge

As lambda decreases, var increases, bias decreases

16 CHAPTER 3. LINEAR REGRESSION

Since the matrix (phi^T)phi+lambdaI is semi-definite positive and

has the property that if is sufficiently large, some of the coefficients are driven to zero, leading to a sparse model. For this reason it can be also used for feature selection, excluding the ones that have coefficient equal to zero.

Generally LASSO performs better in a setting where a relatively small number of predictors have substantial coefficients, and the remaining have coefficients near 0. Ridge performs better when the response is a function of many predictors, all with coefficients of roughly equal size. Since they are not known a priori we use CROSSVALIDATION.

3.6 Bayesian Linear Regression

3.6.1 Bayesian approach

In the previous examples of linear regression we have viewed probabilities in terms of the frequencies of random, repeatable events (frequentist approach). In some cases it's not possible to repeat multiple times some events to define a notion of probability. In such circumstances, the solution is to quantify our expression of uncertainty and make precise revisions of

the uncertainty after acquiring new evidence. This is the Bayesian approach, which can be divided in steps:

1. Formulate the knowledge about the world in a probabilistic way:
   • Define the model that expresses our knowledge quantitatively.
   • The model will have some unknown parameters.
   • Capture our assumptions about unknown parameters by specifying the prior distribution p(w) over those parameters before seeing the data.
2. Observe the data, whose effect is expressed through the conditional probability p(D|w). It can be viewed as a function of the parameter vector (likelihood function).
3. Compute the posterior probability distribution p(w|D) for the parameters, given the observed data. It is the uncertainty in w after is observed.
4. Use the posterior distribution to:
   • Make predictions by averaging over the posterior distribution.
   • Examine/Account for uncertainty in the parameter values.
   • Make decisions by minimizing the expected posterior loss.

Note that

1. real-time learning scenarios or in case of large datasets, because they do not require the whole dataset to be stored or loaded into memory.
2. It is important that the prior and posterior have the same distribution. This fact leads to introducing the concept of conjugate priors: for a given probability distribution, we can seek a prior that is conjugate to the likelihood function, so that the posterior has the same distribution as the prior. For example the prior of a Gaussian is a Gaussian, and the prior of a Beta is a Bernoulli (so it gives another Beta as posterior).
3. data | theta ~ Bernoulli, theta ~ Beta
4. 3.6.3 Predictive Distribution
5. Prediction for a new data point x* (given the training dataset can be done by integrating over the posterior distribution:
6. Z E[p(x*|w,D)p(w|D)dw D)]p(x*|D) = p(x*|w, = p(x | z)=int p(x, y | z)dy = int p(x | y, z)p(y | z)dy
7. which is sometimes called predictive distribution.
8. Note that computing the predictive distribution requires knowledge of the posterior distribution,

which is usually intractable.

3.6.4 Bayesian Linear Regression

In the Bayesian approach the parameters of the model are considered as drawn from some distribution.

Assuming a Gaussian likelihood model, the conjugate prior is Gaussian too

Np(w) = (w|w , S )0 0D,

Given the data the posterior is still Gaussian:

2 2/ N Np(w|t, , ) (w|w , S )N (t| w, I ) = (w|w , S )0 0 N N N

where 0 1B CT tB C1w = S S w +@ AN N 00 2| {z } | {z }

from prior from data when we have a non informative

T1 1 prior the gaussian solution

S = S +0N 2 coincides with the OLS

In a Gaussian distribution the mode coincides with the mean. It follows that w is the MAP estimator.

NIn many cases we may have little idea of what form the distribution should take. We may then seek a

form of prior distribution, called a noninformative prior, which is intended to have as little influence

on the posterior distribution as possible.! 1

In this case the value S and so0T T t1 2 T 1 2 T 1 T 1 T! ) )S 0 + S = ( ) w = ( ) =( ) tN NN 2 22

so, w reduces

to the ML estimator. If w = 0 and S = ⌧ I, then w reduces to the ridge estimate,N 0 0 N2where = 2⌧6 independent and identically distributed18 CHAPTER 3. LINEAR REGRESSION3.6.5 Posterior Predictive DistributionWe are interested in the posterior predictive distribution, which is the distribution over the outputvariable obtained taking into account all the models, each one with its density.probabilityZ of the modelz }| {2 T 2D, N Np(t|x, )= (t|w (x), ) (w|w , S ) dwN NTN 2N= (t|w (x), (x))Nwhere 2 2 T(x) = + (x) S (x)N|{z}N | {z }noise in the uncertainty associatedtarget values with parameter values! 1,In the limit, as N the second term goes to zero.The variance of the predictive distribution arises only from the additive noise governed by parameter .Spans one standard deviationThe predictive uncertainty either side of the mean (reddepends on x and it is line)smallest in the neighborhoodof the data point The level of uncertaintydecreases as more datapoints are observedFigure 3.5: