Appunti Metodi Matematici

Regularization

Ridge smaller norm so I have less freedom to choose the parameter.

Lasso sets some parameter so 0.

In simple terms, regularization is tuning or selecting the preferred level of model complexity so your models are better at predicting

(generalizing). If you don't do this your models may be too complex and overfit or too simple and underfit, either way giving poor predictions.

To regularize you need 2 things:

1. A way of testing how good your models are at prediction, for example using cross-validation or a set of validation data (you can't use the

fitting error for this).

2. A tuning parameter which lets you change the complexity or smoothness of the model (because shrinks the coefficient), or a selection

of models of differing complexity/smoothness.

Basically you adjust the complexity parameter (or change the model) and find the value which gives the best model predictions. (Infatti che e'

un hyperparameter, viene ricavato mediante per esempio cross validation).

Note that the optimized regularization error will not be an accurate estimate of the overall prediction error so after regularization you will finally

have to use an additional validation dataset or perform some additional statistical analysis to get an unbiased prediction error. (quindi divisione

in Training data, Validation data and Test data).

An alternative to using (cross-)validation testing is to use Bayesian Priors or other methods to penalize complexity or non-smoothness, but

these require more statistical sophistication and knowledge of the problem and model features.

Alla fine pero' you penalize your loss function, abbiamo visto che training error e' minore, but avoid overfitting.

Generalization Error and Training Error

true risk = Generalization Error

→