Appunti di Statistical Learning Theory (STALT)

Review

Gaussian distribution

Mean

Variance

Standardization

Statistical learning

• Regression function
• MSE decomposition
• Nearest neighbor averaging
• Linear model
• Model accuracy
• Training/Test MSE
• Bias-variance trade-off
• Bias
• Classification problems
• Conditional class probability
• Misclassification error rate
• Nearest Neighbors

Review 2

• Covariance
• Correlation
• Sample moments
  • Mean
  • Variance

Linear regression

• Estimation with least squares
• Residual
• RSS
• Accuracy of LS
• Standard error
• Confidence interval
• Hypothesis testing
• Regression basics review
• LS criterion
• Matrix formulation
• Confidence intervals
  • σ² is known
  • General case
• Comparing nested models
  • Fisher's F
  • RSE
  • R²
• Multiple linear regression
  • Forward selection
  • Backward selection
  • Qualitative predictors
  • Interactions

Classification

• Using Linear Regression
• Logistic Regression
  • Probability
  • Logit
  • Maximum likelihood
  • Confounding
  • Case-control sampling
• Multinomial Regression
• Discriminant analysis
  • Probability
  • LDA
  • QDA
  • Discriminant score
  • Estimated parameters
  • LDA with p > 1
  • Discriminant score
  • Fisher's discriminant plot
• From g(x) to probabilities
• Types of errors
• ROC plot
• Quadratic DA
• Naive Bayes

Resampling

• Validation set approach
• K-fold cross validation
  • CV
  • LOOCV
• Classification
• Lots of predictors
• Bootstrap
  • Estimating prediction error

Model selection

• Linear model selection
• Feature selection
• Subset selection
  • Best subset selection
  • Stepwise selection
    • Forward
    • Backwards
  • Estimating best error
  • C_p
  • AIC
  • BIC
  • Adjusted R²
  • Validation/CV
• One-standard-error rule
• Shrinkage methods
  • Ridge regression
  • Lasso
• Dimension reduction methods
  • Principal Components Analysis
  • Partial Least Squares

Nonlinear models

• Polynomial Regression
• Step functions
• Piecewise polynomials
• Linear splines
• Cubic splines
  • Natural cubic splines
  • Knot placement
• Smoothing splines
• Local Regression
• Generalized Additive Models

Trees

• Regression problems
  • Tree building
  • Recursive binary splitting
  • Pruning
  • Cost complexity pruning
• Classification problems
  • Gini index
  • Deviance
• Bagging
  • Out-of-bag error estimation
• Boosting
  • B
  • λ
  • X

Support vector machines

• Maximal margin classifier
• Non-separable data
• Feature expansion
• Kernels

Unsupervised learning

• Principal Component Analysis
  • Proportion of Variance Explained
• K-means clustering