Appunti Machine Learning

Appunti Machine Learning

Prof.: Tommaso Di Noia

A.A.: 2021/2022

INDICE

CAPITOLO 1…………………………………………………………………………………5

Machine Learning Introduction

Univariate Linear Regression

Gradient Descent

Batch Gradient Descent

Stochastic Gradient Descent

Mini Batch

Riassunto varianti di Gradient Descents

Min-Max Normalization and Z-score Normalization

Linear Regression Probabilistic Interpretation

Likelihood Function

CAPITOLO 2……………………………………………………………………………….17

Classification

Logistic Regression Probabilistic Interpretation

Cross-Entropy Error Function

Multiclass Classification: One vs All

CAPITOLO 3……………………………………………………………………………….22

Fitting Problem: Bias and Variance

Generalization Error (GER)

L2 and L1 Regularization

CAPITOLO 4………………………………………………………………………………..30

Non Linear Hypothesis: Neural Networks

NN Cost Function for Classification and Regression

Backpropagation Algorithm

Zero Initialization and Random Initializations

Activation Functions

CAPITOLO 5……………………………………………………………………..…………38

Regression Tree

How to build a Regression Tree

Regression Tree with Multiple Features

CAPITOLO 6……………………………………………………………………………….40

Classification Tree

Entropy and Information Gain

CAPITOLO 7………………………………………………………………………………..42

How to choose the best Classification or Regression Tree

Categorical and Numerical values in Classification Trees

Missing Values

Pruning Regression Trees

CAPITOLO 8………………………………………………………………………………..45

Random Forest

Bootstraped Datasets and Bagging Technique

CAPITOLO 9……………………………………………………………………………….47

How to build a ML System

Data Analysis and Preprocessing

Outlier Removal Example (Boxplot)

Normalization

Feature Selection

Evaluation of Hypothesis

Cross Validation

Hold-out Cross Validation

K-fold Cross Validation

Random Subsampling

How to choose next hypothesis

Lambda value

Learning Curves

High Variance Diagnosis

How to measure if a ML System works well

MAE, MSE, RMSE

Confusion Matrix

ROC Curve and AUC

How to measure if the results are significant

Paired t-test

2

R

CAPITOLO 10………………………………………………………………………………60

Support Vector Machines (SVM)

Non-linear separable data: Slack variables and Error Tolerance

Cover’s Theorem

Parameter’s Tuning

Advantages of SVM

CAPITOLO 11………………………………………………………………………………66

Recommender Systems

Collaborative RS

Knowledge-based RS

Collaborative Filtering

Collaboration Filtering Drawbacks

CAPITOLO 12………………………………………………………………………………69

Unsupervised Learning

K-Means

K-Medoids

Gaussian Mixture Models

Expectation-Maximization (EM) Algorithm

Choosing the value of K

Elbow Method

Kullback-Liebler Method

Akaike Information Criterion (AIC)

Bayesian Information Criterion (BIC)

Deviance Information Criterion (DIC)

Silhouette Coefficient

Hierarchical Clustering

DBSCAN

HDBSCAN

CAPITOLO 13………………………………………………………………………………84

The curse of dimensionality

Dimensionality Reduction

PCA (Principal Component Analysis)

SVD (Single Value Decomposition)

CAPITOLO 14………………………………………………………………………………90

Indipendent Component Analysis (ICA)

Cocktail Party Problem

ICA Ambiguities

Kernel PCA

MACHUNE LEARNiNG

LEHlONE /2I0-202

epae o duputiou to diusnd duhot HL Loun u_t

I

So EXPERIENCEo uboto oud tus

uuht oCoRRolepefame-to

Expones eo DT FRoh THE PAS

LEARNING nouiutk epeioudEot aee to

wht ppeot tce peonmouteP)lamenuni

eoup eoo.ChA

SUrERUISED EARNTNGLd a eden.o ka hond on lnduu

2ONSOfERUSED LEARNING-W o wodd to oldt| Poa commc

eekahe ckuleg)

3RENfRCEHENT LEARNINC- eiue etion boul oue

Cou aipu o wotd) o1 gnoin t a [behowtolue lof ow

TE TKIAL ERKGR n22

4 ECOHMENNER SHS T o eloetiooupuz

Aude HL dobetiaCd be

DISCKIHNATIVE HobELS= Tue madls oe Tonud diea a [RAWMG SG

d tla'h1oboh kdtie maolatet rocluca ltla opul oud

al pixel pnio

adpa4de Toke Th tute the potan

LINEAK KEGRESSiON t ovobe (UNiVARIATT

Nube otno eXouplhnioie dlitoret

- INPUr ocbe FEATURES

TARGE

E

Joupbé

OUTUT

Dstite th o l

OLINERk. 6,x

FUNeTON

I n HMETERS

PA

Y

th Jeedut rNokHALT

t senuet)

bet we t adtus pow t s l o i t u o tepopu)

LE 2ONE 14-10-2021

So, oalupadk uud, To HINIKeE oudr CosT FUNCTLON Co

FX poite(xb)ohdmod dIn

J

w.etipu 25 bu uip CNTOUR PLOT

OT

toXimiu

D olop laue Hd dipeusug dlal doda ue

ca ldmaue hok a

CoT ki epuot

GOAL: a e. e)D osFuicro t

oytuceext

Tue

ehanplu

e a p hiou

ole SweLE oholoa

) CoNvERGENCE

rATEel uucd

UPDATE SIHOLTANEOUS

ou

i2

2o t

ta

eusudaki.oo od bu tpozwny uolutmpO oud teup 4)lhan

youupd SiMULTANEOUSLYta Joniobl , e

HU H vaaoha euoRARAMETE

R

okA UalutoHPER PARA METER

t ek0gu tuGRADIENT DESCENT ALGO

Q

Ouieecuunouo t (DERIUATVE

EARNING RATE 2

d

mcdk Toknde

- eduu deoiadle ceald DrERGE

RECAESSIN pkcth

1PRTLALDERIUATIBES 2wopet to ao

dto coupide

e 20 tlue Counpule t a PCATES

LINEARKEGRESSION= luLTIPLE FEATURES

o Cuen ue e t x 4 e eo eume tle boviaotomeda

Nou odit deau oleo A beddBATCH GRADIENT

Rocdouy ke poonzteag

DESCENT PRt che ConuERG: { |

ahose eopleapole o 5UER SAGLE PoNT

a o k Ol cilhd $ToCHASTLC GRADIENT

DESCENT wedat

dote pol

te

CCs bd actic dbK

hale oCte

Tt 2 o d QH2 2eoe 2-2AG PRoBLEH

BATCH ure la Cal

aud STOCHASTIC7

t INIBATCH

od

Vauoses tu oldtlioda

t oluti fu o oUNTIL CoUVERGENCE

Ue

ca Mop tu dponitua'a teiiaun byua acSTOPFING CRli

HAX TTERHTIONa ue

ABioLITE TOLERAN oalop e uou uae_etou Ualue e

KELAWE TOLERANCE ou ocod)T ouplu ou tu etPetia

GRADIENT NORH TOLERANCE= lop whe a onm Prod.al

Ulu aDte u i ood o appnokiug tla ekoliawn, uie u

to we dhun mmaduW

o tlk POLYNOHIAL KEGRESiCA

+6,+6x+6 x

E 4RADENTDESCeNT (Upte

-de

deot

de

DEKCENT

DIENT

GRA

BATEH

Aasopmeulo e udttep Lako e t Lut

ouotopdllepoca

2STOCASTe GRADlEAT NESCENT(SG)

TeRALONE

BHI-BATCH GRaDieNT DE SCENT

U2uo abeuoeu x<m

TERATAion

BATCHGRAbiErSGD

ESÇENT lolouc

poh

dt

ateaupitoeiaddk

C

Baeh u c t o

dnte

betaededla

aue

dotoet pelmotanaeo

piud

oepuiuda

TAce

SGD

Phe peWe TKADIENT SCENTo poluoaR on

wuiuetido dau pounaleno

E2ONE B-910- 2o21

e& d CGRADIENT ESCENT f UULTFE UARABLES

FEATORE SCALING owl tla u podem ed

eatuie CopaniTNE Aea XF-

Cado (HIN-HAX NORMALLATiON t hen D

o ho to et up the add max Ualue

tApnaadk ot veliPud-al oGTuERS)

U

L tuRANNCplhoo4ou Cou uoue

X Xrnecie(

deuod

a en do SCORE NoRHALIEAT ONLA0ou ees o 2ere

by Ta-daid dlou otie t ueir tobu to mcrnoe you

ecoueuo 2teeoebeaateieo

ou- oludatieil kb We-Kx-Xo

-)x

Caupilotiouoepewai

ve):O

Lskc DepeudeFeR

V

ud tle mupmumT sol

ekta ptoded n muleUe:

LoBABILISTI IKTERARETAUDN Liuare

h Epadluai. Hlt apbioxiuol 2ulota baRA

eatce ueciozaod oulpsN

tOUTRUT Co obaled Qa

CPprxialo

o we ea Du KRBABIL TYDASTRIBUTiON tu eno NoRHAL

ALSTRIBUTloN 0 m o Otud ruona 27

Ae Hu Aucdil b

thol Hu2elotio:

zknte

Cai

a 22

e

27

eoua ELIHO EUNCTioN

poronTeuathive u upl, tu ta op

2 . e h 20unuoCoae Htehoads

od Huoromlla

o emoeoudun onauapo o tu coces NDIPENDENT Hal

p2oCty o tTod eveatb Ha tpAeioe Ha pobohi

O U Vy 2

2

tLot ttu opat W atuu te pLtlou DMAXIHOH

Lfe LIkELIHD

CRERoN

LIkELIHOo HCOB

la_

om

uewatoMAXK u ave

t o ax io_otbcae a

uUida a CoNSTA NTH Uoegec

tu

tload

tuot,

toaxuite

i puo

nfou

ietm

tox mox [e)= mu

miw kelkad

Li uaenci dto0adameu tuaKeood o

b2ocobeiatie leapotoAic

do ed

utla iar pwionmiow ouptha p2obliu Gatou M-Vo

CLASIFCATION

k t dlponcth e e iou GiSTe REGRESSiONE atoA

unsloke beconta uatolo ocktiou)

El poulnno hpou

Exelonetio Duuusakiouo deezt (Gea/lo

umor

W ca hoe d

uken an TRESHOL D

FUNCTLON orned C3TTC UNCTIONJ

DEtisid

BouNDAR

NoA RORABILISTIC NTERPRETATUOAlcuur aturo

eou

boloupiug elon 4a ledoupaBUT hoe ee 3 pr

belaugiupto clo Oo

, 2 I koe

Clon) oud 0,rob seloliuo

pottye elo

LiKELIHOD FONCTlON

He e le al ulueaD oud e hoveBERNOULLL

AISTRIBUTION

00e ve duodl Auneu Coupoc Woy

eolclate t i

Ate houiu coupo mloo eoeupde

Colupo.coto Likellocduct.ou

w

malpconoie To ta Colotwcr

Sonc e Comoie acu MAX(KCOHe HAIHIATOANI ncbdu

useluk_a mermmeCiadtipa cud tls uou

uilluude.aod atea)

S-4 (

CROS ETRORYRRR FUKIeT ON

Je)

-()th

T he

hea h

hel=4E

GRADIENT DESENT

DENCZNT

KatusubeH poe thogh te OPDATE C THE WE1GHTSUa ey

uup ormla de

we Eauae Ha duotue c FUNCTONwEt eU hoe

1 8

pooler

AHla eud h.)4he )

AH 0Dd uau aje

ONE NS ALL THULTCLASS CLARRIFTCAT PM

Eawedola hour c a 2 clo00 to ekor

tuo top o k , taiecda, Hdb

Westhen Sun Cloud koiu

Houoouo HA teeobleue

cd

RST od

F ALU i oa Co ubstttuk

thne ut aunloz 234,

tla uou ona

do te orquar oud eroe t la eud yo houe

toue olutii oua). M23)

EIONE 4 240-2024

Flauds On Jpplo uKepul Tt

CEONE5 26-10-2021|

FITTING PRoBLEM

tonl dota opl Renria odelau

Kuoha), oicoios Agmdl f i c Knuralo)

hoHue doa dekoue ue eTAN

Co ASLLTIC APPRDACH L

t u mcdul H EATURES

CoMnp LOU

MSE(Me

(He Squar EnorCud bka olereltent eak

HSE ojat tCRADIEKT DESCaU

ALGO o t

liobA H eoupd Hla beat uoluea tu poromider

I I _core o e oup o eoupdle

c d i l t

HTPERPLANE LuK cauir a kypepanle

tlapom bUalunr co pradek codbol& Lopate R

pokel Uoluin

CRT

Ue da htuaed tTICLAST CLASS IFIER

LoGISTIC REGR. Atk CNE VS AD paeeh /ue

tAA MOLTIPLE BINARY QLASIFIERS Oud we Ca Loie Yu

duAtauid elomiclic

TRAININe $ET

EST SET

KOHL H RALW ING STo o a mcdl, p Htla 2adhed

o o ue TEST Sai tcoupiie HOD

ot e tod.an HOU TO TUNE ou oljoLefumwn 15 )

e

dale(t hale oend.ted mcclul) tLL FTING PAdB(E

e Cootoouplpeaoniol madlel

t diko atopproxi d by Clueon beluoulouz

Tcd 0NDERFITS dot medel predea

abuiku Xctu o HGH ERROR) HIGH BAS

u jolds oouokud buHa mcdil

o luole a Ervn Veyclon 0 WT

t l TRAINING DATA

Le modulOVERFIT dido

e OlER

(ia HIGH VARIANCE

SPECIAL (LED tnoi doto)

dUNGER FITTAG dluia poloed

o e edd to upotoo coupAK

UNDERF ITN =H wodil otm TRAING TEIT dote

OVERFITOUNG eccou u adtl beius to HaHORIEE Thoung dalo 20u

tanLEARNING to panbhize)

IAS Dd VARIANCE

BIAS Sate-okic devidaa u timokor