Appunti Machine Learning

Appunti Machine Learning

Prof.: Tommaso Di Noia

A.A.: 2021/2022

Indice

Capitolo 1…………………………………………………………………………………5

• Machine Learning Introduction
• Univariate Linear Regression
• Gradient Descent
• Batch Gradient Descent
• Stochastic Gradient Descent
• Mini Batch
• Riassunto varianti di Gradient Descents
• Min-Max Normalization and Z-score Normalization
• Linear Regression Probabilistic Interpretation
• Likelihood Function

Capitolo 2……………………………………………………………………………….17

• Classification
• Logistic Regression Probabilistic Interpretation
• Cross-Entropy Error Function
• Multiclass Classification: One vs All

Capitolo 3……………………………………………………………………………….22

• Fitting Problem: Bias and Variance
• Generalization Error (GER)
• L2 and L1 Regularization

Capitolo 4………………………………………………………………………………..30

• Non Linear Hypothesis: Neural Networks
• NN Cost Function for Classification and Regression
• Backpropagation Algorithm
• Zero Initialization and Random Initializations
• Activation Functions

Capitolo 5……………………………………………………………………..…………38

• Regression Tree
• How to build a Regression Tree
• Regression Tree with Multiple Features

Capitolo 6……………………………………………………………………………….40

• Classification Tree
• Entropy and Information Gain

Capitolo 7………………………………………………………………………………..42

• How to choose the best Classification or Regression Tree
• Categorical and Numerical values in Classification Trees
• Missing Values
• Pruning Regression Trees

Capitolo 8………………………………………………………………………………..45

• Random Forest
• Bootstraped Datasets and Bagging Technique

Capitolo 9……………………………………………………………………………….47

• How to build a ML System
• Data Analysis and Preprocessing
• Outlier Removal Example (Boxplot)
• Normalization
• Feature Selection
• Evaluation of Hypothesis
• Cross Validation
• Hold-out Cross Validation
• K-fold Cross Validation
• Random Subsampling
• How to choose next hypothesis
• Lambda value
• Learning Curves
• High Variance Diagnosis
• How to measure if a ML System works well
• MAE, MSE, RMSE
• Confusion Matrix
• ROC Curve and AUC
• How to measure if the results are significant
• Paired t-test
• 2R

Capitolo 10………………………………………………………………………………60

• Support Vector Machines (SVM)
• Non-linear separable data: Slack variables and Error Tolerance
• Cover’s Theorem
• Parameter’s Tuning
• Advantages of SVM

Capitolo 11………………………………………………………………………………66

• Recommender Systems
• Collaborative RS
• Knowledge-based RS
• Collaborative Filtering
• Collaboration Filtering Drawbacks

Capitolo 12………………………………………………………………………………69

• Unsupervised Learning
• K-Means
• K-Medoids
• Gaussian Mixture Models
• Expectation-Maximization (EM) Algorithm
• Choosing the value of K
• Elbow Method
• Kullback-Liebler Method
• Akaike Information Criterion (AIC)
• Bayesian Information Criterion (BIC)
• Deviance Information Criterion (DIC)
• Silhouette Coefficient
• Hierarchical Clustering
• DBSCAN
• HDBSCAN

Capitolo 13………………………………………………………………………………84

• The curse of dimensionality
• Dimensionality Reduction
• PCA (Principal Component Analysis)
• SVD (Single Value Decomposition)

Capitolo 14………………………………………………………………………………90

• Indipendent Component Analysis (ICA)
• Cocktail Party Problem
• ICA Ambiguities
• Kernel PCA

Machune learning

LEHlONE /2I0-202epae o duputiou to diusnd duhot HL Loun u_tISo EXPERIENCEo uboto oud tusuuht oCoRRolepefame-toExpones eo DT FRoh THE PASLEARNING nouiutk epeioudEot aee towht ppeot tce peonmouteP)lamenunieoup eoo.

ChASUrERUISED EARNTNG

Ld a eden.o ka hond on lnduu2ONSOfERUSED LEARNING-W o wodd to oldt| Poa commceekahe ckuleg)3RENfRCEHENT LEARNINC- eiue etion boul oueCou aipu o wotd) o1 gnoin t a [behowtolue lof owTE TKIAL ERKGR n224 ECOHMENNER SHS T o eloetiooupuzAude HL dobetiaCd beDISCKIHNATIVE HobELS= Tue madls oe Tonud diea a [RAWMG SGd tla'h1oboh kdtie maolatet rocluca ltla opul oudal pixel pnioadpa4de Toke Th tute the potanLINEAK KEGRESSiON t ovobe (UNiVARIATTNube otno eXouplhnioie dlitoret- INPUr ocbe FEATURESTARGEEJoupbéOUTUTDstite th o lOLINERk. 6,xFUNeTONI n HMETERSPAYth Jeedut rNokHALTt senuet)bet we t adtus pow t s l o i t u o tepopu)

Lezione 14-10-2021

LE 2ONE 14-10-2021So, oalupadk uud, To HINIKeE oudr CosT FUNCTLON CoFX poite(xb)ohdmod dInJw.etipu 25 bu uip CNTOUR PLOTOTtoXimiuD olop laue Hd dipeusug dlal doda ueca ldmaue hok aCoT ki epuotGOAL: a e. e)D osFuicro toytuceextTueehanplue a p hiouole SweLE oholoa) CoNvERGENCErATEel uucdUPDATE SIHOLTANEOUSoui22o ttaeusudaki.oo od bu tpozwny uolutmpO oud teup 4)lhanyouupd SiMULTANEOUSLYta Joniobl , eHU H vaaoha euoRARAMETERokA UalutoHPER PARA METERt ek0gu tuGRADIENT DESCENT ALGOQOuieecuunouo t (DERIUATVEEARNING RATE 2dmcdk Toknde- eduu deoiadle ceald DrERGERECAESSIN pkcth1PRTLALDERIUATIBES 2wopet to aodto coupidee 20 tlue Counpule t a PCATESLINEARKEGRESSION= luLTIPLE FEATURESo Cuen ue e t x 4 e eo eume tle boviaotomedaNou odit deau oleo A beddBATCH GRADIENTRocdouy ke poonzteagDESCENT PRt che ConuERG: { |ahose eopleapole o 5UER SAGLE PoNTa o k Ol cilhd $ToCHASTLC GRADIENTDESCENT wedatdote polteCCs bd actic dbKhale oCteTt 2 o d QH2 2eoe 2-2AG PRoBLEHBATCH ure la Calaud STOCHASTIC7t INIBATCHodVauoses tu oldtliodat oluti fu o oUNTIL CoUVERGENCEUeca Mop tu dponitua'a teiiaun byua acSTOPFING CRliHAX TTERHTIONa ueABioLITE TOLERAN oalop e uou uae_etou Ualue eKELAWE TOLERANCE ou ocod)T ouplu ou tu etPetiaGRADIENT NORH TOLERANCE= lop whe a onm Prod.alUlu aDte u i ood o appnokiug tla ekoliawn, uie uto we dhun mmaduWo tlk POLYNOHIAL KEGRESiCA+6,+6x+6 xE 4RADENTDESCeNT (Upte-dedeotdeDEKCENTDIENTGRABATEHAasopmeulo e udttep Lako e t Lutouotopdllepoca2STOCASTe GRADlEAT NESCENT(SG)TeRALONEBHI-BATCH GRaDieNT DE SCENTU2uo abeuoeu x<mTERATAionBATCHGRAbiErSGDESÇENT loloucpohdtateaupitoeiaddkCBaeh u c t odntebetaededlaauedotoet pelmotanaeopiudoepuiudaTAceSGDPhe peWe TKADIENT SCENTo poluoaR onwuiuetido dau pounaleno

Lezione B-910-2o21

E2ONE B-910- 2o21e& d CGRADIENT ESCENT f UULTFE UARABLESFEATORE SCALING owl tla u podem edeatuie CopaniTNE Aea XF-Cado (HIN-HAX NORMALLATiON t hen Do ho to et up the add max UaluetApnaadk ot veliPud-al oGTuERS)UL tuRANNCplhoo4ou Cou uoueX Xrnecie(deuoda en do SCORE NoRHALIEAT ONLA0ou ees o 2ereby Ta-daid dlou otie t ueir tobu to mcrnoe youecoueuo 2teeoebeaateieoou- oludatieil kb We-Kx-Xo-)xCaupilotiouoepewaive):OLskc DepeudeFeRVud tle mupmumT solekta ptoded n muleUe:LoBABILISTI IKTERARETAUDN Liuareh Epadluai. Hlt apbioxiuol 2ulota baRAeatce ueciozaod oulpsNtOUTRUT Co obaled QaCPprxialoo we ea Du KRBABIL TYDASTRIBUTiON tu eno NoRHALALSTRIBUTloN 0 m o Otud ruona 27Ae Hu Aucdil bthol Hu2elotio:zknteCaia 22e27eoua ELIHO EUNCTioNporonTeuathive u upl, tu ta op2 . e h 20unuoCoae Htehoadsod Huoromllao emoeoudun onauapo o tu coces NDIPENDENT Halp2oCty o tTod eveatb Ha tpAeioe Ha pobohiO U Vy 22tLot ttu opat W atuu te pLtlou DMAXIHOHLfe LIkELIHDCRERoNLIkELIHOo HCOBla_omuewatoMAXK u avet o ax io_otbcae auUida a CoNSTA NTH Uoegectutloadtuot,toaxuitei puonfouietmtox mox [e)= mumiw kelkadLi uaenci dto0adameu tuaKeood ob2ocobeiatie leapotoAicdo edutla iar pwionmiow ouptha p2obliu Gatou M-VoCLASIFCATIONk t dlponcth e e iou GiSTe REGRESSiONE atoAunsloke beconta uatolo ocktiou)El poulnno hpouExelonetio Duuusakiouo deezt (Gea/loumorW ca hoe duken an TRESHOL DFUNCTLON orned C3TTC UNCTIONJDEtisidBouNDARNoA RORABILISTIC NTERPRETATUOAlcuur aturoeouboloupiug elon 4a ledoupaBUT hoe ee 3 prbelaugiupto clo Oo, 2 I koeClon) oud 0,rob seloliuopottye eloLiKELIHOD FONCTlONHe e le al ulueaD oud e hoveBERNOULLLAISTRIBUTION00e ve duodl Auneu Coupoc Woyeolclate t iAte houiu coupo mloo eoeupdeColupo.coto Likellocduct.ouwmalpconoie To ta ColotwcrSonc e Comoie acu MAX(KCOHe HAIHIATOANI ncbduuseluk_a mermmeCiadtipa cud tls uouuilluude.aod atea)S-4 (CROS ETRORYRRR FUKIeT ONJe)-()thT hehea hhel=4EGRADIENT DESENTDENCZNTKatusubeH poe thogh te OPDATE C THE WE1GHTSUa eyuup ormla dewe Eauae Ha duotue c FUNCTONwEt eU hoe1 8poolerAHla eud h.)4he )AH 0Dd uau ajeONE NS ALL THULTCLASS CLARRIFTCAT PMEawedola hour c a 2 clo00 to ekortuo top o k , taiecda, HdbWesthen Sun Cloud koiuHouoouo HA teeobleuecdRST odF ALU i oa Co ubstttukthne ut aunloz 234,tla uou onado te orquar oud eroe t la eud yo houetoue olutii oua). M23)

Lezione 4 240-2024

EIONE 4 240-2024Flauds On Jpplo uKepul TtCEONE5 26-10-2021|FITTING PRoBLEMtonl dota opl Renria odelauKuoha), oicoios Agmdl f i c Knuralo)hoHue doa dekoue ue eTANCo ASLLTIC APPRDACH Lt u mcdul H EATURESCoMnp LOUMSE(Me(He Squar EnorCud bka olereltent eakHSE ojat tCRADIEKT DESCaUALGO o tliobA H eoupd Hla beat uoluea tu poromiderI I _core o e oup o eoupdlec d i l tHTPERPLANE LuK cauir a kypepanletlapom bUalunr co pradek codbol& Lopate Rpokel UoluinCRTUe da htuaed tTICLAST CLASS IFIERLoGISTIC REGR. Atk CNE VS AD paeeh /uetAA MOLTIPLE BINARY QLASIFIERS Oud we Ca Loie YuduAtauid elomiclicTRAININe $ETEST SETKOHL H RALW ING STo o a mcdl, p Htla 2adhedo o ue TEST Sai tcoupiie HODot e tod.an HOU TO TUNE ou oljoLefumwn 15 )edale(t hale oend.ted mcclul) tLL FTING PAdB(Ee Cootoouplpeaoniol madlelt diko atopproxi d by Clueon beluoulouzTcd 0NDERFITS dot medel predeaabuiku Xctu o HGH ERROR) HIGH BASu jolds oouokud buHa mcdilo luole a Ervn Veyclon 0 WTt l TRAINING DATALe modulOVERFIT didoe OlER(ia HIGH VARIANCESPECIAL (LED tnoi doto)dUNGER FITTAG dluia poloedo e edd to upotoo coupAKUNDERF ITN =H wodil otm TRAING TEIT doteOVERFITOUNG eccou u adtl beius to HaHORIEE Thoung dalo 20utanLEARNING to panbhize)IAS Dd VARIANCEBIAS Sate-okic devidaa u timokor