Analisi statistica predizione voti finali

Statistical learning project

Giulia Bartocci, Matteo Bonamin, Giacomo Griggio

06/09/2023

1

Introduction

In our project we study a binary classification about the grades of students in secondary education of two

Portuguese schools. In particular we want to predict if a student passes the final exam or not. Among all

the features that we will analyze for each student we are curious about the data regarding drinking alcohol

because we want to understand if that affects, and in which measure, the performances at school. We already

know that effect of alcohol on studying is widely recognized as negative, because it is a psychoactive substance

that can negatively influence cognitive functions and mental abilities essential for learning and memorization.

For these reasons, alcohol abuse is strongly discouraged among students and should be avoided, especially

during periods of intensive study, such as during exams or the preparation of important projects. With this

research we aim to highlight how some attributes can positively or negatively affect academic performance and

overall development of scholastic career. To do this we will use various models, starting from a generalized

linear model we will then perform a backward feature selection. After we will use shrinkage methods, like

Ridge and Lasso regression, and later, we will see discriminant analysis (LDA and QDA) to then conclude

with Naive Bayes model and KNN. At the end we will compare the methods to find which one guarantees the

best evaluation metrics.

Libraries

library(knitr)

library(readxl)

library(RColorBrewer)

library(corrplot)

library(vcd)

library(dplyr)

library(car)

library(pROC)

library(glmnet)

library(MASS)

library(e1071)

library(caret)

library(class)

library(dplyr)

Functions definition

In the following section, we defined some functions, designed to perform specific tasks and computations,

that will be used for our analysis, making our code more organized and reusable.

compute_yules_q <- col2) {

function(col1,

val1_1=unique(col1)[1]

val1_2=unique(col1)[2]

val2_1=unique(col2)[1]

val2_2=unique(col2)[2]

a <- val1_1 col2 val2_1)

sum(col1 == & ==

b <- val1_1 col2 val2_2)

sum(col1 == & ==

c <- val1_2 col2 val2_1)

sum(col1 == & ==

d <- val1_2 col2 val2_2)

sum(col1 == & ==

q <- (a d b c) (a d b c)

* - * / * + *

return(q)

} 2

compute_yules_q_yes_no <- col2) {

function(col1,

a <- 'yes' col2 'passed')

sum(col1 == & ==

b <- 'yes' col2 'not passed')

sum(col1 == & ==

c <- 'no' col2 'passed')

sum(col1 == & ==

d <- 'no' col2 'not passed')

sum(col1 == & ==

q <- (a d b c) (a d b c)

* - * / * + *

return(q)

}

compute_model_pred <- test, threshold=0.5) {

function(model,

glm_prob <- test, type = "response")

predict(model,

glm_pred <- rep(0, nrow(test))

glm_pred[glm_prob threshold] <-1

>

return(glm_pred)

}

compute_accuracy_model <- test, threshold=0.5) {

function(model,

glm_prob <- test, type = "response")

predict(model,

glm_pred <- rep(0, nrow(test))

glm_pred[glm_prob threshold] <-1

>

accuracy <- sum(glm_pred==test$output) /nrow(test)

return(accuracy)

}

Loading data

Our dataset includes 649 students with their grades, demographic, social and school related features and it

was collected by using school reports and questionnaires. The source of these data is provided by Cortez

Paulo in 2014 with the title of ‘Student performance’ and published in ‘UCI Machine Learning Repository’.

This dataset is freely available for classification or regression tasks and it can be found on the Kaggle platform

at the following link: https://www.kaggle.com/datasets/whenamancodes/alcohol-effects-on-study

data <- "Users", "matte", "OneDrive",

read.csv(file.path("C:",

"Desktop","Portuguese.csv"))

str(data)

## 'data.frame': 649 obs. of 33 variables:

## $ school : chr "GP" "GP" "GP" "GP" ...

## $ sex : chr "F" "F" "F" "F" ...

## $ age : int 18 17 15 15 16 16 16 17 15 15 ...

## $ address : chr "U" "U" "U" "U" ...

## $ famsize : chr "GT3" "GT3" "LE3" "GT3" ...

## $ Pstatus : chr "A" "T" "T" "T" ...

## $ Medu : int 4 1 1 4 3 4 2 4 3 3 ...

## $ Fedu : int 4 1 1 2 3 3 2 4 2 4 ...

## $ Mjob : chr "at_home" "at_home" "at_home" "health" ...

## $ Fjob : chr "teacher" "other" "other" "services" ...

## $ reason : chr "course" "course" "other" "home" ...

## $ guardian : chr "mother" "father" "mother" "mother" ...

## $ traveltime: int 2 1 1 1 1 1 1 2 1 1 ...

## $ studytime : int 2 2 2 3 2 2 2 2 2 2 ...

## $ failures : int 0 0 0 0 0 0 0 0 0 0 ...

3

## $ schoolsup : chr "yes" "no" "yes" "no" ...

## $ famsup : chr "no" "yes" "no" "yes" ...

## $ paid : chr "no" "no" "no" "no" ...

## $ activities: chr "no" "no" "no" "yes" ...

## $ nursery : chr "yes" "no" "yes" "yes" ...

## $ higher : chr "yes" "yes" "yes" "yes" ...

## $ internet : chr "no" "yes" "yes" "yes" ...

## $ romantic : chr "no" "no" "no" "yes" ...

## $ famrel : int 4 5 4 3 4 5 4 4 4 5 ...

## $ freetime : int 3 3 3 2 3 4 4 1 2 5 ...

## $ goout : int 4 3 2 2 2 2 4 4 2 1 ...

## $ Dalc : int 1 1 2 1 1 1 1 1 1 1 ...

## $ Walc : int 1 1 3 1 2 2 1 1 1 1 ...

## $ health : int 3 3 3 5 5 5 3 1 1 5 ...

## $ absences : int 4 2 6 0 0 6 0 2 0 0 ...

## $ G1 : int 0 9 12 14 11 12 13 10 15 12 ...

## $ G2 : int 11 11 13 14 13 12 12 13 16 12 ...

## $ G3 : int 11 11 12 14 13 13 13 13 17 13 ...

Features description

Here there is a description of the features, i.e. the columns of the dataset:

school:

• The student’s school (binary: ’GP’ - Gabriel Pereira or ’MS’ - Mousinho da Silveira)

sex:

• The student’s sex (binary: ’F’ - female or ’M’ - male)

age:

• The student’s age (numeric: from 15 to 22)

address:

• Student’s home address type (binary: ’U’ - urban or ’R’ - rural)

famsize:

• Family size (binary: ’LE3’ - less or equal to 3 or ’GT3’ - greater than 3)

Pstatus:

• The parent’s cohabitation status (binary: ’T’ - living together or ’A’ - apart)

Medu: Level of mother’s education (numeric: 0 - none, 1 - primary education (4th grade), 2 - 5th to

• 9th grade, 3 - secondary education or 4 - higher education)

Fedu:

• Level of father’s education (numeric: 0 - none, 1 - primary education (4th grade), 2 - 5th to 9th

grade, 3 - secondary education or 4 - higher education)

Mjob:

• The mother’s job (nominal: ’teacher’, ’health’ care related, civil ’services’ (e.g. administrative

or police), ’at home’ or ’other’)

Fjob:

• The father’s job (nominal: ’teacher’, ’health’ care related, civil ’services’ (e.g. administrative or

police), ’at home’ or ’other’)

reason:

• Reason to choose this school (nominal: close to ’home’, school ’reputation’, ’course’ preference

or ’other’)

guardian:

• The student’s guardian (nominal: ’mother’, ’father’ or ’other’)

traveltime:

• Travel time from home to school (numeric: 1 - <15 min., 2 - 15 to 30 min., 3 - 30 min. to

1 hour, or 4 - >1 hour)

studytime:

• Weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3 - 5 to 10 hours, or 4 - >10

hours)

failures:

• Number of past class failures (numeric: n if 1<=n<3, else 4)

schoolsup:

• Extra educational support (binary: yes or no)

4

famsup:

• Family educational support (binary: yes or no)

paid:

• Extra paid classes within the course subject (Math or Portuguese) (binary: yes or no)

activities:

• Extra-curricular activities (binary: yes or no)

nursery:

• Attended nursery school (binary: yes or no)

higher:

• The student wants to take higher education (binary: yes or no)

internet:

• Internet access at home (binary: yes or no)

romantic:

• In a romantic relationship (binary: yes or no)

famrel:

• The quality of family relationships (numeric: from 1 - very bad to 5 - excellent)

freetime:

• Free time after school (numeric: from 1 - very low to 5 - very high)

goout:

• Going out with friends (numeric: from 1 - very low to 5 - very high)

Dalc:

• Workday alcohol consumption (numeric: from 1 - very low to 5 - very high)

Walc:

• Weekend alcohol consumption (numeric: from 1 - very low to 5 - very high)

health;

• Current health status (numeric: from 1 - very bad to 5 - very good)

absences:

• The number of school absences (numeric: from 0 to 93)

G1:

• first period grade (numeric: from 0 to 20)

G2:

• second period grade (numeric: from 0 to 20)

G3:

• final grade (numeric: from 0 to 20, output target)

Data processing

Removing meaningless features

Our original dataset was composed of 33 columns but we decided to remove some because we consider them

not influential for our study. The data is collected from students of two different schools: ‘Gabriel Pereira’

and ‘Mousinho da Silveira’ but we choose to ignore that because we consider both school at the same level.

Therefore, not considering differences between the two schools we exclude the column ‘reason’ too. Since we

intend to analyze only personal information of each student we remove certain features reguarding his/her

family, like: ‘famsize’, ‘Pstatus’, ‘Medu’, ‘Fedu’, ‘Mjob’, ‘Fjob’, ‘guardian’. In our opinion these attributes

have low influence on the school grades.

Missing data

Here we checked if there are any missing value:

sum(is.na(data))

## [1] 0

As we can see our dataset is already complete and we didn’t have to fill it.

Modifying target column

G3

Our target feature, i.e. contains numerical values from 0 to 20 but in order to have a binary classification

problem we divide the output in two classes. In the data source we didn’t find indications about a passing

→ →

grade so we decided to separate them in “grade<12 not passed” and “grade>= 12 passed” considering

the Italian threshold of 60% of the total evaluation. With this choice we obtained a balanced binary output

target that we will see in the following graphics. 5

Obtaining reduced dataset

Here we selected only the features that we want to study:

new_data <- select=c(sex, age, address, traveltime, studytime, failures,

subset(data, schoolsup, famsup, paid, activities, higher, internet,

romantic, famrel, freetime, goout, Dalc, Walc, health,

absences, G1, G2))

new_data$output <- 'not passed', 'passed')

ifelse(data$G3<12,

Considering the target one now we have 23 columns and these are the first 5 rows of our new dataset:

head(new_data,5)

## sex age address traveltime studytime failures schoolsup famsup paid

## 1 F 18 U 2 2 0 yes no no

## 2 F 17 U 1 2 0 no yes no

## 3 F 15 U 1 2 0 yes no no

## 4 F 15 U 1 3 0 no yes no

## 5 F 16 U 1 2 0 no yes no

## activities higher internet romantic famrel freetime goout Dalc Walc health

## 1 no yes no no 4 3 4 1 1 3

## 2 no yes yes no 5 3 3 1 1 3

## 3 no yes yes no 4 3 2 2 3 3

## 4 yes yes yes yes 3 2 2 1 1 5

## 5 no yes no no 4 3 2 1 2 5

## absences G1 G2 output

## 1 4 0 11 not passed

## 2 2 9 11 not passed

## 3 6 12 13 passed

## 4 0 14 14 passed

## 5 0 11 13 passed

Now the dataset is composed only by numerical, ordinal or binary features and we can proceed with the

visualization of the data. 6

Data visualization

Target visualization Output classification

100

80

60

Count 40

20

0 0 1 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

Grades

Output binary classification

300

200

Count 100

50

0 not passed passed

Grades<12 and grades>=12

##

## not passed passed

## 301 348

So we have a total of 301 grades below 12 and 348 students with a final grade equal or higher than 12.

7

Categorical variables visualization

Now we want to represent the plots that shows the relations between categorical attributes and the target

distribution. Gender Address

F R

300 300

M U

200 200

100 100

50 50

0 0

not passed passed not passed passed

Extra educational support Family educational support

no no

300 300

yes yes

200 200

100 100

50 50

0 0

not passed passed not passed passed

Extra paid classes Extra−curricular activities

no no

300 300

yes yes

200 200

100 100

50 50

0 0

not passed passed not passed passed

8

Wants to take higher education Internet access

no no

300 300

yes yes

200 200

100 100

50 50

0 0

not passed passed not passed passed

Romantic relation no

300 yes

200

100

50

0 not passed passed

Observing these plots we cannot obtain a lot of information because almost all of them resulted balanced

between not passed and passed exams. We would like to put in evidence from the ‘higher’ feature that very

few people who doesn’t want to take a higher education obtained a grade better or equal than 12, so this

attribute can be significant for our binary classification.

##

## not passed passed

## no 64 5

## yes 237 343

Ordinal variables visualization

There are some features that are already divided in subsets and have a range from 1 to 5 so we can still

visualize them with these barplots. 9

Travel time Study time

1 1

300 300

2 2

3 3

4 4

200 200

100 100

50 50

0 0

not passed passed not passed passed

Past class failures Family relations quality

0 1

300 300

1 2

2 3

3 4

5

200 200

100 100

50 50

0 0

not passed passed not passed passed

Free time after school Going out with friends

1 1

300 300

2 2

3 3

4 4

5 5

200 200

100 100

50 50

0 0

not passed passed not passed passed

10

Workday alcohol consumption Weekend alcohol consumption

1 1

300 300

2 2

3 3

4 4

5 5

200 200

100 100

50 50

0 0

not passed passed not passed passed

Health status 1

300 2

3

4

5

200

100

50

0 not passed passed

Even here most of the features seems to be equally distributed between the two possible outputs but we want

to pay attention to some of them which result meaningful. This is the distribution of the students following

the study time attribute from the lowest (1) to the highest level (4):

##

## not passed passed

## 1 133 79

## 2 132 173

## 3 24 73

## 4 12 23

We can observe that the 44% of students who didn’t pass the exam have a weekly study time less than 2

hours (i.e. belongs to the category 1). Instead the people who fit in the level 1 are only the 23% of the ones

who obtained a good grade.

Even the attribute related to the past class failures is significant because, as we can see:

##

## not passed passed

## 0 208 341

## 1 65 5

## 2 14 2

## 3 14 0

We notice that 341 out of 348 students, i.e. almost the 98%, who passed the exam has never failed one before.

This percentage goes down to the 69% for the people who didn’t obtain a grade higher or equal than 12 in

the final exam.

Now we analyze the distribution of the workday alcohol consumption:

11

##

## not passed passed

## 1 180 271

## 2 66 55

## 3 31 12

## 4 11 6

## 5 13 4

We can see that the percentage of people with smallest assumption of alcohol (level 1) is:

## not passed

## 0.5980066

## passed

## 0.7787356

I.e. approximately 60% of the students who belong to this level didn’t pass the exam against the 78% who

passed it. It is also interesting to point out that for a total of 17 people who declared a large use of alcohol

only 4 of them obtained a sufficient grade.

Numerical variables visualization

For the numerical features, which have a large range of values, we created boxplots graphics because the

barplots will be useless for these kind of attributes.

Age Absences

22 30

21 25

20 20

19 15

18 10

17