Analisi statistica predizione voti finali

Statistical learning project

Authors and Date

Giulia Bartocci, Matteo Bonamin, Giacomo Griggio
06/09/2023

Introduction

In our project, we study a binary classification about the grades of students in secondary education of two Portuguese schools. In particular, we want to predict if a student passes the final exam or not. Among all the features that we will analyze for each student, we are curious about the data regarding drinking alcohol because we want to understand if that affects, and in which measure, the performances at school.

We already know that the effect of alcohol on studying is widely recognized as negative because it is a psychoactive substance that can negatively influence cognitive functions and mental abilities essential for learning and memorization. For these reasons, alcohol abuse is strongly discouraged among students and should be avoided, especially during periods of intensive study, such as during exams or the preparation of important projects.

With this research, we aim to highlight how some attributes can positively or negatively affect academic performance and the overall development of a scholastic career. To do this, we will use various models, starting from a generalized linear model. We will then perform a backward feature selection. Afterward, we will use shrinkage methods, like Ridge and Lasso regression, and later, we will see discriminant analysis (LDA and QDA) to then conclude with Naive Bayes model and KNN. At the end, we will compare the methods to find which one guarantees the best evaluation metrics.

Libraries

library(knitr)
library(readxl)
library(RColorBrewer)
library(corrplot)
library(vcd)
library(dplyr)
library(car)
library(pROC)
library(glmnet)
library(MASS)
library(e1071)
library(caret)
library(class)
library(dplyr)

Functions definition

In the following section, we defined some functions, designed to perform specific tasks and computations, that will be used for our analysis, making our code more organized and reusable.

compute_yules_q <- col2) {function(col1,val1_1=unique(col1)[1]val1_2=unique(col1)[2]val2_1=unique(col2)[1]val2_2=unique(col2)[2]a <- val1_1 col2 val2_1)sum(col1 == & ==b <- val1_1 col2 val2_2)sum(col1 == & ==c <- val1_2 col2 val2_1)sum(col1 == & ==d <- val1_2 col2 val2_2)sum(col1 == & ==q <- (a d b c) (a d b c)* - * / * + *return(q)}
2compute_yules_q_yes_no <- col2) {function(col1,a <- 'yes' col2 'passed')sum(col1 == & ==b <- 'yes' col2 'not passed')sum(col1 == & ==c <- 'no' col2 'passed')sum(col1 == & ==d <- 'no' col2 'not passed')sum(col1 == & ==q <- (a d b c) (a d b c)* - * / * + *return(q)}
compute_model_pred <- test, threshold=0.5) {function(model,glm_prob <- test, type = "response")predict(model,glm_pred <- rep(0, nrow(test))glm_pred[glm_prob threshold] <-1>return(glm_pred)}
compute_accuracy_model <- test, threshold=0.5) {function(model,glm_prob <- test, type = "response")predict(model,glm_pred <- rep(0, nrow(test))glm_pred[glm_prob threshold] <-1>accuracy <- sum(glm_pred==test$output) /nrow(test)return(accuracy)}

Loading data

Our dataset includes 649 students with their grades, demographic, social, and school-related features. It was collected using school reports and questionnaires. The source of these data is provided by Cortez Paulo in 2014 with the title of ‘Student performance’ and published in ‘UCI Machine Learning Repository’. This dataset is freely available for classification or regression tasks and it can be found on the Kaggle platform at the following link: data <- "Users", "matte", "OneDrive",read.csv(file.path("C:","Desktop","Portuguese.csv"))

str(data)
## 'data.frame': 649 obs. of 33 variables:
## $ school : chr "GP" "GP" "GP" "GP" ...
## $ sex : chr "F" "F" "F" "F" ...
## $ age : int 18 17 15 15 16 16 16 17 15 15 ...
## $ address : chr "U" "U" "U" "U" ...
## $ famsize : chr "GT3" "GT3" "LE3" "GT3" ...
## $ Pstatus : chr "A" "T" "T" "T" ...
## $ Medu : int 4 1 1 4 3 4 2 4 3 3 ...
## $ Fedu : int 4 1 1 2 3 3 2 4 2 4 ...
## $ Mjob : chr "at_home" "at_home" "at_home" "health" ...
## $ Fjob : chr "teacher" "other" "other" "services" ...
## $ reason : chr "course" "course" "other" "home" ...
## $ guardian : chr "mother" "father" "mother" "mother" ...
## $ traveltime: int 2 1 1 1 1 1 1 2 1 1 ...
## $ studytime : int 2 2 2 3 2 2 2 2 2 2 ...
## $ failures : int 0 0 0 0 0 0 0 0 0 0 ...
## $ schoolsup : chr "yes" "no" "yes" "no" ...
## $ famsup : chr "no" "yes" "no" "yes" ...
## $ paid : chr "no" "no" "no" "no" ...
## $ activities: chr "no" "no" "no" "yes" ...
## $ nursery : chr "yes" "no" "yes" "yes" ...
## $ higher : chr "yes" "yes" "yes" "yes" ...
## $ internet : chr "no" "yes" "yes" "yes" ...
## $ romantic : chr "no" "no" "no" "yes" ...
## $ famrel : int 4 5 4 3 4 5 4 4 4 5 ...
## $ freetime : int 3 3 3 2 3 4 4 1 2 5 ...
## $ goout : int 4 3 2 2 2 2 4 4 2 1 ...
## $ Dalc : int 1 1 2 1 1 1 1 1 1 1 ...
## $ Walc : int 1 1 3 1 2 2 1 1 1 1 ...
## $ health : int 3 3 3 5 5 5 3 1 1 5 ...
## $ absences : int 4 2 6 0 0 6 0 2 0 0 ...
## $ G1 : int 0 9 12 14 11 12 13 10 15 12 ...
## $ G2 : int 11 11 13 14 13 12 12 13 16 12 ...
## $ G3 : int 11 11 12 14 13 13 13 13 17 13 ...

Features description

Here is a description of the features, i.e., the columns of the dataset:

• School: The student’s school (binary: ‘GP’ - Gabriel Pereira or ‘MS’ - Mousinho da Silveira)
• Sex: The student’s sex (binary: ‘F’ - female or ‘M’ - male)
• Age: The student’s age (numeric: from 15 to 22)
• Address: Student’s home address type (binary: ‘U’ - urban or ‘R’ - rural)
• Famsize: Family size (binary: ‘LE3’ - less or equal to 3 or ‘GT3’ - greater than 3)
• Pstatus: The parent’s cohabitation status (binary: ‘T’ - living together or ‘A’ - apart)
• Medu: Level of mother’s education (numeric: 0 - none, 1 - primary education (4th grade), 2 - 5th to 9th grade, 3 - secondary education or 4 - higher education)
• Fedu: Level of father’s education (numeric: 0 - none, 1 - primary education (4th grade), 2 - 5th to 9th grade, 3 - secondary education or 4 - higher education)
• Mjob: The mother’s job (nominal: ‘teacher’, ‘health’ care related, civil ‘services’ (e.g., administrative or police), ‘at home’ or ‘other’)
• Fjob: The father’s job (nominal: ‘teacher’, ‘health’ care related, civil ‘services’ (e.g., administrative or police), ‘at home’ or ‘other’)
• Reason: Reason to choose this school (nominal: close to ‘home’, school ‘reputation’, ‘course’ preference or ‘other’)
• Guardian: The student’s guardian (nominal: ‘mother’, ‘father’ or ‘other’)
• Traveltime: Travel time from home to school (numeric: 1 - <15 min., 2 - 15 to 30 min., 3 - 30 min. to 1 hour, or 4 - >1 hour)
• Studytime: Weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3 - 5 to 10 hours, or 4 - >10 hours)
• Failures: Number of past class failures (numeric: n if 1<=n<3, else 4)
• Schoolsup: Extra educational support (binary: yes or no)
• Famsup: Family educational support (binary: yes or no)
• Paid: Extra paid classes within the course subject (Math or Portuguese) (binary: yes or no)
• Activities: Extra-curricular activities (binary: yes or no)
• Nursery: Attended nursery school (binary: yes or no)
• Higher: The student wants to take higher education (binary: yes or no)
• Internet: Internet access at home (binary: yes or no)
• Romantic: In a romantic relationship (binary: yes or no)
• Famrel: The quality of family relationships (numeric: from 1 - very bad to 5 - excellent)
• Freetime: Free time after school (numeric: from 1 - very low to 5 - very high)
• Goout: Going out with friends (numeric: from 1 - very low to 5 - very high)
• Dalc: Workday alcohol consumption (numeric: from 1 - very low to 5 - very high)
• Walc: Weekend alcohol consumption (numeric: from 1 - very low to 5 - very high)
• Health: Current health status (numeric: from 1 - very bad to 5 - very good)
• Absences: The number of school absences (numeric: from 0 to 93)
• G1: First period grade (numeric: from 0 to 20)
• G2: Second period grade (numeric: from 0 to 20)
• G3: Final grade (numeric: from 0 to 20, output target)

Data processing

Removing meaningless features

Our original dataset was composed of 33 columns. Still, we decided to remove some because we consider them not influential for our study. The data is collected from students of two different schools: ‘Gabriel Pereira’ and ‘Mousinho da Silveira,’ but we choose to ignore that because we consider both schools at the same level. Therefore, not considering differences between the two schools, we exclude the column ‘reason’ too.

Since we intend to analyze only personal information of each student, we remove certain features regarding his/her family, like: ‘famsize,’ ‘Pstatus,’ ‘Medu,’ ‘Fedu,’ ‘Mjob,’ ‘Fjob,’ ‘guardian.’ In our opinion, these attributes have low influence on the school grades.

Missing data

Here we checked if there are any missing values:

sum(is.na(data))
## [1] 0

As we can see, our dataset is already complete, and we didn’t have to fill it.

Modifying target column G3

Our target feature, i.e., contains numerical values from 0 to 20 but in order to have a binary classification problem, we divide the output into two classes. In the data source, we didn’t find indications about a passing grade, so we decided to separate them into “grade<12 not passed” and “grade>=12 passed,” considering the Italian threshold of 60% of the total evaluation. With this choice, we obtained a balanced binary output target that we will see in the following graphics.

Obtaining reduced dataset

Here we selected only the features that we want to study:

new_data <- select=c(sex, age, address, traveltime, studytime, failures,subset(data, schoolsup, famsup, paid, activities, higher, internet,romantic, famrel, freetime, goout, Dalc, Walc, health,absences, G1, G2))
new_data$output <- 'not passed', 'passed')ifelse(data$G3<12,

Considering the target one, now we have 23 columns, and these are the first 5 rows of our new dataset:

head(new_data,5)
## sex age address traveltime studytime failures schoolsup famsup paid
## 1 F 18 U 2 2 0 yes no no
## 2 F 17 U 1 2 0 no yes no
## 3 F 15 U 1 2 0 yes no no
## 4 F 15 U 1 3 0 no yes no
## 5 F 16 U 1 2 0 no yes no
## activities higher internet romantic famrel freetime goout Dalc Walc health
## 1 no yes no no 4 3 4 1 1 3
## 2 no yes yes no 5 3 3 1 1 3
## 3 no yes yes no 4 3 2 2 3 3
## 4 yes yes yes yes 3 2 2 1 1 5
## 5 no yes no no 4 3 2 1 2 5
## absences G1 G2 output
## 1 4 0 11 not passed
## 2 2 9 11 not passed
## 3 6 12 13 passed
## 4 0 14 14 passed
## 5 0 11 13 passed

Now the dataset is composed only by numerical, ordinal or binary features and we can proceed with the visualization of the data.

Data visualization

Target visualization

Output classification

100 80 60 Count 40 20 0 0 1 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 Grades

Output binary classification

300 200 Count 100 500 not passed passed Grades<12 and grades>=12

#### not passed passed ## 301 348

So we have a total of 301 grades below 12 and 348 students with a final grade equal or higher than 12.

Categorical variables visualization

Now we want to represent the plots that show the relations between categorical attributes and the target distribution.

Gender Address
F R
300 300
M U
200 200
100 100
50 50
0 0
not passed passed not passed passed
Extra educational support Family educational support
no no
300 300
yes yes
200 200
100 100
50 50
0 0
not passed passed not passed passed
Extra paid classes Extra-curricular activities
no no
300 300
yes yes
200 200
100 100
50 50
0 0
not passed passed not passed passed
Wants to take higher education Internet access
no no
300 300
yes yes
200 200
100 100
50 50
0 0
not passed passed not passed passed
Romantic relation
no
300
yes
200
100
500
not passed passed

Observing these plots, we cannot obtain a lot of information because almost all of them resulted balanced between not passed and passed exams. We would like to put in evidence from the ‘higher’ feature that very few people who doesn’t want to take a higher education obtained a grade better or equal than 12, so this attribute can be significant for our binary classification.

#### not passed passed
## no 64 5
## yes 237 343

Ordinal variables visualization

There are some features that are already divided in subsets and have a range from 1 to 5, so we can still visualize them with these barplots.

Travel time Study time
1 1
300 300
2 2
3 3
4 4
200 200
100 100
50 50
0 0
not passed passed not passed passed
Past class failures Family relations quality
0 1
300 300
1 2
2 3
3 4
5
200 200
100 100
50 50
0 0
not passed passed not passed passed
Free time after school Going out with friends
1 1
300 300
2 2
3 3
4 4
5 5
200 200
100 100
50 50
0 0
not passed passed not passed passed
Workday alcohol consumption Weekend alcohol consumption
1 1
300 300
2 2
3 3
4 4
5 5
200 200
100 100
50 50
0 0
not passed passed not passed passed
Health status
1
300
2
345
200
100
50
0
not passed passed

Even here, most of the features seem to be equally distributed between the two possible outputs, but we want to pay attention to some of them which result meaningful. This is the distribution of the students following the study time attribute from the lowest (1) to the highest level (4):

#### not passed passed
## 1 133 79
## 2 132 173
## 3 24 73
## 4 12 23

We can observe that 44% of students who didn’t pass the exam have a weekly study time less than 2 hours (i.e., belongs to the category 1). Instead, the people who fit in the level 1 are only 23% of the ones who obtained a good grade.

Even the attribute related to the past class failures is significant because, as we can see:

#### not passed passed
## 0 208 341
## 1 65 5
## 2 14 2
## 3 14 0

We notice that 341 out of 348 students, i.e., almost the 98%, who passed the exam has never failed one before. This percentage goes down to 69% for the people who didn’t obtain a grade higher or equal than 12 in the final exam.

Now we analyze the distribution of the workday alcohol consumption:

#### not passed passed
## 1 180 271
## 2 66 55
## 3 31 12
## 4 11 6
## 5 13 4

We can see that the percentage of people with the smallest assumption of alcohol (level 1) is:

## not passed
## 0.5980066
## passed
## 0.7787356

I.e., approximately 60% of the students who belong to this level didn’t pass the exam against the 78% who passed it. It is also interesting to point out that for a total of 17 people who declared a large use of alcohol, only 4 of them obtained a sufficient grade.

Numerical variables visualization

For the numerical features, which have a large range of values, we created boxplot graphics because the barplots will be useless for these kinds of attributes.

Age Absences
22 30
21 25
20 20
19 15
18 10
17