Stima Puntuale (a Handbook for Data Scientists with examples in R)

Statistical Methods for Data Science

Victor Plesco

Contents

Point Estimation 2

Evaluation of an Estimator (Loss Function) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1. Unbiasedness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

2. Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

R - Plots & Proofs (Simulating Efficiency) . . . . . . . . . . . . . . . . . . . . . 3

3. Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

R - Plots & Proofs (Simulating Consistency) . . . . . . . . . . . . . . . . . . . 4

Point Estimation of the Mean (Normal and not Normal IID Samples) . . . . . . . . . . . . . . . . 5

1. The estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

2. Expected value of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3. Variance of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

4. Risk of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

5. Asymptotic normality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Point Estimation of the Variance (Known Mean and Normal IID Samples) . . . . . . . . . . . . . . 7

1. The estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2. Expected value of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

3. Variance of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4. Distribution of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

5. Risk of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8

Point Estimation of the Variance (Unknown Mean and Normal IID Samples) . . . . . . . . . . . . 9

1. The estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

2. Expected value of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

3. Variance of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

4. Distribution of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

5. Risk of the estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1

Point Estimation

Point estimation is the act of choosing a parameter that is our best guess of the true (and unknown)

∈

θ̂ ξ

parameter . Our best guess is called an estimate of .

θ θ̂ θ

0 0

Evaluation of an Estimator (Loss Function)

Making an estimate is an act that produces some consequences. Among the consequences that are usually

θ̂

considered in a parametric decision problem the most relevant one is the estimation error. The estimation

is the difference between the estimate and the true parameter :

error e θ̂ θ 0

= −

e θ̂ θ 0

Of course, the statistican’s goal is to commit the smallest possible estimation error. This preference can

be formalized using loss functions. A ), mapping Θ x Θ into quantifies the loss

loss function L(

θ̂, θ R,

0

incured by estimating with Frequently used loss functions are:

θ θ̂.

0 The absolute error : ) = || − ||

L(

θ̂, θ θ̂ θ

0 0

The squared error : ) = 2

|| − ||

L(

θ̂, θ θ̂ θ

0 0

The expected value of a loss function is called the of the estimator and is denoted by:

statistical risk θ̂

• When the absolute error is used as a loss function, then the risk

= − ||]

R(

θ̂) E[||

θ̂ θ 0

is called the of the estimator.

mean absolute error (MAE)

• When the squared error is used as a loss function, then the risk

= ] = ) + = Squared bias + Variance.

2 2

− || −

R(

θ̂) E[||

θ̂ θ E( θ̂ θ V ar[ θ̂]

0 0

is called the The square root of the mean squared error is called

mean squared error (MSE). root

mean squared error (RMSE).

1. Unbiasedness

If an estimator produces parameters estimates that are on average correct, then it is said to be unbiased.

Let be the true parameter and let be an estimator . is an of if and on