Appunti di Numerical Analysis

PFLXKEIMJCXPMJCX

Hessian matrix of f(x)

DICA JCXTT.CAT

EYMJCNPMJCXTtEIIJCX702MJCX yM3CDVMILX

754

51

Hessian approximation PFLX

We will look for an approximation of Newton's method of getting the second order terms out: we will

get the Gauss-Newton method, to avoid the calculation of the Hessian matrix.

Gauss-Newton method KO

Start with some d'K

direction

Determine descent

step LIFERENCE

1

ceroose solution

Update K

puts d

K Kts

Increment iteration counter

Solution K

Levenberg-Marquardt (LM)

Method Jirk

Jisnd

When a Jacobian is rank-de cient, or nearly so, matrix in (*) is singular. However, because of the

equivalence of (*) with Jkd Mkll

LS ftp.pnl

• There exist in nitely many solutions for in this case.

• In LM the search direction is de ned as the solution to the system:

5IJKT

IS

REGULARIZED

IIITIjiffynumbe .stort

Levenberg-Marquardt (LM)

Method KO

with some d

Determine direction

descent gazza

in

factor

ne

adjust damping solution

size and Update

Choose Step 1 21 dek K Kts

Increment iteration counter

Solution K

steepest descent

• The method uses only rst derivatives in selecting a suitable search direction.

Newton’s

• method (sometimes called Newton-Raphson method) uses rst and second derivatives

and indeed performs better. Evaluation of the Hessian can be computationally expensive. The

Hessian matrix may not be positive de nite.

Gauss-Newton

• method does not require calculation of the second derivatives. However JTJ may

not be positive de nite.

Levenberg–Marquardt

• The algorithm regularizes a nonlinear LS problem, and it blends the

steepest descent method (for λ big) and the Gauss–Newton algorithm (for λ tends to zero).

Fortunately, it inherits the speed advantage of the Gauss–Newton algorithm and the stability of the

steepest descent method

Non-linear data/curve tting (identi cation of parameters)

gli

xli

f

0 argming

• model f is parameterized by parameters 0 401 Op

i i data

i N known

y are

1 points

is of o

nonlinear function

f 0

X a

Orthogonal distance regression

Minimize the mean square distance of data points to graph of f(x,θ)

• Example: orthogonal distance regression with cubic polynomial

Èydi di 711m

yli

fa

argmino 11

a

n

• optimization variables are model parameters θ and N points u(i)

• its term is squared distance of data point to points

Problems minimizing the orthogonal distance between

model and measurements are sometimes referred to

as orthogonal regression problems.

PROBLEM

Estimate the derivatives (slope, curvature, etc.) of a function, given a set of function values at a

discrete set of points.

→ Finite Di erence Formulas

Numerical Di erentiation

The simplest way to numerically compute a derivative is to mimic the formal de nition:

È non

mica mixtoy

un fine

For a linear function u(x)=ax+b the formula is exact.

First Derivative at a point: nite di erence schemes

First Derivative

Centered di erence Milyy

Mite

DX

M i

Forward di erence ni

Din Mitai

Backward di erence

Din mi 1

ti

Local Truncation Error (LTE)

We write a Taylor expansion of u(x) about x=ih

In in

h the in er

il utili 0

1

n

Mia il

hm in 0h

il

a 2

TERM

SECOND ORDER ERROR

oca

m'Cia

Dxmi

FIRST TERMS

ERROR

ORDER

Dini oca

ia

u

Dini 019

il

u GP

L.T.EE

dimostration

Consistency →

The FD formula is consistent if : LTE approaches to zero for 0

ℎ

→

The “speed” in which the error goes to zero as 0 is called the rate of convergence.

ℎ

When the truncation error is of the order of O(h), we say that the method is a rst order method. We

refer to a method as a pth-order method if the truncation error is of the order of O(hp)

Order of CONSISTENCY O(h )

p

Centered scheme (O(h )) is a more accurate formula than forward orbackward (O(h)), that is the LTE

2

decreases more rapidly. Second Derivative

approximation of the second derivative by

Centered Di erence formula

2mi Mi

Dxxelemite 1 2

p

m il 01h2

u

ERRORTERM Dxxmi

ORDER

SECOND

dimostration

Finite Di erence Formulas for k>1

LinearOperators

OULX MIG

th

il

n LINEAR OPERATOR

FORWARD

DUCX G

utili utili LINEAR OPERATOR

BACKWARD

S'MIX E

Il

utili M LINEAR

OPERATOR

CENTERED

Tuini

01mi 8111º

ni Mi Mi

mixa 1 Mite 1

We de ne the linear operators oforder k at = ℎ Ui

0ᵗʰ

01 Omnia

Okui O

mi hi NUI

Vui 01 lui

0 0 1

Skui tuit

81 mi Shui

5 S E

E

Computing second order centered nite di erencing

D'D D'mi

D mi

DeDea

Dxxui mi mini

nie s a

D

DIRMI 1112

MI

112

Dyz E

knits mia_cui 1

ni ni mi

ni 1

4 xith that

il Xi

then such

Xixo

Let E

MIX 0 ME

Xo n TE

un

Numerical problems

• Truncation Error:

error due to the truncation of the Taylor expansion

• Rounding Error:

approximation error in nite arithmetic

In nite arithmetic a numerical evaluation which uses an arbitrarily small value of h does not lead to

a reduction of total error.

Remarks

▪ Rounding errors cause deterioration of the approximation for small values of h.

▪ The value of h which allows a correct evaluation of the formulas depends on the accuracy of the

machine.

▪ If the terms ( +ℎ) are calculated inaccurately then the errors are multiplied by a factor 1 / h, which

grows very quickly for small values of h. Flxith Flxi

DxtFCXI

F S

xith flxith flxi

flxithly

VALUE

COMPUTED APPROXIMATION

DIFIXI ERROR

Di erentiation Via Polynomial Interpolation

Approximating an entire derivative function on an interval [a,b]. The rst stage is to construct an

interpolating polynomial from the data. An approximation of the derivative function can be then

obtained by a direct di erentiation of the interpolant.

Example The Lagrange form of the polynomial interpolation through 3 values (xi,yi)

1. Select a few nearby points, interpolate by a polynomial, and then di erentiate the interpolant

2. Di erentiating the interpolant

3. Assuming uniform x points

4. First derivative of the Lagrange interpolant: evaluate the derivative at several points

2 2

3

Plx 1 2

34

5 5

ya 243

42

2 2

To obtain derivatives of order n the interpolation polynomial must be of degree greater than or equal to

n

Truncation Error

We know that the interpolation error is

flxt TKLEIICX

5 XI

PLHIYI.fm

Compute the error for numerical di erentiation: f'KYI

III

f Chep'Cat f 5 XI

TK 0

them

Knots

the

if is

i of

one ftp

f'CHE f

P'LA 5

TruneILE

Multi-dimensional Derivatives

Extension of the one-dimensional case:

Finite di erence formula to approximate partial derivatives of function u(x, y)

C Mixi ulXite

Ux 95

9

1

ulxi

Uxx 45

Xite

U

2m 45

95 Xi

1 la Mi

Mite

Dxxn 1,5

5

21

Dxyll Mi

Mite Mi

1 1

5 1 1

1,5 1,5

1,5

1

Laplacian Operator (5-points) 9

also points

DI Mi 2 Mi

Uxxthyy 1,5

1,5 5 Miste 21 11,5 1

5

derivate

sommapaffè Mista Miss

Lite 1,5 h

tami

01h

The LTE is

Biharmonic operator

DI 047 Ye ftp.t

e div

Divergence Operator b

I fy bdj bon

Ou

bdiv

Yu

vector if b is a function

divbon Sx bis

84 Syniis of u, we can't

take it o

Sx bissxnis

it

is

i

is

i 2 bitsgfxmitsj bi sjfxni.rs

bits bi

uita Mis mi Mi

15 25

tbitsjuitaj bi

bitejuig sjuijtbi.es i 2

I

SO LOCALI DI

bon Dinis bi

bi

if Dinis

div Dyt

consider

we

this scheme is not symmetric, we don't consider the value of b in some points

It does not use

b((i-1)h,jh) and b(ih,(j-1)h)

Sinis Sfmi

bon

div SyFbi

if bi

consider

we Mi MISE Mi

SÉ MIEI 1

gy

1

where h

h

this scheme is symmetric and su ecentely close to the point

this one is the best, because

rst is local and the second

property is that is symmetric

We can also add another

property: the accuracy 0192

the requiriments is that the

points need to sorround

the central point

Remarks

▪ To increase the order of accuracy of the formulas is necessary to increase the number of points

involved in the calculation of the nite di erence scheme.

▪ Higher precision is equivalent to a higher computational complexity.

▪ To achieve greater accuracy without increasing the order of the formulas we can use extrapolation

techniques such as that of Richardson.

Richardson’s Extrapolation

Richardson extrapolation is a simple, e ective mechanism for generating higher order numerical

methods from lower order ones.

This technique uses the concept of using grids with di erent step size Δx for improving the accuracy

of the approximations.

Example in which we show how to turn a second-order approximation of the second derivative into a

fourth order approximation of the same quantity

EÉFÉFÉ 7

2 920

910

f xi IF

We write the equation for grids of di erent step sizes

2

f 7

Flox 020

910

Xi

1 7

E 921

f F as

Xi

2

This, of course, is still a second-order approximation of the derivative. However, the idea is to combine

[1] with [2] such that the x2 term in the error vanishes.

Δ

Indeed, multiplying eq.[2] by 4 and subtracting [1] from [2]

E

9

Xi OF

of 401 90214

9

0102

FOX 920

f XI

3 1

12021

Xi FLIP 1

The equation can be rewritten as:

E Δ

F

f xi az Xi

The the estimation f

of of

new is

accuracy 4 2

010 of

instead 010

We can continue to eliminate higher order terms of the error using a grid even ner:

Find an approximation of the de nite integral of the real function f(x) with respect to the independent

variable x, evaluated between the limits x = a to x = b. The function f(x) is referred to as the integrand:

effnax

b

Ilfia

Why do we need to provide an approximation of I:

- A complicated continuous function that is