Lab 19: Numerical Differentiation Lab

Type any f(x) and point x₀, then move the step h. The table compares forward, backward, central, Richardson and the second-derivative formula with the exact derivative.

The difference table of the robot data. Read the diagonal for the point you need.

Table derivatives and Richardson

Try this

Run it. Compare with Problems 1 and 2.
Stop the series after Δ³. How much does y'(1.1) change?
Change the Richardson steps to 0.02 and 0.01.

Gradient checking

Try this

Run it. Which ε gives the smallest relative difference?
Introduce a bug: change 2 * r * x to 2 * r in the analytic gradient. Does the check catch it?

Q1 Derive the central-difference formula for

f'(x)

and its error.

Taylor:

f(x \pm h) = f \pm hf' + \frac{h^2}{2}f'' \pm \frac{h^3}{6}f''' + \cdots

. Subtract:

f(x+h) - f(x-h) = 2hf' + \frac{h^3}{3}f''' + \cdots

, so

f' = \frac{f(x+h) - f(x-h)}{2h} - \frac{h^2}{6}f'''(\xi)

: error

O(h^2)

.

Q2 Write the formulas for

y'

and

y''

at

x_0

from Newton's forward formula.

y' = \frac1h\big(\Delta y_0 - \frac12\Delta^2y_0 + \frac13\Delta^3y_0 - \frac14\Delta^4y_0 + \cdots\big)

,

\;y'' = \frac{1}{h^2}\big(\Delta^2y_0 - \Delta^3y_0 + \frac{11}{12}\Delta^4y_0 - \frac56\Delta^5y_0 + \cdots\big)

.

Q3 Why is numerical differentiation called ill-conditioned?

It divides small differences of data by small steps: a data error

\delta

produces a derivative error of about

\delta/h

(first derivative) or

4\delta/h^2

(second). Smaller

h

reduces truncation error but magnifies noise and round-off.

Q4 Use Richardson extrapolation:

D(0.2) = 0.536707

,

D(0.1) = 0.539402

for

(\sin x)'

at 1.

R = 0.539402 + (0.539402 - 0.536707)/3 = 0.540300

; true value

0.540302

.

Q5 What is gradient checking?

Comparing a gradient computed analytically (e.g. by backpropagation) with central-difference estimates

\frac{L(w+\epsilon) - L(w-\epsilon)}{2\epsilon}

for a few parameters; a relative difference around

10^{-7}

or smaller means the gradient code is correct.