Numerical differentiation and integration · Lecture 19

Numerical Differentiation

Slopes from data and from black-box functions: forward, backward and central differences from Taylor series, derivative formulas from Newton's interpolation, Richardson extrapolation, and why differentiation is the one operation that amplifies noise.

Md Mahbub E Noor · Assistant Professor
CSE-3107 Numerical Methods · B.Sc. 3rd Year 1st Semester
Department of CSE, University of Barishal

Why this lecture

The problem: speed and braking from position data

A delivery robot is braking. Its position $y$ (metres) is logged every 0.1 s. What are its speed and deceleration at $t = 1.1$ s?

Speed is $dy/dt$ and acceleration is $d^2y/dt^2$ , but there is no formula for $y(t)$ , only a table
A simple slope between neighbours gives one estimate; which neighbours, and how accurate?
Smaller steps should help, yet real sensors add noise: does a smaller step make it worse?
Same question in ML: the slope of a loss with respect to a weight

$t$ (s)	$y$ (m)
1.0	7.989
1.1	8.403
1.2	8.781
1.3	9.129
1.4	9.451
1.5	9.750
1.6	10.031

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

2 / 21

Today's goals

Derive forward, backward and central difference formulas and their errors from Taylor series
Differentiate tabulated data with Newton's forward and backward formulas
Improve an estimate with Richardson extrapolation
Choose the step size $h$ , balancing truncation and round-off
Explain why differentiation amplifies noise, and what to do about it

Last time

Interpolation: values between table entries.

Today

Slopes and curvature from the same tables, and from functions we can only evaluate.

Numerical differentiation and integration · you are here

L19 · Numerical Differentiation→L20 · Numerical Integration

Builds on: L3, L4, L16

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

3 / 21

Start here

The idea in one picture

The derivative at x is the slope of the tangent there.We cannot see the tangent directly: we only have values of f.Forward difference: the slope of the line to the next point.Central difference: the slope between the two neighbours. Much closer to the tangent.Slope = rise ÷ run. Shrink h and the line turns into the tangent.

In plain words

A derivative is a slope
From a table or a program we can only compute slopes between nearby points
Choosing the points cleverly (central) gives a much better slope

New words today

forward difference: (f(x + h) − f(x)) / h
central difference: (f(x + h) − f(x − h)) / 2h
step size h: distance between the points used
order O(h²): halve h → error ÷ 4

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

4 / 21

Difference formulas from Taylor series

Forward and backward: first order

\begin{aligned} f'(x) &= \frac{f(x+h) - f(x)}{h} - \frac{h}{2}f''(\xi) \\ f'(x) &= \frac{f(x) - f(x-h)}{h} + \frac{h}{2}f''(\xi) \end{aligned}

Central: subtract the two Taylor series, the h² terms cancel

f'(x) = \frac{f(x+h) - f(x-h)}{2h} - \frac{h^2}{6}f'''(\xi)

Second derivative: add the two series

f''(x) = \frac{f(x+h) - 2f(x) + f(x-h)}{h^2} - \frac{h^2}{12}f^{(4)}(\xi)

Forward and backward: error $O(h)$ ; central: $O(h^2)$ , same number of evaluations
Halve $h$ : forward error halves, central error quarters
Second derivative: the discrete Laplacian of Lecture 14's heat plate
All from Lecture 3: $f(x \pm h) = f \pm hf' + \tfrac{h^2}{2}f'' \pm \tfrac{h^3}{6}f''' + \cdots$

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

5 / 21

Derivatives from Newton's formulas (operators)

Operator route (Lecture 15): take the logarithm of E = 1 + Δ

hD = \ln(1 + \Delta) = \Delta - \frac{\Delta^2}{2} + \frac{\Delta^3}{3} - \frac{\Delta^4}{4} + \cdots

Square it for the second derivative

h^2D^2 = \Delta^2 - \Delta^3 + \frac{11}{12}\Delta^4 - \frac{5}{6}\Delta^5 + \cdots

Backward forms, for the end of a table

hD = \nabla + \frac{\nabla^2}{2} + \frac{\nabla^3}{3} + \cdots, \qquad h^2D^2 = \nabla^2 + \nabla^3 + \frac{11}{12}\nabla^4 + \cdots

Apply to $y_0$ : $y'(x_0) = \frac1h\big(\Delta y_0 - \tfrac12\Delta^2y_0 + \tfrac13\Delta^3y_0 - \cdots\big)$
Equivalent: differentiate Newton's forward formula and put $p = 0$
Use the forward form near the start, backward near the end
The first term alone is the forward difference $\Delta y_0/h$

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

6 / 21

The method, step by step

Derivatives from a table, step by step

Check equal spacing; note $h$
Build the difference table
Choose the formula: forward near the start, backward near the end, central in the middle
Read the diagonal through the point $x_0$ where the derivative is wanted
Add the series terms $\frac1h(\Delta - \tfrac12\Delta^2 + \tfrac13\Delta^3 - \cdots)$ until they are negligible; for $y''$ use $\frac{1}{h^2}(\Delta^2 - \Delta^3 + \tfrac{11}{12}\Delta^4 - \cdots)$

Pseudocode

input y[0..n] (step h), i  # point x_i
build differences d[k] = Δᵏ y_i
d1 ← 0;  sign ← +1
for k = 1 … n−i:
    d1 ← d1 + sign·d[k] / k
    sign ← −sign
d2 ← d[2] − d[3] + 11/12·d[4]
      − 5/6·d[5]          # if present
return d1/h,  d2/h²

For a point in the middle of the table, the two-point central formulas are usually enough.

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

7 / 21

Explore: secants approaching the tangent

Orange: forward secant; blue: central secant; green: the true tangent
Move $h$ from $10^{-1}$ to $10^{-5}$ : compare how fast each error falls
Below $10^{-6}$ the errors grow again: round-off
Richardson combines two central estimates into a far better one

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

8 / 21

Problem 1: speed and deceleration at $t = 1.1$

From the robot table, find

y'(1.1)

and

y''(1.1)

with Newton's forward formula (

h = 0.1

).

1

Forward differences at

t = 1.1

:

\Delta y = 0.378

,

\Delta^2 y = -0.030

,

\Delta^3 y = 0.004

,

\Delta^4 y = -0.001

,

\Delta^5 y = 0.003

2

y' = \dfrac{1}{0.1}\Big(0.378 + \dfrac{0.030}{2} + \dfrac{0.004}{3} + \dfrac{0.001}{4} + \dfrac{0.003}{5}\Big) = 10(0.39518)

3

y'' = \dfrac{1}{0.01}\Big(-0.030 - 0.004 + \dfrac{11}{12}(-0.001) - \dfrac56(0.003)\Big) = 100(-0.037417)

Speed

y'(1.1) \approx 3.952

m/s; acceleration

y''(1.1) \approx -3.74

m/s² (braking).

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

9 / 21

Problem 2: central and backward formulas

From the same table, estimate

y'(1.3)

and

y''(1.3)

with the central formulas, and

y'(1.6)

with the backward formula.

1

Central:

y'(1.3) \approx \dfrac{9.451 - 8.781}{0.2} = 3.350

2

y''(1.3) \approx \dfrac{9.451 - 2(9.129) + 8.781}{0.01} = -2.60

3

Backward at 1.6:

\nabla y = 0.281

,

\nabla^2 y = -0.018

,

\nabla^3 y = 0.005

:

\;y' \approx 10\big(0.281 - 0.009 + 0.0017\big) = 2.74

y'(1.3) \approx 3.35

m/s,

y''(1.3) \approx -2.6

m/s²,

y'(1.6) \approx 2.74

m/s: the robot is slowing down.

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

10 / 21

Richardson extrapolation: cancel the leading error

Central difference with step h has error c·h² + O(h⁴)

\begin{aligned} D(h) &= f'(x) + ch^2 + O(h^4) \\ D(h/2) &= f'(x) + \tfrac14 ch^2 + O(h^4) \end{aligned}

Combine to remove the h² term

f'(x) \approx D(h/2) + \frac{D(h/2) - D(h)}{3} \qquad (\text{error } O(h^4))

$\sin$ at 1: $D(0.2) = 0.536707$ , $D(0.1) = 0.539402$
$R = 0.539402 + (0.539402 - 0.536707)/3 = 0.540301$ ; true $\cos 1 = 0.540302$
Error drops from $9\times10^{-4}$ to $1.8\times10^{-6}$ for two extra evaluations
Same as the five-point formula $\frac{-f_2 + 8f_1 - 8f_{-1} + f_{-2}}{12h}$ ; same idea gives Romberg integration (Lecture 20)

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

11 / 21

Classical vs numerical

Same slope, two approaches

Calculus

Classical (differentiate the formula)

$f(x) = x^x$ : write $e^{x\ln x}$ , then $f'(x) = x^x(1 + \ln x)$
At $x = 2$ : $4(1 + 0.693147) = 6.772589$
Exact, but needs a formula and the rules of calculus

vs

Algorithm

Numerical (central difference)

$\dfrac{f(2.001) - f(1.999)}{0.002} = 6.772593$
Needs only values of $f$
Same code for any function, table or program

When the classical approach cannot work

The robot's position table, or a simulation

There is no formula for $y(t)$ to differentiate, only seven numbers. The same holds when $f$ is the output of a simulation or a trained model: finite differences work on values alone (and in ML they check the gradients computed by backpropagation).

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

12 / 21

Derivatives from a table, and Richardson, in Python and C

Forward-series derivative from the difference table
Central first and second derivatives
Richardson extrapolation for $\sin$ at 1
Prints each term, as in a written solution

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

13 / 21

When differentiation goes wrong

Limitations shown failing

Round-off: at $h = 10^{-11}$ the central error is $10^{-6}$ , worse than at $h = 10^{-5}$ ( $10^{-11}$ ); the second derivative is useless (Lecture 4)
Noise: positions with 1 mm noise (sampled every 0.01 s) give speed errors up to 0.2 m/s and acceleration errors larger than the acceleration itself (54 vs 39 m/s²)
Kinks: at a corner (|x| at 0) the derivative does not exist; the central formula quietly reports 0
Dividing by $h$ (or $h^2$ ) multiplies every data error by $1/h$ (or $1/h^2$ )

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

14 / 21

Practice

Practice problem

Problem 3"From the table

x

: 0, 1, 2, 3, 4, 5 and

y

: 4, 8, 15, 7, 6, 2, find

\dfrac{dy}{dx}

and

\dfrac{d^2y}{dx^2}

at

x = 0

."

How to write a complete answer

Build the difference table; top diagonal: $\Delta y_0 = 4$ , $\Delta^2 = 3$ , $\Delta^3 = -18$ , $\Delta^4 = 40$ , $\Delta^5 = -72$
State the formulas: $y' = \frac1h(\Delta - \frac{\Delta^2}{2} + \frac{\Delta^3}{3} - \frac{\Delta^4}{4} + \frac{\Delta^5}{5})$ and $y'' = \frac{1}{h^2}(\Delta^2 - \Delta^3 + \frac{11}{12}\Delta^4 - \frac56\Delta^5)$
Substitute with $h = 1$ : $y'(0) = 4 - 1.5 - 6 - 10 - 14.4 = -27.9$ ; $\;y''(0) = 3 + 18 + 36.67 + 60 = 117.67$
Comment: the differences grow, so the series converges badly and the answer is unreliable; the data are far from smooth

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

15 / 21

When to use it

Numerical differentiation: method profile

Best case

Smooth functions you can evaluate anywhere: central difference with $h \approx 10^{-5}$ gives about 10 digits
Richardson: $O(h^4)$ for two extra evaluations
Smooth, accurate tables: the difference series converges quickly

Worst case

Noisy measurements: errors grow like noise $/h$ or noise $/h^2$
$h$ too small: round-off (Lecture 4)
Corners and jumps: the derivative does not exist

Best suited for

Gradient checking in machine learning
Velocity and acceleration from smooth sensor data
Finite-difference methods for differential equations (heat, waves)
Jacobians for Newton's method when no formula is available

Avoid when

Raw noisy data (smooth or fit first, Lecture 23)
When an exact derivative is available cheaply (use it, or automatic differentiation)
High-order derivatives from tables

Rule of thumb: central differences with $h \approx \varepsilon^{1/3}|x|$ ; Richardson for more accuracy; smooth before differentiating data.

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

16 / 21

Comparative analysis: derivative formulas for $\sin$ at 1, $h = 0.1$

Formula	Points used	Order	Estimate	Error	Use
Forward $\frac{f_1 - f_0}{h}$	2	$O(h)$	0.497364	$4.3\times10^{-2}$	start of a table
Backward $\frac{f_0 - f_{-1}}{h}$	2	$O(h)$	0.581441	$4.1\times10^{-2}$	end of a table, real-time data
Central $\frac{f_1 - f_{-1}}{2h}$	2	$O(h^2)$	0.539402	$9.0\times10^{-4}$	default inside
Five-point / Richardson	4	$O(h^4)$	0.540301	$1.8\times10^{-6}$	high accuracy, smooth $f$
Newton forward series	many	up to $O(h^n)$	—	—	tables, written problems

True value $\cos 1 = 0.540302$ . More points buy higher order, but also more sensitivity to noise in the data.

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

17 / 21

Roadmap

Limitations, and where we fix them

Differentiation works well for smooth functions and badly for noisy data. Its siblings fix or avoid the weaknesses:

What it does well (keep this)

Central differences: $O(h^2)$ for two evaluations
Richardson extrapolation
Formulas from Taylor or from Newton's series
Error control by the choice of $h$

LimitationHow it is overcomeWhere

Integration is the stable inverse: it averages noise

→

Trapezoidal, Simpson, Romberg

Lecture 20 →

Derivatives appear inside equations:

y' = f(x, y)

→

ODE methods (Euler, Runge–Kutta)

Lecture 21 →

Noisy data

→

Fit a smooth model first: least squares

Lecture 23 →

Millions of derivatives of one loss

→

Automatic differentiation; gradient descent

Lecture 25 →

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

18 / 21

Applications

Where this method runs in the real world

Click a card to see where a difference formula is applied.

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

19 / 21

Checkpoint

Checkpoint · think, then discuss with your neighbour

f(1.9) = 3.61

,

f(2.0) = 4.00

,

f(2.1) = 4.41

. What is the central-difference estimate of

f'(2)

?

A

4.0

B

3.9

C

4.1

D

8.0

A.

\dfrac{4.41 - 3.61}{0.2} = 4.0

(exact for

f = x^2

). Forward gives

4.1

, backward

3.9

.

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

20 / 21

Summary

Key points

Forward/backward $O(h)$ , central $O(h^2)$ , second derivative $\frac{f_1 - 2f_0 + f_{-1}}{h^2}$
Tables: $y' = \frac1h(\Delta - \frac{\Delta^2}{2} + \frac{\Delta^3}{3} - \cdots)$ , $y'' = \frac{1}{h^2}(\Delta^2 - \Delta^3 + \frac{11}{12}\Delta^4 - \cdots)$
Richardson: $D(h/2) + \frac{D(h/2) - D(h)}{3}$ gives $O(h^4)$
Differentiation amplifies noise and round-off: choose $h$ carefully, smooth data first

Opening problem: solved

Forward series from the table: speed 3.952 m/s and acceleration −3.74 m/s² at $t = 1.1$ s. Central formulas give 3.35 m/s and −2.6 m/s² at 1.3 s: the robot is braking.

Homework

From $x$ : 0, 0.1, 0.2, 0.3, 0.4 and $y = e^x$ : 1.0000, 1.1052, 1.2214, 1.3499, 1.4918, find $y'(0)$ and $y''(0)$ . Compare with 1.
Use Richardson extrapolation with $h = 0.2$ and $0.1$ to estimate the derivative of $\ln x$ at $x = 2$ .
Derive the central second-derivative formula and its error term from Taylor series.
In the lab, find the best $h$ for the forward and the central formula for $e^x$ at $x = 1$ . Compare with $\sqrt{\varepsilon}$ and $\varepsilon^{1/3}$ .

Next lecture

Numerical integration. Areas, distances and probabilities from the same kind of tables: trapezoidal, Simpson, Romberg and Gauss.

Md Mahbub E Noor

Lecture 19: Numerical Differentiation

University of Barishal

21 / 21

Numerical Differentiation

The problem: speed and braking from position data

Today's goals

The idea in one picture

Difference formulas from Taylor series

Derivatives from Newton's formulas (operators)

Derivatives from a table, step by step

Explore: secants approaching the tangent

Problem 1: speed and deceleration at $t = 1.1$

Problem 2: central and backward formulas

Richardson extrapolation: cancel the leading error

Same slope, two approaches

Classical (differentiate the formula)

Numerical (central difference)

The robot's position table, or a simulation

Derivatives from a table, and Richardson, in Python and C

Limitations shown failing

Practice problem

Numerical differentiation: method profile

Comparative analysis: derivative formulas for $\sin$ at 1, $h = 0.1$

Limitations, and where we fix them

Where this method runs in the real world

🧠 Gradient checking in deep learning

❤️ Heartbeat detection in ECG

🔥 Simulating heat flow

🛰️ Velocity from GPS and IMU

Checkpoint

Summary

Numerical Differentiation

The problem: speed and braking from position data

Today's goals

The idea in one picture

Difference formulas from Taylor series

Derivatives from Newton's formulas (operators)

Derivatives from a table, step by step

Explore: secants approaching the tangent

Problem 1: speed and deceleration at t=1.1t = 1.1t=1.1

Problem 2: central and backward formulas

Richardson extrapolation: cancel the leading error

Same slope, two approaches

Classical (differentiate the formula)

Numerical (central difference)

The robot's position table, or a simulation

Derivatives from a table, and Richardson, in Python and C

Limitations shown failing

Practice problem

Numerical differentiation: method profile

Comparative analysis: derivative formulas for sin⁡\sinsin at 1, h=0.1h = 0.1h=0.1

Limitations, and where we fix them

Where this method runs in the real world

🧠 Gradient checking in deep learning

❤️ Heartbeat detection in ECG

🔥 Simulating heat flow

🛰️ Velocity from GPS and IMU

Checkpoint

Summary

Problem 1: speed and deceleration at $t = 1.1$

Comparative analysis: derivative formulas for $\sin$ at 1, $h = 0.1$