Data fitting and optimization for machine learning · Lecture 25

Numerical Optimization for Machine Learning

Finding the minimum of a function when no formula gives it: golden-section search in one variable, gradient descent and the learning rate, Newton's method with curvature, and Gauss–Newton for nonlinear least squares. This is how machine-learning models are trained.

Md Mahbub E Noor · Assistant Professor
CSE-3107 Numerical Methods · B.Sc. 3rd Year 1st Semester
Department of CSE, University of Barishal

Why this lecture

The problem: who will pass the exam?

Ten students: hours of study $x$ and result $y$ (1 = pass, 0 = fail). Predict the probability of passing with a model $p = \sigma(wx + b)$ , where $\sigma(z) = 1/(1 + e^{-z})$ .

Best $w$ , $b$ : minimise the cross-entropy loss $L(w, b) = -\frac1n\sum \big[y_i\ln p_i + (1 - y_i)\ln(1 - p_i)\big]$
Setting $\partial L/\partial w = \partial L/\partial b = 0$ gives nonlinear equations with no closed-form solution
Least squares (Lectures 23–24) does not apply: the model is not linear in $w$ , $b$
Neural networks: the same problem with millions of parameters

hours $x$	result $y$
0.5, 1.0, 1.5	0, 0, 0
2.0, 2.5	1, 0
3.0, 3.5	1, 0
4.0, 4.5, 5.0	1, 1, 1
$w$ , $b$ = ?	boundary = ?

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

2 / 26

Today's goals

State optimality conditions and the link between minimisation and root finding
Apply golden-section search to a one-variable function
Apply gradient descent; explain the learning rate through stability
Apply Newton's method for minimisation and Gauss–Newton for nonlinear least squares
Train a logistic-regression classifier; choose a method for a given problem

Last time

Root finding (bisection, Newton, Lectures 6–10); iteration and convergence (Lecture 14); Euler stability (Lectures 21–22); least squares (Lectures 23–24).

Today

Minimise any smooth function by iteration: bracket it, walk downhill, or use curvature.

Data fitting and optimization for machine learning · you are here

L23 · Least Squares and Linear Regression→L24 · Polynomial, Exponential and Trigonometric Fitting→L25 · Numerical Optimization for Machine Learning

Builds on: L6, L9, L10, L14, L24

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

3 / 26

Start here

The idea in one picture

A function of two variables drawn as a map: each ring is one height.Start anywhere. We want the bottom of the valley.Feel which way is steepest downhill and take a step that way.Look again from the new place, step again.The steps get shorter as the ground flattens.Keep going …… until the bottom: the minimum.If the steps are too long, each one jumps across the valley and lands higher. The learning rate matters.

In plain words

Training a model = finding the lowest point of an error surface
We cannot see the whole surface, only the slope where we stand
So step downhill, look again, and repeat

New words today

gradient ∇f: the direction of steepest uphill
learning rate η: the length of each step
minimum: a point with no downhill direction left
loss function: the error being minimised
Hessian H: the table of second derivatives (curvature)
convex: bowl-shaped: only one minimum

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

4 / 26

Working principle: a minimum is where the slope is zero and the curve bends up

One variable: necessary and sufficient conditions

f'(x^{*}) = 0, \qquad f''(x^{*}) > 0

Several variables: gradient zero, Hessian positive definite

\nabla f(\mathbf{x}^{*}) = \mathbf{0}, \qquad H = \Big[\frac{\partial^2 f}{\partial x_i\partial x_j}\Big] \text{ positive definite}

Minimising $f$ = finding a root of $f'$ : bisection and Newton (Lectures 6–10) apply to $f'$
But $f'$ may be unknown or expensive: golden section uses only values of $f$
Many variables: walk downhill along $-\nabla f$ (gradient descent)
Local minimum ≠ global minimum unless $f$ is convex

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

5 / 26

Golden-section search: shrink a bracket without derivatives

f unimodal on [a, b]; two interior points, g = (√5 − 1)/2 ≈ 0.618

c = b - g(b - a), \qquad d = a + g(b - a)

Keep the part that must contain the minimum

f(c) < f(d) \;\Rightarrow\; [a, d], \qquad f(c) \ge f(d) \;\Rightarrow\; [c, b]

Like bisection (Lecture 6), but compares two values instead of checking a sign
The kept interior point is reused: one new $f$ value per step, because $g^2 = 1 - g$
Bracket width × 0.618 each step: from 3 to $10^{-4}$ in 22 steps
Needs only a unimodal $f$ (one dip): no derivatives, very robust

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

6 / 26

The method, step by step

Golden-section search, step by step

Choose a bracket $[a, b]$ in which $f$ has a single minimum
Place $c = b - 0.618(b - a)$ and $d = a + 0.618(b - a)$ ; evaluate $f(c)$ , $f(d)$
If $f(c) < f(d)$ : $b \leftarrow d$ , $d \leftarrow c$ , new $c$ ; otherwise $a \leftarrow c$ , $c \leftarrow d$ , new $d$
Evaluate $f$ only at the new point
Stop when $b - a <$ tolerance; answer $(a + b)/2$

Pseudocode

input f, a, b, tol
g ← (√5 − 1)/2
c ← b − g(b − a);  d ← a + g(b − a)
while b − a > tol:
    if f(c) < f(d):
        b ← d;  d ← c
        c ← b − g(b − a)
    else:
        a ← c;  c ← d
        d ← a + g(b − a)
return (a + b)/2

Store $f(c)$ and $f(d)$ so that each step costs one evaluation.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

7 / 26

Problem 1: minimise $f(x) = x^4 - 3x^3 + 2$ on $[0, 3]$

1

c = 3 - 0.618(3) = 1.1459

,

\;d = 1.8541

;

\;f(c) = -0.790

,

f(d) = -5.304

2

f(c) > f(d)

: keep

[1.1459, 3]

; new

d = 2.2918

,

f(d) = -6.525

3

Again

f(c) > f(d)

: keep

[1.8541, 3]

; new

d = 2.5623

,

f = -5.363

4

f(c) < f(d)

: keep

[1.8541, 2.5623]

, then

[2.1246, 2.5623]

, …

5

After 22 steps:

x^{*} = 2.2500

,

f(x^{*}) = -6.5430

Check:

f'(x) = 4x^3 - 9x^2 = x^2(4x - 9) = 0

at

x = 2.25

,

f''(2.25) = 20.25 > 0

: the method found the minimum using values of

f

only.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

8 / 26

Gradient descent: step downhill, scaled by the learning rate

Learning rate (step size) η > 0

\mathbf{x}_{k+1} = \mathbf{x}_k - \eta\,\nabla f(\mathbf{x}_k)

On f(w) = (w − 3)² + 1 the error is multiplied by (1 − 2η) each step

w_{k+1} - 3 = (1 - 2\eta)(w_k - 3)

$-\nabla f$ is the direction of steepest descent
From $w_0 = 0$ : $\eta = 0.1$ : $0.6, 1.08, 1.46, \dots$ (slow); $\eta = 0.5$ : 3 in one step; $\eta = 0.9$ : overshoots but converges
$\eta = 1$ : jumps $0, 6, 0, 6$ for ever; $\eta = 1.1$ : diverges
It is Euler's method (Lecture 21) for $\mathbf{x}' = -\nabla f$ ; stable when $\eta < 2/\lambda_{\max}(H)$ , as in Lecture 22

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

9 / 26

The method, step by step

Gradient descent, step by step

Start at $\mathbf{x}_0$ ; choose $\eta$ (try 0.1, then adjust by factors of 3)
Compute the gradient $\mathbf{g} = \nabla f(\mathbf{x}_k)$
Step: $\mathbf{x}_{k+1} = \mathbf{x}_k - \eta\mathbf{g}$
Stop when $\|\mathbf{g}\|$ or the change in $f$ is below a tolerance
If $f$ increases, $\eta$ is too large: halve it. Momentum: $\mathbf{v} \leftarrow \beta\mathbf{v} - \eta\mathbf{g}$ , $\;\mathbf{x} \leftarrow \mathbf{x} + \mathbf{v}$

Pseudocode

input ∇f, x0, η, tol, kmax
x ← x0;  v ← 0
for k = 1 … kmax:
    g ← ∇f(x)
    if ‖g‖ < tol: stop
    v ← β·v − η·g    # β = 0: plain GD
    x ← x + v
return x

Machine learning: $\nabla f$ from a random mini-batch of data = stochastic gradient descent.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

10 / 26

Problem 2: gradient descent on a long, narrow valley

Minimise

f(x, y) = \tfrac12(x^2 + 10y^2)

from

(10, 1)

.

\nabla f = (x, 10y)

; the Hessian has eigenvalues 1 and 10.

1

\eta = 0.1

:

(10, 1) \to (10 - 1, 1 - 1) = (9, 0) \to (8.1, 0) \to (7.29, 0) \dots

2

Along

x

the error shrinks only by

1 - 0.1(1) = 0.9

per step: 153 steps to

\|\mathbf{x}\| < 10^{-6}

3

Larger

\eta

is limited by the steep direction:

|1 - 10\eta| < 1 \Rightarrow \eta < 0.2

; at 0.2

y

flips sign for ever, at 0.21 it diverges

4

Best fixed rate

\eta = 2/(1 + 10)

: 81 steps. Momentum (

\eta = 0.1

,

\beta = 0.5

): 41 steps

Slow convergence is set by the condition number

\lambda_{\max}/\lambda_{\min} = 10

of the Hessian, just as for iterative linear solvers (Lecture 14).

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

11 / 26

Explore: gradient descent, momentum and Newton

Move η: zig-zag across the valley, then divergence above 0.2
Momentum smooths the zig-zag
Newton jumps to the minimum of a quadratic in one step
Rosenbrock's banana valley: GD crawls; Newton reaches $(1, 1)$ in 6 steps

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

12 / 26

Newton's method for minimisation: use the curvature

One variable: Newton (Lecture 9) applied to f′

x_{k+1} = x_k - \frac{f'(x_k)}{f''(x_k)}

Several variables: solve a linear system with the Hessian each step

H(\mathbf{x}_k)\,\boldsymbol{\delta} = -\nabla f(\mathbf{x}_k), \qquad \mathbf{x}_{k+1} = \mathbf{x}_k + \boldsymbol{\delta}

Fits a parabola (quadratic model) at $\mathbf{x}_k$ and jumps to its minimum
Quadratic convergence: $f = x^4 - 3x^3 + 2$ from 3: $2.5,\ 2.291667,\ 2.251462,\ 2.250002,\ 2.25$
Exact in one step for a quadratic $f$ (any condition number)
Cost: second derivatives and an $n \times n$ solve; can head for a maximum or saddle if $H$ is not positive definite

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

13 / 26

Gauss–Newton: nonlinear least squares by repeated linear least squares

Minimise S(p) = Σ rᵢ², residuals rᵢ = yᵢ − f(xᵢ; p); Jacobian Jᵢⱼ = ∂f(xᵢ; p)/∂pⱼ

\big(J^{T}J\big)\,\boldsymbol{\delta} = J^{T}\mathbf{r}, \qquad \mathbf{p} \leftarrow \mathbf{p} + \boldsymbol{\delta}

Bacteria (Lecture 24): f = a e^{bt}, columns of J

\frac{\partial f}{\partial a} = e^{bt_i}, \qquad \frac{\partial f}{\partial b} = a\,t_i\,e^{bt_i}

Each step: linearise the model (Newton, Lecture 10), solve the normal equations (Lecture 23)
From the log-fit start $(100.29, 0.40767)$ : $(100.436, 0.407153)$ in 2 steps
From a poor start $(50, 0.3)$ : 94.7, 85.4, 96.4, 100.32, 100.436: converged by step 5
Levenberg–Marquardt adds $\lambda I$ to $J^{T}J$ for safety far from the answer

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

14 / 26

Problem 3: train the pass/fail classifier by gradient descent

1

Gradients:

\dfrac{\partial L}{\partial w} = \frac1n\sum (p_i - y_i)x_i

,

\;\dfrac{\partial L}{\partial b} = \frac1n\sum (p_i - y_i)

(simple, like least squares)

2

Start

w = b = 0

: all

p_i = 0.5

,

L = \ln 2 = 0.6931

; gradient

(-0.475, 0)

3

Step 1 (

\eta = 0.5

):

w = 0.2375

,

b = 0

,

L = 0.6458

4

After 100 steps

L = 0.445

; after 1000:

w = 1.3533

,

b = -3.7214

,

L = 0.4335

5

Newton's method on the same loss: 5 iterations to

w = 1.35341

,

b = -3.72188

p = \sigma(1.353x - 3.722)

: 50% chance at $x = 3.722/1.353 \approx 2.75$ hours; 27% after 2 hours, 73% after 3.5 hours.

Training log, learning rate 0.5

step	$w$	$b$	loss
0	0	0	0.6931
1	0.2375	0	0.6458
10	0.3493	−0.5260	0.5802
100	1.0211	−2.6795	0.4450
1000	1.3533	−3.7214	0.4335

Gradient descent: cheap steps, many of them. Newton: 5 expensive steps.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

15 / 26

Classical vs numerical

Same goal, two approaches

Calculus

Classical (solve ∇f = 0 by hand)

$f = x^4 - 3x^3 + 2$ : $x^2(4x - 9) = 0 \Rightarrow x = 2.25$
Least squares: $\nabla S = 0$ is linear (Lecture 23)
Needs equations that can be solved exactly

vs

Algorithm

Numerical (iterate downhill)

Golden section: 22 values of $f$ , no derivatives
Gradient descent or Newton: any differentiable $f$
Same code for 2 or $10^{9}$ parameters

When the classical approach cannot work

Training a neural network: minimise a loss over millions of weights

$\nabla L = 0$ is a huge system of nonlinear equations with many solutions; nobody writes it down. Stochastic gradient descent (with momentum: Adam) takes millions of small steps, each using the gradient of a small batch of data computed by backpropagation.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

16 / 26

Golden section, Newton and gradient descent in code

Golden section on $[0, 3]$ with one new $f$ per step
Newton's method on $f'$ from $x_0 = 3$
Lab: logistic regression trained by gradient descent
Compare steps and accuracy of the two methods

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

17 / 26

When optimisation goes wrong

Limitations shown failing

Local minima: from $(2, 1)$ GD stops at $x = 1.347$ , $f = -2.62$ ; the deeper valley at $x = -1.473$ has $f = -5.44$
Learning rate: too small crawls, too large diverges (bowl presets)
Ill-conditioning: narrow valleys (Rosenbrock) make GD zig-zag for thousands of steps
Newton finds any stationary point, including maxima and saddles; golden section needs a single dip

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

18 / 26

Practice

Practice problem

Problem 4"For

f(x, y) = x^2 + 3y^2 - 2x

, perform two steps of gradient descent from

(0, 1)

with

\eta = 0.1

, then one step of Newton's method from

(0, 1)

. Compare with the exact minimum."

How to write a complete answer

$\nabla f = (2x - 2,\; 6y)$ ; at $(0, 1)$ : $(-2, 6)$
Step 1: $(0, 1) - 0.1(-2, 6) = (0.2, 0.4)$ , $f = 0.12$ ; $\;$ Step 2: gradient $(-1.6, 2.4)$ → $(0.36, 0.16)$ , $f = -0.5136$
Newton: $H = \begin{bmatrix} 2 & 0 \\ 0 & 6 \end{bmatrix}$ , $\;\boldsymbol{\delta} = -H^{-1}\nabla f = (1, -1)$ → $(1, 0)$
Exact minimum $(1, 0)$ , $f = -1$ : Newton is exact in one step because $f$ is quadratic; GD needs many steps

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

19 / 26

When to use it

Optimisation methods: profile

Best case

Golden section: one variable, no derivatives, guaranteed for unimodal $f$
Gradient descent: huge numbers of parameters, cheap steps, works with noisy (stochastic) gradients
Newton / Gauss–Newton: few parameters, high accuracy in few steps

Worst case

Golden section: several variables, functions with many dips
Gradient descent: ill-conditioned valleys; sensitive to η
Newton: Hessian too large to form; non-convex regions

Best suited for

Tuning one hyperparameter (golden section on validation error)
Training ML models: linear, logistic, neural networks (SGD, Adam)
Curve fitting and calibration (Gauss–Newton, Levenberg–Marquardt)

Avoid when

Problems with a closed-form solution (linear least squares)
Non-smooth or discrete objectives (use specialised methods)
Constrained problems without modification (Lagrange multipliers, linear programming)

Rule of thumb: few variables and smooth: Newton or quasi-Newton; many variables or huge data: (stochastic) gradient descent with a tuned learning rate.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

20 / 26

Comparative analysis: optimisation methods

Method	Uses	Convergence	Cost per step ( $n$ variables)	Problem 1/2 result
Golden section	$f$ only	linear, ×0.618	1 evaluation	22 steps to $10^{-4}$
Newton (1-D)	$f'$ , $f''$	quadratic	2 derivative values	4 steps to $10^{-6}$
Gradient descent	$\nabla f$	linear, set by condition number	$O(n)$	153 steps (bowl, $\eta = 0.1$ )
GD + momentum	$\nabla f$	faster linear	$O(n)$	41 steps
Newton ( $n$ -D)	$\nabla f$ , $H$	quadratic	$O(n^3)$ solve	1 step (quadratic $f$ )
Gauss–Newton	residuals, $J$	near quadratic (small residuals)	$O(mn^2)$	2–5 steps (bacteria)

The trade-off is the same as in root finding (Lecture 10): cheap, slow, robust steps against expensive, fast, fragile ones.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

21 / 26

Beyond the classroom

The whole course inside a machine-learning library

🔢

Floating point (L2–5)

Training in 16-bit precision; loss of significance in softmax; why log-sum-exp is used

🎯

Roots and Newton (L6–10)

Newton and quasi-Newton optimisers; solving $\nabla f = 0$

🧮

Linear systems (L11–14)

Least squares, Cholesky in Gaussian processes, conditioning of features

📈

Interpolation (L15–18)

Splines for features and learning-rate schedules; Runge = overfitting

∫

Derivatives and integrals (L19–20)

Gradient checking with finite differences; expectations by quadrature

🌀

ODEs (L21–22)

Gradient descent = Euler; learning-rate limit = stability; neural ODEs

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

22 / 26

Roadmap

End of the course: where these methods lead

Every method in this course has a bigger sibling. These are the natural next steps.

What you can now do

Know how computers store numbers and lose digits
Solve equations, linear systems and ODEs
Interpolate, differentiate and integrate data
Fit models and minimise functions

LimitationHow it is overcomeWhere

Millions of parameters, noisy gradients

→

SGD, Adam, backpropagation

Machine learning

Constraints (

x \ge 0

, budgets)

→

Lagrange multipliers, linear programming

Operations research

Equations in space and time (heat, waves, fluids)

→

Finite differences and finite elements

Numerical PDEs

Huge sparse matrices and eigenvalues

→

Krylov methods (CG, GMRES), SVD

Numerical linear algebra

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

23 / 26

Applications

Where this method runs in the real world

Click a card to see optimisation at work.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

24 / 26

Checkpoint

Checkpoint · think, then discuss with your neighbourGradient descent on

f(w) = 2w^2

(

f'(w) = 4w

). For which learning rates does it converge?

A

0 < \eta < 0.5

B

0 < \eta < 1

C

0 < \eta < 2

Devery

\eta > 0

A.

w_{k+1} = w_k - 4\eta w_k = (1 - 4\eta)w_k

; converges when

|1 - 4\eta| < 1

, i.e.

0 < \eta < 0.5

. At

\eta = 0.25

it reaches 0 in one step (

\eta = 1/f''

, the Newton step).

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

25 / 26

Summary

Key points

Minimum: $\nabla f = 0$ and $H$ positive definite; local ≠ global
Golden section: bracket × 0.618 per step, values of $f$ only
Gradient descent $\mathbf{x} \leftarrow \mathbf{x} - \eta\nabla f$ ; needs $\eta < 2/\lambda_{\max}$ ; speed set by the condition number; momentum helps
Newton: $H\boldsymbol\delta = -\nabla f$ , quadratic convergence; Gauss–Newton: $(J^{T}J)\boldsymbol\delta = J^{T}\mathbf{r}$ for nonlinear least squares

Opening problem: solved

Gradient descent finds $w = 1.353$ , $b = -3.722$ : $p = \sigma(1.353x - 3.722)$ . A student needs about 2.75 hours of study for a 50% chance of passing, and 73% after 3.5 hours.

Homework

Golden-section search for the minimum of $f(x) = x^2 - 4\sin x$ on $[0, 3]$ : three steps by hand, then finish in the lab.
Gradient descent on $f(x, y) = (x - 1)^2 + 4(y + 2)^2$ from $(0, 0)$ : two steps with $\eta = 0.1$ . What is the largest stable $\eta$ ?
Newton's method for the minimum of $f(x) = e^{x} - 2x$ from $x_0 = 0$ : three iterations.
Train the classifier of Problem 3 in the lab with $\eta = 0.1$ , 0.5 and 2: compare the loss after 100 steps.

After this course

Machine learning, optimisation, numerical PDEs and numerical linear algebra all build on these 25 lectures. Keep the labs: every method here is a few lines of code you can now read, check and trust.

Md Mahbub E Noor

Lecture 25: Optimization for machine learning

University of Barishal

26 / 26

Numerical Optimization for Machine Learning

The problem: who will pass the exam?

Today's goals

The idea in one picture

Working principle: a minimum is where the slope is zero and the curve bends up

Golden-section search: shrink a bracket without derivatives

Golden-section search, step by step

Problem 1: minimise $f(x) = x^4 - 3x^3 + 2$ on $[0, 3]$

Gradient descent: step downhill, scaled by the learning rate

Gradient descent, step by step

Problem 2: gradient descent on a long, narrow valley

Explore: gradient descent, momentum and Newton

Newton's method for minimisation: use the curvature

Gauss–Newton: nonlinear least squares by repeated linear least squares

Problem 3: train the pass/fail classifier by gradient descent

Same goal, two approaches

Classical (solve ∇f = 0 by hand)

Numerical (iterate downhill)

Training a neural network: minimise a loss over millions of weights

Golden section, Newton and gradient descent in code

Limitations shown failing

Practice problem

Optimisation methods: profile

Comparative analysis: optimisation methods

The whole course inside a machine-learning library

Floating point (L2–5)

Roots and Newton (L6–10)

Linear systems (L11–14)

Interpolation (L15–18)

Derivatives and integrals (L19–20)

ODEs (L21–22)

End of the course: where these methods lead

Where this method runs in the real world

🧠 Training neural networks

🎛️ Tuning a hyperparameter

✉️ Spam filters and medical tests

📷 Camera calibration and 3-D reconstruction

Checkpoint

Summary

Numerical Optimization for Machine Learning

The problem: who will pass the exam?

Today's goals

The idea in one picture

Working principle: a minimum is where the slope is zero and the curve bends up

Golden-section search: shrink a bracket without derivatives

Golden-section search, step by step

Problem 1: minimise f(x)=x4−3x3+2f(x) = x^4 - 3x^3 + 2f(x)=x4−3x3+2 on [0,3][0, 3][0,3]

Gradient descent: step downhill, scaled by the learning rate

Gradient descent, step by step

Problem 2: gradient descent on a long, narrow valley

Explore: gradient descent, momentum and Newton

Newton's method for minimisation: use the curvature

Gauss–Newton: nonlinear least squares by repeated linear least squares

Problem 3: train the pass/fail classifier by gradient descent

Same goal, two approaches

Classical (solve ∇f = 0 by hand)

Numerical (iterate downhill)

Training a neural network: minimise a loss over millions of weights

Golden section, Newton and gradient descent in code

Limitations shown failing

Practice problem

Optimisation methods: profile

Comparative analysis: optimisation methods

The whole course inside a machine-learning library

Floating point (L2–5)

Roots and Newton (L6–10)

Linear systems (L11–14)

Interpolation (L15–18)

Derivatives and integrals (L19–20)

ODEs (L21–22)

End of the course: where these methods lead

Where this method runs in the real world

🧠 Training neural networks

🎛️ Tuning a hyperparameter

✉️ Spam filters and medical tests

📷 Camera calibration and 3-D reconstruction

Checkpoint

Summary

Problem 1: minimise $f(x) = x^4 - 3x^3 + 2$ on $[0, 3]$