No. 
PDF 
Video 
 Week 1 
1 
Class #01 
Introduction To Optimization 
1a 

Course Overview 
1b 

Examples: Fermat's Problem 
1c 

Open Sets; Interior And Boundary Points 
1d 

Minimizers and Minima 
2 
Class #02 
Minimizing By Approximation 
2a 

Approximation Of Data 
2b 

Polynomial Models 
2c 

Vandermonde Matrix For Quadratics 
2d 

Quadratic Examples 
3 
Class #03 
Minimizing By Approximation 
3a 

Stationarity 
3b 

Conditions for Stationarity 
3c 

Convexity; Strict Convexity 
3d 

Gradient Inequality And Convexity 
 Week 2 
4 
Class #04 
Scalar Minimization 
4a 

Searching For A Scalar Minimizer 
4b 

Searching With A Fixed Stepsize 
4c 

Searching With a Variable Stepsize 
4d 

Armijo Backtracking 
5 
Class #05 
Functions With A Vector Argument 
5a 

Functions of Multiple Variables 
5b 

The 1Form as Linear Algebra 
5c 

Directional Derivative and the Chain Rule 
5d 

Gradient 1Form And Jacobian Matrix 
5e 

Linear Forms And Quadratic Forms 
5f 

Level Curves 
6 
Class #06 
Stationary Points 
6a 

Stationarity Example 
6b 

Conditions For Stationarity 
6c 

Second Derivative And Hessian Matrix 
6d 

Eigenvalues Of A Hessian Matrix 
 Week 3 
7 
Test #1 
Basic Scalar Optimization 
8 
Class #08 
Methods Using Steepest Descent 
8a 

Introduction To Descent 
8b 

Descent Directions 
8c 

FixedStepsize Descent 
8d 

Backtracking Descent 
9 
Class #09 
Newton's Method 
9a 

Scaling Methods 
9b 

Manual Scaling 
9c 

Newton's Method 
9d 

Damped Newton's Method 
 Week 4 
10 
Class #10 
Nonlinear Least Squares 
10a 

GPS As Vector Optimization 
10b 

Descent Methods for NLS Problems 
10c 

The LevenbergMarquardt Algorithm 
10d 

FermatWeber Problems 
11 
Class #11 
Linear Algebra For Neural Networks 
11a 

New Matrix New Products 
11b 

Quantization And The Heaviside Function 
11c 

Vectorization Of Matrices 
11d 

Kronecker Product 
11e 

Layer With Two Neurons 
11f 

Hadamard Product 
12 
Class #12 
Single Artificial Neuron 
12a 

Terms And Symbols For Neural Networks 
12b 

Activation Functions 
12c 

Steepest Descent For One Observation 
12d 

Batches Of Data 
 Week 5 
13 
Test #2 
Adaptive Vector Optimization 
14 
Class #14 
Artificial Neural Networks 
14a 

Simple Neural Network 
14b 

Network Formulation 
14c 

Network Vectorization 
14d 

Steepest Descent For Simple Network 
14e 

Batches Of Data And ExclusiveOr Example 
15 
Class #15 
BackPropagation Of Scale Factors 
15a 

Fundamental Algorithm Of Neural Networks 
15b 

Hadamard Product For Simplification 
15c 

Chain Rule And BackPropagation 
15d 

Example Computations 
 


Holiday Special 
 Week 6 
16 
Class #16 
Convexity And Level Sets 
16a 

Monotonicity And Convexity 
16b 

Convex Functions 
16c 

Convex Sets 
16d 

Level Sets 
16e 

Gradient Inequality 
17 
Class #17 
Constrained Optimization 
17a 

Constraint Properties For A Minimizer 
17b 

Linear Constraints 
17c 

Linear Objective Functions 
17d 

Convex Problems 
18 
Class #18 
Lagrange Multipliers 
18a 

Objective Gradient And Property Gradient 
18b 

Quadratic Objective And Quadratic Constraint 
18c 

Linear Objective And NonConvex Constraint 
18d 

Existence Of Lagrange Multipliers 
 Week 7 
19 
Class #19 
The Lagrange Equations 
19a 

Single Linear Constraint 
19b 

Quadratic Objective With Linear Constraints 
19c 

Example Mechanical System 
19d 

Matrix Form Of Quadratic Problems 
20 
Class #20 
Dual Formulation of Lagrange Equations 
20a 

Primal Form And Feasible Set 
20b 

MinMax And MaxMin Expressions 
20c 

Closed Form For Minimization Step 
20d 

Dual Formulation For Quadratic Problems 
21 
Class #21 
Quadratic Examples Of Dual Formulation 
21a 

Dual Formulation And Smaller Matrices 
21b 

SquaredNorm Objective Function 
21c 

Example Mechanical System 
21d 

3D Example 
21e 

Extra: Hessian Matrix 
 Week 8 
22 
Test #3 
Nonlinear Least Squares and Neural Networks 
23 
Class #23 
KKT Conditions For Constrained Optimization 
23a 

Linear Inequality Constraints 
23b 

Examples Of Linear Inequalities 
23c 

KKT Background And Examples 
23d 

Definition Of A KKT Point 
24 
Class #24 
Geometry At KKT Points 
24a 

Algebraic Interpretation Of Lagrange Multipliers 
24b 

KKT Geometry Of Inactive Linear Inequalities 
24c 

KKT Geometry Of Active Linear Inequalities 
24d 

KKT Geometry Of Linear Equality 
 Week 9 
25 
Class #25 
Constrained Least Squares 
25a 

Ordinary Least Squares Problem 
25b 

Constrained Least Squares Problem 
25c 

KKT Conditions For Constrained Least Squares 
26 
Class #26 
Tikhonov Regularization 
26a 

Objective With Quadratic Constraint 
26b 

Parameters For Tikhonov Regularization 
26c 

Discrete Total Variation Problem 
26d 

Tikhonov Regularization For Denoising 
27 
Class #27 
The Lasso And Related Regularization 
27a 

Regularization For Standardized Data 
27b 

Ridge Regression And Lasso Regularization 
27c 

Lasso Selects Variables And Constrains Regression 
27d 

Elastic Net Blends Ridge Regression And Lasso 
 Week 10 
28 
Test #4 
KKT Conditions and Constrained Least Squares 
29 
Class #29 
The Support Vector Machine (SVM) 
29a 

Hyperplane Classification 
29b 

Hyperplane Optimization Using Support Vectors 
29c 

Hyperplane Margins 
29d 

Primal Formulation Of SVM 
30 
Class #30 
Primal And Dual Formulations Of The SVM 
30a 

Design Matrix And Label Matrix 
30b 

Vectorization Of SVM Primal Formulation 
30c 

Dual Formulation Of SVM 
 Week 11 
31 
Class #31 
Slack Variables And Dual Formulations 
31a 

Separable Data Not Linearly Separable 
31b 

Slack Variables And Inequality Constraints 
31c 

Slack Variables In SVM Primal Formulation 
31d 

Slack Variables In SVM Dual Formulation 
32 
Class #32 
Gram Matrix For Nonlinear SVM 
32a 

Nonlinearly Separable Data 
32b 

Kernel Functions 
32c 

Gram Matrix And The Kernel Trick 
33 
Class #33 
The Kernel Trick For SVM 
33a 

Convex Problem For Lagrange Multipliers 
33b 

Classifying Observations And The Kernel Trick 
33c 

Examples Using Gaussian Kernels 
 Week 12 
34 
Test #5 
The Lasso and Support Vector Machines 
35 
Class #35 
Course Summary 
  
  

Symbols 
Symbols used in these notes 