11. Optimization Algorithms

If you read the book in sequence up to this point you already used a number of advanced optimization algorithms to train deep learning models. They were the tools that allowed us to continue updating model parameters and to minimize the value of the loss function, as evaluated on the training set. Indeed, anyone content with treating optimization as a black box device to minimize objective functions in a simple setting might well content oneself with the knowledge that there exists an array of incantations of such a procedure (with names such as “Adam”, “NAG”, or “SGD”).

To do well, however, some deeper knowledge is required. Optimization algorithms are important for deep learning. On one hand, training a complex deep learning model can take hours, days, or even weeks. The performance of the optimization algorithm directly affects the model’s training efficiency. On the other hand, understanding the principles of different optimization algorithms and the role of their parameters will enable us to tune the hyperparameters in a targeted manner to improve the performance of deep learning models.

In this chapter, we explore common deep learning optimization algorithms in depth. Almost all optimization problems arising in deep learning are nonconvex. Nonetheless, the design and analysis of algorithms in the context of convex problems has proven to be very instructive. It is for that reason that this section includes a primer on convex optimization and the proof for a very simple stochastic gradient descent algorithm on a convex objective function.

• 11.1. Optimization and Deep Learning
  • 11.1.1. Optimization and Estimation
  • 11.1.2. Optimization Challenges in Deep Learning
  • 11.1.3. Summary
  • 11.1.4. Exercises
• 11.2. Convexity
  • 11.2.1. Basics
  • 11.2.2. Properties
  • 11.2.3. Constraints
  • 11.2.4. Summary
  • 11.2.5. Exercises
• 11.3. Gradient Descent
  • 11.3.1. Gradient Descent in One Dimension
  • 11.3.2. Multivariate Gradient Descent
  • 11.3.3. Adaptive Methods
  • 11.3.4. Summary
  • 11.3.5. Exercises
• 11.4. Stochastic Gradient Descent
  • 11.4.1. Stochastic Gradient Updates
  • 11.4.2. Dynamic Learning Rate
  • 11.4.3. Convergence Analysis for Convex Objectives
  • 11.4.4. Stochastic Gradients and Finite Samples
  • 11.4.5. Summary
  • 11.4.6. Exercises
• 11.5. Minibatch Stochastic Gradient Descent
  • 11.5.1. Vectorization and Caches
  • 11.5.2. Minibatches
  • 11.5.3. Reading the Dataset
  • 11.5.4. Implementation from Scratch
  • 11.5.5. Concise Implementation
  • 11.5.6. Summary
  • 11.5.7. Exercises
• 11.6. Momentum
  • 11.6.1. Basics
  • 11.6.2. Practical Experiments
  • 11.6.3. Theoretical Analysis
  • 11.6.4. Summary
  • 11.6.5. Exercises
• 11.7. Adagrad
  • 11.7.1. Sparse Features and Learning Rates
  • 11.7.2. Preconditioning
  • 11.7.3. The Algorithm
  • 11.7.4. Implementation from Scratch
  • 11.7.5. Concise Implementation
  • 11.7.6. Summary
  • 11.7.7. Exercises
• 11.8. RMSProp
  • 11.8.1. The Algorithm
  • 11.8.2. Implementation from Scratch
  • 11.8.3. Concise Implementation
  • 11.8.4. Summary
  • 11.8.5. Exercises
• 11.9. Adadelta
  • 11.9.1. The Algorithm
  • 11.9.2. Implementation
  • 11.9.3. Summary
  • 11.9.4. Exercises
• 11.10. Adam
  • 11.10.1. The Algorithm
  • 11.10.2. Implementation
  • 11.10.3. Yogi
  • 11.10.4. Summary
  • 11.10.5. Exercises
• 11.11. Learning Rate Scheduling
  • 11.11.1. Toy Problem
  • 11.11.2. Trackers
  • 11.11.3. Policies
  • 11.11.4. Summary
  • 11.11.5. Exercises