12. Computational Performance

In deep learning, datasets are usually large and model computation is complex. Therefore, we are always very concerned about computing performance. This chapter will focus on the important factors that affect computing performance: imperative programming, symbolic programming, asynchronous programing, automatic parallel computation, and multi-GPU computation. By studying this chapter, you should be able to further improve the computing performance of the models that have been implemented in the previous chapters, for example, by reducing the model training time without affecting the accuracy of the model.

• 12.1. Automatic Parallelism
  • 12.1.1. Parallel Computation on CPUs and GPUs
  • 12.1.2. Parallel Computation and Communication
  • 12.1.3. Summary
  • 12.1.4. Exercises
• 12.2. Hardware
  • 12.2.1. Computers
  • 12.2.2. Memory
  • 12.2.3. Storage
  • 12.2.4. CPUs
  • 12.2.5. GPUs and other Accelerators
  • 12.2.6. Networks and Buses
  • 12.2.7. Summary
  • 12.2.8. More Latency Numbers
  • 12.2.9. Exercises
• 12.3. Concise Implementation for Multiple GPUs
  • 12.3.1. A Toy Network
  • 12.3.2. Parameter Initialization and Logistics
  • 12.3.3. Training
  • 12.3.4. Experiments
  • 12.3.5. Summary
  • 12.3.6. Exercises
  • 12.3.7. Summary
  • 12.3.8. Exercises
• 12.4. Parameter Servers
  • 12.4.1. Data Parallel Training
  • 12.4.2. Ring Synchronization
  • 12.4.3. Multi-Machine Training
  • 12.4.4. (key,value) Stores
  • 12.4.5. Summary
  • 12.4.6. Exercises