5. Deep Learning Computation

Alongside giant datasets and powerful hardware, great software tools have played an indispensable role in the rapid progress of deep learning. Starting with the pathbreaking Theano library released in 2007, flexible open-source tools have enabled researchers to rapidly prototype models, avoiding repetitive work when recycling standard components while still maintaining the ability to make low-level modifications. Over time, deep learning’s libraries have evolved to offer increasingly coarse abstractions. Just as semiconductor designers went from specifying transistors to logical circuits to writing code, neural networks researchers have moved from thinking about the behavior of individual artificial neurons to conceiving of networks in terms of whole layers, and now often design architectures with far coarser blocks in mind.

So far, we have introduced some basic machine learning concepts, ramping up to fully-functional deep learning models. In the last chapter, we implemented each component of a multilayer perceptron from scratch and even showed how to leverage high-level APIs to roll out the same models effortlessly. To get you that far that fast, we called upon the libraries, but skipped over more advanced details about how they work. In this chapter, we will peel back the curtain, digging deeper into the key components of deep learning computation, namely model construction, parameter access and initialization, designing custom layers and blocks, reading and writing models to disk, and leveraging GPUs to achieve dramatic speedups. These insights will move you from end user to power user, giving you the tools needed to reap the benefits of a mature deep learning library while retaining the flexibility to implement more complex models, including those you invent yourself! While this chapter does not introduce any new models or datasets, the advanced modeling chapters that follow rely heavily on these techniques.

• 5.1. Layers and Blocks
  • 5.1.1. A Custom Block
  • 5.1.2. The Sequential Block
  • 5.1.3. Executing Code in the forward Method
  • 5.1.4. Summary
  • 5.1.5. Exercises
• 5.2. Parameter Management
  • 5.2.1. Parameter Access
  • 5.2.2. Parameter Initialization
  • 5.2.3. Tied Parameters
  • 5.2.4. Summary
  • 5.2.5. Exercises
• 5.3. Custom Layers
  • 5.3.1. Layers without Parameters
  • 5.3.2. Layers with Parameters
  • 5.3.3. Summary
  • 5.3.4. Exercises
• 5.4. File I/O
  • 5.4.1. Loading and Saving Tensors
  • 5.4.2. Model Parameters
  • 5.4.3. Summary
  • 5.4.4. Exercises
• 5.5. GPUs
  • 5.5.1. Computing Devices
  • 5.5.2. Tensors and GPUs
  • 5.5.3. Neural Networks and GPUs
  • 5.5.4. Summary
  • 5.5.5. Exercises