6. Convolutional Neural Networks

In earlier chapters, we came up against image data, for which each example consists of a 2D grid of pixels. Depending on whether we’re handling black-and-white or color images, each pixel location might be associated with either one or multiple numerical values, respectively. Until now, our way of dealing with this rich structure was deeply unsatisfying. We simply discarded each image’s spatial structure by flattening them into 1D vectors, feeding them through a (fully connected) MLP. Because these networks invariant to the order of the features we could get similar results regardless of whether we preserve an order corresponding to the spatial structure of the pixels or if we permute the columns of our design matrix before fitting the MLP’s parameters. Preferably, we would leverage our prior knowledge that nearby pixels are typically related to each other, to build efficient models for learning from image data.

This chapter introduces convolutional neural networks (CNNs), a powerful family of neural networks that were designed for precisely this purpose. CNN-based architectures are now ubiquitous in the field of computer vision, and have become so dominant that hardly anyone today would develop a commercial application or enter a competition related to image recognition, object detection, or semantic segmentation, without building off of this approach.

Modern ConvNets, as they are called colloquially owe their design to inspirations from biology, group theory, and a healthy dose of experimental tinkering. In addition to their sample efficiency in achieving accurate models, convolutional neural networks tend to be computationally efficient, both because they require fewer parameters than dense architectures and because convolutions are easy to parallelize across GPU cores. Consequently, practitioners often apply CNNs whenever possible, and increasingly they have emerged as credible competitors even on tasks with 1D sequence structure, such as audio, text, and time series analysis, where recurrent neural networks are conventionally used. Some clever adaptations of CNNs have also brought them to bear on graph-structured data and in recommender systems.

First, we will walk through the basic operations that comprise the backbone of all convolutional networks. These include the convolutional layers themselves, nitty-gritty details including padding and stride, the pooling layers used to aggregate information across adjacent spatial regions, the use of multiple channels (also called filters) at each layer, and a careful discussion of the structure of modern architectures. We will conclude the chapter with a full working example of LeNet, the first convolutional network successfully deployed, long before the rise of modern deep learning. In the next chapter, we will dive into full implementations of some popular and comparatively recent CNN architectures whose designs represent most of the techniques commonly used by modern practitioners.

• 6.1. From Dense Layers to Convolutions
  • 6.1.1. Invariances
  • 6.1.2. Constraining the MLP
  • 6.1.3. Convolutions
  • 6.1.4. Waldo Revisited
  • 6.1.5. Summary
  • 6.1.6. Exercises
• 6.2. Convolutions for Images
  • 6.2.1. The Cross-Correlation Operator
  • 6.2.2. Convolutional Layers
  • 6.2.3. Object Edge Detection in Images
  • 6.2.4. Learning a Kernel
  • 6.2.5. Cross-Correlation and Convolution
  • 6.2.6. Summary
  • 6.2.7. Exercises
• 6.3. Padding and Stride
  • 6.3.1. Padding
  • 6.3.2. Stride
  • 6.3.3. Summary
  • 6.3.4. Exercises
• 6.4. Multiple Input and Output Channels
  • 6.4.1. Multiple Input Channels
  • 6.4.2. Multiple Output Channels
  • 6.4.3. \(1\times 1\) Convolutional Layer
  • 6.4.4. Summary
  • 6.4.5. Exercises
• 6.5. Pooling
  • 6.5.1. Maximum Pooling and Average Pooling
  • 6.5.2. Padding and Stride
  • 6.5.3. Multiple Channels
  • 6.5.4. Summary
  • 6.5.5. Exercises
• 6.6. Convolutional Neural Networks (LeNet)
  • 6.6.1. LeNet
  • 6.6.2. Data Acquisition and Training
  • 6.6.3. Summary
  • 6.6.4. Exercises