7. Modern Convolutional Neural Networks

Now that we understand the basics of wiring together convolutional neural networks, we will take you through a tour of modern deep learning. In this chapter, each section will correspond to a significant neural network architecture that was at some point (or currently) the base model upon which an enormous amount of research and projects were built. Each of these networks was at briefly a dominant architecture and many were at one point winners or runners-up in the famous ImageNet competition, which has served as a barometer of progress on supervised learning in computer vision since 2010.

These models include AlexNet, the first large-scale network deployed to beat conventional computer vision methods on a large-scale vision challenge; the VGG network, which makes use of a number of repeating blocks of elements; the network in network (NiN) which convolves whole neural networks patch-wise over inputs; the GoogLeNet, which makes use of networks with parallel concatenations; residual networks (ResNet), which are the most popular go-to architecture today, and densely connected networks (DenseNet), which are expensive to compute but have set some recent benchmarks.

• 7.1. Deep Convolutional Neural Networks (AlexNet)
  • 7.1.1. Learning Feature Representation
  • 7.1.2. AlexNet
  • 7.1.3. Reading the Dataset
  • 7.1.4. Training
  • 7.1.5. Summary
  • 7.1.6. Exercises
• 7.2. Networks Using Blocks (VGG)
  • 7.2.1. VGG Blocks
  • 7.2.2. VGG Network
  • 7.2.3. Model Training
  • 7.2.4. Summary
  • 7.2.5. Exercises
• 7.3. Network in Network (NiN)
  • 7.3.1. NiN Blocks
  • 7.3.2. NiN Model
  • 7.3.3. Data Acquisition and Training
  • 7.3.4. Summary
  • 7.3.5. Exercises
• 7.4. Networks with Parallel Concatenations (GoogLeNet)
  • 7.4.1. Inception Blocks
  • 7.4.2. GoogLeNet Model
  • 7.4.3. Data Acquisition and Training
  • 7.4.4. Summary
  • 7.4.5. Exercises
• 7.5. Batch Normalization
  • 7.5.1. Training Deep Networks
  • 7.5.2. Batch Normalization Layers
  • 7.5.3. Implementation from Scratch
  • 7.5.4. Using a Batch Normalization LeNet
  • 7.5.5. Concise Implementation
  • 7.5.6. Controversy
  • 7.5.7. Summary
  • 7.5.8. Exercises
• 7.6. Residual Networks (ResNet)
  • 7.6.1. Function Classes
  • 7.6.2. Residual Blocks
  • 7.6.3. ResNet Model
  • 7.6.4. Data Acquisition and Training
  • 7.6.5. Summary
  • 7.6.6. Exercises
• 7.7. Densely Connected Networks (DenseNet)
  • 7.7.1. Function Decomposition
  • 7.7.2. Dense Blocks
  • 7.7.3. Transition Layers
  • 7.7.4. DenseNet Model
  • 7.7.5. Data Acquisition and Training
  • 7.7.6. Summary
  • 7.7.7. Exercises