3. Linear Neural Networks

Before we get into the details of deep neural networks, we need to cover the basics of neural network training. In this chapter, we will cover the entire training process, including defining simple neural network architectures, handling data, specifying a loss function, and training the model. In order to make things easier to grasp, we begin with the simplest concepts. Fortunately, classic statistical learning techniques such as linear and softmax regression can be cast as linear neural networks. Starting from these classic algorithms, we will introduce you to the basics, providing the basis for more complex techniques in the rest of the book.

• 3.1. Linear Regression
  • 3.1.1. Basic Elements of Linear Regression
  • 3.1.2. The Normal Distribution and Squared Loss
  • 3.1.3. From Linear Regression to Deep Networks
  • 3.1.4. Summary
  • 3.1.5. Exercises
• 3.2. Linear Regression Implementation from Scratch
  • 3.2.1. Generating the Dataset
  • 3.2.2. Reading the Dataset
  • 3.2.3. Initializing Model Parameters
  • 3.2.4. Defining the Model
  • 3.2.5. Defining the Loss Function
  • 3.2.6. Defining the Optimization Algorithm
  • 3.2.7. Training
  • 3.2.8. Summary
  • 3.2.9. Exercises
• 3.3. Concise Implementation of Linear Regression
  • 3.3.1. Generating the Dataset
  • 3.3.2. Reading the Dataset
  • 3.3.3. Defining the Model
  • 3.3.4. Defining the Loss Function
  • 3.3.5. Defining the Optimization Algorithm
  • 3.3.6. Instantiate Configuration and Trainer
  • 3.3.7. Initializing Model Parameters
  • 3.3.8. Metrics
  • 3.3.9. Training
  • 3.3.10. Saving Your Model
  • 3.3.11. Summary
  • 3.3.12. Exercises
• 3.4. Softmax Regression
  • 3.4.1. Classification Problems
  • 3.4.2. Loss Function
  • 3.4.3. Information Theory Basics
  • 3.4.4. Model Prediction and Evaluation
  • 3.4.5. Summary
  • 3.4.6. Exercises
• 3.5. The Image Classification Dataset
  • 3.5.1. Getting the Dataset
  • 3.5.2. Reading a Minibatch
  • 3.5.3. Summary
  • 3.5.4. Exercises
• 3.6. Implementation of Softmax Regression from Scratch
  • 3.6.1. Initializing Model Parameters
  • 3.6.2. The Softmax
  • 3.6.3. The Model
  • 3.6.4. The Loss Function
  • 3.6.5. Classification Accuracy
  • 3.6.6. Model Training
  • 3.6.7. Prediction
  • 3.6.8. Summary
  • 3.6.9. Exercises
• 3.7. Concise Implementation of Softmax Regression
  • 3.7.1. Initializing Model Parameters
  • 3.7.2. The Softmax
  • 3.7.3. Optimization Algorithm
  • 3.7.4. Instantiate Configuration
  • 3.7.5. Initializing Trainer
  • 3.7.6. Metrics
  • 3.7.7. Training
  • 3.7.8. Exercises