14. Natural Language Processing: Pretraining

Humans need to communicate. Out of this basic need of the human condition, a vast amount of written text has been generated on an everyday basis. Given rich text in social media, chat apps, emails, product reviews, news articles, research papers, and books, it becomes vital to enable computers to understand them to offer assistance or make decisions based on human languages.

Natural language processing studies interactions between computers and humans using natural languages. In practice, it is very common to use natural language processing techniques to process and analyze text (human natural language) data, such as language models in Section 8.3 and machine translation models in Section 9.5.

To understand text, we can begin with its representation, such as treating each word or subword as an individual text token. As we will see in this chapter, the representation of each token can be pretrained on a large corpus, using word2vec, GloVe, or subword embedding models. After pretraining, representation of each token can be a vector, however, it remains the same no matter what the context is. For instance, the vector representation of “bank” is the same in both “go to the bank to deposit some money” and “go to the bank to sit down”. Thus, many more recent pretraining models adapt representation of the same token to different contexts. Among them is BERT, a much deeper model based on the Transformer encoder. In this chapter, we will focus on how to pretrain such representations for text, as highlighted in Fig. 14.1.

Fig. 14.1 Pretrained text representations can be fed to various deep learning architectures for different downstream natural language processing applications. This chapter focuses on the upstream text representation pretraining.

As shown in Fig. 14.1, the pretrained text representations can be fed to a variety of deep learning architectures for different downstream natural language processing applications. We will cover them in chap_nlp_app.

• 14.1. Word Embedding (word2vec)
  • 14.1.1. Why Not Use One-hot Vectors?
  • 14.1.2. The Skip-Gram Model
  • 14.1.3. The Continuous Bag of Words (CBOW) Model
  • 14.1.4. Summary
  • 14.1.5. Exercises
• 14.2. Approximate Training
  • 14.2.1. Negative Sampling
  • 14.2.2. Hierarchical Softmax
  • 14.2.3. Summary
  • 14.2.4. Exercises
• 14.3. The Dataset for Pretraining Word Embedding
  • 14.3.1. Reading and Preprocessing the Dataset
  • 14.3.2. Subsampling
  • 14.3.3. Loading the Dataset
  • 14.3.4. Putting All Things Together
  • 14.3.5. Summary
  • 14.3.6. Exercises
• 14.4. Word Embedding with Global Vectors (GloVe)
  • 14.4.1. The GloVe Model
  • 14.4.2. Understanding GloVe from Conditional Probability Ratios
  • 14.4.3. Summary
  • 14.4.4. Exercises
• 14.5. Subword Embedding
  • 14.5.1. fastText
  • 14.5.2. Byte Pair Encoding
  • 14.5.3. Summary
  • 14.5.4. Exercises