We’ve explored the world of tokenization, transforming text into a format understandable by Large Language Models (LLMs). Now, we embark on a critical step: generating input-target pairs, the fuel for LLM training.
Understanding Input-Target Pairs
Recall from Episode 1 that LLMs excel at predicting the next word in a sequence. Input-target pairs provide them with bite-sized examples to learn from. Each pair consists of two parts:
- Input (x): This represents a sequence of tokens, like a mini-sentence (eg, „the quick brown fox“).
- Target (y): This is the next word that follows the input sequence (eg, „jumps“).
By creating numerous input-target pairs using a sliding window approach, we provide the LLM with diverse contexts to learn from. Imagine an LLM being shown flashcards with sentence snippets and their missing next words. These pairs progressively equip the LLM to grasp the flow and logic of language.
Looking Ahead
With these input-target pairs in hand, the next chapter will introduce embeddings – a powerful tool for bridging the gap between discrete tokens and the LLM’s internal understanding of language. Embeddings will transform tokens into numerical vectors, allowing the LLM to process and analyze language in a way more akin to the human brain. Stay tuned for this exciting revelation!
In this section, we design a data loader that uses a sliding window technique to get the input-target pairings shown in Figure 1 from the training dataset.
with open("the-verdict.txt", "r", encoding="utf-8") as f:
raw_text = f.read()
enc_text = tokenizer.encode(raw_text) print(len(enc_text)) After applying the BPE tokenizer, running the code above will yield 5145 , which is the total number of tokens in the training set. For demonstration reasons, we then eliminate the first 50 tokens from the dataset because doing so produces a marginally more engaging text passage in the following steps:
enc_sample = enc_text[50:] Making two variables, x and y, with x holding the input tokens and y holding the targets, which are the inputs shifted by one, is among the simplest and most natural approaches to generate the input-target pairs for the next word prediction task:
context_size = 4 #A
x = enc_sample[:context_size]
y = enc_sample[1:context_size+1]
print(f"x: {x}")
print(f"y: {y}")
# The number of tokens in the input is determined by the context size. When the code above is run, the following output is printed:
x: [290, 4920, 2241, 287]
y: [4920, 2241, 287, 257]The next-word prediction tasks shown before in Figure 1 can then be created by processing the inputs along with the targets, which are the inputs shifted by one position, as follows:
for i in range(1, context_size + 1):
context = enc_sample[:i]
desired = enc_sample[i]
print(context, "--->", desired)
Output:
[290] ----> 4920
[290, 4920] ----> 2241
[290, 4920, 2241] ----> 287
[290, 4920, 2241, 287] ----> 257The input to an LLM would receive is everything to the left of the arrow (—->), and the target token ID, which the LLM is meant to anticipate, is represented by the token ID on the right side of the arrow. To demonstrate, let’s rewrite the previous code with the token IDs changed to text:
for i in range(1, context_size + 1):
context = enc_sample[:i]
desired = enc_sample[i]
print(tokenizer.decode(context), "--->", tokenizer.decode([desired]))
The outputs that follow illustrate how the input and outputs appear in text format:
and ----> established
and established ----> himself
and established himself ----> in
and established himself in ----> aAs we indicated at the beginning of this chapter, there is just one more thing to do before we can convert the tokens into embeddings: create an effective data loader that loops through the input dataset and provides the inputs and targets as PyTorch tensors. Specifically, what we want is two tensors returned, as shown in Figure 2 : an input tensor with the text that the LLM sees and a target tensor with the targets that the LLM needs to forecast.
For illustration purposes, Figure 2 displays the tokens in string format; however, since the BPE tokenizer’s encode method completes tokenization and conversion into token IDs in a single step, the code implementation will work directly with token IDs. We will use the built-in Dataset and DataLoader classes from PyTorch for the effective data loader implementation.
import torch
from torch.utils.data import Dataset, DataLoader
class GPTDatasetV1(Dataset):
def __init__(self, txt, tokenizer, max_length, stride):
self.tokenizer = tokenizer
self.input_ids = []
self.target_ids = []
token_ids = tokenizer.encode(txt) # A
for i in range(0, len(token_ids) - max_length, stride): # B
input_chunk = token_ids[i:i + max_length]
target_chunk = token_ids[i + 1:i + max_length + 1]
self.input_ids.append(torch.tensor(input_chunk))
self.target_ids.append(torch.tensor(target_chunk))
def __len__(self): # C
return len(self.input_ids)
def __getitem__(self, idx): # D
return self.input_ids[idx], self.target_ids[idx]
#A Tokenize the entire text
#B Use a sliding window to chunk the book into overlapping sequences of max_length
#C Return the total number of rows in the dataset
#D Return a single row from the dataset
Depending on the PyTorch Dataset class, the GPTDatasetV1 class in the code above specifies how individual rows are collected from the dataset. Each row is made up of a number of token IDs (depending on a max_length) allocated to an input_chunk tensor. The related targets are contained in the target_chunk tensor. I suggest continuing reading to discover what happens to the data given by this dataset when it is combined with a PyTorch DataLoader; this will add further understanding and insight.The GPTDatasetV1 will be used by the following code to load the inputs in batches using a PyTorch DataLoader:
def create_dataloader_v1(txt, batch_size=4, max_length=256, stride=128, shuffle=True, drop_last=True):
tokenizer = tiktoken.get_encoding("gpt2") # A
dataset = GPTDatasetV1(txt, tokenizer, max_length, stride) # B
dataloader = DataLoader(
dataset, batch_size=batch_size, shuffle=shuffle, drop_last=drop_last) # C
return dataloader
#A Initialize the tokenizer
#B Create dataset
#C drop_last=True drops the last batch if it is shorter than the specified batch_size to prevent loss spikes during trainingTo gain an understanding of the interaction between the GPTDatasetV1 class and the create_dataloader function, let’s test the dataloader with a batch size of 1 for an LLM with a context size of 4:
with open("the-verdict.txt", "r", encoding="utf-8") as f:
raw_text = f.read()
dataloader = create_dataloader(
raw_text, batch_size=1, max_length=4, stride=1, shuffle=False
)
data_iter = iter(dataloader) # A
first_batch = next(data_iter)
print(first_batch)
#A dataloader is converted into a Python iterator using the built-in next() method in Python to retrieve the next entry.The following is printed when the previous code is executed:
[tensor([[ 40, 367, 2885, 1464]]), tensor([[ 367, 2885, 1464, 1807]])]Two tensors are present in the first_batch variable: the input token IDs are stored in the first tensor, and the target token IDs are stored in the second tensor. Each of the two tensors includes four token IDs because the max_length is set to four. Keep in mind that 4 is a pretty tiny input size and was simply selected for demonstration. LLMs are frequently trained with input sizes of at least 256. Let’s retrieve another batch from this dataset to demonstrate what stride=1 means:
second_batch = next(data_iter)
print(second_batch)The contents of the second batch are as follows:
[tensor([[ 367, 2885, 1464, 1807]]), tensor([[2885, 1464, 1807, 3619]])]The second batch’s token IDs are displaced by one position in comparison to the first batch, as can be seen if we compare the two (for instance, the second ID in the first batch’s input is 367, which is the first ID of the second batch’s input). As shown in Figure 3 , the stride setting determines how many positions the inputs shift between batches to simulate a sliding window method.
Experimenting with Data Loaders (Optional):
This section is for those curious to delve deeper into data loaders. We’ve been using a batch size of 1 for illustration, but training often utilizes larger batches. While requiring more memory, larger batches can lead to more efficient model updates.
Exploring Different Settings
If you’re familiar with deep learning, you know that hyperparameters like batch size significantly impact training. Here’s an opportunity to experiment with the data loader using different settings:
Try max_length=2 and stride=2.
Try max_length=8 and stride=2.These variations will change how the data loader extracts input-target pairs, potentially influencing the LLM’s learning process.
Batch sizes and overfitting
The provided code demonstrates using a batch size greater than 1. Note that the stride is increased to 5 (matching max_length + 1). This ensures we utilize the entire dataset without skipping any words and avoids overlap between batches. Overlap could lead to increased overfitting, where the model memorizes specific training examples instead of generalizing well.
dataloader = create_dataloader(raw_text, batch_size=8, max_length=4,
stride=5)
data_iter = iter(dataloader)
inputs, targets = next(data_iter)
print("Inputs:\n", inputs)
print("\nTargets:\n", targets)The following is printed as a result:
Inputs:
tensor([[ 40, 367, 2885, 1464],
[ 3619, 402, 271, 10899],
[ 257, 7026, 15632, 438],
[ 257, 922, 5891, 1576],
[ 568, 340, 373, 645],
[ 5975, 284, 502, 284],
[ 326, 11, 287, 262],
[ 286, 465, 13476, 11]])
Targets:
tensor([[ 367, 2885, 1464, 1807],
[ 402, 271, 10899, 2138],
[ 7026, 15632, 438, 2016],
[ 922, 5891, 1576, 438],
[ 340, 373, 645, 1049],
[ 284, 502, 284, 3285],
[ 11, 287, 262, 6001],
[ 465, 13476, 11, 339]])Looking Ahead
The intriguing idea of embeddings will be examined in the upcoming sections. Our discrete token IDs will become continuous vector representations—the LLM language—thanks to these embeddings. Await the demonstration of how these embeddings enable LLMs to understand and analyze human language!
