From Raw Text to LLM Training Data

The 3-step pretraining pipeline: raw text becoming training data. Other models set each one up their own way — other tokenizers, other ways to pack windows, rotary positions instead of a lookup — but the principles hold.

Step 1: Text to token ids

A network consumes numbers, not strings. Each token gets an integer id from a vocabulary: encoding is a dictionary lookup, decoding the inverse. Spaces and punctuation are tokens like any other, so the ids give your text back exactly.

The word-level option is there to fail. Number every unique word in a passage and the vocabulary closes around it — the KeyError is that design showing through. <|unk|> stops the crash without fixing it: every unseen name, number and typo lands on one shared id, which the model can neither read nor write back out. Each inflected form — run, running, ran — also needs a row of its own, so the table grows with the corpus instead of converging, and most rows are seen too rarely to ever train.

Byte-pair encoding — the other three picks, and what real models use — has no closed vocabulary to fall out of. Ids 0–255 are reserved for the 256 values a byte can hold, and text is UTF-8 bytes before it is anything else, so nothing is unencodable. Coverage stops being a risk and size becomes a dial: fewer ids, smaller embedding matrix, longer sequences. GPT-2 ships no <|unk|> at all — its one reserved marker is <|endoftext|>, inserted by hand where concatenated documents end. Figures from here on are its, listed as GPT-2 on the picker and r50k_base beside it.

Merging could run forever, so size is not something BPE arrives at: the vocabulary is a budget set before training, and the merge loop stops when it is spent.

50,257 = 256 byte values + 50,000 merges + 1 special token

The ids run in that order: 0–255 the bytes, 256 onward the merges as they were learned. So id 256 is t, the most frequent pair in the corpus, and the last ids are rare words joined at the end of training. 50256 is <|endoftext|>, which no merge rule can produce. Later encodings raise the budget to 100,277 and 200,019, trading embedding rows for fewer tokens per document. That number comes back in step 3.

Step 2: Slice ids into input–target pairs

The objective is next-token prediction: given a window of tokens, predict the one that follows. So the training data is the id sequence cut by a sliding window, target y being input x shifted one position right. Drag the controls, or click any token to jump the window there.

• One window yields context_size training signals — y[0] from x[:1], y[1] from x[:2], and so on up to y[context_size - 1] from the whole of x.
• stride = context size → windows never overlap. Fewer samples, less redundancy.
• stride 1 → the most samples the text can give, at the cost of near-duplicates.

Targets exist only in training. Here the next token is already in the text, so it can be used as the answer. Nobody labels anything — the corpus supplies its own targets, which is what self-supervised means. At inference the model has to produce it.

Everything here shows one window at a time, for clarity. Real training stacks many windows into a batch and runs them together — wrap this step in a PyTorch Dataset and DataLoader and you have that machinery. The steps do not change: the same slicing, the same lookup, several samples at once. batch_size is the leading dimension of every tensor from here on.

Step 3: Embed the ids

An id is still just an index — 287 means nothing more than 286. The model's first layer looks up row id of a trainable token embedding matrix and gets a dense vector. A lookup, nothing more: mathematically a one-hot vector times a weight matrix, minus the waste. The rows here are freshly initialized — the moment before training starts.

The same id always fetches the same row, so GPT-2 adds a second embedding indexed by position and sums the two element by element. Read any column straight down: the token value t^(i) plus the position value p^(i) is the input embeddings row x^(i) underneath, and that sum is what the transformer actually receives. Switch position embeddings off to see what it would get without them — the same row for a token wherever it lands in the window.

The [batch_size, context_size, dims] tensor that comes out is what enters the transformer. The real GPT-2 uses 768 dimensions, so its embedding matrix is 50,257 × 768 — 38.6M parameters, nearly a third of the 124M model, before a single transformer block. Flip the encoding and watch that count move: embedding rows are where a large vocabulary gets paid for.

Takeaway

Nothing in these three steps is intelligent. A dictionary lookup, a window slid along an array, a row fetched and added to another. The pipeline holds two learnable things — the token embedding matrix and the positional one — and you just watched both start as noise.

What it produces is data, and nobody labeled any of it. The text supplies its own targets, one per position — a 1,024-token window is 1,024 predictions, not one. That is why pretraining scales to whatever text you can collect, instead of stopping at whatever humans can annotate.

Every setting here is a trade. Raise the vocabulary size and each document costs fewer tokens, but the embedding matrix grows by another 768-number row. Shorten the stride and the same corpus yields more samples, each one more like the last. None of these are learned — you fix them before the run, and the run is expensive.

The whole window moves at once, so all 1,024 of those predictions come out of a single forward pass rather than one after another. That is what the transformer architecture actually bought — scaling.

That [batch_size, 1024, 768] tensor is where the transformer starts. Its core module, attention, takes it as an input and builds a context vector per token. Nothing in that computation depends on a token's position. the dog bites the man and the man bites the dog are the same five tokens reordered, so dog comes out with the identical context vector in both — one representation, whether it bites or is bitten. Order has to be in the vectors already, which is what p^(i) is for. Switch it off at step 3 and attention gets a bag of tokens, which training then has to work around.

References

Sebastian Raschka, Build a Large Language Model (From Scratch) (Manning, 2024). Chapter 2 walks this same ground on paper and is the best from-first-principles treatment of it I know of.