The two linear layers each transformer block runs on every token independently: a projection up to four times the width, a nonlinear activation, and a projection back down.

A technique that stabilizes training by making the next layer's input scale independent of how large the values from earlier layers grew. It runs identically in training and at inference.

Regularization by removal: a random subset of a tensor's elements zeroed on every training step, the survivors scaled back up, and a different sub-network trained each time.

Attention on your own text: every sequence token projected to queries, keys, and values, scored against every other, then masked so nothing reads ahead, and split across heads.

The GPT-2 style pretraining data pipeline, running on your own text: split into tokens and mapped to ids, sliced into input–target pairs, embedded as vectors.

Before a language model reads a single word, your text is chopped into tokens. Paste anything into the playground and watch exactly how a GPT-style tokenizer sees it.