Dropout is a regularization method for neural networks: it keeps a model from
overfitting, from memorizing its training data instead of learning what generalizes beyond
it. During training it zeroes each element of a tensor independently with probability p
and divides the survivors by 1 − p. A new mask is drawn on every training step, so every
step trains a different sub-network. At inference it does nothing at all.
The division is the half that is easy to miss. Without it a layer's output would be smaller
in training than at inference, and every layer downstream would meet a scale it was never
trained on. Dividing by 1 − p restores that scale on average — at p = 0.5 an element is
zeroed half the time and doubled the rest — so no single step reproduces the value that was
there, but its expected value is unchanged and inference has nothing to undo. This is
inverted dropout, what torch.nn.Dropout(p) does.
Everything else in this series is part of what the model computes. Dropout is not: take it out of a trained model and every output is unchanged, because inference already runs with it off. Its only job is to make training harder.
On the attention weights
Dropout falls on activations: a layer's outputs, not the weights, which are learned. The attention weights are activations too, but constrained ones — softmax leaves every row positive and summing to 1, which is what makes a context vector a weighted average of the value vectors and not merely a weighted sum.
The card above applies dropout to the causal attention weights from the
attention chapter, after the softmax and before they produce the
context vectors. The row sums do not survive it: nothing renormalizes a row after the draw, so
a sum lands anywhere from 0 to 1/(1 − p), and some of the tokens the row attended to
contribute nothing while the rest count for more than the softmax gave them. The weighted
average is gone; what is left is a weighted sum.
On ordinary activations
Everywhere else in the model, dropout falls on a [T, d] matrix, one vector of dimension d
per token: the mask and the rescale, nothing more. One of these places is the embedding sum,
before the first transformer block.
The other two sit on a sublayer — the attention module or the feed-forward network, each wrapped in a residual connection. The sublayer's output is added to the shortcut, the residual stream as it stood before the sublayer ran, and dropout masks that output before the addition. The shortcut is never masked, so where an element is zeroed the stream keeps the value it already had.
Training and inference
Dropout runs in training mode only. In inference mode every value passes through unchanged — no mask and no rescale. A model that generates while still in training mode samples a different sub-network on every forward pass, so it answers the same question differently every time, and answers worse on average than the complete model would.
Why it helps, and when it is skipped
No neuron can count on any other being present. One that is only useful while some specific other is active contributes nothing on the steps where that one is dropped, so training favors features that work on their own. This is the co-adaptation argument from Srivastava et al. And because every step trains a different sub-network, the complete model at inference stands in for an ensemble of all of them.
Whether that is worth having depends on the run. Dropout earns its place where memorization is a real risk: a small dataset, passed over many times. That is fine-tuning, not a pretraining run that sees trillions of tokens once — which is why many recent recipes set the pretraining rate to zero and bring dropout back only to fine-tune. It is a hyperparameter, not a fixed part of the architecture.
References
Sebastian Raschka, Build a Large Language Model (From Scratch) (Manning, 2024). Chapter 3 applies dropout to the attention weights, chapter 4 to the rest of the block.
Nitish Srivastava et al., Dropout: A Simple Way to Prevent Neural Networks from Overfitting (JMLR, 2014). The paper the method comes from.