Meet the Decoder: GPT-2 Writes One Token at a Time

The other half of the transformer family: the decoder. How GPT-2 generates text — next-token prediction, the autoregressive loop run by hand so you can see the probabilities of one step, then generate() doing the same thing — and why a model that continues text cannot answer a question.

Everything you’ve run so far reads. DistilBERT classifies, MiniLM embeds, your search engine ranks — all of it takes text in and hands back a label or a vector. None of it ever wrote a sentence. This chapter meets the half of the transformer family that does: the decoder, the architecture behind GPT and every chatbot you’ve used. We’ll start where it started, with GPT-2 — small, ungated, and simple enough to understand completely — and we will watch it take one step, by hand, before letting the library run the loop.

It’s the same block you read in chapter 14, pointed at a different job.

▶ Open the companion notebook in Colab

Run against: PyTorch 2.2.2 and transformers 4.57.6 on CPU (macOS), with distilbert/distilgpt2.

Encoder reads, decoder writes

Back in chapter 13 the transformer split into two flavors, and the difference is just what each word is allowed to look at:

  • An encoder (BERT) lets every word see the whole sentence, both directions. Perfect for understanding a fixed piece of text.
  • A decoder (GPT) lets each word see only the words before it — never ahead. That one restriction is what makes generation possible.

Why does “only look back” enable writing? Because generating text is a guessing game played left to right: given everything so far, predict the next token. A decoder is trained to do exactly that — read a run of text and output a probability for every possible next token. If it could peek ahead, the game would be rigged; blinding it to the future is what forces it to actually predict.

One step, by hand

The whole chapter is in one forward pass, so let’s look at one. Load the small GPT-2 the way you loaded DistilBERT in chapter 17 — AutoModelForCausalLM is the task class for generation; “causal” is the formal word for the look-back rule — and push a prompt through it:

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("distilbert/distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2").eval()

inputs = tok("The best thing about PyTorch is", return_tensors="pt")
with torch.no_grad():
    logits = model(**inputs).logits
print(logits.shape)   # torch.Size([1, 8, 50257])

Read that shape with chapter 5’s eyes: one input, eight token positions, and 50,257 scores at each position — one per token in GPT-2’s vocabulary. The classifier in chapter 17 gave you two logits per input; this gives you a full vocabulary’s worth, at every position. The position we care about is the last one, because the score there is the model’s guess at what comes after the prompt:

next_logits = logits[0, -1]              # the last position: 50,257 scores
probs = next_logits.softmax(dim=-1)      # chapter 6's softmax, over the whole vocabulary
top = torch.topk(probs, 5)
for p, i in zip(top.values, top.indices):
    print(f"{tok.decode(i)!r:>8}  {p.item():.3f}")
  ' that'  0.642
    ' it'  0.071
   ' the'  0.067
   ' its'  0.036
   ' you'  0.024

There is the entire mechanism on five lines: a probability distribution over the next token, and the model is 64% sure the next one is ” that”. (The leading space is part of the token — GPT-2’s vocabulary bakes word boundaries into its pieces, which is why the decoded strings start with one.) Everything the next chapter calls “decoding” is a rule for choosing from this list. The simplest rule is to take the top one.

Generation is a loop

Here’s the part that surprises people: a model doesn’t write a sentence in one shot. It produces one token, sticks it on the end of the input, and runs again — over and over. That’s the autoregressive loop, and having seen one step, you can write it:

ids = inputs["input_ids"]
for _ in range(5):
    with torch.no_grad():
        nxt = model(ids).logits[0, -1].argmax()        # the top token at the last position
    ids = torch.cat([ids, nxt.view(1, 1)], dim=1)      # append it, go round again
print(tok.decode(ids[0]))
# The best thing about PyTorch is that it's a very

Five turns, five tokens, each one chosen from a fresh distribution computed over the text so far. That is what every language model does, including the giant ones — the loop is the same, the model inside it is bigger. The library’s generate runs this loop for you, and you can check it is the same loop:

out = model.generate(**inputs, max_new_tokens=5, do_sample=False)
print(tok.decode(out[0]) == tok.decode(ids[0]))   # True

max_new_tokens=5 says “run the loop five times”; do_sample=False says “take the top token,” which is what the hand-written loop did. The loop stops early if the model emits its end-of-text token or runs out of budget.

The easy button

With the mechanism in hand, the three-line version is the one you’ll use day to day — pipeline, the way you ran sentiment in chapter 7, now pointed at "text-generation":

from transformers import pipeline

gen = pipeline("text-generation", model="distilbert/distilgpt2")
out = gen("The best thing about PyTorch is", max_new_tokens=30)
print(out[0]["generated_text"])

distilgpt2 is the distilled, 82-million-parameter GPT-2 — ungated and small enough to run on a plain CPU. What comes back is your prompt with thirty generated tokens tacked on. You gave it a beginning; it wrote a continuation.

Run that cell three times and you get three different continuations. That is not the loop above, which was greedy and deterministic. GPT-2’s config carries a set of task-specific parameters — {"text-generation": {"do_sample": true, "max_length": 50}} — and the pipeline applies them, so by default it samples the next token rather than taking the top one. model.generate ignores those parameters and is greedy unless told otherwise. Neither default is wrong; the lesson is that the pipeline and the raw call are the same loop with different dice, and the next chapter is about the dice.

It continues — it does not converse

Run GPT-2 a few times and you’ll notice something: it’s good at continuing text and clueless about answering you. Ask it a question and watch what the greedy loop does:

print(gen("What is the capital of France?", max_new_tokens=25, do_sample=False)[0]["generated_text"])
# What is the capital of France?
#
#
#
# ...

Twenty-five newlines. The most likely continuation of a lone question on a web page, as far as GPT-2 learned, is blank space — the question sat at the end of a paragraph and nothing followed. That’s not a bug — GPT-2 was trained only to predict the next token on web text, so it does the most natural continuation, not the helpful one. It has no idea you want an answer.

That gap — from “plausible continuation” to “helpful answer” — is the whole story of the next two chapters. First we’ll learn to control how it picks tokens (which fixes a lot of the wandering), then we’ll meet the instruction-tuned models that were specifically taught to answer rather than ramble.

Gotchas

  • GPT-2 autocompletes; it doesn’t follow instructions. Treat it as a text continuer. For question-answering and chat you want an instruction-tuned model (chapter 24) — feeding GPT-2 a question and expecting an answer is the most common disappointment here.
  • AutoModelForCausalLM, not AutoModel. The plain backbone gives hidden states; the ...ForCausalLM class adds the head that turns them into 50,257 scores. Generation needs the head.
  • The logits you want are at the last position. logits[0, -1] is the next-token distribution; logits[0, 0] is the model’s guess at what follows the first token, which it already knows.
  • The pipeline samples, generate doesn’t. Same model, different defaults, for the reason above. Pass do_sample explicitly and both behave the same.
  • max_new_tokens controls cost. It’s how many loop turns to run. Bigger means longer output and longer waits — generation time scales with the number of tokens.
  • It will happily make things up. A model that predicts plausible next words has no notion of truth. Confident, fluent, and wrong is a normal output — never read generated text as fact.

What’s next

You’ve met the decoder, watched one step of it produce a distribution over 50,257 tokens, and run the loop yourself. But we glossed over the most interesting step: at each turn the model hands you that whole distribution — and how you choose from it completely changes the result, from robotic repetition to wild invention.

Next: Greedy, Random, or Wise: How a Model Picks the Next Word.

Comments