Meet the Decoder: GPT-2 Writes One Token at a Time

Series: Practical PyTorch: Running Models — LLMs — Part 3 of 6

Everything you’ve run so far reads. DistilBERT classifies, MiniLM embeds, your search engine ranks — all of it takes text in and hands back a label or a vector. None of it ever wrote a sentence. This post meets the half of the transformer family that does: the decoder, the architecture behind GPT and every chatbot you’ve used. We’ll start where it started, with GPT-2 — small, ungated, and simple enough to understand completely.

It’s the same block you read in the post on reading a real transformer, pointed at a different job.

▶ Open the companion notebook in Colab

Encoder reads, decoder writes

Back in the attention and the transformer post the transformer split into two flavors, and the difference is just what each word is allowed to look at:

• An encoder (BERT) lets every word see the whole sentence, both directions. Perfect for understanding a fixed piece of text.
• A decoder (GPT) lets each word see only the words before it — never ahead. That one restriction is what makes generation possible.

Why does “only look back” enable writing? Because generating text is a guessing game played left to right: given everything so far, predict the next word. A decoder is trained to do exactly that — read a run of text and output a probability for every possible next token. If it could peek ahead, the game would be rigged; blinding it to the future is what forces it to actually predict.

Generation is a loop

Here’s the part that surprises people: a model doesn’t write a sentence in one shot. It produces one token, sticks it on the end of the input, and runs again — over and over. That’s the autoregressive loop:

1. Feed in the prompt → the model outputs scores for the next token.
2. Pick a token (the next post is all about how you pick).
3. Append it to the text.
4. Go back to step 1, now with the longer text.

Repeat until it emits a “stop” token or hits your length limit. Every word a language model writes came out of this same loop, one turn at a time — including the giant ones. GPT-2 is the small, legible version of the exact machine inside the big chatbots.

Running GPT-2

The easy button first — pipeline, the way you ran sentiment in the pipelines post, now pointed at "text-generation":

from transformers import pipeline

gen = pipeline("text-generation", model="distilgpt2")
out = gen("The best thing about PyTorch is", max_new_tokens=30)
print(out[0]["generated_text"])

distilgpt2 is the distilled, 82-million-parameter GPT-2 — ungated and small enough to run on a plain CPU. max_new_tokens=30 says “run the loop 30 times.” What comes back is your prompt with thirty generated tokens tacked on. You gave it a beginning; it wrote a continuation.

To see the loop isn’t magic, drop one level — same from_pretrained move as the beyond-pipelines post:

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained("distilgpt2")
model = AutoModelForCausalLM.from_pretrained("distilgpt2")

inputs = tok("The best thing about PyTorch is", return_tensors="pt")
output_ids = model.generate(**inputs, max_new_tokens=30)
print(tok.decode(output_ids[0], skip_special_tokens=True))

Two names to notice. AutoModelForCausalLM is the task class for generation — “causal” is the formal word for “each token depends only on the ones before it,” the look-back rule from earlier. And model.generate is the autoregressive loop, implemented for you: it runs the model, picks a token, appends, and repeats, all inside that one call. You hand it tokenized input (the tokenizer from the tokens and model inputs post) and it hands back token IDs to decode.

It continues — it does not converse

Run GPT-2 a few times and you’ll notice something: it’s good at continuing text and clueless about answering you. Ask it “What is the capital of France?” and it might continue with another question, or wander off topic. That’s not a bug — GPT-2 was trained only to predict the next token on web text, so it does the most natural continuation, not the helpful one.

That gap — from “plausible continuation” to “helpful answer” — is the whole story of the next two posts. First we’ll learn to control how it picks tokens (which fixes a lot of the wandering), then we’ll meet the instruction-tuned models that were specifically taught to answer rather than ramble.

Gotchas

• GPT-2 autocompletes; it doesn’t follow instructions. Treat it as a text continuer. For question-answering and chat you want an instruction-tuned model (Part 5) — feeding GPT-2 a question and expecting an answer is the most common disappointment here.
• AutoModelForCausalLM, not AutoModel. The plain backbone gives hidden states; the ...ForCausalLM class adds the head that predicts the next token. Generation needs the head.
• max_new_tokens controls cost. It’s how many loop turns to run. Bigger means longer output and longer waits — generation time scales with the number of tokens.
• Output changes between runs (sometimes). With default settings GPT-2 is fairly deterministic, but once you turn on sampling (next post) the same prompt gives different text each time. That’s expected, not broken.
• It will happily make things up. A model that predicts plausible next words has no notion of truth. Confident, fluent, and wrong is a normal output — never read generated text as fact.

What’s next

You’ve met the decoder and watched it write, one token at a time, through the autoregressive loop. But we glossed over the most interesting step: at each turn the model hands you a probability for every possible next token — and how you choose from that distribution completely changes the result, from robotic repetition to wild invention.

Next: Greedy, Random, or Wise: How a Model Picks the Next Word, where you take the wheel on generation.

---

Target keyword(s): gpt-2 text generation, autoregressive generation, AutoModelForCausalLM, how language models generate text.