Greedy, Random, or Wise: How a Model Picks the Next Word

A beginner-friendly guide to decoding and sampling in text generation — greedy vs sampling, temperature, top-k, top-p, and repetition penalty — the handful of knobs that turn robotic or unhinged output into something good.

Series: Practical PyTorch: Running Models — LLMs — Part 4 of 6

Last post, every turn of the generation loop ended with the model handing you a probability for every possible next token. We skipped the obvious question: which one do you actually pick? It turns out this choice — called decoding — matters as much as the model itself. The same GPT-2, on the same prompt, can sound like a broken record or a caffeinated poet depending entirely on how you sample. This post is the handful of knobs that control it.

Open the companion notebook in Colab

The naive choice: always take the top token

The simplest rule is “pick the single most likely token every time.” It’s called greedy decoding, and it’s just argmax from the logits and softmax post applied at each step:

from transformers import pipeline

gen = pipeline("text-generation", model="distilgpt2")
print(gen("My favorite hobby is", max_new_tokens=40, do_sample=False)[0]["generated_text"])

do_sample=False is greedy. It sounds safe — always the best guess — but it has a notorious failure: it gets stuck. Greedy text loops (“I love it. I love it. I love it.”) because the highest-probability path often circles back on itself. Always taking the safest next word makes for dull, repetitive writing. Real language needs a little chance.

Sampling: roll the dice, but loaded

The fix is to sample — treat the probabilities as odds and draw a token at random, so likely words come up often and unlikely ones occasionally. Turn it on with do_sample=True:

print(gen("My favorite hobby is", max_new_tokens=40, do_sample=True)[0]["generated_text"])

Now the output varies run to run, and it stops looping. But pure sampling can also wander into nonsense — sometimes it draws a genuinely bad token. The next three knobs shape how the dice are loaded.

Temperature — how wild the dice are

temperature scales how sharply the model favors its top guesses. Low temperature (≈0.2) makes it timid and focused — close to greedy. High temperature (≈1.2) flattens the odds so rarer words get a real chance — more creative, more risk of nonsense. Around 0.7 is the usual happy medium.

gen("Write a tagline for a coffee shop:", max_new_tokens=25, do_sample=True, temperature=0.7)

Think of it as a creativity dial: turn it down for focused, predictable text; turn it up for variety.

Top-k and top-p — shrink the pool first

Rather than risk the whole vocabulary, you can let the model only sample from its best candidates:

  • top_k=50 — consider just the 50 highest-probability tokens, ignore the long tail.
  • top_p=0.9 (“nucleus sampling”) — consider the smallest set of tokens whose probabilities add up to 90%. The pool grows when the model is unsure and shrinks when it’s confident.
gen("Write a tagline for a coffee shop:",
    max_new_tokens=25, do_sample=True, temperature=0.8, top_p=0.9)

top_p is the modern default most people reach for. The two combine with temperature: shrink the pool, then sample from it with whatever wildness you set.

Repetition penalty — stop the broken record

repetition_penalty (try ≈1.2) gently lowers the odds of tokens that already appeared, which curbs the looping that plagues greedy decoding. A light touch helps; too much makes the model avoid normal repeated words like “the.”

A sensible default

You rarely tune all of these. A good general-purpose recipe for readable, varied text:

gen("Once upon a time,",
    max_new_tokens=60,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
    repetition_penalty=1.15)

That’s the difference between output you’d paste into a demo and output you’d quietly delete. Same model — the knobs did the work.

Gotchas

  • Greedy ≠ “best.” do_sample=False feels principled but produces flat, looping text. For anything human-facing, sample.
  • The settings only apply with do_sample=True. temperature, top_k, and top_p are ignored in greedy mode — a common “why is nothing changing?” trap.
  • High temperature breaks coherence. Past ~1.3 the model starts emitting word salad. If output goes haywire, turn temperature down first.
  • Sampling makes runs non-reproducible. Same prompt, different text every time — that’s the point. For a reproducible demo, set a seed (transformers.set_seed(0)) or go greedy.
  • These are the same knobs everywhere. temperature and top_p are exactly the parameters you’ll set on a hosted chat API. Learn them on GPT-2 and they transfer straight to the big models.
  • More tokens, more time. max_new_tokens is the main cost lever — generation time grows with output length, on every model.

What’s next

You can now drive generation — focused or freewheeling, on demand. But GPT-2, however well you sample it, still only continues text; it was never taught to answer you. Closing that gap — from plausible continuation to helpful reply — took a second kind of training, and it’s exactly what separates a chatbot from autocomplete.

Next: From Autocomplete to Assistant: Instruction-Tuned and Chat Models, where a base model learns to follow instructions and hold a conversation.

Target keyword(s): temperature top-p sampling, decoding strategies llm, top_k top_p generation, repetition penalty transformers.

Comments