From Words to Vectors: How a Model Turns Text into Numbers

Series: Practical PyTorch: Running Models — Language — Part 1 of 9

You’ve spent the whole vision arc handing models tensors — a photo is already a grid of numbers, so a CNN can get straight to work. Text isn’t so lucky. A model only ever does arithmetic: it multiplies, adds, and compares numbers. It cannot multiply the word “cat.” So before we reach the transformer next post and let words look at each other, there’s a quieter step to settle: how does a word become something a model can compute with at all?

The answer comes in two moves — give every known word a number, then turn that number into a vector. This post walks both, math-free, on a vocabulary small enough to print.

▶ Open the companion notebook in Colab

A vocabulary is just a numbered list

A model doesn’t know every word in English. It knows a fixed list — its vocabulary — decided before training and frozen from then on. Every word the model will ever handle has a slot in that list, and the slot’s position is its ID.

Real vocabularies run to tens of thousands of entries, and nobody writes them by hand: a separate tokenizer training step builds the list from a big corpus, keeping the words and word-fragments common enough to earn a slot (that’s Part 4). Ours we’ll just type out; twelve entries is enough to see the whole idea:

vocab = ["the", "a", "cat", "dog", "sat", "ran",
         "on", "mat", "sofa", "and", "big", "small"]

stoi = {word: i for i, word in enumerate(vocab)}
print(stoi["cat"])   # 2

That stoi (“string to integer”) dictionary is the entire vocabulary, really: a lookup from word to slot number. "the" is 0, "cat" is 2, "sofa" is 8. Nothing clever yet — it’s a coat check, handing each word a numbered ticket.

A sentence becomes a row of IDs

With the lookup in hand, turning a sentence into numbers is one line — split it into words, swap each for its ID:

sentence = "the cat sat on the mat"
ids = [stoi[word] for word in sentence.split()]
print(ids)   # [0, 2, 4, 6, 0, 7]

Six words in, six numbers out. Notice "the" became 0 both times it appeared — the same word always maps to the same ID, which is exactly what you want. The model now has something it can hold in a tensor.

This toy version has two obvious cracks, and real tokenizers exist to seal them. Hand it "kitten" and stoi["kitten"] blows up with a KeyError — our vocabulary never heard of it. And splitting on spaces is crude: punctuation, capitalization, and rare words all need handling. The real tokenizers you’ll meet in Part 4 fix both by breaking text into subword pieces, so they can always compose a word they’ve never seen from fragments they know. The principle is identical to what you just did, though — text in, integer IDs out.

But IDs aren’t meaningful numbers

Here’s the trap. We have numbers now, but they’re the wrong kind of numbers. "cat" is 2 and "sofa" is 8 — that does not make a sofa “four times” a cat, or place "on" (6) halfway between them in meaning. The IDs are arbitrary labels; their numeric values are noise. Feed them straight into arithmetic and the model would read a fiction that “mat” (7) is one more than “on” (6).

What we actually want is a number for each word that carries some sense of the word — and one number per word is far too cramped to hold meaning. So we give each word a small list of numbers instead: a vector. “cat” and “dog” should end up with similar vectors because they behave similarly; “cat” and “sofa” should end up far apart. That richer representation is called an embedding.

Three steps: a word picks its ID from the vocabulary, and the ID picks its row from the embedding table.

The mechanism is a second lookup table. Picture a grid with one row per vocabulary word, and a handful of numbers across each row. To embed a word, you take its ID and grab that row. ID 2 → row 2 → the vector for “cat.” Looking up an embedding is nothing more exotic than reading a row out of a table by its number.

Build the lookup in PyTorch

PyTorch hands you that table as a layer, nn.Embedding. You tell it how many rows (the vocabulary size) and how wide each row is (the vector length):

import torch
import torch.nn as nn

embed = nn.Embedding(num_embeddings=len(vocab), embedding_dim=4)

print(embed.weight.shape)   # torch.Size([12, 4])

There’s the grid: twelve rows, four numbers each. embed.weight is the table — every word’s vector stacked up, 12 × 4 numbers the model will learn. Now feed it the sentence’s IDs:

x = torch.tensor(ids)        # [0, 2, 4, 6, 0, 7]
vectors = embed(x)
print(vectors.shape)         # torch.Size([6, 4])

Read that shape as “six words, each now a vector of four numbers.” The sentence that started as a string is finally a tensor a model can do arithmetic on. And because the lookup is pure indexing, the two 0s in the input come back as the same row — vectors[0] and vectors[4] are identical, both the vector for “the.”

Our toy uses width 4 so it fits on the page. Real models go wide: the transformer block next post sets d_model=128, and full-size language models use 768, 1024, or more. That number — the embedding dimension — is just how many numbers each word gets to carry. Bigger means more room for nuance.

Where the meaning comes from

One honest caveat. Print a vector from our embed and it’s gibberish — nn.Embedding starts the whole table as random numbers, so right now “cat” and “dog” are no closer than “cat” and “sofa.” The table is a blank set of coordinates, not a map. We never filled it in, and on this twelve-word toy we can’t: there’s no text here to learn from.

So where do the numbers come from? Training — though never the table on its own. The embedding layer rides along inside a bigger model that’s been set a language task, like predict the next word or fill in a blanked-out word. To get good at that task, the model is pushed toward one trick: give words that appear in similar contexts similar vectors. “cat” and “dog” both turn up near “fed the,” “chased the,” and “my pet,” so a model that parks their two rows close together predicts a little better, and training nudges them there. Do that across billions of words and the random grid becomes a map: “cat” and “dog” as neighbors, “big” and “small” on opposite sides. You know a word by the company it keeps, and the embedding table is where that company gets recorded.

Real vocabularies get their vectors the same way. The embedding layer inside DistilBERT is an nn.Embedding exactly like ours — same lookup, just 30000 × 768 instead of 12 × 4 — learned jointly with the rest of the model on a mountain of text. There’s no separate step for it: the same pre-training run that teaches the model language is what drags the embedding rows into place, and because the vocabulary is locked in beforehand, pre-training only has to learn one vector per existing slot. You rarely train one yourself: you load a pretrained model and its table arrives already meaningful. Part 8 is all about reading that finished geometry: measuring which vectors sit close, and using it to search by meaning.

For now the point is narrower and structural: every word has become a vector you can look up, and that’s the input the rest of the architecture is built on.

What’s next

So when the next post says “each word is a vector” and runs attention over a sentence, you’ll know exactly what that vector is and where it came from: a word picked its ID from the vocabulary, and the ID picked its row from the embedding table. Words are numbers now — rows you can print.

Which means we’re finally ready for the idea those rows exist to serve. Attention lets every word’s vector reach out and gather from every other word’s vector, and a stack of that is a transformer.

Next: Teaching a Model to Read: Attention and the Transformer, the one new idea that took models from seeing to reading.

---

Target keyword(s): what is an embedding, token IDs explained, nn.Embedding pytorch, vocabulary tokenization beginner, words to vectors.