Type a Vibe, Get a Track: Text-to-Music with MusicGen

Series: Practical PyTorch: Running Models — Audio — Part 6 of 8

In Part 5 you watched Encodec chop a waveform into a grid of discrete tokens — sound turned into something that looks, suspiciously, like the integer IDs a language model reads. This is the post where that suspicion pays off. Type a sentence describing a vibe, and a model writes you a piece of music to match. The trick underneath is the one you just learned: a transformer predicts Encodec tokens, one after another, and the codec decodes them into a waveform. It’s a text model — it just speaks in audio.

▶ Open the companion notebook in Colab

The model is MusicGen, from Meta. A heads-up before we start: this one leans on a GPU. Generation is autoregressive, meaning the model predicts each audio token from the ones before it, so on a CPU it’s painfully slow, the kind of slow where you go make coffee and it’s still not done. In Colab, flip Runtime → Change runtime type → GPU before you run anything here.

The easy button — a music pipeline

You know the shape by now. Same pipeline, a task you haven’t used yet: "text-to-audio". Install transformers and point it at the smallest MusicGen checkpoint:

!pip install -q transformers

from transformers import pipeline
import numpy as np

music = pipeline("text-to-audio", model="facebook/musicgen-small")
out = music("lo-fi hip hop with a mellow piano and vinyl crackle")
print(out["sampling_rate"])                    # 32000

Then play it right in the notebook:

from IPython.display import Audio

Audio(np.squeeze(out["audio"]), rate=out["sampling_rate"])

That’s a full track from one English sentence. The output is a dict, same as the TTS pipeline in Part 2: out["audio"] holds the waveform and out["sampling_rate"] is 32000, so MusicGen works at 32 kHz, twice the 16 kHz of the speech models. The np.squeeze is housekeeping: the audio comes back with an extra length-1 dimension (a leftover channel axis), and Audio wants a flat array, so we squeeze it down.

The prompt is the whole interface. “Lo-fi hip hop with a mellow piano and vinyl crackle” gets you exactly the genre, instrument, and texture you named. Try your own — the model responds to genre, instrument, mood, and tempo words, and concrete is better than vague. “Driving synthwave with a punchy bassline” beats “good music” every time.

The honest way — processor, model, generate

The pipeline is a fine default, but it hides the line that explains everything. Let’s run MusicGen by hand and watch the language-model machinery underneath.

from transformers import AutoProcessor, MusicgenForConditionalGeneration
import torch

device = "cuda" if torch.cuda.is_available() else "cpu"
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small").to(device)

inputs = processor(
    text=["upbeat 8-bit chiptune for a boss fight"],
    padding=True,
    return_tensors="pt",
).to(device)

audio = model.generate(**inputs, max_new_tokens=256)   # ~5 s (≈50 tokens/sec)

Two objects, the same split you’ve seen all series. The processor turns your prompt into the tensors the model conditions on; the text is wrapped in a list because MusicGen can generate a batch of clips at once, and padding=True lines them up if you pass several. The model is the generator. And .to(device) puts both the model and the inputs on the GPU, because mixing model on GPU with inputs on CPU is an error, not a guess.

Now look hard at that one line: model.generate(**inputs, max_new_tokens=256). That’s the same generate you’d call on a text model. It’s the same autoregressive loop — predict the next token, append it, predict the next — except the tokens it’s predicting are Encodec audio codes, not words. The model emits a sequence of those codes; Encodec’s decoder, the one you met in Part 5, turns the codes back into a waveform. The whole thing is a language model whose vocabulary happens to be sound.

Which makes max_new_tokens the most important dial here. It’s literally how many audio tokens to predict, and since MusicGen runs at roughly 50 tokens per second of audio, 256 tokens is about five seconds of music. Want a longer clip? Ask for more tokens. We’ll spend Part 7 on this and the other knobs, but the intuition is right here: token count is clip length.

Playback is the same as before, but you fetch the sample rate from the model’s config instead of a pipeline dict:

from IPython.display import Audio

sr = model.config.audio_encoder.sampling_rate          # 32000
Audio(audio[0].cpu().numpy(), rate=sr)

audio[0] pulls the first clip out of the batch, .cpu() brings it back from the GPU, and .numpy() hands Audio the array it wants. The sample rate lives at model.config.audio_encoder.sampling_rate because it’s a property of the Encodec codec doing the decoding: 32 kHz, mono. That’s the path: text → conditioning → predicted audio tokens → decoded waveform, the exact loop from Part 5 with a prompt bolted to the front.

Try prompts that are specific

The model rewards detail. Stack a genre, an instrument or two, a mood, and a tempo, and you steer it hard:

prompts = [
    "epic orchestral film score, soaring strings and timpani",
    "warm bossa nova with nylon guitar and brushed drums",
    "dark ambient drone, slow and cinematic, no drums",
]

Run each through the same music(prompt) call (pipeline) or feed the whole list to the processor at once (by hand) and you’ve got three very different tracks from three sentences. Naming what you don’t want works too — “no drums” actually lands. This is the fun part of the post, and it’s worth ten minutes of just typing vibes and listening to what comes back.

A word on the checkpoint. facebook/musicgen-small is the place to start: it’s the fastest and lightest, and on a Colab GPU it generates a short clip in seconds. There are bigger siblings — musicgen-medium and musicgen-large — that sound noticeably better at the cost of more memory and slower generation. Start small, get the loop working, then trade up only if you need the quality.

Gotchas

• You really want a GPU. Autoregressive generation on CPU is brutally slow for music — minutes for a few seconds of audio. Set the Colab runtime to GPU before you run a thing.
• max_new_tokens ≈ seconds × 50. That’s your length dial. 256 tokens is roughly five seconds; double it for ten. It’s a useful rule of thumb, not an exact contract — Part 7 goes deeper.
• Small vs. medium vs. large is a speed-for-quality trade. musicgen-small is the right first move. Reach for the bigger checkpoints once you’ve outgrown it, and expect the slowdown.
• Output is mono at 32 kHz. One channel, 32,000 samples per second. The rate lives at model.config.audio_encoder.sampling_rate, not at a fixed 16000 like the speech models — grab it from there so you never play it back at the wrong speed.
• The first run downloads weights. Even small is hundreds of megabytes, and the larger checkpoints are much more. A first cell that hangs for a minute is fetching the model, not broken; later runs hit the cache.

Final thoughts

The headline writes itself — you typed a sentence and got a song — but the part worth keeping is quieter. MusicGen isn’t a special “music architecture.” It’s a transformer doing next-token prediction, the same as the text models you’ve run all year, pointed at the Encodec vocabulary from Part 5. The prompt conditions it, generate runs the loop, and the codec decodes the result into sound. Once you see that, “generate music” stops being a separate magic trick and becomes a familiar one wearing a new hat. Next we pick up the controls — how long, in what key, following which melody — and start actually steering the band.

Next: Part 7 — Steering the Band: Length, Melody, and Sampling — turning the dials on MusicGen to control how long it plays and how closely it follows you.

---

Target keyword(s): text-to-music, musicgen, generate music from text pytorch.