Trying out the facebook/musicgen-small sound generation model#

Facebook’s musicgen is a model that generates snippets of audio from a text description - it’s effectively a Stable Diffusion for music.

It turns out it’s pretty easy to run it using Python, thanks to the Hugging Face transformers library.

Here’s the code that worked for me. First, install the dependencies:

1
pip install scipy transformers

The following will download the small model - around 2GB - and store it in ~/.cache/huggingface/hub/models--facebook--musicgen-small the first time you run it.

1
from transformers import AutoProcessor, MusicgenForConditionalGeneration
2
import scipy
3

4
processor = AutoProcessor.from_pretrained("facebook/musicgen-small")
5
model = MusicgenForConditionalGeneration.from_pretrained("facebook/musicgen-small")
6

7
def save(prompt, filename, num_tokens=1503):
8
    inputs = processor(
9
        text=[prompt],
10
        padding=True,
11
        return_tensors="pt",
12
    )
13
    audio_values = model.generate(**inputs, max_new_tokens=num_tokens)
14
    sampling_rate = model.config.audio_encoder.sampling_rate
15
    scipy.io.wavfile.write(filename, rate=sampling_rate, data=audio_values[0, 0].numpy())

Then you can use that save() function like this to generate and save an audio sample:

1
save("trumpet mariachi frenetic excitement", "trumpet_mariachi.wav")

Here’s the audio that generated:

https://static.simonwillison.net/static/2023/trumpet_mariachi.wav