Whisper
Typing up interviews and voice notes takes hours. Whisper turns audio into text, and can translate speech into English, on your own computer if you are comfortable with some technical setup.
How you use it: Install two pieces of free software, then run one command on your audio file to get the text.
In plain English
OpenAI's README calls Whisper a general-purpose speech recognition model trained on a large and varied set of audio. It can transcribe speech, translate speech into English, identify the spoken language, and detect where voice is present.
You install it as a Python package and you also need ffmpeg on the computer so it can read audio files. The command-line example transcribes flac, mp3, and wav files. A Python API is documented in the same README.
The README lists several model sizes and the video memory they need, from about 1 GB for the lighter ones up to about 10 GB for the largest. It says the turbo model is a faster English transcription option and is not trained for translation. For translation into English, it tells you to use a multilingual model. Code and model weights are MIT licensed.
Practical ways to use it
Example 1
Get a transcript of a recording
The README's command transcribes an audio file with the turbo model, which it says works well for English.
Example 2
Transcribe speech that is not English
The README shows a command that sets the language, and says to pick a multilingual model rather than turbo.
Example 3
Translate a recording into English
The README shows a translate task with a multilingual model such as medium. It warns that turbo will not do this job.
Example 4
Name the language in a clip
The introduction says the model can do spoken language identification, alongside transcription and translation.
Who it is for
Developer. The interface in this repository is a Python package and a command. You should be fine pasting terminal commands and installing ffmpeg.
Difficulty: Developer setup. pip and ffmpeg are enough for a first transcript. Choosing a model size is about how much memory you have.
Getting started
- 1
Install ffmpeg
The README gives commands for Ubuntu, Arch, macOS Homebrew, and Windows via Chocolatey or Scoop. Whisper needs ffmpeg to open audio files.
- 2
Install the Python package
The README shows pip install -U openai-whisper. It expects a recent PyTorch. If the install complains about Rust, the README links the Rust getting-started page.
- 3
Transcribe one English clip
Run whisper on your file with --model turbo. The README says that default choice works well for English transcription.
- 4
Switch models when the task changes
Use a multilingual model, not turbo, for non-English transcription or for translation into English. The README's translation example uses the medium model and the translate task.
What you need first
- Accounts
- None. The README installs the package from PyPI and runs the model on your machine. This is not the paid OpenAI audio API.
- Hardware
- Python, ffmpeg, and enough memory for the model you pick. The README table is the source for those memory figures. Your machine will differ from the A100 they used for speed notes.
- Cost
- Code and weights are MIT licensed, so you are not paying OpenAI per minute for this repository. You pay with your own computer's time and electricity.
Cautions
- Turbo is for English transcription. The README says it is not trained for translation and will not follow a translate task the way the multilingual models will.
- Accuracy varies by language. The README says so and points at the paper. Do not treat a first transcript as a legal record.
- The model runs locally, which keeps the audio off OpenAI's API, and it still uses your disk and GPU. A long file can take a while on a laptop.
Questions people ask
What does Whisper do?
It transcribes speech from audio files, can translate speech into English, and can identify the spoken language. You run it on your own computer.
Who is Whisper for?
People who can use Python and a terminal. It is a command and a library, not a consumer recording app.
What do I need first?
Python, ffmpeg, and pip install -U openai-whisper. No OpenAI account is required for this repository.
What should I watch out for?
Do not use the turbo model for translation. Pick a model size your memory can hold, and proofread the transcript.