Transcribe audio on a CPU VPS with whisper.cpp

What whisper.cpp is

whisper.cpp is a C/C++ implementation of OpenAI's Whisper speech recognition model. It runs on CPUs without a GPU, so it works on our plans, and audio stays on your server. Speed depends on the model size and your CPU. Smaller models are faster and less accurate, and larger ones are slower and need more memory.

Typical uses are meeting notes, podcasts and voice memos that you do not want to upload to a third party. Check the licence of the model you download and the terms for any audio you process.

nproc
free -h
df -h /

Install the build tools

You compile whisper.cpp from source. Install a compiler, CMake, git and ffmpeg. On a plan with 1 to 2 GB of memory, add swap before compiling, because the build can run out of memory (see the swap guide).

sudo apt update
sudo apt install build-essential cmake git ffmpeg

Build it

Clone the repository, then configure and build in release mode. Limit the number of parallel jobs on a small server so the build does not exhaust memory. The build takes a few minutes.

cd ~
git clone https://github.com/ggml-org/whisper.cpp
cd whisper.cpp
cmake -B build
cmake --build build -j 2 --config Release
ls build/bin/

Download a model

The repository includes a script that downloads converted models. The base English model is a good start: it is small, roughly 140 MB, and the English-only variants are a little more accurate for English audio than the multilingual ones of the same size. For other languages use the multilingual models without .en. Larger models such as small and medium are more accurate but slower and use more memory.

cd ~/whisper.cpp
sh ./models/download-ggml-model.sh base.en
ls -lh models/

Convert audio and transcribe

Whisper.cpp expects 16 kHz mono WAV audio. Convert any common format with ffmpeg first. Then run the transcription, and ask for a text file as well as printed output. The repository includes a sample file you can use to test the setup.

cd ~/whisper.cpp
./build/bin/whisper-cli -m models/ggml-base.en.bin -f samples/jfk.wav

ffmpeg -i ~/meeting.mp3 -ar 16000 -ac 1 -c:a pcm_s16le ~/meeting.wav
cd ~/whisper.cpp
time ./build/bin/whisper-cli -m models/ggml-base.en.bin -f ~/meeting.wav -otxt -of ~/meeting
cat ~/meeting.txt

Measure speed on your plan

The time command shows how long transcription takes. Compare it with the audio length: if a 10-minute file takes 5 minutes, you transcribe at twice real time. Try the next model size up and compare the accuracy against the extra time. Speed on shared vCPUs varies, so test at different times and do not rely on figures from other hardware. Give the program more threads with -t if your plan has spare cores.

ffprobe -v error -show_entries format=duration -of csv=p=0 ~/meeting.wav
cd ~/whisper.cpp
time ./build/bin/whisper-cli -m models/ggml-base.en.bin -f ~/meeting.wav -t 3 -otxt -of ~/meeting

Transcribe a folder as a batch job

A small script processes every audio file in a folder, skips those already done and writes a text file beside each one. Run it manually or from a systemd timer. Only one job should run at a time on a small server, and a lock keeps overlapping runs apart.

cat > ~/transcribe.sh <<'EOF'
#!/bin/bash
set -euo pipefail
exec 9>/tmp/transcribe.lock
flock -n 9 || { echo "already running"; exit 0; }
cd ~/audio
for f in *.mp3 *.m4a *.wav; do
  [ -e "$f" ] || continue
  out="${f%.*}"
  [ -e "$out.txt" ] && continue
  ffmpeg -nostdin -loglevel error -y -i "$f" -ar 16000 -ac 1 -c:a pcm_s16le /tmp/in.wav
  ~/whisper.cpp/build/bin/whisper-cli -m ~/whisper.cpp/models/ggml-base.en.bin -f /tmp/in.wav -otxt -of "$out"
  echo "done: $f"
done
EOF
chmod 755 ~/transcribe.sh
mkdir -p ~/audio

Handle private recordings carefully

Recordings can contain personal information, so keep the audio and the transcripts in folders only you can read, delete them when finished, and make sure you have the right to transcribe the recording. Remember that automatic transcripts contain mistakes, particularly with names, numbers and accents, so check them before relying on them.

chmod 700 ~/audio
ls -ld ~/audio