TroubleshootingWritten by GLWS12 min readUbuntu 24.04 LTS / AlmaLinux 9
Commands run on your server unless a section says otherwise. Replace example usernames, addresses and paths with your own values. Keep a working session open when changing remote access.

Two different things can run out

The error "No space left on device" can mean the disk is full or that the filesystem has run out of inodes, which are the records that track files. A disk with plenty of free space can still report the error if it holds millions of tiny files. Check both before deleting anything.

Both systems
df -h
df -i
In df -h look at the Use% for the filesystem mounted at /. In df -i look at IUse%. Either one at 100% is the problem.

Find the biggest directories

Work down from the top, staying on one filesystem with -x so you do not count other disks. The commands list the largest items at each level. Do the same for the next level inside the biggest directory. Common culprits are /var/log, /var/lib/docker, database directories, backups and /home.

Both systems
sudo du -xh / --max-depth=1 2>/dev/null | sort -h | tail -n 10
sudo du -xh /var --max-depth=1 2>/dev/null | sort -h | tail -n 10
Largest individual files
sudo find / -xdev -type f -size +200M -exec ls -lh {} + 2>/dev/null | sort -k5 -h | tail -n 15

Find where the inodes went

If inodes are exhausted, count files per directory. Mail queues, session files, caches and runaway temporary files are typical causes. The loop counts entries below each directory in /var, then you drill into the largest.

Both systems
for d in /var/* /tmp /home/*; do [ -d "$d" ] && echo "$(sudo find "$d" -xdev 2>/dev/null | wc -l) $d"; done | sort -n | tail -n 8

Deleted files that are still using space

If df says the disk is full but du finds far less, a process may still hold a deleted file open. The space is returned only when that process closes it or restarts. This often happens after someone deletes a log that a running service is still writing.

Both systems
sudo lsof +L1 2>/dev/null | head -n 15
Install lsof if missing: Ubuntu
sudo apt install lsof
Install lsof if missing: AlmaLinux
sudo dnf install lsof
Restart the service that owns the file, such as sudo systemctl restart nginx, to release the space. To empty a live log without breaking the service, use truncate -s 0 on it instead of deleting it.

Clean up logs safely

The systemd journal can grow to gigabytes. Check its size, then limit it. Rotate and compress application logs with logrotate rather than deleting them by hand. Never remove files in /var/log that services are writing without checking that they will recreate them.

Both systems
sudo journalctl --disk-usage
sudo journalctl --vacuum-size=200M
ls -lhS /var/log | head -n 12
Make the limit permanent: add to /etc/systemd/journald.conf, then restart journald
sudo sed -i 's/^#\?SystemMaxUse=.*/SystemMaxUse=200M/' /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

Clean up packages and old kernels

Package caches and old kernels take hundreds of megabytes. Remove them with the package manager so that dependencies stay consistent. Keep the running kernel and one older one.

Ubuntu only
sudo apt clean
sudo apt autoremove --purge
uname -r
AlmaLinux only
sudo dnf clean all
sudo dnf remove --oldinstallonly --setopt installonly_limit=2 kernel-core
uname -r

Clean up Docker data

Docker images, stopped containers, build cache and logs accumulate. Check what uses space, then remove only what you do not need. Pruning images removes anything not used by a container, so make sure your containers are running first, or you will have to download images again.

Both systems
sudo docker system df
sudo docker container prune
sudo docker image prune -a
sudo docker builder prune
Do not run docker volume prune unless you are certain no volume holds data you need. Volumes contain databases and application data, and pruning cannot be undone.

Stop it returning

Set up a disk alert (see the monitoring guide), limit journal size, make sure log rotation is configured for every application, and keep an eye on backups that pile up on the same disk. Consider a larger plan or a separate data volume if growth is genuine. As an emergency measure, ext4 reserves a small share of blocks for root, which is how you can still log in and clean up when the disk looks full.

Both systems
sudo logrotate --debug /etc/logrotate.conf 2>&1 | tail -n 15
df -h /

Official documentation

These upstream references provide more detail on the commands and workflows in this guide.