Why was my Linux process "Killed"? Diagnose out-of-memory failures

What "Killed" means

When a command ends with the single word "Killed", or a service disappears without an error of its own, the process was sent SIGKILL (signal 9, exit code 137). The most common sender on a server is the kernel's out-of-memory killer, which picks a process to end when memory and swap run out. Other causes are a container or systemd memory limit, or a person or script running kill -9. You need to find out which one.

Confirm it in the kernel log

The kernel records every out-of-memory kill, with the victim's name and a memory table. Search the kernel log, using journalctl if your user cannot read dmesg. If the log is empty, the kill came from elsewhere: a cgroup limit, a container runtime or a person.

Missing log lines do not rule out OOM: logs may have rotated, and a container or service can hit its cgroup memory limit while the host still has free RAM. For containers, inspect OOMKilled and the exit code together; exit 137 alone can also mean an intentional SIGKILL.

sudo journalctl -k --since "7 days ago" | grep -i -E "out of memory|killed process|oom"
sudo dmesg -T 2>/dev/null | grep -i -E "out of memory|killed process" | tail -n 10

systemctl status myapp --no-pager | grep -i -E "result|status"
sudo journalctl -u myapp --since "1 day ago" --no-pager | grep -i -E "oom|killed"

Read the log line

A typical line looks like "Out of memory: Killed process 1234 (mysqld) total-vm:... anon-rss:... shmem-rss:..." The name is the victim, which is not always the cause: the kernel chooses the process using the most memory, so a small leak elsewhere can lead to killing your database. Note the time, then look at what was running.

Check current memory and swap

See how much memory is used, how much is available, and whether swap exists and is used. Available memory is what matters, as Linux uses spare memory for caches. A server with no swap and a small amount of memory runs out quickly.

free -h
swapon --show
vmstat 5 5
cat /proc/meminfo | grep -E "MemTotal|MemAvailable|SwapTotal|SwapFree"

Find what is using the memory

List the processes by resident memory. Check them when the server is busy, and watch them over time to catch growth: a process whose memory keeps rising is a leak. Note that many worker processes, such as PHP-FPM children, add up.

ps -eo pid,user,rss,pmem,comm --sort=-rss | head -n 12
ps -eo comm,rss | awk '{a[$1]+=$2} END {for (k in a) printf "%8.0f MB  %s\n", a[k]/1024, k}' | sort -rn | head
watch -n 5 'ps -eo pid,rss,comm --sort=-rss | head -n 6'

Containers and Kubernetes

A container killed for exceeding its memory limit shows as OOMKilled, even when the host has plenty of memory. Check the state in Docker or Kubernetes, and raise the limit or reduce the application's use.

sudo docker inspect --format '{{.Name}} oom={{.State.OOMKilled}} exit={{.State.ExitCode}}' $(sudo docker ps -aq)
sudo docker stats --no-stream

kubectl get pods -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name} {.status.containerStatuses[0].lastState.terminated.reason}{"\n"}{end}' | grep OOMKilled

/glws-oom oom=true exit=137

Fix 1: tune the biggest users

Reduce the memory that your own services are allowed to take. For PHP-FPM, lower pm.max_children so that children times average size fits in memory. For MariaDB, set a smaller innodb_buffer_pool_size. For Node.js, set a heap limit. For builds, run fewer parallel jobs. These changes address the cause, and they cost nothing.

# PHP-FPM: check average child size, then set pm.max_children accordingly
ps -C php-fpm8.3 -o rss= | awk '{s+=$1; n++} END {print s/n/1024 " MB average over " n " processes"}'

# Node.js build or app with a heap cap
NODE_OPTIONS=--max-old-space-size=1024 npm run build

# Fewer parallel compile jobs
make -j2

Fix 2: add swap and a safety net

Swap gives short spikes somewhere to go, instead of triggering a kill. It does not fix a workload that needs more memory, but it prevents surprise failures. See the swap guide for how to add it. You can also protect a critical service so the kernel prefers other victims, and cap a noisy one so it cannot take everything.

[Service]
# protect this service from the OOM killer
OOMScoreAdjust=-500
# or contain a service that tends to grow
# MemoryHigh=700M
# MemoryMax=900M

sudo systemctl daemon-reload
sudo systemctl restart myapp
systemctl show myapp -p OOMScoreAdjust -p MemoryMax

Fix 3: reduce what runs, or use a bigger plan

If the services you need do not fit, remove what you do not need, move heavy work such as builds to another machine, or move to a plan with more memory. The guide on what fits in 1 GB and the one on moving to a bigger plan help here. Alert on low available memory, as the monitoring guide describes, and after any change watch the kernel log for a week to confirm kills have stopped.

sudo journalctl -k --since "7 days ago" | grep -ci "killed process"
free -m