Keep long-running SSH tasks alive with tmux

The problem tmux solves

When your SSH connection drops, the programs running in that terminal receive a hangup signal and normally stop. A long upgrade, backup, build or data import can die halfway. tmux is a terminal multiplexer: it runs sessions on the server that keep going when you disconnect, and you reattach to them later and see everything as you left it.

Install it

tmux is in the standard repositories of both systems and is installed by default on some images.

sudo apt install tmux

sudo dnf install tmux

tmux -V

Start, detach and reattach

Start a named session, run your command, then detach with the key sequence Control-b followed by d. The command keeps running. Reconnect over SSH any time and attach again by name. Use named sessions so you can tell them apart.

tmux new -s upgrade
# run your long task here, then press Ctrl-b and then d to detach

tmux ls
tmux attach -t upgrade

tmux new -d -s backup 'sudo /usr/local/bin/restic-backup 2>&1 | tee ~/backup.log'
tmux attach -t backup

ssh deploy@YOUR_SERVER -t "tmux new -A -s main"

Windows and panes

One session can hold several windows (like tabs) and split panes. All keys start with the prefix Control-b. Press the prefix, release it, then press the key.

The most useful ones: c creates a window, n and p move to the next and previous window, , renames a window, % splits left and right, " splits top and bottom, arrow keys move between panes, z zooms a pane to full screen and back, x closes a pane, and d detaches.

tmux list-windows
tmux list-panes
tmux kill-session -t upgrade

Scroll back and copy

The normal scroll wheel does not scroll tmux history unless you turn on mouse support. Press the prefix and then [ to enter copy mode, move with the arrow keys or Page Up and Page Down, and press q to leave. With mouse mode on, you can scroll with the wheel and select text with the mouse.

cat > ~/.tmux.conf <<'EOF'
set -g mouse on
set -g history-limit 50000
set -g base-index 1
setw -g pane-base-index 1
set -g status-style bg=black,fg=green
EOF
tmux source-file ~/.tmux.conf

Log and check on a task

For long jobs, write the output to a file as well, so you can read the result even if the session scrolls away or you forget to look. The status line can show you when a task is done if you end the command with a visible message.

With a pipeline, Bash normally reports the status of its last command, tee here. Enable pipefail so a failed migration gives a failing pipeline status, then save that status immediately. A complete log does not prove that the migration succeeded.

tmux new -s migrate
set -o pipefail
time ./migrate.sh 2>&1 | tee ~/migrate.log
status=$?
printf 'FINISHED with status %s\n' "$status"

tail -f ~/migrate.log

What tmux does not do

tmux sessions end when the server reboots, so tmux is not for services that must survive restarts or crashes. For those, use a systemd service, as in the service guides. tmux is also not a way to run something as a different user or to protect a process from the out-of-memory killer. Use it for interactive and one-off work.

Alternatives are screen, an older multiplexer with similar ideas, and nohup or systemd-run for simple background jobs without a terminal.

sudo systemd-run --unit=oneoff --property=Nice=10 /usr/local/bin/long-job.sh
journalctl -u oneoff -f

Sharing a session and cleaning up

Two people can attach to the same session, which is useful for pair debugging or support: both see the same screen. Only share with people you trust, since they type as your user. Kill finished sessions to keep the server tidy.

tmux attach -t upgrade
tmux ls
tmux kill-session -t upgrade
tmux kill-server