Backing up a Linux server with restic

Before you start

A Linux server with a sudo account
A destination: a NAS reachable over SFTP (SSH), a mounted disk, or S3-compatible storage
Having read the 3-2-1 rule to know where this backup fits in your strategy

In plain words

restic works like a safe with numbered drawers: every night it files away a snapshot of your folders, but only copies what changed since the day before. Everything is locked with a key that only you hold, and a caretaker (the systemd timer) comes round every night to do the job and throw away snapshots that are too old.

restic is a free and open-source backup tool, shipped as a single binary, available on Linux, Windows and macOS. It encrypts everything before sending it, deduplicates the data (after the first backup, only changed blocks are sent) and keeps a history that you prune according to your own rules.

We are going to back up a server's /etc and /home to a NAS over SFTP, every night.

1. Install restic

sudo apt install restic          # Debian, Ubuntu
sudo dnf install restic          # Fedora, RHEL (EPEL repository)
restic version

2. Prepare access to the NAS

restic connects to the NAS over SFTP with an SSH key without a passphrase, since the backup runs unattended. Create a dedicated key for the root account, which will run the backup:

sudo ssh-keygen -t ed25519 -f /root/.ssh/restic_nas -N "" -C "restic@$(hostname)"
sudo ssh-copy-id -i /root/.ssh/restic_nas.pub backup@nas.home.lan

On the NAS side, use a dedicated account (backup) that can only access the backup folder.

Then declare the key for this host:

sudo nano /root/.ssh/config

Host nas.home.lan
    User backup
    IdentityFile /root/.ssh/restic_nas

Test: sudo ssh nas.home.lan must log in without asking anything.

3. Create the encrypted repository

The repository password encrypts all the data. Generate a strong one and store it in a file readable by root only:

sudo install -d -m 700 /etc/restic
openssl rand -base64 32 | sudo tee /etc/restic/password > /dev/null
sudo chmod 600 /etc/restic/password

Danger

Without this password, the backup cannot be recovered. Copy it into your password manager right away. If the server dies, it is the only thing you will need to restore.

Create a variables file, used by the commands and by the scheduled job:

sudo nano /etc/restic/env

RESTIC_REPOSITORY=sftp:nas.home.lan:/volume1/backups/server
RESTIC_PASSWORD_FILE=/etc/restic/password

Initialize the repository:

sudo bash -c 'set -a; . /etc/restic/env; restic init'

4. First backup

sudo bash -c 'set -a; . /etc/restic/env; restic backup /etc /home --exclude-caches --tag daily'

The first backup sends everything. The following ones only transfer the differences and are much faster.

List the snapshots:

sudo bash -c 'set -a; . /etc/restic/env; restic snapshots'

Databases and applications

Copying the files of a database while it is being written can produce an inconsistent backup. First make an export (pg_dump, mysqldump, sqlite3 base.db ".backup export.db") into a folder included in the backup, or stop the service during the copy.

5. Automate with a systemd timer

The service, which backs up, prunes old snapshots and checks the repository:

sudo nano /etc/systemd/system/restic-backup.service

[Unit]
Description=restic backup to the NAS
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
EnvironmentFile=/etc/restic/env
# restic cache: speeds up later backups considerably
CacheDirectory=restic
Environment=RESTIC_CACHE_DIR=/var/cache/restic
ExecStart=/usr/bin/restic backup /etc /home --exclude-caches --tag daily
ExecStart=/usr/bin/restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
ExecStart=/usr/bin/restic check
Nice=10
IOSchedulingClass=idle

The timer, every night at 2 a.m. with a random delay:

sudo nano /etc/systemd/system/restic-backup.timer

[Unit]
Description=Daily restic backup

[Timer]
OnCalendar=*-*-* 02:00
RandomizedDelaySec=30min
Persistent=true

[Install]
WantedBy=timers.target

Persistent=true runs the backup at boot if the server was off at the scheduled time.

Enable and test:

sudo systemctl daemon-reload
sudo systemctl enable --now restic-backup.timer
sudo systemctl start restic-backup.service     # run now to test
systemctl list-timers restic-backup.timer
journalctl -u restic-backup.service -n 50

The retention policy

The forget command keeps:

Rule	Keeps
`--keep-daily 7`	The latest backup of each of the last 7 days that have one
`--keep-weekly 4`	The latest of each of the last 4 weeks that have one
`--keep-monthly 12`	The latest of each of the last 12 months that have one

A day without a backup (server switched off) does not count: restic looks further back to keep 7 real days.

--prune then actually deletes the data no longer used by any snapshot.

6. Test a restore

A backup is only worth something if you know how to restore it. Restore into a temporary folder, never over the files in place:

sudo bash -c 'set -a; . /etc/restic/env; restic restore latest --target /tmp/restore --include /etc/hostname'
cat /tmp/restore/etc/hostname

restic recreates the full tree under the target folder: /etc/hostname becomes /tmp/restore/etc/hostname.

To recover a file by browsing, mount the repository (requires the fuse3 package):

sudo mkdir -p /mnt/restic
sudo bash -c 'set -a; . /etc/restic/env; restic mount /mnt/restic'
# in another terminal: sudo ls /mnt/restic/snapshots/

Tip

Once a month, also check part of the data itself, not just the repository structure: restic check --read-data-subset=5%

Going further

Second destination (3-2-1 rule): add an off-site repository (S3 storage, a NAS at a relative's home) with a second service, or copy the repository with restic copy.
Alert on failure: add OnFailure= to the service, pointing to a unit that sends a notification (ntfy, e-mail), or a ping to a service such as Healthchecks.
Ransomware protection: enable read-only snapshots on the NAS side, so a compromised server cannot delete its own history.

Backup / Tutorial