Backup setup

Server setup
← Previous	Next →
Log stats	PiHole

I use Borg for taking backups of my two machines every night. It's controlled by a SystemD timer. The deduplication of backups allows many backups to be made in not much space. Borg allows easy recovery by allowing archives to be mounted as a drive.

The desktop and server machine each have two repositories. Each machine creates a backup locally and one on the other machine. I do it this way so there are no clashes with concurrent backups writing to the same repository from different machines.

Create users

Backups are safer if they use non-privileged users to do the backups. I create these users for the OS and databases.

OS

On each machine, create backup users. These will be used as the target of backups to remote machines.

Give these users login shells and home directories.

For each user, create an SSH key and transfer it to the other machine with ssh-copy-id. Once you've set up that both root and backup on each machine can SSH into backup on the other machine, you can disable the passwords for each backup user.

MySQL

For instance, create the backup user in MySQL. This user has read-only access to all databases, and does not require a password. Create the user by entering the following commands:

root@desktop:~# mysql -u 'root' -p 
mysql> grant select, lock tables, process on *.* to 'backup'@'localhost';
mysql> quit;

This automatically creates the user, with no password, and gives them permissions to read all MySQL databases.

Postgres

For Postgres, it's a bit more complicated. Create the user, connect to the database to back up, and grant permissions.

root@desktop:~# sudo -u postgres psql
postgres=# create user backup with login encrypted password 'mypassword';
CREATE ROLE
postgres=# \c dmarc
psql (9.3.13, server 9.1.18)
You are now connected to database "dmarc" as user "postgres".
postgres=# grant connect on database dmarc to backup;
GRANT
postgres=# grant select on all tables in schema public to backup;
GRANT
postgres=# grant select on all sequences in schema public to backup;
GRANT

Then create the file /root/.pgpass with contents

# hostname:port:database:username:password
*:*:*:backup:mypassword

(giving the password you've used) and give it the correct permissions:

root@desktop:~# chmod 0600 ~/.pgpass

Install Borg

I currently use Borg 1.26. It needs a library to support mounting backups via FUSE.

root@desktop:~# apt install borgbackup python3-llfuse

Creating archives

As the backup user, create two repositories on each machine, one for each machine being backed up. Remember to check and write down the repository encryption keys!

root@desktop:~# mkdir -p /backup/from_desktop
root@desktop:~# mkdir -p /backup/from_server
root@desktop:~# sudo -u backup borg init --encryption=repokey /backup/from_desktop
root@desktop:~# sudo -u backup borg init --encryption=repokey /backup/from_server

Do the same on the server, creating the two repositories.

On each machine, create the file /etc/borg/backup_patterns that defines what files are backed up. This allows the same files to be backed up to each destination.

P sh

R /home/
- /home/*/.cache
- /home/*/TresoritDrive
- /home/*/.kde/share/apps/nepomuk
- /home/*/.thunderbird
- /home/*/.local/share
- /home/*/.rvm
- /home/*/.vagrant.d
- /home/*/anaconda3
- /home/*/.ghcup
- /home/**/machine-learning-datasets

R /opt/photos
R /opt/music
R /opt/sources

R /etc
R /lib/systemd
R /var/www

To do the backups from the desktop, create the files /etc/borg/backup_to_desktop and /etc/borg/backup_to_server. Below is the content of /etc/borg/backup_to_desktop. /etc/borg/backup_to_server is identical except for the line at the top which becomes export BORG_REPO=backup@server:/backup/from_desktop and the appropriate passphrase.

#!/bin/sh
 
# the repo and its passphrase
export BORG_REPO=/backup/from_desktop
export BORG_PASSPHRASE='long complex passphrase'
 
# some helpers and error handling:
info() { printf "\n%s %s\n\n" "$( date )" "$*" >&2; }
trap 'echo $( date ) Backup interrupted >&2; exit 2' INT TERM
 
info "Starting backup"
 
# backup the directories
borg create \
  --verbose --filter AME \
  --list --stats --show-rc \
  --one-file-system \
  --compression zstd,22 --exclude-caches \
  --patterns-from /etc/borg/backup_patterns \
  ::'{hostname}-daily-{now}'  2>&1
 
backup_exit=$?
 
info "Pruning repository"
 
# prune the repo
borg prune \
    --list \
    --prefix '{hostname}-daily-' \
    --show-rc \
    --keep-daily 7 \
    --keep-weekly 4 \
    --keep-monthly 24 2>&1
 
prune_exit=$?

# Compact the repo, for when version 1.2 is available
# borg compact 
# compact_exit=$? 
 
# use highest exit code as exit code
global_exit=$(( backup_exit > prune_exit ? backup_exit : prune_exit ))
# global_exit=$(( compact_exit > global_exit ? compact_exit : global_exit ))
 
if [ ${global_exit} -eq 1 ];
then
    info "Backup,  Prune, or Compact finished with a warning"
fi
 
if [ ${global_exit} -gt 1 ];
then
    info "Backup, Prune, or Compact finished with an error"
fi
 
exit ${global_exit}

Backing up the server requires an extra step, as I want to create database dumps that are backed up first. That means there are four Bash scripts in /etc/borg: backup_starter coordinates the backup. It calls create_db_backups to dump the databases, then the two backup_to_server and backup_to_desktop scripts.

backup_starter is simple:

#!/bin/bash

/etc/borg/create_db_backups

/etc/borg/backup_to_server  &
/etc/borg/backup_to_desktop &

wait

create_db_backups does the database dumps, as well as creating a list of manually-installed packages.

#!/bin/bash

export DB_BACKUP_DIR=/backup/db_backups

rm ${DB_BACKUP_DIR}/* || true

(
nice comm -23 \
  <(aptitude search '~i !~M' -F '%p' | sed "s/ *$//" | sort -u) \
  <(gzip -dc /var/log/installer/initial-status.gz | sed -n 's/^Package: //p' | sort -u) \
  > "${DB_BACKUP_DIR}/manually-installed-packages.txt"
) &

# Create database backups 
for db in thisdb thatdb ; do
  (
    nice mysqldump --user='backup' --opt --databases $db > "${DB_BACKUP_DIR}/${db}-dump.sql"
  ) & 
done

(
    pg_dump -h 127.0.0.1 -U backup -Fc dmarc  > "${DB_BACKUP_DIR}/dmarc-dump"
) & 


for wiki in this_wiki that_wiki ; do
  (
    php /usr/share/mediawiki/maintenance/dumpBackup.php --full --quiet \
      --conf /var/www/${wiki}/mediawiki/LocalSettings.php \
      > "${DB_BACKUP_DIR}/${wiki}.dump.xml"
  ) &
done

wait

backup_to_server and backup_to_desktop are as above, but include the /backup/db_backups directory in the patterns file.

All the Bash scripts need to be executable.

Services and timers

The backups are run by SystemD services and timers. On the server, create /lib/systemd/system/borg_backup.service and /lib/systemd/system/borg_backup.timer.

/lib/systemd/system/borg_backup.service

[Unit]
Description=Borg Backup to server and desktop
 
[Service]
Type=simple
Nice=19
IOSchedulingClass=2
IOSchedulingPriority=7
ExecStart=/etc/borg/backup_starter
# User=backup
# Group=backup

/lib/systemd/system/borg_backup.timer

[Unit]
Description=Borg Backup Timer

[Timer]
WakeSystem=false
OnCalendar=*-*-* 1:00:00
RandomizedDelaySec=10min
 
[Install]
WantedBy=timers.target

The desktop machine has two service and timer files, one for each backup script: the backup to the desktop and backup to the server run as separate services.

Once you've created the service and timer files, tell SystemD about them and start the timers.

root@desktop:~# systemctl daemon-reload
root@desktop:~# systemctl enable borg_backup_to_desktop.timer
root@desktop:~# systemctl restart timers.target

Restoring files

The easiest way is to use borg list to view the backups in each repository, and borg mount to mount the repository as a filesystem. You can then extract the files you want fairly easily.