> ## Documentation Index
> Fetch the complete documentation index at: https://www.latitude.sh/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Backing up a bare metal server to Object storage with restic

> Encrypted, deduplicated server backups in an S3-compatible bucket, with scheduled runs and tested restores

[restic](https://restic.net/) is an open-source backup program that deduplicates and encrypts your data before it leaves the server, and it speaks the S3 API natively. That makes it a good fit for [Object storage](https://www.latitude.sh/docs/storage/object-storage.md): your server writes straight to a bucket, and Latitude.sh does not charge for the traffic in either direction (see [Egress](https://www.latitude.sh/docs/storage/egress.md)).

Everything that matters happens on the server. restic splits your files into chunks, drops the chunks it has already stored, encrypts what is left, and packs it into objects. The bucket only ever sees encrypted data, which is why the repository password matters as much as the access key.

## What you'll do

The guide covers a full backup lifecycle on a single server, in four phases. Do them in order: the repository has to exist before the first backup, and a restore is only worth automating once you have tested it manually.

> [accordion] Backing up more than one server
>
> The steps below set up one server. To cover several, go through them once per host, with a single change when you reach [Step 4](#step-4-configure-credentials): append the hostname to the repository URL before running `restic init`.
>
> ```ini theme={null}
> RESTIC_REPOSITORY=s3:$LSH_S3_ENDPOINT_URL/$BUCKET_BACKEND_NAME/web-01
> ```
>
> That gives each host its own repository inside the same bucket, reachable with the same access key. Pointing every host at one shared repository instead would hand each of them a key that can read and delete the whole fleet's backups.

## Prerequisites

* A [bare metal server](https://www.latitude.sh/docs/servers/deploying-a-server.md) you can [reach over SSH](https://www.latitude.sh/docs/servers/logging-into-your-server.md) as a user with `sudo`. The commands below assume you run them as `root`
* A [verified team](https://www.latitude.sh/docs/account-setup/verify-account.md), which is required to create access keys
* The [Latitude.sh CLI](https://www.latitude.sh/docs/cli.md) installed and logged in on your workstation. Steps 1 and 2 use the `lsh s3` command group; if `lsh s3 --help` does not recognize it, run `lsh update` first
* A place to store the repository password that is **not** the server you are backing up (a password manager, for example)

> [!WARNING]
>
> restic encrypts the repository with a password. If you lose that password, the
> backups cannot be decrypted by anyone, including Latitude.sh. Store it
> somewhere other than the machine being backed up.

## Step 1: Create a bucket

Create a dedicated bucket for backups so its lifecycle and credentials stay separate from application data.

Run this step from your workstation, where `lsh` is logged in. Bucket administration goes through the Latitude.sh API, so it uses your `lsh` session and needs no S3 credentials. To do it from the dashboard instead, see [Creating a bucket](https://www.latitude.sh/docs/storage/object-storage.md#creating-a-bucket).

**1. Set the values the commands will use**

The commands in this step and the next read three values from your shell. Export them once, substituting the placeholders, so every command below can be pasted as is:

```shell theme={null}
export LSH_PROJECT=<PROJECT_ID_OR_SLUG>
export SITE=<SITE_SLUG>
export BUCKET=<team>-<hostname>-backups
```

* **`LSH_PROJECT`** — the project that will own the bucket, by ID or slug. `lsh projects list` prints both for every project; the dashboard shows the ID under [project settings](https://www.latitude.sh/docs/projects.md#project-settings). The CLI reads this variable itself, so the `--project` flag below is there only to make the choice explicit.
* **`SITE`** — the site where the bucket will live. `lsh regions list` prints the slugs (`DAL`, `SAO`, `NYC`), and [Regions & Locations](https://www.latitude.sh/docs/regions-locations.md) explains how sites relate to regions. Pick a site that is **not** the same location as the server you are backing up: if the server and its backups sit in one facility, a single site-level incident takes out both.
* **`BUCKET`** — the display name of the bucket. Names are globally unique, so a generic `backups` is likely taken; `<team>-<hostname>-backups` avoids collisions and still reads clearly a year from now. See [Bucket naming rules](https://www.latitude.sh/docs/storage/object-storage.md#bucket-naming-rules) for the full constraints.

These variables live in this shell only. If you open a new terminal before finishing [Step 2](#step-2-create-an-access-key), export them again.

**2. Create the bucket**

> [tab] Without object lock (recommended)
>
> ```shell theme={null}
> lsh s3 mb "s3://$BUCKET" --project "$LSH_PROJECT" --region "$SITE" --storage-class standard --versioning --no-access-key
> ```

> [tab] With object lock
>
> Object lock (WORM) keeps every object version immutable until its retention expires, even for the bucket owner. Use it for backups that a stolen access key must not be able to destroy, or for a regulatory retention period. It cannot be added to an existing bucket, so it has to be decided here.
>
> ```shell theme={null}
> lsh s3 mb "s3://$BUCKET" --project "$LSH_PROJECT" --region "$SITE" --storage-class standard --versioning --no-access-key \
>   --locking --retention-mode GOVERNANCE --retention-days 30
> ```
>
> * **`--locking`** — turns object lock on.
> * **`--retention-mode`** — decides who can lift the lock early. **GOVERNANCE** lets users with specific permissions do it; **COMPLIANCE** lets nobody, not even you, and the CLI asks you to confirm that before creating the bucket. See [Versioning and object lock](https://www.latitude.sh/docs/storage/object-storage.md#versioning-and-object-lock).
> * **`--retention-days`** — how long each object version stays locked, counted from the moment it is written. Set it to the minimum time a backup must remain recoverable.
>
> The other flags are the same as in the default command and are explained below.
>
> The rest of the guide works unchanged. restic never modifies an object in place, and deleting one in a versioned bucket only hides it behind a delete marker, which the lock allows. The one difference shows up in [Step 10](#step-10-apply-a-retention-policy): the data that `restic forget --prune` deletes stays in the bucket as locked versions, and keeps taking up space, until its retention expires. In COMPLIANCE mode nothing can remove it earlier. Expect the bucket to shrink only after that window.

What each flag decides:

* **`--storage-class standard`** — backups are sequential and write-mostly, so the low latency that `high_performance` is built for does not shorten a nightly run. The two classes are also priced differently: see the [storage pricing page](https://www.latitude.sh/pricing/storage). Standard is the default; the flag is there to make the choice visible.
* **`--versioning`** — preserves every version of an object, so an accidental overwrite or delete inside the repository can be rolled back.
* **`--no-access-key`** — skips the prompt `mb` shows after creating a bucket, which offers to create an access key and store it in your `lsh` profile. A key stored there is for running `lsh s3` commands from your workstation. restic needs its own key on the server, which you create in [Step 2](#step-2-create-an-access-key).

**3. Read back the bucket**

```shell theme={null}
lsh s3 stat "s3://$BUCKET"
```

Check that **Versioning** says `yes` and **Locking** matches the tab you used, then keep two lines from the output. You will export them on the server in [Step 4](#step-4-configure-credentials):

* **Bucket name (backend)** — the name the bucket has on the S3 endpoint. It can differ from the display name you chose, and it is the one restic needs.
* **Endpoint** — the S3 API endpoint for this bucket. Buckets of different storage classes are reached at different endpoints.

You can run this command again at any time to read both values back.

## Step 2: Create an access key

Access keys are project-scoped S3 credentials, separate from the API token `lsh` logged in with. Make a dedicated one for the backup job, scoped to this bucket only, so revoking it later touches nothing else. To do it from the dashboard instead, see [Access keys](https://www.latitude.sh/docs/storage/object-storage.md#access-keys).

Still on your workstation, with the variables from [Step 1](#step-1-create-a-bucket) exported:

```shell theme={null}
lsh s3 access-keys create --bucket "$BUCKET=rw" --name restic-backup
```

`--bucket "$BUCKET=rw"` creates a **limited** key that reaches only this bucket, with **read-write** permission. restic needs to write new data and to delete data when you prune, so `readonly` will not work for the full lifecycle. Storage class, site and project are inferred from the bucket.

`--name` is capped at 25 characters by the API, which is why it does not carry the hostname. If you back up several servers into this bucket, one key serves all of them.

The output includes an **Access Key ID** and a **Secret Access Key** line. Keep both for [Step 4](#step-4-configure-credentials).

> [!WARNING]
>
> The secret is displayed only once, right after the key is created. Neither the
> API nor the CLI can show it again. If you lose it, create a new key.

> [!NOTE]
>
> Do not add `--save`. It stores the key in your workstation's `lsh` profile and
> hides the secret, which is the opposite of what you need here: this key belongs
> on the server, in the environment file you will write in Step 4.

## Step 3: Install restic on the server

[Log in to the server over SSH](https://www.latitude.sh/docs/servers/logging-into-your-server.md). Every remaining step in this guide runs there, not on your workstation.

> [!NOTE]
>
> Everything from here runs as `root`. If you logged in as another user, switch
> once with `sudo -i` rather than prefixing each command: several commands below
> are chained with `&&`, where a single `sudo` would apply only to the first half.

Most distributions package restic, though often a few versions behind:

```shell Ubuntu / Debian theme={null}
apt-get update && apt-get install -y restic
```

```shell AlmaLinux / Rocky theme={null}
dnf install -y epel-release && dnf install -y restic
```

```shell Oracle Linux 10 theme={null}
dnf install -y oracle-epel-release-el10 && dnf install -y restic
```

On the RHEL family restic ships in EPEL, not in the base repositories, which is why those commands enable EPEL before installing.

> [accordion] Installing the official binary instead
>
> Use this to run a current release, or if your distribution has no restic package. Check the [releases page](https://github.com/restic/restic/releases) for the latest version, without the `v`, and export it once:
>
> ```shell theme={null}
> export RESTIC_VERSION=0.19.1
> ```
>
> Then run the tab for your architecture. `uname -m` tells you which: `x86_64` for the first, `aarch64` for the second.
>
> ```shell x86_64 theme={null}
> curl -fsSLO "https://github.com/restic/restic/releases/download/v${RESTIC_VERSION}/restic_${RESTIC_VERSION}_linux_amd64.bz2"
> bzip2 -d "restic_${RESTIC_VERSION}_linux_amd64.bz2"
> install -m 755 "restic_${RESTIC_VERSION}_linux_amd64" /usr/local/bin/restic
> ```
>
> ```shell arm64 theme={null}
> curl -fsSLO "https://github.com/restic/restic/releases/download/v${RESTIC_VERSION}/restic_${RESTIC_VERSION}_linux_arm64.bz2"
> bzip2 -d "restic_${RESTIC_VERSION}_linux_arm64.bz2"
> install -m 755 "restic_${RESTIC_VERSION}_linux_arm64" /usr/local/bin/restic
> ```
>
> The binary lands in `/usr/local/bin/restic`, which matters again in [Step 9](#step-9-schedule-it-with-systemd).

Confirm the install before going further:

```shell theme={null}
restic version
```

It prints the restic version, the Go version it was built with, and the platform. If the shell reports `command not found`, the install did not land on your `PATH`.

> [accordion] Upgrading restic later
>
> `restic self-update` exists only in the official binary. Distribution packages are built without it and answer `unknown command "self-update" for "restic"`, so upgrade those through the package manager:
>
> > [tab] Ubuntu / Debian
> >
> > ```shell theme={null}
> > apt-get update && apt-get install --only-upgrade restic
> > ```
>
> > [tab] AlmaLinux / Rocky / Oracle Linux
> >
> > ```shell theme={null}
> > dnf upgrade restic
> > ```

## Step 4: Configure credentials

restic reads its configuration from environment variables. Keeping them in one file means the secret never has to appear in a `restic` command line, or in the systemd unit you will write later.

**1. Generate the repository password**

```shell theme={null}
openssl rand -base64 32 > /root/.restic-password
chmod 600 /root/.restic-password
```

Print it and copy it into your password manager now:

```shell theme={null}
cat /root/.restic-password
```

The file stays on the server so restic can read it unattended, but that copy dies with the server. The one in your password manager is what makes the backups recoverable.

**2. Write the environment file**

Four values from your workstation go in this file. Export them on the server first, substituting the placeholders. The variable names are the ones the CLI reads for its own S3 commands, so they carry over if you ever use `lsh s3` against this bucket:

```shell theme={null}
export BUCKET_BACKEND_NAME=<BUCKET_BACKEND_NAME>
export LSH_S3_ENDPOINT_URL=<S3_API_ENDPOINT>
export LSH_S3_ACCESS_KEY_ID=<LSH_S3_ACCESS_KEY_ID>
export LSH_S3_SECRET_ACCESS_KEY=<LSH_S3_SECRET_ACCESS_KEY>
```

* **`BUCKET_BACKEND_NAME`** and **`LSH_S3_ENDPOINT_URL`** — the **Bucket name (backend)** and **Endpoint** lines that `lsh s3 stat` printed in [Step 1](#step-1-create-a-bucket). The backend name is not the display name you chose.
* **`LSH_S3_ACCESS_KEY_ID`** and **`LSH_S3_SECRET_ACCESS_KEY`** — the **Access Key ID** and **Secret Access Key** from [Step 2](#step-2-create-an-access-key).

Then write the file. The heredoc expands those variables, so it can be pasted as is:

```shell theme={null}
cat > /root/.restic-env <<EOF
RESTIC_REPOSITORY=s3:$LSH_S3_ENDPOINT_URL/$BUCKET_BACKEND_NAME
RESTIC_PASSWORD_FILE=/root/.restic-password
AWS_ACCESS_KEY_ID=$LSH_S3_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY=$LSH_S3_SECRET_ACCESS_KEY
EOF
chmod 600 /root/.restic-env
```

The repository URL is the `s3:` prefix, then the endpoint exactly as the CLI prints it, then the backend bucket name as the last path segment.

> [!NOTE]
>
> The `export` line with the secret lands in your shell history. If you would
> rather it never did, create the file with an editor instead, or clear the
> history afterwards.

**3. Load it into your shell**

```shell theme={null}
set -a && . /root/.restic-env && set +a
```

Check that it took, before restic has a chance to fail for a reason you would have to go looking for:

```shell theme={null}
echo "$RESTIC_REPOSITORY"
echo "$AWS_ACCESS_KEY_ID"
```

Both should print the values you just wrote. An empty line means the file did not load, usually a typo in the path. Neither of these is the secret, so it is safe to print them.

Every `restic` command in the rest of this guide assumes these variables are loaded. They live in this shell only: open a new SSH session and you have to load them again.

## Step 5: Initialize the repository

```shell theme={null}
restic init
```

This writes the repository structure and the encrypted key material into the bucket. Run it once per repository, not once per backup.

Now confirm restic can actually reach it:

```shell theme={null}
restic cat config
```

Expect `repository <id> opened`, then the repository's own metadata. That line is the signal, not the metadata: a wrong endpoint, bucket name, key, or password fails the command outright. See [Troubleshooting](#troubleshooting) if it does.

## Step 6: Run the first backup

Decide what to back up. On a server, the useful set is usually configuration, application data, and databases — not the whole root filesystem. Reinstalling packages is cheap; recovering `/etc` and your data is not.

**1. Write an exclude file**

These are paths that either cannot be restored or are not worth storing: kernel and runtime filesystems, scratch space, caches, and sockets.

```shell theme={null}
cat > /root/.restic-excludes <<'EOF'
/dev
/proc
/sys
/run
/tmp
/var/tmp
/var/cache
/var/lib/docker
/swapfile
*.sock
EOF
```

Adjust the list to the server. `/var/lib/docker`, for example, only matters if you run Docker, and its images are rebuildable anyway.

**2. Dump any databases**

Skip this if the server does not run one. A file-level copy of a running database can be inconsistent, so dump it to disk first and let restic pick up the dump. The example is PostgreSQL; other databases have their own dump tools, and the same rules apply: run the dump as the database's system user, into a directory only that user and `root` can read.

```shell theme={null}
mkdir -p /var/backups/postgres && chown postgres:postgres /var/backups/postgres && chmod 700 /var/backups/postgres
runuser -u postgres -- pg_dumpall --clean --file=/var/backups/postgres/all.sql
```

The dump lands under `/var/backups`, which is already in the backup command of the next step.

> [!NOTE]
>
> This step is for a database you run on the server yourself. A
> [Managed Database](https://www.latitude.sh/docs/managed-databases.md) lives outside the server and has its own
> [scheduled backups](https://www.latitude.sh/docs/managed-databases/backups-and-restoration.md), so it does not
> go in this list.

**3. Back up**

```shell theme={null}
restic backup /etc /home /srv /var/backups \
  --one-file-system \
  --exclude-caches \
  --exclude-file=/root/.restic-excludes \
  --tag manual
```

The first three exist on any server, and `/var/backups` is where the dump from the previous step lands. Add the ones that hold your data: `/var/www` for a web root, whatever else applies.

> [!WARNING]
>
> If you dumped a database, `/var/backups` has to stay in this list. The dump
> is only protected if the path that holds it is backed up, and nothing warns
> you when it is not: the backup succeeds without it.

A path that is not there is not fatal. restic prints `<path> does not exist, skipping` and backs up the rest, so a list that is slightly wrong still produces a snapshot. Read that line when it appears, though: it is also what you would see if the data you meant to protect is somewhere else.

`--one-file-system` keeps restic from walking into other mounts, such as an attached [block volume](https://www.latitude.sh/docs/storage/block-storage.md) or a network filesystem. Drop it if you do want those included.

The first run uploads everything. Later runs only upload changed data, because restic splits files into content-addressed chunks and skips those already in the repository.

## Step 7: Verify the backup

Check the repository after the first run.

```shell theme={null}
restic snapshots
```

Lists one row per snapshot under `ID`, `Time`, `Host`, `Tags`, and `Paths`. After a first backup you should see exactly one row. An empty list means nothing was stored.

```shell theme={null}
restic check
```

Validates the repository structure and metadata without downloading every pack file. It should end with `no errors were found`. Any other ending names what is damaged.

```shell theme={null}
restic check --read-data-subset=5%
```

The same check, plus downloading and hashing a sample of the pack files, so it also catches corruption in the stored data rather than just the index. Ends with the same line, takes considerably longer, and is worth running periodically rather than after every backup.

## Step 8: Restore

Practice this before you need it. If you are back in a new SSH session, load the environment file from [Step 4](#step-4-configure-credentials) first, or every command below fails with `Fatal: Please specify repository location`:

```shell theme={null}
set -a && . /root/.restic-env && set +a
```

**1. Restore everything to a scratch directory**

> [tab] Ubuntu / Debian
>
> ```shell theme={null}
> restic restore latest --target /tmp/restore
> ```

> [tab] AlmaLinux / Rocky / Oracle Linux
>
> ```shell theme={null}
> restic restore latest --target /tmp/restore --exclude-xattr security.selinux
> ```
>
> These distributions run SELinux: the kernel labels every restored file and blocks restic from changing that label. Without `--exclude-xattr security.selinux` the restore writes all the data and still ends in `Fatal: There were N errors`.

restic creates the target directory, reports progress, and ends with a summary of how many files and how much data it restored. Run it twice against the same target and the second summary reports most files as `skipped`: restic leaves files that already match the snapshot alone.

The real check is the result on disk. Under the target you get the paths you backed up, rebuilt as a tree:

```shell theme={null}
ls /tmp/restore
```

You get one top-level directory per path you backed up in [Step 6](#step-6-run-the-first-backup), so `etc`, `home`, `srv`, `var`, and whichever others you kept. Spot-check a file you recognize, for example `cat /tmp/restore/etc/hostname`.

If the snapshot holds the database dump from Step 6, load it back as `root`, through stdin:

```shell theme={null}
sudo -u postgres psql postgres < /tmp/restore/var/backups/postgres/all.sql
```

restic creates the target directory so that only `root` can enter it, which is why `psql -f` run as the `postgres` user fails with `Permission denied`. Errors about roles or databases that already exist are expected when a `--clean` dump is loaded on the server it came from.

Restoring to `/tmp/restore` rather than `/` is what makes this safe to practice: nothing live is overwritten.

**2. Restore a single path**

> [tab] Ubuntu / Debian
>
> ```shell theme={null}
> restic restore latest --target /tmp/restore-ssh --include /etc/ssh
> ```

> [tab] AlmaLinux / Rocky / Oracle Linux
>
> ```shell theme={null}
> restic restore latest --target /tmp/restore-ssh --include /etc/ssh --exclude-xattr security.selinux
> ```

Same command, narrowed, and pointed at a fresh target so the result is unambiguous. Only the included subtree is written, so `etc/ssh` is all you get:

```shell theme={null}
ls /tmp/restore-ssh/etc/ssh
```

`/etc/ssh` is used here because it is guaranteed to be in the snapshot: you reached this server over SSH.

> [!WARNING]
>
> If `--include` matches nothing, restic reports `Restored 0 files/dirs (0 B)`,
> does not create the target directory at all, and still exits successfully. An
> empty result means the pattern is wrong, not the repository.

> [accordion] Browsing snapshots as a filesystem (optional)
>
> `restic mount` exposes the repository as a read-only filesystem. It is not part of the backup lifecycle, and the two restores above already cover recovery. It earns its place when you need one file and do not know which snapshot holds it, because you can look instead of guessing.
>
> Mounting needs the `fusermount` binary from FUSE. Whether it is already there depends on the image, not on restic: the Ubuntu and AlmaLinux images ship it, the Oracle Linux one does not. Check before mounting:
>
> ```shell theme={null}
> command -v fusermount
> ```
>
> If that prints nothing, `restic mount` would fail with `fusermount: executable file not found in $PATH`. On the RHEL family, install it with:
>
> ```shell theme={null}
> dnf install -y fuse
> ```
>
> It has to be `fuse`, not `fuse3`: only the first one provides `fusermount`.
>
> Create the mountpoint first, because restic does not:
>
> ```shell theme={null}
> mkdir -p /mnt/restic
> restic mount /mnt/restic
> ```
>
> The command prints `Now serving the repository at /mnt/restic` and then blocks, holding the terminal. Leave it running and open a second SSH session to browse. That session starts as your login user, so switch to `root` again with `sudo -i` before continuing:
>
> ```shell theme={null}
> ls /mnt/restic/snapshots/
> ```
>
> Each snapshot is a directory, alongside a `latest` shortcut to the newest one. Inside, the tree looks like the server did, and ordinary commands work on it, so you can copy a single file straight out:
>
> ```shell theme={null}
> cp /mnt/restic/snapshots/latest/etc/hostname /tmp/
> ```
>
> Back in the first session, `Ctrl+C` unmounts.

## Step 9: Schedule it with systemd

Everything so far has been manual, which means the backup is only as current as the last time someone remembered to run it.

systemd is the service manager already running on your server, and scheduling with it takes two files: a **service** unit that defines what to run, and a **timer** unit that defines when to run it. The timer starts the service on a schedule the way cron would, with two differences that matter here: every run is recorded in the system journal, so you can read what happened afterwards, and a run missed while the server was off is caught up on the next boot.

You will write both files, tell systemd to pick them up, then trigger one run manually to confirm it works before leaving it to the schedule.

**1. Create the service unit**

systemd runs this unit with no shell profile and no `PATH` of its own, so every binary has to be named by absolute path, and a path that does not exist fails the whole unit. Two things in the unit below depend on your server:

* **The `restic` path.** A packaged install lands in `/usr/bin/restic`, the official binary in `/usr/local/bin/restic`.
* **The backup paths.** The same list you settled on in [Step 6](#step-6-run-the-first-backup), including `/var/backups` if you dump a database. The environment file carries your credentials, not your path choices, so they have to be spelled out again here.

Ask the server which restic it has, and keep the answer in a variable:

```shell theme={null}
export RESTIC_BIN=$(which restic)
echo "$RESTIC_BIN"
```

It prints `/usr/bin/restic` for a packaged install, or `/usr/local/bin/restic` for the official binary. Then create the unit, substituting your own backup paths. The heredoc fills in the restic path from the variable:

```shell theme={null}
cat > /etc/systemd/system/restic-backup.service <<EOF
[Unit]
Description=restic backup to Latitude.sh Object storage
Wants=network-online.target
After=network-online.target

[Service]
Type=oneshot
Environment=HOME=/root
EnvironmentFile=/root/.restic-env
ExecStart=$RESTIC_BIN backup /etc /home /srv /var/backups --one-file-system --exclude-caches --exclude-file=/root/.restic-excludes --tag scheduled
Nice=10
IOSchedulingClass=idle
EOF
```

What the directives do:

* **`EnvironmentFile`** loads the credentials from [Step 4](#step-4-configure-credentials), which is how the timer authenticates without a shell.
* **`Environment=HOME=/root`** is there because systemd starts the unit without a `HOME`. Without it, restic logs `unable to open cache` and re-downloads metadata it could have reused.
* **`Nice`** and **`IOSchedulingClass`** keep the backup from competing with your workload for CPU and disk.

If you dump a database, add a line above `ExecStart` so the scheduled run refreshes the dump instead of storing the one you made manually, getting staler every night. It is the same command as in [Step 6](#step-6-run-the-first-backup), with the absolute paths that `which runuser` and `which pg_dumpall` print:

```ini theme={null}
ExecStartPre=/usr/sbin/runuser -u postgres -- /usr/bin/pg_dumpall --clean --file=/var/backups/postgres/all.sql
```

> [!WARNING]
>
> Check that `/var/backups` is in the `ExecStart` line above before you move
> on. With the dump line in place and the path missing, every scheduled run
> refreshes the dump and then backs up everything except it, and the unit
> still reports `SUCCESS`.

**2. Create the timer**

```shell theme={null}
cat > /etc/systemd/system/restic-backup.timer <<'EOF'
[Unit]
Description=Daily restic backup

[Timer]
OnCalendar=daily
RandomizedDelaySec=1h
Persistent=true

[Install]
WantedBy=timers.target
EOF
```

`Persistent=true` runs a missed backup after a reboot. `RandomizedDelaySec` spreads the load when you run this across a fleet.

**3. Load and arm the timer**

```shell theme={null}
systemctl daemon-reload
```

Makes systemd read the two new files. Silent on success.

```shell theme={null}
systemctl enable --now restic-backup.timer
```

Prints a `Created symlink` line, which is systemd confirming the timer is enabled at boot and started now.

**4. Run it once manually**

Do not wait for the schedule to find out the unit is broken.

```shell theme={null}
systemctl start restic-backup.service
```

This runs a real backup and returns only when it finishes, printing nothing. No output is the good outcome; an error here means the unit failed to start at all.

```shell theme={null}
systemctl status restic-backup.service
```

Look for `status=0/SUCCESS` on the `Main PID:` line. `Active: inactive (dead)` above it is expected and not a failure: a `Type=oneshot` unit is meant to run and exit, so it sits inactive between runs.

```shell theme={null}
journalctl -u restic-backup.service -n 50
```

Shows what restic actually did on that run, including the new snapshot and the files it processed. If `systemctl start` failed instead, these same two commands say why: see [Troubleshooting](#troubleshooting).

**5. Confirm the schedule**

```shell theme={null}
systemctl list-timers restic-backup.timer
```

One row, with `NEXT` and `LEFT` showing when the next run fires. `LAST` and `PASSED` stay `-` until the timer itself triggers the service: starting it manually does not count as a timer activation. An empty table means the timer is not enabled, so revisit the `enable` command above.

## Step 10: Apply a retention policy

The timer now adds a snapshot every night and never removes one, so the repository grows forever. A retention policy decides which snapshots are worth keeping: recent ones in full detail, older ones thinned out to one per week, then one per month.

Two separate things do that work, and the distinction explains the commands below:

* **`forget`** marks snapshots as no longer wanted. It is fast, because it only drops the snapshot's entry, not the data behind it.
* **`prune`** deletes the data that no remaining snapshot references. It is what actually frees space in the bucket, and it is slow, because it rewrites pack files.

Deduplication is why they are separate: a chunk shared by five snapshots can only be deleted once nothing points at it, so the data cannot go until you have decided which snapshots are staying.

```shell theme={null}
restic forget \
  --keep-daily 7 \
  --keep-weekly 5 \
  --keep-monthly 12 \
  --prune
```

That keeps the last 7 daily snapshots, 5 weekly, and 12 monthly, and drops the rest. `--prune` runs the second phase right after the first.

Add `--dry-run` first to see which snapshots the policy would remove, without removing anything.

The real run prints the policy it applied, then a `keep` table of every snapshot that survived it. The `Reasons` column there is worth reading: it names the rule that saved each one, so a snapshot kept as `daily snapshot`, `weekly snapshot`, and `monthly snapshot` at once is filling all three roles. Snapshots on their way out appear in a second table below, and `--prune` then reports what it repacked and deleted.

> [!NOTE]
>
> Right after setting this up, nothing is removed: with one or two snapshots,
> every one still matches the policy.

Confirm the state it left behind:

```shell theme={null}
restic snapshots
restic check
```

`restic snapshots` should list exactly the snapshots the policy said to keep, and nothing else. `restic check` should still end with `no errors were found`, which is what tells you prune removed data without leaving the repository inconsistent.

Run this on a schedule too, or the repository grows anyway. Give it a timer of its own rather than adding it to the nightly backup unit, built the same way as [Step 9](#step-9-schedule-it-with-systemd), for two reasons: prune is the heavy half and does not need to run daily, and a prune that fails would otherwise mark your backup as failed even though the backup itself succeeded.

## Wrap-up

You now have:

* A versioned bucket in a different site from the server, reached with an access key scoped to that bucket alone
* An encrypted restic repository in it, with the password stored off the server
* A nightly systemd timer that backs up your chosen paths and logs every run to the journal
* A retention policy that keeps the repository from growing forever, and a restore you have already tested once

Keep the repository password and the access key in your password manager, and repeat the restore from [Step 8](#step-8-restore) from time to time. A backup counts only once it has been restored.

## Troubleshooting

> [accordion] systemctl start failed because the control process exited with error code
>
> Run `systemctl status restic-backup.service` and
> `journalctl -u restic-backup.service -n 50`. The usual causes:
>
> * **`status=203/EXEC`** — systemd could not execute the binary named in
>   `ExecStart` or `ExecStartPre`. The path is wrong or the file is not there.
>   Compare it against `which restic`.
> * **restic ran and exited non-zero** — the journal carries restic's own
>   message. A path in the backup list that does not exist on the server is
>   the common one.

> [accordion] Please specify repository location (-r or --repository-file)
>
> restic did not find `RESTIC_REPOSITORY` in the environment. The variables
> from [Step 4](#step-4-configure-credentials) live in the shell that loaded
> them: a new SSH session, or a fresh `sudo -i`, starts without them. Load
> the file again:
>
> ```shell theme={null}
> set -a && . /root/.restic-env && set +a
> ```

> [accordion] xattr.LRemove ... security.selinux: permission denied
>
> Hundreds of these lines during a restore, ending in `Fatal: There were N
>         errors`, mean the host runs SELinux. The data was restored; what failed is
> restic trying to change the SELinux label the kernel put on each new file.
> Run the restore again with `--exclude-xattr security.selinux`, as in
> [Step 8](#step-8-restore).

> [accordion] FATAL: role root does not exist
>
> `pg_dumpall` was run as `root`. PostgreSQL authenticates local connections
> by system user, and there is no database role called `root`. Run the dump
> as the `postgres` user with `runuser`, as in
> [Step 6](#step-6-run-the-first-backup), and make sure the output directory
> belongs to that user.

> [accordion] Is there a repository at the following location?
>
> restic could not find a repository at `RESTIC_REPOSITORY`. Check that
> `restic init` has been run, that the last path segment of the URL is the
> **Bucket name (backend)** that `lsh s3 stat "s3://$BUCKET"` prints, not
> the display name, and that the endpoint is the **Endpoint** from the same
> output. Buckets of different storage classes are reached at different
> endpoints, so one copied from another bucket will not resolve.

> [accordion] 403 Forbidden or SignatureDoesNotMatch
>
> Usually the secret key. Because it is shown only once, a truncated paste is
> common. It can also mean the access key is scoped elsewhere — keys apply to
> a specific storage class and region, and a limited key only reaches the
> buckets it was created with. Run
> `lsh s3 access-keys get restic-backup` on your workstation and
> confirm the **Buckets** line lists this bucket with `rw`.

> [accordion] Errors mentioning region
>
> Some S3 clients need an explicit region for a non-AWS endpoint. Set
> `AWS_DEFAULT_REGION` in `/root/.restic-env` to the **Signing region** that
> `lsh s3 stat "s3://$BUCKET"` prints, or pass it per command with
> `-o s3.region=<region>`.

> [accordion] repository is already locked
>
> A previous run left a stale lock, usually after a crash or a killed process.
> Confirm no backup is actually running, then clear it:
>
> ```shell theme={null}
> restic unlock
> ```

> [accordion] Operation counts look high
>
> restic groups chunks into pack files of roughly 16 MB rather than storing one
> object per file, so a backup of many small files does not create a matching
> number of objects. To see what the bucket is costing, run
> `lsh s3 metrics "s3://$BUCKET"` on your workstation: it prints the
> storage consumed in the current billing period and the estimated cost. The
> [storage pricing page](https://www.latitude.sh/pricing/storage) lists what
> each class of operation costs.

## Next steps

* Read the [restic documentation](https://restic.readthedocs.io/) for append-only setups, `rest-server`, and backend tuning


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.