What you’ll do
The guide covers a full backup lifecycle on a single server, in four phases. Do them in order: the repository has to exist before the first backup, and a restore is only worth automating once you have tested it manually.Backing up more than one server
Backing up more than one server
restic init.Prerequisites
- A bare metal server you can reach over SSH as a user with
sudo. The commands below assume you run them asroot - A verified team, which is required to create access keys
- The Latitude.sh CLI installed and logged in on your workstation. Steps 1 and 2 use the
lsh s3command group; iflsh s3 --helpdoes not recognize it, runlsh updatefirst - A place to store the repository password that is not the server you are backing up (a password manager, for example)
Step 1: Create a bucket
Create a dedicated bucket for backups so its lifecycle and credentials stay separate from application data. Run this step from your workstation, wherelsh is logged in. Bucket administration goes through the Latitude.sh API, so it uses your lsh session and needs no S3 credentials. To do it from the dashboard instead, see Creating a bucket.
Set the values the commands will use
LSH_PROJECT— the project that will own the bucket, by ID or slug.lsh projects listprints both for every project; the dashboard shows the ID under project settings. The CLI reads this variable itself, so the--projectflag below is there only to make the choice explicit.SITE— the site where the bucket will live.lsh regions listprints the slugs (DAL,SAO,NYC), and Regions & Locations explains how sites relate to regions. Pick a site that is not the same location as the server you are backing up: if the server and its backups sit in one facility, a single site-level incident takes out both.BUCKET— the display name of the bucket. Names are globally unique, so a genericbackupsis likely taken;<team>-<hostname>-backupsavoids collisions and still reads clearly a year from now. See Bucket naming rules for the full constraints.
Create the bucket
- Without object lock (recommended)
- With object lock
--storage-class standard— backups are sequential and write-mostly, so the low latency thathigh_performanceis built for does not shorten a nightly run. The two classes are also priced differently: see the storage pricing page. Standard is the default; the flag is there to make the choice visible.--versioning— preserves every version of an object, so an accidental overwrite or delete inside the repository can be rolled back.--no-access-key— skips the promptmbshows after creating a bucket, which offers to create an access key and store it in yourlshprofile. A key stored there is for runninglsh s3commands from your workstation. restic needs its own key on the server, which you create in Step 2.
Read back the bucket
yes and Locking matches the tab you used, then keep two lines from the output. You will export them on the server in Step 4:- Bucket name (backend) — the name the bucket has on the S3 endpoint. It can differ from the display name you chose, and it is the one restic needs.
- Endpoint — the S3 API endpoint for this bucket. Buckets of different storage classes are reached at different endpoints.
Step 2: Create an access key
Access keys are project-scoped S3 credentials, separate from the API tokenlsh logged in with. Make a dedicated one for the backup job, scoped to this bucket only, so revoking it later touches nothing else. To do it from the dashboard instead, see Access keys.
Still on your workstation, with the variables from Step 1 exported:
--bucket "$BUCKET=rw" creates a limited key that reaches only this bucket, with read-write permission. restic needs to write new data and to delete data when you prune, so readonly will not work for the full lifecycle. Storage class, site and project are inferred from the bucket.
--name is capped at 25 characters by the API, which is why it does not carry the hostname. If you back up several servers into this bucket, one key serves all of them.
The output includes an Access Key ID and a Secret Access Key line. Keep both for Step 4.
--save. It stores the key in your workstation’s lsh profile and
hides the secret, which is the opposite of what you need here: this key belongs
on the server, in the environment file you will write in Step 4.Step 3: Install restic on the server
Log in to the server over SSH. Every remaining step in this guide runs there, not on your workstation.root. If you logged in as another user, switch
once with sudo -i rather than prefixing each command: several commands below
are chained with &&, where a single sudo would apply only to the first half.Installing the official binary instead
Installing the official binary instead
v, and export it once:uname -m tells you which: x86_64 for the first, aarch64 for the second./usr/local/bin/restic, which matters again in Step 9.command not found, the install did not land on your PATH.
Upgrading restic later
Upgrading restic later
restic self-update exists only in the official binary. Distribution packages are built without it and answer unknown command "self-update" for "restic", so upgrade those through the package manager:- Ubuntu / Debian
- AlmaLinux / Rocky / Oracle Linux
Step 4: Configure credentials
restic reads its configuration from environment variables. Keeping them in one file means the secret never has to appear in arestic command line, or in the systemd unit you will write later.
Generate the repository password
Write the environment file
lsh s3 against this bucket:BUCKET_BACKEND_NAMEandLSH_S3_ENDPOINT_URL— the Bucket name (backend) and Endpoint lines thatlsh s3 statprinted in Step 1. The backend name is not the display name you chose.LSH_S3_ACCESS_KEY_IDandLSH_S3_SECRET_ACCESS_KEY— the Access Key ID and Secret Access Key from Step 2.
s3: prefix, then the endpoint exactly as the CLI prints it, then the backend bucket name as the last path segment.export line with the secret lands in your shell history. If you would
rather it never did, create the file with an editor instead, or clear the
history afterwards.Load it into your shell
restic command in the rest of this guide assumes these variables are loaded. They live in this shell only: open a new SSH session and you have to load them again.Step 5: Initialize the repository
repository <id> opened, then the repository’s own metadata. That line is the signal, not the metadata: a wrong endpoint, bucket name, key, or password fails the command outright. See Troubleshooting if it does.
Step 6: Run the first backup
Decide what to back up. On a server, the useful set is usually configuration, application data, and databases — not the whole root filesystem. Reinstalling packages is cheap; recovering/etc and your data is not.
Write an exclude file
/var/lib/docker, for example, only matters if you run Docker, and its images are rebuildable anyway.Dump any databases
root can read./var/backups, which is already in the backup command of the next step.Back up
/var/backups is where the dump from the previous step lands. Add the ones that hold your data: /var/www for a web root, whatever else applies.A path that is not there is not fatal. restic prints <path> does not exist, skipping and backs up the rest, so a list that is slightly wrong still produces a snapshot. Read that line when it appears, though: it is also what you would see if the data you meant to protect is somewhere else.--one-file-system keeps restic from walking into other mounts, such as an attached block volume or a network filesystem. Drop it if you do want those included.Step 7: Verify the backup
Check the repository after the first run.ID, Time, Host, Tags, and Paths. After a first backup you should see exactly one row. An empty list means nothing was stored.
no errors were found. Any other ending names what is damaged.
Step 8: Restore
Practice this before you need it. If you are back in a new SSH session, load the environment file from Step 4 first, or every command below fails withFatal: Please specify repository location:
Restore everything to a scratch directory
- Ubuntu / Debian
- AlmaLinux / Rocky / Oracle Linux
skipped: restic leaves files that already match the snapshot alone.The real check is the result on disk. Under the target you get the paths you backed up, rebuilt as a tree:etc, home, srv, var, and whichever others you kept. Spot-check a file you recognize, for example cat /tmp/restore/etc/hostname.If the snapshot holds the database dump from Step 6, load it back as root, through stdin:root can enter it, which is why psql -f run as the postgres user fails with Permission denied. Errors about roles or databases that already exist are expected when a --clean dump is loaded on the server it came from.Restoring to /tmp/restore rather than / is what makes this safe to practice: nothing live is overwritten.Restore a single path
- Ubuntu / Debian
- AlmaLinux / Rocky / Oracle Linux
etc/ssh is all you get:/etc/ssh is used here because it is guaranteed to be in the snapshot: you reached this server over SSH.Browsing snapshots as a filesystem (optional)
Browsing snapshots as a filesystem (optional)
restic mount exposes the repository as a read-only filesystem. It is not part of the backup lifecycle, and the two restores above already cover recovery. It earns its place when you need one file and do not know which snapshot holds it, because you can look instead of guessing.Mounting needs the fusermount binary from FUSE. Whether it is already there depends on the image, not on restic: the Ubuntu and AlmaLinux images ship it, the Oracle Linux one does not. Check before mounting:restic mount would fail with fusermount: executable file not found in $PATH. On the RHEL family, install it with:fuse, not fuse3: only the first one provides fusermount.Create the mountpoint first, because restic does not:Now serving the repository at /mnt/restic and then blocks, holding the terminal. Leave it running and open a second SSH session to browse. That session starts as your login user, so switch to root again with sudo -i before continuing:latest shortcut to the newest one. Inside, the tree looks like the server did, and ordinary commands work on it, so you can copy a single file straight out:Ctrl+C unmounts.Step 9: Schedule it with systemd
Everything so far has been manual, which means the backup is only as current as the last time someone remembered to run it. systemd is the service manager already running on your server, and scheduling with it takes two files: a service unit that defines what to run, and a timer unit that defines when to run it. The timer starts the service on a schedule the way cron would, with two differences that matter here: every run is recorded in the system journal, so you can read what happened afterwards, and a run missed while the server was off is caught up on the next boot. You will write both files, tell systemd to pick them up, then trigger one run manually to confirm it works before leaving it to the schedule.Create the service unit
PATH of its own, so every binary has to be named by absolute path, and a path that does not exist fails the whole unit. Two things in the unit below depend on your server:- The
resticpath. A packaged install lands in/usr/bin/restic, the official binary in/usr/local/bin/restic. - The backup paths. The same list you settled on in Step 6, including
/var/backupsif you dump a database. The environment file carries your credentials, not your path choices, so they have to be spelled out again here.
/usr/bin/restic for a packaged install, or /usr/local/bin/restic for the official binary. Then create the unit, substituting your own backup paths. The heredoc fills in the restic path from the variable:EnvironmentFileloads the credentials from Step 4, which is how the timer authenticates without a shell.Environment=HOME=/rootis there because systemd starts the unit without aHOME. Without it, restic logsunable to open cacheand re-downloads metadata it could have reused.NiceandIOSchedulingClasskeep the backup from competing with your workload for CPU and disk.
ExecStart so the scheduled run refreshes the dump instead of storing the one you made manually, getting staler every night. It is the same command as in Step 6, with the absolute paths that which runuser and which pg_dumpall print:Create the timer
Persistent=true runs a missed backup after a reboot. RandomizedDelaySec spreads the load when you run this across a fleet.Load and arm the timer
Created symlink line, which is systemd confirming the timer is enabled at boot and started now.Run it once manually
status=0/SUCCESS on the Main PID: line. Active: inactive (dead) above it is expected and not a failure: a Type=oneshot unit is meant to run and exit, so it sits inactive between runs.systemctl start failed instead, these same two commands say why: see Troubleshooting.Confirm the schedule
NEXT and LEFT showing when the next run fires. LAST and PASSED stay - until the timer itself triggers the service: starting it manually does not count as a timer activation. An empty table means the timer is not enabled, so revisit the enable command above.Step 10: Apply a retention policy
The timer now adds a snapshot every night and never removes one, so the repository grows forever. A retention policy decides which snapshots are worth keeping: recent ones in full detail, older ones thinned out to one per week, then one per month. Two separate things do that work, and the distinction explains the commands below:forgetmarks snapshots as no longer wanted. It is fast, because it only drops the snapshot’s entry, not the data behind it.prunedeletes the data that no remaining snapshot references. It is what actually frees space in the bucket, and it is slow, because it rewrites pack files.
--prune runs the second phase right after the first.
Add --dry-run first to see which snapshots the policy would remove, without removing anything.
The real run prints the policy it applied, then a keep table of every snapshot that survived it. The Reasons column there is worth reading: it names the rule that saved each one, so a snapshot kept as daily snapshot, weekly snapshot, and monthly snapshot at once is filling all three roles. Snapshots on their way out appear in a second table below, and --prune then reports what it repacked and deleted.
restic snapshots should list exactly the snapshots the policy said to keep, and nothing else. restic check should still end with no errors were found, which is what tells you prune removed data without leaving the repository inconsistent.
Run this on a schedule too, or the repository grows anyway. Give it a timer of its own rather than adding it to the nightly backup unit, built the same way as Step 9, for two reasons: prune is the heavy half and does not need to run daily, and a prune that fails would otherwise mark your backup as failed even though the backup itself succeeded.
Wrap-up
You now have:- A versioned bucket in a different site from the server, reached with an access key scoped to that bucket alone
- An encrypted restic repository in it, with the password stored off the server
- A nightly systemd timer that backs up your chosen paths and logs every run to the journal
- A retention policy that keeps the repository from growing forever, and a restore you have already tested once
Troubleshooting
systemctl start failed because the control process exited with error code
systemctl start failed because the control process exited with error code
systemctl status restic-backup.service and
journalctl -u restic-backup.service -n 50. The usual causes:status=203/EXEC— systemd could not execute the binary named inExecStartorExecStartPre. The path is wrong or the file is not there. Compare it againstwhich restic.- restic ran and exited non-zero — the journal carries restic’s own message. A path in the backup list that does not exist on the server is the common one.
Please specify repository location (-r or --repository-file)
Please specify repository location (-r or --repository-file)
RESTIC_REPOSITORY in the environment. The variables
from Step 4 live in the shell that loaded
them: a new SSH session, or a fresh sudo -i, starts without them. Load
the file again:xattr.LRemove ... security.selinux: permission denied
xattr.LRemove ... security.selinux: permission denied
Fatal: There were N errors, mean the host runs SELinux. The data was restored; what failed is
restic trying to change the SELinux label the kernel put on each new file.
Run the restore again with --exclude-xattr security.selinux, as in
Step 8.FATAL: role root does not exist
FATAL: role root does not exist
pg_dumpall was run as root. PostgreSQL authenticates local connections
by system user, and there is no database role called root. Run the dump
as the postgres user with runuser, as in
Step 6, and make sure the output directory
belongs to that user.Is there a repository at the following location?
Is there a repository at the following location?
RESTIC_REPOSITORY. Check that
restic init has been run, that the last path segment of the URL is the
Bucket name (backend) that lsh s3 stat "s3://$BUCKET" prints, not
the display name, and that the endpoint is the Endpoint from the same
output. Buckets of different storage classes are reached at different
endpoints, so one copied from another bucket will not resolve.403 Forbidden or SignatureDoesNotMatch
403 Forbidden or SignatureDoesNotMatch
lsh s3 access-keys get restic-backup on your workstation and
confirm the Buckets line lists this bucket with rw.Errors mentioning region
Errors mentioning region
AWS_DEFAULT_REGION in /root/.restic-env to the Signing region that
lsh s3 stat "s3://$BUCKET" prints, or pass it per command with
-o s3.region=<region>.repository is already locked
repository is already locked
Operation counts look high
Operation counts look high
lsh s3 metrics "s3://$BUCKET" on your workstation: it prints the
storage consumed in the current billing period and the estimated cost. The
storage pricing page lists what
each class of operation costs.Next steps
- Read the restic documentation for append-only setups,
rest-server, and backend tuning