Refactor Docker backups into modular pipeline
This commit is contained in:
Codex authored and 
howard committed
1 parent
3ba4b7f738
commit
261e88f5dc
12 files changed
+480
-848
No files matched your search
@@ -1,83 +1,154 @@
|
||||
# Docker Backup Scripts
|
||||
# Zorruno Docker Backup Scripts
|
||||
|
||||
Simple shell scripts for backing up Docker-related data and pruning old backups.
|
||||
Modular backup scripts for Docker Compose servers such as Panda and Beaver.
|
||||
Each server has its own untracked `.env` and `containers.txt`, while all script
|
||||
logic remains common and can be updated through Git.
|
||||
|
||||
## Files
|
||||
## Pipeline
|
||||
|
||||
- `docker-backup.sh` - create backups
|
||||
- `dockerbackup_prune.sh` - prune old backups
|
||||
- `containers.txt` - list of containers or backup targets
|
||||
- `.env.example` - example environment file
|
||||
`backup.sh` runs these stages in order:
|
||||
|
||||
## Reqirements
|
||||
1. `stages/10-containers.sh` — stop, archive, pull, start and verify projects.
|
||||
2. `methods/20-rclone.sh` — copy archives to an rclone destination when enabled.
|
||||
3. `methods/30-smb.sh` — copy archives to a mounted SMB share when enabled.
|
||||
4. `stages/90-prune.sh` — retain the newest configured number locally.
|
||||
|
||||
docker-compose
|
||||
Every stage can also be run directly. The old `docker-backup.sh` name invokes
|
||||
the container archive stage, while `dockerbackup_prune.sh` invokes the local
|
||||
prune stage. Use the new `backup.sh` for the complete pipeline.
|
||||
|
||||
## BACKUPS: Example use docker-backup.sh
|
||||
## Requirements
|
||||
|
||||
Configure the backup script, docker-backup.sh
|
||||
Set up your list of containers in container.txt . The script assumes each container is in a subdirectory of one main docker directory
|
||||
- Bash 4 or later
|
||||
- Docker with `docker compose`, or legacy `docker-compose`
|
||||
- GNU `tar` with zstd support
|
||||
- `flock`
|
||||
- `rclone` only when the rclone method is enabled
|
||||
- `rsync` and `mountpoint` only when the SMB method is enabled
|
||||
|
||||
Edit the location of where you want docker backups stored in docker-backup.sh (default is "/dockerbackups") or use
|
||||
The account running the scripts must be allowed to control Docker and read all
|
||||
files under `/dockervolumes`.
|
||||
|
||||
./docker-backup.sh -s /home/me/mycontainers
|
||||
## Installation on Beaver
|
||||
|
||||
**Actions:**
|
||||
- It will recurse though the directories if they are listed in containers.txt
|
||||
- It will run docker-compose to take down the container, pull a new version if there is one, then bring the container back up.
|
||||
- it will create a timestamped archive of each container in the backup directory, under a directory with that container name.
|
||||
|
||||
```
|
||||
Usage: ./docker-backup.sh [-f containers_list_file] [-s source_base_dir] [-t] [-v] [-h]
|
||||
|
||||
Flags:
|
||||
-t Test (dry run; no backups performed)
|
||||
-v Verbose (debug logging to stderr)
|
||||
-f FILE File containing list of container names
|
||||
Default: '$containersfile'
|
||||
-s DIR Base directory containing container/project folders
|
||||
Default: '$SOURCE_BASE_DIR'
|
||||
-h Show this help
|
||||
```bash
|
||||
sudo git clone https://home.fox.co.nz/gitea/zorruno/zorruno-backupdockerscript.git /backupdockerscripts
|
||||
cd /backupdockerscripts
|
||||
sudo cp .env.example .env
|
||||
sudo cp containers.txt containers.local.txt
|
||||
sudo chmod 600 .env
|
||||
sudo chmod +x backup.sh docker-backup.sh dockerbackup_prune.sh stages/*.sh methods/*.sh
|
||||
```
|
||||
|
||||
## PRUNE and REMOTE BACKUP dockerbackup_prune.sh
|
||||
Edit `.env`. Beaver's Docker source is already represented by:
|
||||
|
||||
Set up the env file using the example.
|
||||
```cp .env.example .env```
|
||||
|
||||
You don't need to worry about the rclone info (backups to google drive) it you won't be using that.
|
||||
It will just backup up the file locally. Feel free to adjust for other remote backup systems.
|
||||
|
||||
If you run .dockerbackup_prune.sh it will prune the number of backup archives that are already there (based on your prune numbers), make a .zst archive of all container backups
|
||||
then finally uploades to google drive and deletes the large archive (if that's what you set).
|
||||
Then it will prune the number of archive files for each
|
||||
|
||||
**local archive only (no pruning)**
|
||||
```./dockerbackup_prune.sh```
|
||||
|
||||
**local archive + real prune, keep new archive locally. Default prune is 5 archives)**
|
||||
```./dockerbackup_prune.sh --apply```
|
||||
|
||||
**local archive + upload + prune dry-run (no actual prune), keep new archive locally (default location is /tmp)**
|
||||
```./dockerbackup_prune.sh --rclone```
|
||||
|
||||
**full remote archive, google drive upload + real prune (default 5) + delete new local archive**
|
||||
```./dockerbackup_prune.sh --rclone --apply```
|
||||
|
||||
**Full remote workflow with actual prune leaving 3 archives of each container and final full archive delete.**
|
||||
**Specify location of docker archives, final archive dir and remote google drive folder.**
|
||||
```./dockerbackup_prune.sh --root /mydockerarchives --/mybigbackupdolder --keep 3 --rclone /dockergooglebackup --apply
|
||||
|
||||
Usage: $0 [--apply] [--quiet] [--dry-run] [--keep N] [--root PATH] [--source PATH] [--outdir PATH] [--remote PATH]
|
||||
|
||||
Flags:
|
||||
--rclone Enable rclone upload flow
|
||||
--apply Actually delete pruned files; with --rclone also delete the new local archive
|
||||
--quiet Minimal console output (no live progress)
|
||||
--dry-run Simulate everything (no backup, no upload, no deletes)
|
||||
--keep N Keep newest N .zst files per subfolder (default ${KEEP_COUNT})
|
||||
--root PATH Root folder to prune (default ${PRUNE_ROOT})
|
||||
--source PATH Folder to archive (default ${SOURCE_DIR})
|
||||
--outdir PATH Where to write the .zst (default ${OUT_DIR})
|
||||
--remote PATH rclone destination (default ${RCLONE_REMOTE})
|
||||
```bash
|
||||
SOURCE_BASE_DIR="/dockervolumes"
|
||||
BACKUP_HOSTNAME="beaver"
|
||||
```
|
||||
|
||||
Choose and create a local staging path for `BACKUP_ROOT`. `/dockerbackups` is
|
||||
the default, but it should be on a filesystem with enough free space to hold
|
||||
at least `KEEP_COUNT` copies of every listed project.
|
||||
|
||||
Edit Beaver's untracked `containers.local.txt`. Each active line can be either a running
|
||||
container name or a directory name beneath `/dockervolumes`. Blank lines and
|
||||
text after `#` are ignored. If several entries resolve to the same Compose
|
||||
project directory, that project is backed up only once. Keeping this host-local
|
||||
file out of Git prevents a pull on Panda or Beaver from replacing its list.
|
||||
|
||||
## Configuration
|
||||
|
||||
Copy `.env.example` to `.env`; `.env` is ignored by Git. Important values:
|
||||
|
||||
| Variable | Purpose |
|
||||
|---|---|
|
||||
| `SOURCE_BASE_DIR` | Parent of Compose project directories |
|
||||
| `BACKUP_ROOT` | Local archive staging and retention tree |
|
||||
| `CONTAINERS_FILE` | Absolute path, or path relative to this repository |
|
||||
| `BACKUP_HOSTNAME` | Host folder/name embedded in archive filenames |
|
||||
| `BACKUP_METHODS` | Ordered method names, default `rclone smb` |
|
||||
| `KEEP_COUNT` | Local archives retained per Compose project |
|
||||
| `PULL_IMAGES` | Pull images before restarting (`1` or `0`) |
|
||||
|
||||
### Rclone
|
||||
|
||||
```bash
|
||||
RCLONE_ENABLED=1
|
||||
RCLONE_REMOTE="gdrive:dockerbackups"
|
||||
```
|
||||
|
||||
Archives are copied beneath `<remote>/<BACKUP_HOSTNAME>/`. The method uses
|
||||
`rclone copy`, so it does not delete older destination files.
|
||||
|
||||
### SMB
|
||||
|
||||
Mount the SMB share through `/etc/fstab` or a systemd mount unit; do not put SMB
|
||||
credentials in this repository. Then configure:
|
||||
|
||||
```bash
|
||||
SMB_ENABLED=1
|
||||
SMB_MOUNT="/mnt/dockerbackups"
|
||||
SMB_HOST_SUBDIR=1
|
||||
```
|
||||
|
||||
The script refuses to copy unless `SMB_MOUNT` is an actual mount point. This
|
||||
prevents a failed network mount from silently filling the server's root disk.
|
||||
|
||||
## Running
|
||||
|
||||
Run the complete pipeline:
|
||||
|
||||
```bash
|
||||
sudo /backupdockerscripts/backup.sh
|
||||
```
|
||||
|
||||
Preview all stages without stopping containers, copying, or deleting:
|
||||
|
||||
```bash
|
||||
sudo /backupdockerscripts/backup.sh --dry-run
|
||||
```
|
||||
|
||||
Run individual parts:
|
||||
|
||||
```bash
|
||||
sudo stages/10-containers.sh --dry-run
|
||||
sudo methods/20-rclone.sh --dry-run --force
|
||||
sudo methods/30-smb.sh --dry-run --force
|
||||
sudo stages/90-prune.sh --dry-run
|
||||
```
|
||||
|
||||
`--force` runs a transfer method even when its `*_ENABLED` value is false. It
|
||||
still requires the method's destination to be configured and available.
|
||||
|
||||
## Adding another backup method
|
||||
|
||||
Add an executable script named `methods/NN-name.sh`, for example
|
||||
`methods/40-restic.sh`. It should load `lib/common.sh`, accept `--dry-run`, and
|
||||
return a non-zero status on failure. Add its short name to `.env`:
|
||||
|
||||
```bash
|
||||
BACKUP_METHODS="rclone smb restic"
|
||||
```
|
||||
|
||||
The main pipeline discovers the numbered file from the short name and runs it
|
||||
after the earlier methods. A failed method stops the pipeline, so pruning will
|
||||
not run after a failed transfer.
|
||||
|
||||
## Archive and restore format
|
||||
|
||||
Archives are stored as:
|
||||
|
||||
```text
|
||||
<BACKUP_ROOT>/<project>/<host>.<project>.<date_time>.tar.zst
|
||||
```
|
||||
|
||||
Each archive contains the project's top-level directory rather than absolute
|
||||
paths. Example restore into an empty temporary directory:
|
||||
|
||||
```bash
|
||||
mkdir -p /tmp/docker-restore
|
||||
tar --zstd -xf backup.tar.zst -C /tmp/docker-restore
|
||||
```
|
||||
|
||||
Inspect and verify restored content before replacing a live project.
|
||||
Reference in new issue
Block a user