✅ Beyond 'Set and Forget': Continuous Backup Validation

December 17, 2025

Verifying Borg Backups Daily - repository checks, archive validation, and live health dashboards across multi-cloud nodes.

The Problem

borgmatic has become one of those tools I almost take for granted. It quietly runs every night, backing up my entire fleet with incredible efficiency - routinely hitting 99%+ deduplication and keeping costs low at rsync.net.

I’ve never actually needed to restore anything major (thankfully), but the backups are there, my safety net.

That’s the catch - they’re only a safety net if they actually work.

What if an archive is corrupted? What if incremental changes introduce silent errors? What if the repository itself degrades over time? In a real disaster, discovering a backup is unrecoverable is the last thing anyone wants.

Most backup systems are "set and forget" - great until the moment you can’t forget anymore, and you actually need them.

The Solution

Borg ships with powerful built-in verification commands that most people (including me, until recently) rarely run:

  • borg check --repository-only validates the overall repository structure
  • borg check --archives-only confirms the integrity of individual archives (I check the last three)
  • borg info --json gives detailed stats on original, compressed, and deduplicated sizes

These checks are fast, non-destructive, and perfect for automation.

I built a lightweight Python pipeline that runs them daily across my nodes. The script parses the output, logs structured JSON, and feeds a live dashboard on backups.beane.me.

The result? Not just backups - proven recoverable backups, with real-time visibility into health and efficiency (including that near-magical 99.8% deduplication).

It took a focused sprint, but the peace of mind is priceless. I now know my safety net is intact, every single day.

Backup Verification Dashboard

Architecture Overview

                             ┌──────────────────────┐
                             │   rsync.net Repo     │
                             │ (encrypted Borg repo)│
                             └──────────^───────────┘
                                        │
                  ┌─────────────────────┼─────────────────────┐
                  │                                           │
   ┌──────────────┴──────────────┐               ┌────────────┴────────────────┐
   │       Triton Node           │               │       Zephyrus Node         │
   │                             │               │                             │
   │  Borgmatic (nightly backups)│               │  Borgmatic (nightly backups)│
   │                             │               │                             │
   │  verify_borgmatic.py        │               │  verify_borgmatic.py        │
   │   (daily: source check,     │               │   (daily: source check,     │
   │   repo integrity, archive   │               │   repo integrity, archive   │
   │   verification, stats)      │               │   verification, stats)      │
   └──────────────^──────────────┘               └──────────────^──────────────┘
                  │                                           │
                  │  verification_log.json (sanitized)        │
                  │                                           │
                  └──────────────> HTTP fetch <───────────────┘
                                        │
                                        ▼
                             ┌──────────────────────┐
                             │       Triton         │
                             │  html_update.py      │
                             │ (every hour: fetch   │
                             │ remote logs, render) │
                             └──────────^───────────┘
                                        │
                                        ▼
                             ┌─────────────────────┐
                             │   backups.beane.me  │
                             │   (live dashboard)  │
                             └─────────────────────┘

Implementation

For the core of the verification, we utilized the built-in borg check commands - first for a quick check, then to verify the integrity of the last 3:

# Repository structure check (fast)
borg check --repository-only $REPO

# Archive integrity for last 3
borg check --archives-only $REPO::$(borg list --last 3 --short $REPO)

Since we're aiming for full automation, the archive verification loop is written in Python.

def verify_borg_archives(repo: str, last_n: int = 3) -> dict:
    # List recent archives
    success, stdout, _ = run_command(["borg", "list", "--short", "--last", str(last_n), repo])
    archives = [line.strip() for line in stdout.strip().split('\n') if line.strip()]

    for archive_name in archives:
        success, _, _ = run_command(["borg", "check", "--archives-only", f"{repo}::{archive_name}"])
        # Log result

We can then extract the stats, which is how we got our dedupe number - a staggering 99.8% on one of the nodes - meaning only 0.2% of those backups are actually stored on disk!

info = json.loads(run_command(["borg", "info", "--json", f"{repo}::{latest_archive}"])[1])
stats = info['archives'][0]['stats']
compression_ratio = (1 - stats['compressed_size'] / stats['original_size']) * 100
dedup_ratio = (1 - stats['deduplicated_size'] / stats['compressed_size']) * 100

When we fetch our dashboard, we're able to use multi-node aggregation (since some JSON is stored locally, running on Triton, and other JSON is pulled remotely - currently on Zephyrus.

def fetch_remote_verification(url):
    log = requests.get(url).json()
    latest = {}
    for entry in reversed(log):
        if entry.get("check_type") not in latest:
            latest[entry["check_type"]] = entry
    return latest

$REPO is used rather than actual repo at rsync.net - just for sanitization.

Challenges

Even a "smooth" project like this had a few hurdles worth calling out - mostly around coordination, timing, and keeping things secure/scalable.

  1. Timing & Coordination Across Nodes

    2. The biggest practical challenge was ensuring the dashboard always showed fresh data, without race conditions.

    • Verification runs daily on each node at 30 minutes past midnight)
    • The original HTML rendering script ran every 10 minutes, on the hour.
    • Solution: Shift rendering to every hour at :40 (10 minutes after verification completes), giving the remote nodes time to finish and expose updated JSON.
  2. Secure Multi-Node Aggregation Without Credential Exposure

    3. I wanted centralized visibility but did not want to share Borg credentials, or SSH keys across nodes.

    • Early idea: Rsync logs privately - blocked by no root SSH access from remote nodes.
    • Final approach: Sanitize sensitive fields (repository path: "rsync.net (hidden credentials)") and expose logs publicly via HTTPS.
    • Remote fetch in html_update.py processes them identically to local logs. Trade-off accepted: minimal exposure, maximum simplicity, and future scalability.
  3. Parsing Borgmatic Config Safely

    4. Extracting repo/path/passphrase automatically required careful YAML parsing - edge cases like missing keys or legacy formats could crash the script.

    • Added robust fallbacks and warnings.
    • Ensured passphrase handling never logs or exposes secrets.
  4. Keeping the Pipeline Lightweight

    5. Borg checks are fast, but running full archive scans too frequently could burden rsync.net.

    • Stuck to --repository-only and --archives-only on the last 3 - quick metadata checks, no heavy data transfer.
    • Daily verification strikes the balance between early detection and courtesy to the provider.

None of these were showstoppers; however, solving them cleanly made our system more robust and easier to extend to future nodes.

Results

Since launch (mid-December 2025), the verification pipeline has been running flawlessly across Triton and Zephyrus.

Key Metrics

  • Verification Frequency: Daily (30 minutes past midnight - 00:30) - running now, with full automation and no failures.
  • Check Coverage:
    • Source directories: 12/12 (Triton), 3/3 Zephyrus consistently accessible
    • Repository integrity: Verified every run
    • Archives: Last 3 validated every run
  • Efficiency Stats (from latest archive on Triton):
    • 90,551 files backed up
    • Original size: 2.37 GB
    • Compressed: 1.01 GB (58.2% reduction)
    • Deduplicated: 1.95 MB (99.8% deduplication)

Dashboard Impact

  • Live at backups.beane.me
  • Multi-node health cards update hourly
  • Clear visual status (checkmark for verified, timestamps, compression/dedup ratios)
  • No stale data - rendering shifted to :40 to sync with verification completion

Peace of Mind Bonus

  • Early detection window: issues would surface same-day, not weeks later.
  • Zero manual intervention required since deployment.

The pipeline didn't add checks - it turned backups from a background task into transparent, proven reliability.

Your data, your proof. Verified, recoverable, and always ready.
Back to blog index