🛠️ Gitea on Kubernetes with Automated Backups

Self-hosting Git with Gitea on Kubernetes - resulting in a resilient, private instance powered by PostgreSQL and protected with daily automated Borgmatic backups.

The Problem

git is an incredible resource, but when you get passionate about self-hosting like me; you begin to realize it can be infinitely more powerful for different projects.

Relying on a service like GitHub means in essence your repos live on all their services. A self-hosted alternative, however, allows you to keep projects fully private - as well as potentially being more cost-effective if you have the infrastructure to deploy it.

The Solution

Gitea is the perfect solution for a self-hosted Git - free and lightweight, the only real cost comes from the infrastructure that powers it.

Not only that, but you can choose your own database (PostgreSQL in this case), caching layer (Valkey), as well as a fitting backup strategy (Borgmatic, daily backups using pgdump - which live at rsync.net).

Architecture Overview

                   ┌────────────────────────┐
                   │          Gitea         │
                   │ (https://git.beane.me) │
                   └───────────^────────────┘
                               │
             ┌─────────────────┼─────────────────┐
             │                                   │
   ┌─────────┴─────────┐               ┌─────────┴──────────┐
   │   PostgreSQL DB   │               │   Valkey Cluster   │
   │   (stateful data) │               │   (caching layer)  │
   └───────────────────┘               └────────────────────┘
             │                                   
   ┌─────────v─────────┐               ┌─────────────────────┐
   │ Borgmatic Jobs    │               │   rsync.net Backup  │
   │ (pgdump nightly)  │──────────────>│   (offsite storage) │
   └───────────────────┘               └─────────────────────┘

Implementation

Namespace isolation is used within Kubernetes, as well as persistent volumes covering Valkey clusters, PostgreSQL data, and Gitea storage.

• gitea-shared-storage (10GB)
• data-gitea-postgresql (10GB)
• valkey-data-gitea-cluster-* (8 GB each) - three Valkey cluster volumes

Kubernetes was chosen for the orchestration behind it all; simply because of the scability and resiliency it provides as the instance grows.

For Gitea deployment, we used Helm charts because it is perfect for applications like this - all manifests are bundled into a reusable chart, so a complex application like this can be deployed with a single command:

helm install gitea gitea-charts/gitea --namespace gitea --create-namespace

PostgreSQL was utilized instead of SQLite because it provides production-grade persistence. We also went with a Valkey cluster instead of Redis, as a caching layer for our instance.

For backups, we went with borgmatic as usual - scheduling it with a Kubernetes CronJob. A nightly pgdump runs, resulting in a compressed archive (dump) that we can back up straight to rsync.net for recovery if ever needed.

We also integrated CrowdSec - using the file in /var/log/containers relevant to our NGINX ingress. Fail2Ban is in effect as well, as with every other VM I run, just to protect against anyone trying to gain SSH access or similar. Strict network policies and ingress rules are in use for safety and protection - to keep the containers truly "contained".

Challenges

Upon first deployment, it went pretty smoothly except for a little bit of an issue with the cron running the pgdump to backup with Borgmatic.

Borgmatic was running, but the issue was it was running before (or as) the pgdump finished. Luckily, it was a pretty simple fix.

Here's the relevant portion of the CronJob that solved the race condition issue by dumping the PostgreSQL in an initContainer before Borgmatic runs:

initContainers:
  - name: pg-dump
    image: postgres:17
    command:
      - /bin/sh
      - -c
      - |
        mkdir -p /var/lib/borgmatic/pgdump
        pg_dump -h gitea-postgresql -U gitea -F c \
          -f /var/lib/borgmatic/pgdump/gitea.dump gitea
    env:
      - name: PGPASSWORD
        valueFrom:
          secretKeyRef:
            name: gitea-postgresql
            key: password
    volumeMounts:
      - name: borgmatic-data
        mountPath: /var/lib/borgmatic

containers:
  - name: borgmatic
    image: ghcr.io/borgmatic-collective/borgmatic:latest
    command: ["borgmatic", "--verbosity", "1"]
    env:
      - name: BORG_RSH
        value: ssh -i /etc/borgmatic/keys/borgmatic_rsync -o StrictHostKeyChecking=no

Results

• Resilient Git in action at git.beane.me
• Automated backups verified daily
• Fully hardened against instrusion and brute-force attempts

Now we have our own private Gitea deployment, running on a blazing fast 8-core 9950X VM - 1ms ping to our central server.