Automated Node Health Failover

Self‑healing node failover with minute‑level checks to guarantee API and endpoint availability.

The Problem

While downtime is rare with the providers I tend to use, it's always possible to have an unexpected outage or have a service go down when you least expect it. I try to mitigate this as much as possible, but it's an always-present worry.

You never know when unexpected maintenance could take place; or a server could go down due to a failure or network outage. It doesn't happen often, but when it does, problems can surface and data can be lost.

Especially with the APIs that are running on our infrastructure now, it's important to make sure they're available as much as possible - and that's where node failover comes into play.

The Goal

When designing this health pipeline, I wanted to ensure we avoided as much server downtime or network issues as possible - keeping endpoints accessible and always available.

Easier said than done, but at the same time, a little bit of automation never hurt. Checking these endpoints in real-time, each minute would ensure our APIs, endpoints, as well as static sites stay functioning even when one server goes down.

The entire failover engine runs on a tiny <$7/year VPS, so the design had to be lightweight, predictable, and resilient without relying on expensive redundancy. The goal wasn't to throw hardware at the problem, it was to engineer reliability through automation.

Architecture Diagram

┌────────────┐
│   Users    │
└─────┬──────┘
      │
      ▼
┌─────────────────────┐
│   DNS (A/AAAA)      │◄────────────┐
│   Records           │             │
└──────────┬──────────┘             │
           │                        │
           ▼                        │
┌─────────────────────┐         ┌─────────────────────┐
│     PRIMARY         │         │       MIRROR        │
│     Node            │◄────────┤       Node          │
└──────────┬──────────┘         └──────────┬──────────┘
           │                               │
           ▼                               ▼
       ┌──────────────────────────────────────────┐
       │             Health Check Engine          │
       │     • HTTP(s) probes                     │
       │     • Timeout / threshold logic          │
       │     • Dual-stack (A + AAAA) validation   │
       └──────────────────────┬───────────────────┘
                              ▼
       ┌──────────────────────────────────────────┐
       │           Failover Orchestrator          │
       │     • State evaluation                   │
       │     • Lock file (anti-flap)              │
       │     • Primary ↔ Mirror switching         │
       │     • DNS record updates                 │
       │     • Guaranteed AAAA recreation         │
       └──────────────────────┬───────────────────┘
                              ▼
       ┌──────────────────────────────────────────┐
       │           Observability Layer            │
       │     • Structured logs (JSON)             │
       │     • Alerts (Discord webhook)           │
       └──────────────────────────────────────────┘

Health Validation Pipeline

To ensure our services were running and accessible, a few different functions were introduced in python for health validation:

HTTP(s) probe with a health endpoint on multiple services - checking for 200 codes to verify service was healthy.
intel function written for semantic validation, checking for consistency and ensuring the data is available on the Intel API.
threats function also written for validation using the Threats API, to be sure data is not only fresh but certain fields are included in the response.
Timeouts were included CF_TIMEOUT for both the Cloudflare API and the http function, HTTP_TIMEOUT.
Dual stack awareness was also a big part of it - some endpoints include both A/AAAA records (IPv4/IPv6), while some only operate using A records (IPv4).

Failover Orchestration Logic

The failover engine implemented evaluates node health every minute and makes the decision whether traffic should remain on the primary node, or shift to the mirror. The orchestration layer blends simple rules with safety checks to avoid flapping and ensure predictable behavior.

Primary to mirror switching only happens when the primary fails validation and the mirror is confirmed healthy.
Cooldown enforcement prevents rapid back-and-forth switching by requiring a minimum interval between changes.
State tracking stores the last switch on disk, giving the system memory across runs.
Lock file override allows maintenance windows where failover is intentionally paused.
IPv4 and IPv6 alignment ensures both A and AAAA records always point to the same active node.
Retry logic handles transient DNS update failures gracefully.

primary_ok = check_health(service, "primary")
mirror_ok = check_health(service, "mirror")

if primary_ok:
    target = "primary"
elif mirror_ok:
    target = "mirror"
else:
    log("Both endpoints unhealthy — skipping", "ERROR")
    return

The full orchestration logic - including cooldown handling, state persistence, IPv6 lifecycle management, and DNS retries - is open source on GitHub.

DNS Automation Layer

The DNS layer is where the failover turns the health decisions into real traffic movement. Every minute (when running in systemd or cron), the failover system evaluates the active node and ensures Cloudflare's A and AAAA records reflect the correct target. The logic is intentionally conservative, designed to avoid flapping and respect DNS propagation retries.

Cloudflare record queries: Each cycle begins by fetching the current A and AAAA records for the service. This establishes the ground truth before any changes are made.
Atomic A and AAAA updates: When a switch is required, the system updates the IPv4 and IPv6 records in a coordinated way, ensuring both address families always point to the same active node.
Handling missing or stale records: If a record is missing, the system recreates it. If a record is stale or mismatched, it is updated in place. This keeps the DNS state consistent even after manual edits or unexpected drift.
IPv6 removal during total outage: When both nodes are unhealthy, the system removes the AAAA record entirely. This prevents clients from being routed to a dead IPv6 endpoint while IPv4 fallback remains available.
TTL-aware behavior: All updates respect a short, predictable TTL, allowing changes to propagate quickly without overwhelming resolvers or causing unnecessary churn.

record = get_dns_record(service["dns_name"], zone_id, headers)
aaaa = get_aaaa_record(service["dns_name"], zone_id, headers)

ipv4_change = (record["content"] != desired_ip)
ipv6_change = (aaaa and aaaa["content"] != desired_ipv6)

Full DNS automation logic, including retries, IPv6 management, and stale record recovery - is available in the public repository.

Observability & Alerting

The failover engine emits structured, timestamped logs on every health check cycle, making it easy to trace decisions and understand system behavior during an outage. When a switch occurs, a Discord webhook sends a real-time alert summarizing the event - including which node failed, which node was promoted, and the exact IPs involved.

Structured logs provide a clear timeline of health checks, decisions, and DNS updates.
Discord webhook alerts deliver immediate visibility into failover events
Reported context includes primary/mirror health, chosen target, IPv4/IPv6, and switch reason
Incident clarity improves dramatically because every decision is logged with intent, not just outcome

An example from a failover event:

[2025-01-10 14:02:11 UTC] [INFO] Checking service: example-service
[2025-01-10 14:02:11 UTC] [INFO] Primary healthy: False
[2025-01-10 14:02:11 UTC] [INFO] Mirror healthy: True
[2025-01-10 14:02:11 UTC] [INFO] example-service: updating IPv4 → MIRROR (5.6.7.8)
[2025-01-10 14:02:12 UTC] [INFO] example-service: updating IPv6 → MIRROR (2001:db8::2)
[2025-01-10 14:02:12 UTC] [INFO] example-service: DNS updated → MIRROR

Real-World Scenario

To validate the system before running it in production, I ran a simulated failure on the primary health endpoint used for one of our sites. A quick NGINX configuration update to throw a 503, rather than the data expected; allowed us to ensure it was working properly. The endpoint was taken offline, and within the next minute the failover service detected the failure, confirmed the mirror was healthy, and shifted the traffic.

DNS records were updated, and the system continued responding normally - just as it would during a real-world incident.

[2026-01-10 00:55:01 UTC] [INFO] Checking service: threats
[2026-01-10 00:55:02 UTC] [INFO] Primary healthy: False
[2026-01-10 00:55:02 UTC] [INFO] Mirror healthy: True
[2026-01-10 00:55:06 UTC] [INFO] threats: updating IPv4 → MIRROR (5.6.7.8)
[2026-01-10 00:55:06 UTC] [INFO] threats: updating IPv6 → MIRROR (2001:db8::2)

Results & Impact

The failover engine now provides a predictable, self-healing layer across my infrastructure (currently running on five endpoints). Minute-level checks catch issues early, DNS updates propagate quickly (1 min TTL), and services stay online even when a node is taken down for testing. The system has eliminated manual intervention during simulated failures and dramatically increased confidence in the availability of the APIs and dashboards that power my platform.

Lessons Learned

Building this failover engine reinforced a few core principles: simple health checks aren't enough, statefulness prevents chaos, and DNS automation needs guardrails for safety just as much as logic. Designing for failure, rather than reacting to it - consistently leads to better decisions, cleaner logs, and a system that can behave predictably under simulated stress.

What's Next

The next step is expanding this system into multi-region failover, allowing services to shift not just between nodes, but between geographic regions. With the orchestration logic, health checks, and DNS automation already in place, extending the platform to a second region would provide true regional redundancy and push availability even further.