Drift‑Detection And Fleet‑Wide Enforcement

The creation of a distributed, self-healing enforcement system to eliminate drift between CrowdSec and nftables across my fleet.

The Problem

Running multiple nodes against a central CrowdSec LAPI, I started to notice something unsettling: drift wasn't an edge case, it was practically a built-in behavior.

Each node's nftables state slowly diverged from the decisions coming out of CrowdSec. And with no unified visibility across the fleet, I had no reliable way to know which bans were actually being enforced. That's not a great feeling when you're focused on security and consistency.

I'm a huge believer in CrowdSec, but the gap between "decisions issued" and "decisions enforced by the kernel" became too wide to ignore. So I built a system that closes that gap: a backend agent that detects and heals drift autonomously, and a dashboard that visualizes enforcement state across the entire fleet.

Architecture Overview (Mermaid)

Here's how the system works end-to-end:

High-level architecture of the backend and fleet dashboard (frontend)

Node Layer: CrowdSec engine, LAPI, nftables bouncer, and the threat-engine-agent
Sync Layer: SSH + rsync pulling JSON snapshots to the dashboard host
Control Plane: Flask API, dashboard UI, and Caddy reverse proxy

Live dashboard: fleet.beane.me

The Agent: Autonomous Drift Detection, and Healing

Drift classification (membership vs metadata)

This is the real heart of the agent's intelligence:

missing = sorted(truth_set - reality_set)
extra = sorted(reality_set - truth_set)

if missing or extra:
    drift = {
        "reason": "membership_mismatch",
        "missing_count": len(missing),
        "extra_count": len(extra),
    }
    return False, len(reality_set), drift

I also implemented some TTL drift logic, so if there is a discrepency between the kernel ban length vs length generated by the threat-decision-engine, it'll be rectified.

if self._has_ttl_drift(siem_seconds, kernel_seconds):
    metadata_drift.append(ip)

Heal logic (targeted and safe)

If I determine there is drift due to metadata_drift, I flush our CrowdSec sets and restart the firewall bouncer.

if drift.get("reason") == "metadata_drift":
    self._flush_crowdsec_sets()
subprocess.run(["systemctl", "restart", "crowdsec-firewall-bouncer"], check=True)

Post-heal verification

Of course, no heal would be complete without verifying that it correctly took place.

post_is_synced, post_count, post_drift = auditor.reconcile()
heal_result = "success" if post_is_synced else "failed"

The Dashboard: Fleet-Wide State Visualization

The reporter state snapshot is passed from the agent to the dashboard:

{
    "node_name": "Zephyrus",
    "last_check": "2026-03-11T01:33:06.369481+00:00",
    "status": "HEALTHY",
    "is_synced": true,
    "kernel_ban_count": 139,
    "last_error": null,
    "missing_count": 0,
    "extra_count": 0,
    "metadata_drift_count": 0,
    "last_drift_fingerprint": "",
    "last_heal_result": "success",
    "last_heal_timestamp": "2026-03-11T01:03:10.172229+00:00",
    "uptime_seconds": 54909,
    "checks_total": 615
}

I also keep 24-hour rolling buckets, so I can generate a clean Plotly graph for each node.

{
    "drift_events_24h": 36,
    "drift_heal_attempts_24h": 36,
    "heals_24h": 36,
    "heal_failures_24h": 1
}

In Operation: A Real Drift/Heal Cycle

Here's a real example of the agent catching and correcting drift in the wild:

2026-03-10 16:24:30,988 - INFO - LAPI Local Parser: Found 141 active bans.
2026-03-10 16:24:30,995 - WARNING - Metadata Drift detected (count=1).
2026-03-10 16:24:30,996 - WARNING - Desync detected (kernel=141, missing=0, extra=0, metadata=1).
2026-03-10 16:24:30,996 - WARNING - Healing approved.
2026-03-10 16:24:31,060 - INFO - Heal: Flushed CrowdSec nft sets to clear stale metadata.
2026-03-10 16:24:31,262 - INFO - Heal: Bouncer restarted successfully.
2026-03-10 16:24:31,263 - INFO - Post-heal verification in 5s (pre_fingerprint=c297f74e11fd3685).

This single sequence captures the entire lifecycle:

The agent parses LAPI truth (141 active bans)
Detects metadata drift (TTL mismatch)
Confirms a desync
Approves healing
Flushes stale nftables sets
Restarts the bouncer
Schedules post-heal verification

It's the whole system coming together - thinking, acting, and validating, ensuring drift is corrected.

Lessons Learned

Over the course of this project, I realized a few crucial things:

State contracts simplify distributed systems
Healing loops need guardrails
Operator UX matters
Observability is narrative

What's Next

There are certainly a few directions I could move in going forward, but I think a few potential candidates are very strong:

Anomaly detection with the potential to spot patterns like repeated drift on the same node, or too frequent healing
Manual reconciliation when necessary, to force a drift and heal cycle on demand
Possible integration with my other tools, like threat-decision-engine, for end-to-end visibility and a more unified decision timeline

Final Thoughts

This project turned a set of isolated nodes into a distributed, observable, and self-healing enforcement platform. It's the most alive my infrastructure has ever felt - and the dashboard (frontend) perfectly encapsulates the magic behind the backend.