Building a threat-decision engine that ingests distributed signals from our API and produces enforceable, explainable security decisions.
The Problem
Enterprise-grade observability is expensive, and heavy
Traditional SIEMs assume a few things:
- Massive ingestion pipelines
- Constant indexing, sorting, and aggressive normalization
- Perfect historical recall
For large enterprises, this makes sense. For my infrastructure, it's unnecessary overhead.
In addition, most security decisions don't require perfect historical recall - they require timely, high-confidence signals.
The goal of our project was to detect real threats, correlate behavior, and act safely and decisively, without the enterprise cost or bloat.
Architecture Overview
[ Fleet Nodes ]
└─ CrowdSec Agents
└─ Scenario Hits
↓
[ Threat Signal API ]
↓
[ Threat Decision Engine ]
- Multi-node Correlation
- Confidence Scoring
- Severity Modeling
- TTL & State (SQLite)
- Explainability
↓
[ CrowdSec Enforcement ]
↓
[ Firewall Bouncer ]
[ Cloudflare Bouncer ]Philosophy
The difference between most SIEMs and this design is simple: it's not about collecting data, it's about knowing when to act.
Only high-confidence, high-severity threats result in enforcement. Everything else is tracked temporarily and expires naturally.
Decision-first security is the philosophy of our project, and particularly why it ended up so lightweight.
Resource Usage
Because it was written to be lightweight, it doesn't require much to run. The same $6.75/year VM that powers our node-health failover also runs the core of this system.
In some cases, efficiency can replace scale. With the combination of external APIs and well-orchestrated cron, we've replaced heavy infrastructure with efficient tooling.
Even if it seems like a strange concept, I view boring logs as a sign of success.
Open-Source
The entire repository that powers this project can be found on GitHub, featuring the Tech Stack, Project Layout, and the Dashboard (also viewable at observe.beane.me - running every 20 minutes).
I'm a huge believer in open source - and the idea that if it helps you, it could also help others.
A few of my projects are still private, because they're more tuned to my infrastructure, such as threats.beane.me and intel.beane.me because they feature more moving parts, like aggregation and different areas running on multiple servers - but in time I may try to get them published on Git as well.
Lessons Learned
In my experience, smaller systems like this generally fail less often, which is a large reason why the project took the direction it did.
With everything I've come to realize, highly orchestrated systems may fail less often on paper - but when they do, failures are harder to trace and resolve.
Cost constraints also in turn force better design. They drive efficiency, and a control panel such as this should be boring. Having it "just work" means you'll spend far less time debugging it and it frees up time for everything else.