Aggregating alerts across multiple servers - keeping it clean, sanitized, and ensuring we can track attackers no matter which node they came from.
The Problem
While having one server feed into threats.beane.me was a great start, I certainly couldn't stop there. The more clean data and info we have on our infrastructure, the better in my eyes. I decided log aggregation was the next best, viable step that would allow more data to be fed in with CrowdSec decisions across multiple servers.
I've also been hitting the 500/alerts per month in CrowdSec's online interface pretty quick, since that was introduced - so that was another reason to get this implemented.
The Goal
Realizing the goal of this project was pretty clear - I was interested in getting alerts both sanitized and available for viewing on threats.beane.me, while aggregating between multiple servers so it wasn't just limited to Triton. Even though it's just two servers at the moment, it opened the doors to allow me to easily add more, as needed.
Architecture Overview
[ Triton Server ] [ Iris Server ]
+------------------+ +------------------+
| CrowdSec Agent | | CrowdSec Agent |
| + Python script | | + Python script |
| crowdsec.py | | crowdsec.py |
| → outputs | | → outputs |
| crowdsec_alerts_triton.json | crowdsec_alerts_iris.json
+------------------+ +------------------+
| |
| (rsync over SSH) | (rsync over SSH)
+---------------------------+--------------+
|
v
[ Aggregation Server ]
(threats.beane.me host)
+---------------------------------+
| Receives both JSON files |
| |
| Python aggregator |
| aggregate_alerts.py |
| → merges & sanitizes |
| → outputs |
| crowdsec_alerts_all.json |
+---------------------------------+
|
v
[ Static Page Generator ]
+---------------------------------+
| Original Python script |
| render_threats.py |
| → reads crowdsec_alerts_all.json
| → builds static HTML page |
| with encapsulated threats |
| → writes to site directory |
| (served at threats.beane.me)|
+---------------------------------+
Solution
I approached the problem in three seperate stages:
- Fetch and sanitized alerts
- Deployed Python code on Iris to generate alerts for the pipeline
- Renamed output file from
crowdsec_alerts.jsontocrowdsec_alerts_server.json - Aggregate and deduplicate
- Created a new Python script to sanitize and merge JSON alerts from each server
- Aggregated output written to
crowdsec_alerts_all.json - Ensure JSON is clean and structured
crowdsec_alerts_all.jsonbecame the new input for the HTML rendering Python script.- Enabled scalable threat rendering across multiple servers.
Python Highlights (Aggregation)
def aggregate_alerts(default_source=None):
combined = []
for path in glob.glob("crowdsec_alerts_*.json"):
if path == OUTPUT_FILE:
continue # skip the merged output file
records = load_json_file(path)
# Derive source tag from filename if not provided
src_name = default_source or os.path.splitext(os.path.basename(path))[0].replace("crowdsec_alerts_", "")
for r in records:
r.setdefault("source", src_name)
combined.extend(records)
return combinedResults
After adding multiple servers to the pipeline, and developing the aggregation script in Python:
- Alerts are aggregated every 10 minutes, from multiple nodes now
- “Top Troublemaker” is now shown at the bottom of the page, with a direct link to AbuseIPDB for confirmation
- Python scripts are modular, reusable, and production-ready, able to be deployed on more servers in the future
Next Steps
- Expand the dashboard to historical trends, across servers in the network (hint: intel.beane.me, coming soon)
- Continue refining Python for error handling and logging improvements