Detecting & Blocking Anomalous Traffic with Cloud Anomaly Detector

try { if(localStorage) { let currentUser = localStorage.getItem('current_user'); if (currentUser) { currentUser = JSON.parse(currentUser); if (currentUser.id === 1238224) { document.getElementById('article-show-container').classList.add('current-user-is-article-author'); } } } } catch (e) { console.error(e); } Joseph Joshua Posted on Apr 28 Detecting & Blocking Anomalous Traffic with Cloud Anomaly Detector #aws #python #devops #programming A lightweight, containerized anomaly detection system that monitors traffic in real time, detects abuse patterns, and automatically blocks malicious IPs at the host firewall level. I built a real-time anomaly detection system that monitors nginx access logs, computes adaptive rolling baselines per time window, detects traffic anomalies using statistical methods (z-score + spike multipliers), and automatically blocks malicious IPs using host-level iptables rules. The system includes Slack alerts and a live dashboard for observability and debugging. 🧠 Background / Motivation Modern systems face constant threats such as: DDoS attacks Credential stuffing API abuse and scraping bots Sudden traffic spikes that degrade service Most production solutions rely on expensive managed WAFs or cloud security tools. I wanted to build a low-cost, self-hosted anomaly detection engine that runs entirely on a VPS using logs, statistics, and system-level enforcement. Constraints: Must be containerized (Docker-based) Must run on low-cost VPS infrastructure Must use logs (not packet inspection tools) Must enforce bans at host level (not only inside containers) Must provide real-time visibility and debugging 🏗️ What I Built A full-stack anomaly detection pipeline composed of: Detector Service (Python) Baseline Engine (rolling statistical model) Blocker Service (iptables enforcement on host) Dashboard (real-time monitoring UI) Slack Alerting System (incident notifications) ⚙️ How It Works Nginx logs every request in structured JSON format. { "ip": "1.2.3.4", "endpoint": "/", "status": 200, "timestamp": 1710000000 } Enter fullscreen mode Exit fullscreen mode 🔄 From Logs to Detection Once nginx writes request logs, the detector continuously processes them in real time. Each incoming log entry goes through the following pipeline: Parse JSON log entry Extract IP, timestamp, and status code Update per-second counters Feed values into rolling baseline engine Evaluate anomaly conditions This pipeline runs continuously with minimal latency, ensuring near real-time detection. 📉 Rolling Baseline Behavior The system does not rely on fixed thresholds. Instead, it learns traffic behavior over time. For each time window, the baseline tracks: Average request rate (mean) Variance (standard deviation) Traffic distribution per second This allows the system to adapt dynamically to traffic changes. Example behavior: Normal traffic period → stable baseline Gradual increase → baseline adjusts slightly Sudden spike → deviation becomes statistically significant ⚠️ Anomaly Decision Process Every second, the detector evaluates: Current request rate vs baseline mean Z-score deviation Spike multiplier threshold Error rate deviation If any condition exceeds configured thresholds, the IP or system state is flagged. This ensures: Low false positives during normal usage Fast reaction to sudden abuse patterns 🚫 Blocking Execution Flow When an anomaly is confirmed, the system does not block immediately inside the application layer. Instead, it uses a…

This excerpt is published under fair use for community discussion. Read the full article at DEV Community.