marco/ids.alfacom.it
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
495e845a79
ids.alfacom.it/replit.md
marco370 495e845a79 Update log format to include timestamps and filter incoming connections 
Correct the rsyslog template to include timestamps in logs, ensuring compatibility with the Python parser. This change also refactors the log filtering to capture only incoming connections, significantly reducing log volume.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: c2f849f9-105f-452a-bdc3-a956d102c54b
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/6ZTQSoP
2025-11-22 08:54:21 +00:00

7.0 KiB
Raw Blame History

IDS - Intrusion Detection System

Overview

This project is a full-stack web application designed as an Intrusion Detection System (IDS) for MikroTik routers, leveraging Machine Learning. Its primary purpose is to monitor network traffic, detect anomalies indicative of intrusions, and automatically block malicious IP addresses across multiple routers. The system aims to provide real-time monitoring, efficient anomaly detection, and streamlined management of network security for MikroTik environments.

User Preferences

Operazioni Git e Deployment

  • IMPORTANTE: L'agente NON deve usare comandi git (push-gitlab.sh) perché Replit blocca le operazioni git
  • Workflow corretto:
    1. Utente riporta errori/problemi dal server AlmaLinux
    2. Agente risolve problemi e modifica file su Replit
    3. Utente esegue manualmente: ./push-gitlab.sh per commit+push
    4. Utente esegue sul server: ./update_from_git.sh o ./update_from_git.sh --db
    5. Utente testa e riporta risultati all'agente
    6. Ripeti fino a funzionamento completo

Linguaggio

  • Tutte le risposte dell'agente devono essere in italiano
  • Codice e documentazione tecnica: inglese
  • Commit message: italiano

System Architecture

The IDS features a React-based frontend for real-time monitoring, detection visualization, and whitelist management, utilizing ShadCN UI and TanStack Query. The backend comprises a Python FastAPI service for ML analysis (Isolation Forest with 25 targeted features), MikroTik API management, and a detection engine scoring anomalies from 0-100 with five risk levels. A Node.js (Express) backend handles API requests from the frontend and manages the PostgreSQL database.

Workflow:

  1. Log Collection: MikroTik Routers send syslog data (UDP:514) to RSyslog, which is then parsed by syslog_parser.py and stored in the network_logs table in PostgreSQL.
  2. Training: The Python ML component extracts 25 features from network logs and trains an Isolation Forest model.
  3. Detection: Real-time analysis of network logs is performed using the trained ML model, assigning a risk score.
  4. Auto-Block: Critical IPs (score >= 80) are automatically blocked across all configured MikroTik routers in parallel via their REST API.

Key Features:

  • ML Analyzer: Isolation Forest with 25 features.
  • MikroTik Manager: Parallel communication with 10+ routers via API REST.
  • Detection Engine: Scoring 0-100 with 5 risk levels (Normal, Basso, Medio, Alto, Critico).
  • Form Validation: Improved validation using react-hook-form and Zod.
  • Database Migrations: Automated SQL migrations applied via update_from_git.sh --db.
  • Microservices: Separation of concerns with dedicated Python ML backend and Node.js API backend.

External Dependencies

  • React: Frontend framework.
  • FastAPI: Python web framework for the ML backend.
  • PostgreSQL: Primary database for storing router configurations, network logs, detections, and whitelist entries.
  • MikroTik API REST: Used for communication with MikroTik routers for configuration and IP blocking.
  • ShadCN UI: Frontend component library.
  • TanStack Query: Data fetching library for the frontend.
  • Isolation Forest: Machine Learning algorithm for anomaly detection.
  • RSyslog: Log collection daemon.
  • Drizzle ORM: Used for database schema definition and synchronization in the Node.js backend.
  • Neon Database: Cloud-native PostgreSQL service (used in Replit environment).
  • pg (Node.js driver): Standard PostgreSQL driver for Node.js (used in AlmaLinux environment).
  • psycopg2: PostgreSQL adapter for Python.

Recent Updates (Novembre 2025)

📊 Log Format Fix - Timestamp Integration (22 Nov 2025 - 10:30)

  • Problema: RSyslog salvava log senza timestamp, parser Python falliva
  • Soluzione: Template rsyslog corretto per includere timestamp BSD 
    # PRIMA: template(... string="%msg%\n")  ❌
# ADESSO: template(... string="%TIMESTAMP% %HOSTNAME% %msg%\n")  ✅
  • Formato log supportato: 
    Nov 22 08:15:30 FIBRA forward: in:<pppoe-user> out:sfp-xxx, connection-state:new proto TCP (SYN), 10.0.254.77:53783->52.213.60.221:443, len 64
  • Compatibilità: Parser Python 100% compatibile con:
    • ✅ Log "forward" e "detected-ddos forward"
    • ✅ Interfacce in/out: in:<pppoe-xxx> out:sfp-xxx
    • ✅ src-mac opzionale
    • ✅ TCP flags: (SYN), (ACK,PSH), etc.
    • ✅ NAT info opzionale
  • Migrazione: Vedi deployment/MIGRATION_INCOMING_LOGS.md
  • Benefici:
    • ⚡ Volume log ridotto 50-70% (solo connessioni in ingresso)
    • 🔒 Parser funzionante con timestamp corretto
    • 📊 Database popolato correttamente

🚀 Database Versioning System (22 Nov 2025 - 10:00)

  • Feature: Sistema intelligente di versioning per migrazioni database
  • Problema risolto: update_from_git.sh rieseguiva tutte le migrazioni SQL ad ogni update (lento)
  • Soluzione:
    • Tabella schema_version traccia versione corrente database
    • Migrazioni SQL numerate sequenzialmente (001, 002, 003, etc.)
    • Script apply_migrations.sh applica solo migrazioni mancanti
    • Integrato in workflow update automatico
  • Benefici:
    • ⚡ Update 10x più veloce (salta migrazioni già applicate)
    • 🔒 Sicuro: previene re-esecuzione migrazioni
    • 📊 Tracciabilità: storico migrazioni applicate
  • Documentazione: database-schema/README.md

Fix Recenti (Novembre 2025)

🚨 Database Full - Auto-Cleanup Fix (21 Nov 2025 - 18:00)

  • Problema: Database PostgreSQL pieno con 417 MILIONI di log accumulati
    • Syslog parser ha processato 417.7M righe senza limite di retention
    • Errore: could not extend file: No space left on device
    • Tutte le tabelle vuote perché database non accetta più scritture
  • Causa: Nessuna pulizia automatica dei vecchi log (retention infinita)
  • Soluzione:
    • Script cleanup_old_logs.sql: Mantiene solo ultimi 7 giorni di network_logs
    • Script cleanup_database.sh: Wrapper per esecuzione manuale/cron
    • Script setup_cron_cleanup.sh: Configura cron job giornaliero (ore 03:00)
  • Fix Immediato sul Server: 
    # 1. Pulisci manualmente log vecchi
psql $DATABASE_URL << 'EOF'
DELETE FROM network_logs WHERE timestamp < NOW() - INTERVAL '7 days';
VACUUM FULL network_logs;
EOF

# 2. Setup pulizia automatica giornaliera
sudo /opt/ids/deployment/setup_cron_cleanup.sh
  • Risultato Atteso:
    • Database ridotto da centinaia di GB a pochi GB
    • Retention 7 giorni sufficiente per training ML
    • Pulizia automatica previene saturazione futura

✅ Database Driver Fix - Dual Mode Neon/PostgreSQL (21 Nov 2025 - 17:40)

  • Problema: Frontend Node.js falliva con errore 500 su tutte le query database
  • Causa: @neondatabase/serverless usa WebSocket ed è compatibile SOLO con Neon Cloud, non con PostgreSQL locale
  • Soluzione: Dual-mode driver in server/db.ts con auto-detection ambiente
  • Risultato: Funziona su Replit (Neon) e AlmaLinux (PostgreSQL standard) ✅