Go to file
marco370 3425521215 Update list fetching to handle new Spamhaus format and IP matching
Update Spamhaus parser to support NDJSON format and fix IP matching errors by ensuring database migrations are applied.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 11e93061-1fe5-4624-8362-9202aff893d7
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/rDib6Pq
2026-01-02 11:48:33 +00:00
attached_assets Update list fetching to handle new Spamhaus format and IP matching 2026-01-02 11:48:33 +00:00
client Fix how public lists are managed by correcting API request parameters 2025-11-26 15:27:19 +00:00
database-schema 🚀 Release v1.0.95 2025-11-27 18:29:37 +00:00
deployment Update database to correctly handle IP address and CIDR data types 2025-11-27 18:27:10 +00:00
extracted_idf Saved progress at the end of the loop 2025-11-11 09:15:10 +00:00
python_ml Update list fetching to handle new Spamhaus format and IP matching 2026-01-02 11:48:33 +00:00
server Implement list synchronization by fetching and saving IP addresses 2025-11-27 18:02:24 +00:00
shared Add full CIDR support for IP address matching in lists 2025-11-26 09:54:57 +00:00
.env.example Update application to use environment variables and force IPv4 2025-11-17 16:48:22 +00:00
.gitignore Add ability to export database schema structure to Git 2025-11-17 17:07:31 +00:00
.replit Update list fetching to handle new Spamhaus format and IP matching 2026-01-02 11:48:33 +00:00
components.json Extracted stack files 2025-11-11 09:12:50 +00:00
deploy-to-gitlab.sh Add ability to export database schema structure to Git 2025-11-17 17:07:31 +00:00
design_guidelines.md Extracted stack files 2025-11-11 09:12:50 +00:00
drizzle.config.ts Extracted stack files 2025-11-11 09:12:50 +00:00
git.env.example Add comprehensive installation and configuration guides for the IDS system 2025-11-17 15:19:13 +00:00
GUIDA_INSTALLAZIONE.md Add comprehensive installation and configuration guides for the IDS system 2025-11-17 15:19:13 +00:00
main.py Add functionality to manage and sync public blacklists and whitelists 2025-11-26 09:21:43 +00:00
MIKROTIK_API_FIX.md Update MikroTik API connection to use correct REST API port 2025-11-25 18:13:31 +00:00
package-lock.json Add a navigation sidebar and dashboard to the IDS system 2025-11-15 11:16:44 +00:00
package.json Add file management and upload capabilities for IDF Mikrotik projects 2025-11-15 10:54:17 +00:00
postcss.config.js Extracted stack files 2025-11-11 09:12:50 +00:00
push-gitlab.sh 🚀 Release v1.0.1 2025-11-17 08:45:53 +00:00
pyproject.toml Add functionality to manage and sync public blacklists and whitelists 2025-11-26 09:21:43 +00:00
README.md Add a navigation sidebar and dashboard to the IDS system 2025-11-15 11:16:44 +00:00
replit.md Add full CIDR support for IP address matching in lists 2025-11-26 09:54:57 +00:00
RISPOSTA_DEPLOYMENT.md Add comprehensive deployment and configuration guides for the IDS system 2025-11-15 11:30:55 +00:00
tailwind.config.ts Extracted stack files 2025-11-11 09:12:50 +00:00
tsconfig.json Extracted stack files 2025-11-11 09:12:50 +00:00
uv.lock Add functionality to manage and sync public blacklists and whitelists 2025-11-26 09:21:43 +00:00
version.json 🚀 Release v1.0.95 2025-11-27 18:29:37 +00:00
vite.config.ts Extracted stack files 2025-11-11 09:12:50 +00:00

🛡️ IDS - Intrusion Detection System

Sistema di rilevamento intrusioni moderno per router MikroTik, basato su Machine Learning.

🎯 Caratteristiche Principali

  • ML Efficiente: Solo 25 feature mirate (non 150+) per analisi veloce e accurata
  • Detection Real-time: Rilevamento anomalie in <2 secondi
  • Multi-Router: Gestione parallela di 10+ router MikroTik tramite API REST
  • Auto-Block: Blocco automatico IP anomali con timeout configurabile
  • Dashboard Web: Monitoring real-time completo
  • PostgreSQL: Database performante per analisi time-series

🏗️ Architettura

┌─────────────────┐
│ Router MikroTik │ ──(Syslog)──▶ ┌──────────────┐
│   (10+ router)   │               │  PostgreSQL  │
└─────────────────┘               │   Database   │
                                  └──────┬───────┘
                                         │
                    ┌────────────────────┼────────────────────┐
                    │                    │                    │
             ┌──────▼─────┐     ┌───────▼────┐      ┌───────▼────┐
             │  Python ML  │     │  FastAPI   │      │   React    │
             │   Analyzer  │     │  Backend   │      │  Dashboard │
             └─────────────┘     └────────────┘      └────────────┘
                    │                    │                    │
                    └────────────────────┼────────────────────┘
                                         │
                              ┌──────────▼──────────┐
                              │  MikroTik Manager   │
                              │   (API REST)        │
                              └─────────────────────┘
                                         │
                    ┌────────────────────┼────────────────────┐
                    ▼                    ▼                    ▼
            ┌───────────┐        ┌───────────┐        ┌───────────┐
            │  Router 1 │        │  Router 2 │        │  Router N │
            └───────────┘        └───────────┘        └───────────┘

🚀 Quick Start

1. Setup Backend Python

cd python_ml
pip install -r requirements.txt
python main.py

Il backend FastAPI partirà su http://0.0.0.0:8000

2. Setup Frontend (già configurato)

Il frontend React è già in esecuzione tramite il workflow "Start application". Accedi alla dashboard web all'URL del tuo Repl.

3. Configurazione Router MikroTik

Sul router MikroTik, abilita l'API REST:

/ip service
set api-ssl disabled=no
set www-ssl disabled=no

Poi aggiungi i router tramite la dashboard web oppure:

INSERT INTO routers (name, ip_address, username, password, api_port, enabled)
VALUES ('Router 1', '192.168.1.1', 'admin', 'password', 443, true);

📊 Come Funziona

1. Raccolta Dati

I log arrivano tramite Syslog dai router MikroTik e vengono salvati in PostgreSQL nella tabella network_logs.

2. Training ML

curl -X POST http://localhost:8000/train \
  -H "Content-Type: application/json" \
  -d '{
    "max_records": 10000,
    "hours_back": 24,
    "contamination": 0.01
  }'

Il sistema estrae 25 feature mirate:

  • Volume: bytes/sec, packets, connessioni
  • Temporali: burst, intervalli, pattern orari
  • Protocolli: diversità, entropia, TCP/UDP ratio
  • Port Scanning: porte uniche, sequenziali
  • Comportamentali: varianza dimensioni, azioni bloccate

3. Detection Real-time

curl -X POST http://localhost:8000/detect \
  -H "Content-Type: application/json" \
  -d '{
    "max_records": 5000,
    "hours_back": 1,
    "risk_threshold": 60.0,
    "auto_block": true
  }'

Il modello Isolation Forest assegna:

  • Risk Score (0-100): livello di pericolosità
  • Confidence (0-100): certezza del rilevamento
  • Anomaly Type: ddos, port_scan, brute_force, botnet, suspicious

4. Auto-Block

IP con risk_score >= 80 (CRITICO) vengono bloccati automaticamente su tutti i router via API REST con timeout 1h.

🎚️ Livelli di Rischio

Score Livello Azione
85-100 🔴 CRITICO Blocco immediato
70-84 🟠 ALTO Blocco + monitoring
60-69 🟡 MEDIO Monitoring
40-59 🔵 BASSO Logging
0-39 🟢 NORMALE Nessuna azione

📚 API Endpoints

  • GET /health - Health check
  • POST /train - Training modello ML
  • POST /detect - Detection anomalie
  • POST /block-ip - Blocco manuale IP
  • POST /unblock-ip - Sblocco IP
  • GET /stats - Statistiche sistema

Documentazione completa: http://localhost:8000/docs

🔧 Configurazione Automatica

Training Automatico (ogni 12h)

0 */12 * * * curl -X POST http://localhost:8000/train

Detection Continua (ogni 5 minuti)

*/5 * * * * curl -X POST http://localhost:8000/detect \
  -H "Content-Type: application/json" \
  -d '{"auto_block": true, "risk_threshold": 75}'

🆚 Vantaggi vs Sistema Precedente

Aspetto Sistema Vecchio Nuovo IDS
Feature ML 150+ 25 (mirate)
Velocità Training ~5 min ~10 sec
Velocità Detection Lento <2 sec
Comunicazione Router SSH (lento) API REST (veloce)
Falsi Negativi Alti Bassi
Multi-Router Sequenziale Parallelo
Database MySQL PostgreSQL

🔍 Troubleshooting

Troppi Falsi Positivi?

Aumenta risk_threshold (es. da 60 a 75)

Non Rileva Attacchi?

  • Diminuisci contamination nel training (es. da 0.01 a 0.02)
  • Abbassa risk_threshold (es. da 75 a 60)

Connessione Router Fallita?

  • Verifica API REST abilitata: /ip service print
  • Controlla firewall: porta 443 deve essere aperta
  • Test: curl -u admin:password https://ROUTER_IP/rest/system/identity

📁 Struttura Progetto

.
├── python_ml/              # Backend Python ML
│   ├── ml_analyzer.py      # Analisi ML (25 feature)
│   ├── mikrotik_manager.py # Gestione router API REST
│   ├── main.py             # FastAPI backend
│   └── requirements.txt    # Dipendenze Python
├── client/                 # Frontend React
│   └── src/
│       └── pages/          # Pagine dashboard
├── server/                 # Backend Node.js
│   ├── db.ts               # Database PostgreSQL
│   ├── routes.ts           # API routes
│   └── storage.ts          # Storage interface
└── shared/
    └── schema.ts           # Schema database Drizzle ORM

🔐 Sicurezza

  • Password router NON in chiaro nel codice
  • Timeout automatico sui blocchi (default 1h)
  • Whitelist per IP fidati
  • Logging completo di tutte le azioni
  • Database PostgreSQL con connessione sicura

📝 Note Importanti

  • Whitelist: IP in whitelist non vengono mai bloccati
  • Timeout: Blocchi hanno timeout (default 1h), poi scadono automaticamente
  • Parallelo: Sistema blocca su tutti i router simultaneamente (veloce)
  • Performance: Analizza 10K log in <2 secondi

📖 Documentazione

🤝 Supporto

Per problemi o domande:

  1. Controlla questa documentazione
  2. Verifica i log di debug (python_ml/main.py)
  3. Testa la connessione database e router
  4. Verifica i modelli addestrati (python_ml/models/)

IDS - Intrusion Detection System v1.0.0
Sistema moderno e performante per proteggere la tua rete MikroTik