Go to file
marco370 dc91096d9d Improve log processing and add automated tasks
Update log processing to use the correct timestamp field and introduce cron jobs for automated model training and anomaly detection.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: f0653fd5-fc94-4fcb-8d7e-2a0e90fc81bf
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 18:11:49 +00:00
attached_assets Improve log processing and add automated tasks 2025-11-17 18:11:49 +00:00
client Add a navigation sidebar and dashboard to the IDS system 2025-11-15 11:16:44 +00:00
database-schema Add database tables for storing detection, training, and whitelist information 2025-11-17 18:03:30 +00:00
deployment Provide initial setup instructions for deploying the application 2025-11-17 17:15:31 +00:00
extracted_idf Saved progress at the end of the loop 2025-11-11 09:15:10 +00:00
python_ml Improve log processing and add automated tasks 2025-11-17 18:11:49 +00:00
server Add a navigation sidebar and dashboard to the IDS system 2025-11-15 11:16:44 +00:00
shared Update deployment instructions and network log schema 2025-11-17 17:42:17 +00:00
.env.example Update application to use environment variables and force IPv4 2025-11-17 16:48:22 +00:00
.gitignore Add ability to export database schema structure to Git 2025-11-17 17:07:31 +00:00
.replit Improve log processing and add automated tasks 2025-11-17 18:11:49 +00:00
components.json Extracted stack files 2025-11-11 09:12:50 +00:00
deploy-to-gitlab.sh Add ability to export database schema structure to Git 2025-11-17 17:07:31 +00:00
design_guidelines.md Extracted stack files 2025-11-11 09:12:50 +00:00
drizzle.config.ts Extracted stack files 2025-11-11 09:12:50 +00:00
git.env.example Add comprehensive installation and configuration guides for the IDS system 2025-11-17 15:19:13 +00:00
GUIDA_INSTALLAZIONE.md Add comprehensive installation and configuration guides for the IDS system 2025-11-17 15:19:13 +00:00
package-lock.json Add a navigation sidebar and dashboard to the IDS system 2025-11-15 11:16:44 +00:00
package.json Add file management and upload capabilities for IDF Mikrotik projects 2025-11-15 10:54:17 +00:00
postcss.config.js Extracted stack files 2025-11-11 09:12:50 +00:00
push-gitlab.sh 🚀 Release v1.0.1 2025-11-17 08:45:53 +00:00
README.md Add a navigation sidebar and dashboard to the IDS system 2025-11-15 11:16:44 +00:00
replit.md Improve intrusion detection system with functional updates and database fixes 2025-11-17 17:54:18 +00:00
RISPOSTA_DEPLOYMENT.md Add comprehensive deployment and configuration guides for the IDS system 2025-11-15 11:30:55 +00:00
tailwind.config.ts Extracted stack files 2025-11-11 09:12:50 +00:00
tsconfig.json Extracted stack files 2025-11-11 09:12:50 +00:00
update_from_git.sh Update script to apply multiple database migration scripts 2025-11-17 17:44:11 +00:00
version.json 🚀 Release v1.0.13 2025-11-17 18:06:53 +00:00
vite.config.ts Extracted stack files 2025-11-11 09:12:50 +00:00

🛡️ IDS - Intrusion Detection System

Sistema di rilevamento intrusioni moderno per router MikroTik, basato su Machine Learning.

🎯 Caratteristiche Principali

  • ML Efficiente: Solo 25 feature mirate (non 150+) per analisi veloce e accurata
  • Detection Real-time: Rilevamento anomalie in <2 secondi
  • Multi-Router: Gestione parallela di 10+ router MikroTik tramite API REST
  • Auto-Block: Blocco automatico IP anomali con timeout configurabile
  • Dashboard Web: Monitoring real-time completo
  • PostgreSQL: Database performante per analisi time-series

🏗️ Architettura

┌─────────────────┐
│ Router MikroTik │ ──(Syslog)──▶ ┌──────────────┐
│   (10+ router)   │               │  PostgreSQL  │
└─────────────────┘               │   Database   │
                                  └──────┬───────┘
                                         │
                    ┌────────────────────┼────────────────────┐
                    │                    │                    │
             ┌──────▼─────┐     ┌───────▼────┐      ┌───────▼────┐
             │  Python ML  │     │  FastAPI   │      │   React    │
             │   Analyzer  │     │  Backend   │      │  Dashboard │
             └─────────────┘     └────────────┘      └────────────┘
                    │                    │                    │
                    └────────────────────┼────────────────────┘
                                         │
                              ┌──────────▼──────────┐
                              │  MikroTik Manager   │
                              │   (API REST)        │
                              └─────────────────────┘
                                         │
                    ┌────────────────────┼────────────────────┐
                    ▼                    ▼                    ▼
            ┌───────────┐        ┌───────────┐        ┌───────────┐
            │  Router 1 │        │  Router 2 │        │  Router N │
            └───────────┘        └───────────┘        └───────────┘

🚀 Quick Start

1. Setup Backend Python

cd python_ml
pip install -r requirements.txt
python main.py

Il backend FastAPI partirà su http://0.0.0.0:8000

2. Setup Frontend (già configurato)

Il frontend React è già in esecuzione tramite il workflow "Start application". Accedi alla dashboard web all'URL del tuo Repl.

3. Configurazione Router MikroTik

Sul router MikroTik, abilita l'API REST:

/ip service
set api-ssl disabled=no
set www-ssl disabled=no

Poi aggiungi i router tramite la dashboard web oppure:

INSERT INTO routers (name, ip_address, username, password, api_port, enabled)
VALUES ('Router 1', '192.168.1.1', 'admin', 'password', 443, true);

📊 Come Funziona

1. Raccolta Dati

I log arrivano tramite Syslog dai router MikroTik e vengono salvati in PostgreSQL nella tabella network_logs.

2. Training ML

curl -X POST http://localhost:8000/train \
  -H "Content-Type: application/json" \
  -d '{
    "max_records": 10000,
    "hours_back": 24,
    "contamination": 0.01
  }'

Il sistema estrae 25 feature mirate:

  • Volume: bytes/sec, packets, connessioni
  • Temporali: burst, intervalli, pattern orari
  • Protocolli: diversità, entropia, TCP/UDP ratio
  • Port Scanning: porte uniche, sequenziali
  • Comportamentali: varianza dimensioni, azioni bloccate

3. Detection Real-time

curl -X POST http://localhost:8000/detect \
  -H "Content-Type: application/json" \
  -d '{
    "max_records": 5000,
    "hours_back": 1,
    "risk_threshold": 60.0,
    "auto_block": true
  }'

Il modello Isolation Forest assegna:

  • Risk Score (0-100): livello di pericolosità
  • Confidence (0-100): certezza del rilevamento
  • Anomaly Type: ddos, port_scan, brute_force, botnet, suspicious

4. Auto-Block

IP con risk_score >= 80 (CRITICO) vengono bloccati automaticamente su tutti i router via API REST con timeout 1h.

🎚️ Livelli di Rischio

Score Livello Azione
85-100 🔴 CRITICO Blocco immediato
70-84 🟠 ALTO Blocco + monitoring
60-69 🟡 MEDIO Monitoring
40-59 🔵 BASSO Logging
0-39 🟢 NORMALE Nessuna azione

📚 API Endpoints

  • GET /health - Health check
  • POST /train - Training modello ML
  • POST /detect - Detection anomalie
  • POST /block-ip - Blocco manuale IP
  • POST /unblock-ip - Sblocco IP
  • GET /stats - Statistiche sistema

Documentazione completa: http://localhost:8000/docs

🔧 Configurazione Automatica

Training Automatico (ogni 12h)

0 */12 * * * curl -X POST http://localhost:8000/train

Detection Continua (ogni 5 minuti)

*/5 * * * * curl -X POST http://localhost:8000/detect \
  -H "Content-Type: application/json" \
  -d '{"auto_block": true, "risk_threshold": 75}'

🆚 Vantaggi vs Sistema Precedente

Aspetto Sistema Vecchio Nuovo IDS
Feature ML 150+ 25 (mirate)
Velocità Training ~5 min ~10 sec
Velocità Detection Lento <2 sec
Comunicazione Router SSH (lento) API REST (veloce)
Falsi Negativi Alti Bassi
Multi-Router Sequenziale Parallelo
Database MySQL PostgreSQL

🔍 Troubleshooting

Troppi Falsi Positivi?

Aumenta risk_threshold (es. da 60 a 75)

Non Rileva Attacchi?

  • Diminuisci contamination nel training (es. da 0.01 a 0.02)
  • Abbassa risk_threshold (es. da 75 a 60)

Connessione Router Fallita?

  • Verifica API REST abilitata: /ip service print
  • Controlla firewall: porta 443 deve essere aperta
  • Test: curl -u admin:password https://ROUTER_IP/rest/system/identity

📁 Struttura Progetto

.
├── python_ml/              # Backend Python ML
│   ├── ml_analyzer.py      # Analisi ML (25 feature)
│   ├── mikrotik_manager.py # Gestione router API REST
│   ├── main.py             # FastAPI backend
│   └── requirements.txt    # Dipendenze Python
├── client/                 # Frontend React
│   └── src/
│       └── pages/          # Pagine dashboard
├── server/                 # Backend Node.js
│   ├── db.ts               # Database PostgreSQL
│   ├── routes.ts           # API routes
│   └── storage.ts          # Storage interface
└── shared/
    └── schema.ts           # Schema database Drizzle ORM

🔐 Sicurezza

  • Password router NON in chiaro nel codice
  • Timeout automatico sui blocchi (default 1h)
  • Whitelist per IP fidati
  • Logging completo di tutte le azioni
  • Database PostgreSQL con connessione sicura

📝 Note Importanti

  • Whitelist: IP in whitelist non vengono mai bloccati
  • Timeout: Blocchi hanno timeout (default 1h), poi scadono automaticamente
  • Parallelo: Sistema blocca su tutti i router simultaneamente (veloce)
  • Performance: Analizza 10K log in <2 secondi

📖 Documentazione

🤝 Supporto

Per problemi o domande:

  1. Controlla questa documentazione
  2. Verifica i log di debug (python_ml/main.py)
  3. Testa la connessione database e router
  4. Verifica i modelli addestrati (python_ml/models/)

IDS - Intrusion Detection System v1.0.0
Sistema moderno e performante per proteggere la tua rete MikroTik