# IDS - Intrusion Detection System

## Overview
This project is a full-stack web application designed as an Intrusion Detection System (IDS) for MikroTik routers, leveraging Machine Learning. Its primary purpose is to monitor network traffic, detect anomalies indicative of intrusions, and automatically block malicious IP addresses across multiple routers. The system aims to provide real-time monitoring, efficient anomaly detection, and streamlined management of network security for MikroTik environments.

## User Preferences
### Operazioni Git e Deployment
- **IMPORTANTE**: L'agente NON deve usare comandi git (push-gitlab.sh) perché Replit blocca le operazioni git
- **Workflow corretto**:
  1. Utente riporta errori/problemi dal server AlmaLinux
  2. Agente risolve problemi e modifica file su Replit
  3. **Utente esegue manualmente**: `./push-gitlab.sh` per commit+push
  4. **Utente esegue sul server**: `./update_from_git.sh` o `./update_from_git.sh --db`
  5. Utente testa e riporta risultati all'agente
  6. Ripeti fino a funzionamento completo

### Linguaggio
- Tutte le risposte dell'agente devono essere in **italiano**
- Codice e documentazione tecnica: inglese
- Commit message: italiano

## System Architecture
The IDS features a React-based frontend for real-time monitoring, detection visualization, and whitelist management, utilizing ShadCN UI and TanStack Query. The backend comprises a Python FastAPI service for ML analysis (Isolation Forest with 25 targeted features), MikroTik API management, and a detection engine scoring anomalies from 0-100 with five risk levels. A Node.js (Express) backend handles API requests from the frontend and manages the PostgreSQL database.

**Workflow:**
1.  **Log Collection**: MikroTik Routers send syslog data (UDP:514) to RSyslog, which is then parsed by `syslog_parser.py` and stored in the `network_logs` table in PostgreSQL.
2.  **Training**: The Python ML component extracts 25 features from network logs and trains an Isolation Forest model.
3.  **Detection**: Real-time analysis of network logs is performed using the trained ML model, assigning a risk score.
4.  **Auto-Block**: Critical IPs (score >= 80) are automatically blocked across all configured MikroTik routers in parallel via their REST API.

**Key Features:**
-   **ML Analyzer**: Isolation Forest with 25 features.
-   **MikroTik Manager**: Parallel communication with 10+ routers via API REST.
-   **Detection Engine**: Scoring 0-100 with 5 risk levels (Normal, Basso, Medio, Alto, Critico).
-   **Form Validation**: Improved validation using react-hook-form and Zod.
-   **Database Migrations**: Automated SQL migrations applied via `update_from_git.sh --db`.
-   **Microservices**: Separation of concerns with dedicated Python ML backend and Node.js API backend.

## External Dependencies
-   **React**: Frontend framework.
-   **FastAPI**: Python web framework for the ML backend.
-   **PostgreSQL**: Primary database for storing router configurations, network logs, detections, and whitelist entries.
-   **MikroTik API REST**: Used for communication with MikroTik routers for configuration and IP blocking.
-   **ShadCN UI**: Frontend component library.
-   **TanStack Query**: Data fetching library for the frontend.
-   **Isolation Forest**: Machine Learning algorithm for anomaly detection.
-   **RSyslog**: Log collection daemon.
-   **Drizzle ORM**: Used for database schema definition and synchronization in the Node.js backend.
-   **Neon Database**: Cloud-native PostgreSQL service (used in Replit environment).
-   **pg (Node.js driver)**: Standard PostgreSQL driver for Node.js (used in AlmaLinux environment).
-   **psycopg2**: PostgreSQL adapter for Python.

## Fix Recenti (Novembre 2025)

### 🚨 Database Full - Auto-Cleanup Fix (21 Nov 2025 - 18:00)
- **Problema**: Database PostgreSQL pieno con **417 MILIONI di log** accumulati
  - Syslog parser ha processato 417.7M righe senza limite di retention
  - Errore: `could not extend file: No space left on device`
  - Tutte le tabelle vuote perché database non accetta più scritture
- **Causa**: Nessuna pulizia automatica dei vecchi log (retention infinita)
- **Soluzione**:
  - Script `cleanup_old_logs.sql`: Mantiene solo ultimi 7 giorni di `network_logs`
  - Script `cleanup_database.sh`: Wrapper per esecuzione manuale/cron
  - Script `setup_cron_cleanup.sh`: Configura cron job giornaliero (ore 03:00)
- **Fix Immediato sul Server**:
  ```bash
  # 1. Pulisci manualmente log vecchi
  psql $DATABASE_URL << 'EOF'
  DELETE FROM network_logs WHERE timestamp < NOW() - INTERVAL '7 days';
  VACUUM FULL network_logs;
  EOF
  
  # 2. Setup pulizia automatica giornaliera
  sudo /opt/ids/deployment/setup_cron_cleanup.sh
  ```
- **Risultato Atteso**:
  - Database ridotto da centinaia di GB a pochi GB
  - Retention 7 giorni sufficiente per training ML
  - Pulizia automatica previene saturazione futura

### ✅ Database Driver Fix - Dual Mode Neon/PostgreSQL (21 Nov 2025 - 17:40)
- **Problema**: Frontend Node.js falliva con errore 500 su tutte le query database
- **Causa**: `@neondatabase/serverless` usa WebSocket ed è compatibile SOLO con Neon Cloud, non con PostgreSQL locale
- **Soluzione**: Dual-mode driver in `server/db.ts` con auto-detection ambiente
- **Risultato**: Funziona su Replit (Neon) e AlmaLinux (PostgreSQL standard) ✅