.replit

@@ -14,6 +14,22 @@ run = ["npm", "run", "start"]
 localPort = 5000
 externalPort = 80
 
+[[ports]]
+localPort = 41303
+externalPort = 3002
+
+[[ports]]
+localPort = 43471
+externalPort = 3003
+
+[[ports]]
+localPort = 43803
+externalPort = 3000
+
+[[ports]]
+localPort = 45059
+externalPort = 3001
+
 [env]
 PORT = "5000"
 

MIKROTIK_API_FIX.md

@@ -2,63 +2,33 @@
 
 ## 🐛 PROBLEMA RISOLTO
 
-**Errore**: Timeout connessione API MikroTik - router non rispondeva a richieste HTTP.
+**Errore**: "All connection attempts failed" quando si tenta di bloccare IP sui router MikroTik.
 
-**Causa Root**: Confusione tra **API Binary** (porta 8728) e **API REST** (porta 80/443).
+**Causa Root**: Bug nel file `python_ml/mikrotik_manager.py` - la porta API non veniva usata nella connessione HTTP.
 
-## 🔍 API MikroTik: Binary vs REST
+### Bug Originale (Riga 36)
-
+```python
-MikroTik RouterOS ha **DUE tipi di API completamente diversi**:
+base_url=f"http://{router_ip}"  # ❌ Porta non specificata!
-
-| Tipo | Porta | Protocollo | RouterOS | Compatibilità |
-|------|-------|------------|----------|---------------|
-| **Binary API** | 8728 | Proprietario RouterOS | Tutte | ❌ Non HTTP (libreria `routeros-api`) |
-| **REST API** | 80/443 | HTTP/HTTPS standard | **>= 7.1** | ✅ HTTP con `httpx` |
-
-**IDS usa REST API** (httpx + HTTP), quindi:
-- ✅ **Porta 80** (HTTP) - **CONSIGLIATA**
-- ✅ **Porta 443** (HTTPS) - Se necessario SSL
-- ❌ **Porta 8728** - API Binary, NON REST (timeout)
-- ❌ **Porta 8729** - API Binary SSL, NON REST (timeout)
-
-## ✅ SOLUZIONE
-
-### 1️⃣ Verifica RouterOS Versione
-
-```bash
-# Sul router MikroTik (via Winbox/SSH)
-/system resource print
 ```
 
-**Se RouterOS >= 7.1** → Usa **REST API** (porta 80/443)  
+Il codice si connetteva sempre a:
-**Se RouterOS < 7.1** → REST API non esiste, usa API Binary
+- `http://185.203.24.2` (porta 80 HTTP standard)
 
-### 2️⃣ Configurazione Porta Corretta
+Invece di:
+- `http://185.203.24.2:8728` (porta API REST MikroTik)
+- `https://185.203.24.2:8729` (porta API-SSL REST MikroTik)
 
-**Per RouterOS 7.14.2 (Alfabit):**
+### Fix Applicato
-
+```python
-```sql
+protocol = "https" if use_ssl or port == 8729 else "http"
--- Database: Usa porta 80 (REST API HTTP)
+base_url=f"{protocol}://{router_ip}:{port}"  # ✅ Porta corretta!
-UPDATE routers SET api_port = 80 WHERE name = 'Alfabit';
 ```
 
-**Porte disponibili**:
+Ora il codice:
-- **80** → REST API HTTP (✅ CONSIGLIATA)
+1. ✅ Usa la porta configurata nel database (`api_port`)
-- **443** → REST API HTTPS (se SSL richiesto)
+2. ✅ Auto-rileva SSL se porta = 8729
-- ~~8728~~ → API Binary (non compatibile)
+3. ✅ Supporta certificati self-signed (`verify=False`)
-- ~~8729~~ → API Binary SSL (non compatibile)
+4. ✅ Include porta nella URL di connessione
-
-### 3️⃣ Test Manuale
-
-```bash
-# Test connessione porta 80
-curl http://185.203.24.2:80/rest/system/identity \
-  -u admin:password \
-  --max-time 5
-
-# Output atteso:
-# {"name":"AlfaBit"}
-```
 
 ---
 
@@ -75,37 +45,59 @@ psql $DATABASE_URL -c "SELECT name, ip_address, api_port, username, enabled FROM
 ```
 name          | ip_address    | api_port | username | enabled
 --------------+---------------+----------+----------+---------
-Alfabit       | 185.203.24.2  | 80       | admin    | t
+Router Main   | 185.203.24.2  | 8728     | admin    | t
+Router Office | 10.0.1.1      | 8729     | admin    | t
 ```
 
 **Verifica**:
-- ✅ `api_port` = **80** (REST API HTTP)
+- ✅ `api_port` = **8728** (HTTP) o **8729** (HTTPS)
 - ✅ `enabled` = **true**
 - ✅ `username` e `password` corretti
 
-**Se porta errata**:
+### 2️⃣ Testa Connessione Manualmente
-```sql
--- Cambia porta da 8728 a 80
-UPDATE routers SET api_port = 80 WHERE ip_address = '185.203.24.2';
-```
-
-### 2️⃣ Testa Connessione Python
 
 ```bash
 # Su AlmaLinux
 cd /opt/ids/python_ml
 source venv/bin/activate
 
-# Test connessione automatico (usa dati dal database)
+# Test connessione (sostituisci con IP/porta reali)
-python3 test_mikrotik_connection.py
+python3 << 'EOF'
-```
+import asyncio
+from mikrotik_manager import MikroTikManager
 
-**Output atteso**:
+async def test():
-```
+    manager = MikroTikManager()
-✅ Connessione OK!
+    
-✅ Trovati X IP in lista 'ddos_blocked'
+    # Test router (SOSTITUISCI con dati reali dal database)
-✅ IP bloccato con successo!
+    result = await manager.test_connection(
-✅ IP sbloccato con successo!
+        router_ip="185.203.24.2",
+        username="admin",          # Dal database
+        password="your_password",  # Dal database
+        port=8728                  # Dal database
+    )
+    
+    print(f"Connessione: {'✅ OK' if result else '❌ FALLITA'}")
+    
+    if result:
+        # Test blocco IP
+        print("\nTest blocco IP 1.2.3.4...")
+        blocked = await manager.add_address_list(
+            router_ip="185.203.24.2",
+            username="admin",
+            password="your_password",
+            ip_address="1.2.3.4",
+            list_name="ddos_test",
+            comment="Test IDS API Fix",
+            timeout_duration="5m",
+            port=8728
+        )
+        print(f"Blocco: {'✅ OK' if blocked else '❌ FALLITO'}")
+    
+    await manager.close_all()
+
+asyncio.run(test())
+EOF
 ```
 
 ---
@@ -167,31 +159,26 @@ curl http://localhost:8000/health
 
 ### Connessione Ancora Fallisce?
 
-#### A. Verifica Servizio WWW su Router
+#### A. Verifica Firewall su Router
-
-**REST API usa servizio `www` (porta 80) o `www-ssl` (porta 443)**:
-
 ```bash
-# Sul router MikroTik (via Winbox/SSH)
+# Sul router MikroTik (via winbox/SSH)
 /ip service print
 
-# Verifica che www sia enabled:
+# Verifica che api o api-ssl sia enabled:
-# 0  www         80   *       ← REST API HTTP
+# 0  api         8728  *
-# 1  www-ssl    443   *       ← REST API HTTPS
+# 1  api-ssl     8729  *
 ```
 
 **Fix su MikroTik**:
-```bash
-# Abilita servizio www per REST API
-/ip service enable www
-/ip service set www port=80 address=0.0.0.0/0
-
-# O con SSL (porta 443)
-/ip service enable www-ssl
-/ip service set www-ssl port=443
 ```
+# Abilita API REST
+/ip service enable api
+/ip service set api port=8728
 
-**NOTA**: `api` (porta 8728) è **API Binary**, NON REST!
+# O con SSL
+/ip service enable api-ssl
+/ip service set api-ssl port=8729
+```
 
 #### B. Verifica Firewall AlmaLinux
 ```bash
@@ -202,20 +189,15 @@ sudo firewall-cmd --reload
 
 #### C. Test Connessione Raw
 ```bash
-# Test TCP connessione porta 80
+# Test TCP connessione porta 8728
-telnet 185.203.24.2 80
+telnet 185.203.24.2 8728
 
-# Test REST API con curl
+# O con curl
-curl -v http://185.203.24.2:80/rest/system/identity \
+curl -v http://185.203.24.2:8728/rest/system/identity \
   -u admin:password \
   --max-time 5
-
-# Output atteso:
-# {"name":"AlfaBit"}
 ```
 
-**Se timeout**: Servizio `www` non abilitato sul router
-
 #### D. Credenziali Errate?
 ```sql
 -- Verifica credenziali nel database
@@ -255,57 +237,33 @@ Dopo il deployment, verifica che:
 
 ---
 
-## 📊 CONFIGURAZIONE CORRETTA
+## 📊 PARAMETRI API CORRETTI
 
-| Parametro | Valore (RouterOS >= 7.1) | Note |
+| Router Config | HTTP | HTTPS (SSL) |
-|-----------|--------------------------|------|
+|--------------|------|-------------|
-| **api_port** | **80** (HTTP) o **443** (HTTPS) | ✅ REST API |
+| **api_port** | 8728 | 8729 |
-| **Servizio Router** | `www` (HTTP) o `www-ssl` (HTTPS) | Abilita su MikroTik |
+| **Protocollo** | http | https |
-| **Endpoint** | `/rest/system/identity` | Test connessione |
+| **Endpoint** | `/rest/ip/firewall/address-list` | `/rest/ip/firewall/address-list` |
-| **Endpoint** | `/rest/ip/firewall/address-list` | Gestione blocchi |
+| **Auth** | Basic (username:password) | Basic (username:password) |
-| **Auth** | Basic (username:password base64) | Header Authorization |
+| **Verify SSL** | N/A | False (self-signed certs) |
-| **Verify SSL** | False | Self-signed certs OK |
 
 ---
 
 ## 🎯 RIEPILOGO
 
-### ❌ ERRATO (API Binary - Timeout)
+**Prima** (BUG):
-```bash
+```
-# Porta 8728 usa protocollo BINARIO, non HTTP REST
+http://185.203.24.2/rest/...  ❌ Porta 80 (HTTP standard) - FALLISCE
-curl http://185.203.24.2:8728/rest/...
-# Timeout: protocollo incompatibile
 ```
 
-### ✅ CORRETTO (API REST - Funziona)
+**Dopo** (FIX):
-```bash
-# Porta 80 usa protocollo HTTP REST standard
-curl http://185.203.24.2:80/rest/system/identity \
-  -u admin:password
-
-# Output: {"name":"AlfaBit"}
 ```
-
+http://185.203.24.2:8728/rest/...  ✅ Porta 8728 (API REST) - FUNZIONA
-**Database configurato**:
+https://185.203.24.2:8729/rest/... ✅ Porta 8729 (API-SSL) - FUNZIONA
-```sql
--- Router Alfabit configurato con porta 80
-SELECT name, ip_address, api_port FROM routers;
--- Alfabit | 185.203.24.2 | 80
 ```
 
 ---
 
-## 📝 CHANGELOG
+**Fix applicato**: 25 Novembre 2024  
-
+**Versione ML Backend**: 2.0.0 (Hybrid Detector)  
-**25 Novembre 2024**:
+**Test richiesto**: ✅ Connessione + Blocco IP manuale
-1. ✅ Identificato problema: porta 8728 = API Binary (non HTTP)
-2. ✅ Verificato RouterOS 7.14.2 supporta REST API
-3. ✅ Configurato router con porta 80 (REST API HTTP)
-4. ✅ Test curl manuale: `{"name":"AlfaBit"}` ✅
-5. ✅ Router inserito in database con porta 80
-
-**Test richiesto**: `python3 test_mikrotik_connection.py`
-
-**Versione**: IDS 2.0.0 (Hybrid Detector)  
-**RouterOS**: 7.14.2 (stable)  
-**API Type**: REST (HTTP porta 80)

attached_assets/Pasted--journalctl-u-ids-list-fetcher-n-50-no-pager-Jan-02-15-_1767364928312.txt

@@ -1,51 +0,0 @@
- journalctl -u ids-list-fetcher -n 50 --no-pager
-Jan 02 15:30:01 ids.alfacom.it ids-list-fetcher[9296]:   Skipped (whitelisted):       0
-Jan 02 15:30:01 ids.alfacom.it ids-list-fetcher[9296]: ============================================================
-Jan 02 15:30:01 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 15:30:01 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.
-Jan 02 15:40:00 ids.alfacom.it systemd[1]: Starting IDS Public Lists Fetcher Service...
-Jan 02 15:40:00 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:00 ids.alfacom.it ids-list-fetcher[9493]: [2026-01-02 15:40:00] PUBLIC LISTS SYNC
-Jan 02 15:40:00 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:00 ids.alfacom.it ids-list-fetcher[9493]: Found 2 enabled lists
-Jan 02 15:40:00 ids.alfacom.it ids-list-fetcher[9493]: [15:40:00] Downloading Spamhaus from https://www.spamhaus.org/drop/drop_v4.json...
-Jan 02 15:40:00 ids.alfacom.it ids-list-fetcher[9493]: [15:40:00] Downloading AWS from https://ip-ranges.amazonaws.com/ip-ranges.json...
-Jan 02 15:40:00 ids.alfacom.it ids-list-fetcher[9493]: [15:40:00] Parsing AWS...
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: [15:40:01] Found 9548 IPs, syncing to database...
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: [15:40:01] ✓ AWS: +0 -0 ~9511
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: [15:40:01] Parsing Spamhaus...
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: [15:40:01] Found 1468 IPs, syncing to database...
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: [15:40:01] ✓ Spamhaus: +0 -0 ~1464
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: SYNC SUMMARY
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: Success: 2/2
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: Errors:  0/2
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: Total IPs Added:   0
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: Total IPs Removed: 0
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: RUNNING MERGE LOGIC
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ERROR:merge_logic:Failed to cleanup detections: operator does not exist: inet = text
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: LINE 9:                             d.source_ip::inet = wl.ip_inet
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:                                                       ^
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ERROR:merge_logic:Failed to sync detections: operator does not exist: inet = text
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: LINE 29:                             bl.ip_inet = wl.ip_inet
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:                                                 ^
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: Traceback (most recent call last):
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:   File "/opt/ids/python_ml/merge_logic.py", line 264, in sync_public_blacklist_detections
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:     cur.execute("""
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: psycopg2.errors.UndefinedFunction: operator does not exist: inet = text
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: LINE 29:                             bl.ip_inet = wl.ip_inet
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:                                                 ^
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: Merge Logic Stats:
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:   Created detections:          0
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:   Cleaned invalid detections:  0
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]:   Skipped (whitelisted):       0
-Jan 02 15:40:01 ids.alfacom.it ids-list-fetcher[9493]: ============================================================
-Jan 02 15:40:01 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 15:40:01 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.

attached_assets/Pasted--journalctl-u-ids-list-fetcher-n-50-no-pager-Jan-02-17-_1767370468601.txt

@@ -1,51 +0,0 @@
- journalctl -u ids-list-fetcher -n 50 --no-pager
-Jan 02 17:10:02 ids.alfacom.it ids-list-fetcher[2139]: ============================================================
-Jan 02 17:10:02 ids.alfacom.it ids-list-fetcher[2139]: ============================================================
-Jan 02 17:10:02 ids.alfacom.it ids-list-fetcher[2139]: RUNNING MERGE LOGIC
-Jan 02 17:10:02 ids.alfacom.it ids-list-fetcher[2139]: ============================================================
-Jan 02 17:10:12 ids.alfacom.it ids-list-fetcher[2139]: INFO:merge_logic:Bulk sync complete: {'created': 0, 'cleaned': 0, 'skipped_whitelisted': 0}
-Jan 02 17:10:12 ids.alfacom.it ids-list-fetcher[2139]: Merge Logic Stats:
-Jan 02 17:10:12 ids.alfacom.it ids-list-fetcher[2139]:   Created detections:          0
-Jan 02 17:10:12 ids.alfacom.it ids-list-fetcher[2139]:   Cleaned invalid detections:  0
-Jan 02 17:10:12 ids.alfacom.it ids-list-fetcher[2139]:   Skipped (whitelisted):       0
-Jan 02 17:10:12 ids.alfacom.it ids-list-fetcher[2139]: ============================================================
-Jan 02 17:10:12 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 17:10:12 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.
-Jan 02 17:12:35 ids.alfacom.it systemd[1]: Starting IDS Public Lists Fetcher Service...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [2026-01-02 17:12:35] PUBLIC LISTS SYNC
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: Found 4 enabled lists
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Downloading Spamhaus from https://www.spamhaus.org/drop/drop_v4.json...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Downloading AWS from https://ip-ranges.amazonaws.com/ip-ranges.json...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Downloading Google Cloud from https://www.gstatic.com/ipranges/cloud.json...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Downloading Google globali from https://www.gstatic.com/ipranges/goog.json...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Parsing AWS...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Found 9548 IPs, syncing to database...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] ✓ AWS: +0 -0 ~9548
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Parsing Google globali...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] ✗ Google globali: No valid IPs found in list
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Parsing Google Cloud...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] ✗ Google Cloud: No valid IPs found in list
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Parsing Spamhaus...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] Found 1468 IPs, syncing to database...
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: [17:12:35] ✓ Spamhaus: +0 -0 ~1468
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: SYNC SUMMARY
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: Success: 2/4
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: Errors:  2/4
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: Total IPs Added:   0
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: Total IPs Removed: 0
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: RUNNING MERGE LOGIC
-Jan 02 17:12:35 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:45 ids.alfacom.it ids-list-fetcher[2279]: INFO:merge_logic:Bulk sync complete: {'created': 0, 'cleaned': 0, 'skipped_whitelisted': 0}
-Jan 02 17:12:45 ids.alfacom.it ids-list-fetcher[2279]: Merge Logic Stats:
-Jan 02 17:12:45 ids.alfacom.it ids-list-fetcher[2279]:   Created detections:          0
-Jan 02 17:12:45 ids.alfacom.it ids-list-fetcher[2279]:   Cleaned invalid detections:  0
-Jan 02 17:12:45 ids.alfacom.it ids-list-fetcher[2279]:   Skipped (whitelisted):       0
-Jan 02 17:12:45 ids.alfacom.it ids-list-fetcher[2279]: ============================================================
-Jan 02 17:12:45 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 17:12:45 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.

attached_assets/Pasted-curl-X-POST-http-localhost-8000-detect-H-Content-Type-a_1767368587291.txt

@@ -1,4 +0,0 @@
-curl -X POST http://localhost:8000/detect \
-  -H "Content-Type: application/json" \
-  -d '{"max_records": 5000, "hours_back": 1, "risk_threshold": 80, "auto_block": true}'
-{"detections":[{"source_ip":"108.139.210.107","risk_score":98.55466848373413,"confidence_level":"high","action_recommendation":"auto_block","anomaly_type":"ddos","reason":"High connection rate: 403.7 conn/s","log_count":1211,"total_packets":1211,"total_bytes":2101702,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":95.0},{"source_ip":"216.58.209.54","risk_score":95.52801848493884,"confidence_level":"high","action_recommendation":"auto_block","anomaly_type":"brute_force","reason":"High connection rate: 184.7 conn/s","log_count":554,"total_packets":554,"total_bytes":782397,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":95.0},{"source_ip":"95.127.69.202","risk_score":93.58280514393482,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"brute_force","reason":"High connection rate: 93.7 conn/s","log_count":281,"total_packets":281,"total_bytes":369875,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":75.0},{"source_ip":"95.127.72.207","risk_score":92.50694363471318,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"brute_force","reason":"High connection rate: 76.3 conn/s","log_count":229,"total_packets":229,"total_bytes":293439,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":75.0},{"source_ip":"95.110.183.67","risk_score":86.42278405656512,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"brute_force","reason":"High connection rate: 153.0 conn/s","log_count":459,"total_packets":459,"total_bytes":20822,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":75.0},{"source_ip":"54.75.71.86","risk_score":83.42037059381207,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"brute_force","reason":"High connection rate: 58.0 conn/s","log_count":174,"total_packets":174,"total_bytes":25857,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":75.0},{"source_ip":"79.10.127.217","risk_score":82.32814469102843,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"brute_force","reason":"High connection rate: 70.0 conn/s","log_count":210,"total_packets":210,"total_bytes":18963,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":75.0},{"source_ip":"142.251.140.100","risk_score":76.61422108557721,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"botnet","reason":"Anomalous pattern detected (botnet)","log_count":16,"total_packets":16,"total_bytes":20056,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:53","confidence":75.0},{"source_ip":"142.250.181.161","risk_score":76.3802033958719,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"botnet","reason":"Anomalous pattern detected (botnet)","log_count":15,"total_packets":15,"total_bytes":5214,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:51","confidence":75.0},{"source_ip":"142.250.180.131","risk_score":72.7723405111559,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"suspicious","reason":"Anomalous pattern detected (suspicious)","log_count":8,"total_packets":8,"total_bytes":5320,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:53","confidence":75.0},{"source_ip":"157.240.231.60","risk_score":72.26853648050493,"confidence_level":"medium","action_recommendation":"manual_review","anomaly_type":"botnet","reason":"Anomalous pattern detected (botnet)","log_count":16,"total_packets":16,"total_bytes":4624,"first_seen":"2026-01-02T16:41:51","last_seen":"2026-01-02T16:41:54","confidence":75.0}],"total":11,"blocked":0,"message":"Trovate 11 anomalie"}[root@ids python_ml]# 

attached_assets/Pasted-journalctl-u-ids-list-fetcher-n-50-no-pager-Jan-02-12-5_1767354943964.txt

@@ -1,51 +0,0 @@
-journalctl -u ids-list-fetcher -n 50 --no-pager
-Jan 02 12:50:02 ids.alfacom.it ids-list-fetcher[5900]: ============================================================
-Jan 02 12:50:02 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 12:50:02 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.
-Jan 02 12:54:56 ids.alfacom.it systemd[1]: Starting IDS Public Lists Fetcher Service...
-Jan 02 12:54:56 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:56 ids.alfacom.it ids-list-fetcher[6290]: [2026-01-02 12:54:56] PUBLIC LISTS SYNC
-Jan 02 12:54:56 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:56 ids.alfacom.it ids-list-fetcher[6290]: Found 2 enabled lists
-Jan 02 12:54:56 ids.alfacom.it ids-list-fetcher[6290]: [12:54:56] Downloading Spamhaus from https://www.spamhaus.org/drop/drop_v4.json...
-Jan 02 12:54:56 ids.alfacom.it ids-list-fetcher[6290]: [12:54:56] Downloading AWS from https://ip-ranges.amazonaws.com/ip-ranges.json...
-Jan 02 12:54:56 ids.alfacom.it ids-list-fetcher[6290]: [12:54:56] Parsing AWS...
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: [12:54:57] Found 9548 IPs, syncing to database...
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: [12:54:57] ✓ AWS: +0 -0 ~9511
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: [12:54:57] Parsing Spamhaus...
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: [12:54:57] Found 1468 IPs, syncing to database...
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: [12:54:57] ✗ Spamhaus: ON CONFLICT DO UPDATE command cannot affect row a second time
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: HINT:  Ensure that no rows proposed for insertion within the same command have duplicate constrained values.
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: SYNC SUMMARY
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: Success: 1/2
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: Errors:  1/2
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: Total IPs Added:   0
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: Total IPs Removed: 0
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: RUNNING MERGE LOGIC
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ERROR:merge_logic:Failed to cleanup detections: operator does not exist: inet = text
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: LINE 9:                             d.source_ip::inet = wl.ip_inet
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:                                                       ^
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ERROR:merge_logic:Failed to sync detections: operator does not exist: text <<= text
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: LINE 30:                             OR bl.ip_inet <<= wl.ip_inet
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:                                                    ^
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: Traceback (most recent call last):
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:   File "/opt/ids/python_ml/merge_logic.py", line 264, in sync_public_blacklist_detections
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:     cur.execute("""
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: psycopg2.errors.UndefinedFunction: operator does not exist: text <<= text
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: LINE 30:                             OR bl.ip_inet <<= wl.ip_inet
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:                                                    ^
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: Merge Logic Stats:
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:   Created detections:          0
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:   Cleaned invalid detections:  0
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]:   Skipped (whitelisted):       0
-Jan 02 12:54:57 ids.alfacom.it ids-list-fetcher[6290]: ============================================================
-Jan 02 12:54:57 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 12:54:57 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.

attached_assets/Pasted-journalctl-u-ids-list-fetcher-n-50-no-pager-Jan-02-16-1_1767366961971.txt

@@ -1,51 +0,0 @@
-journalctl -u ids-list-fetcher -n 50 --no-pager
-Jan 02 16:11:31 ids.alfacom.it ids-list-fetcher[10401]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 16:11:31 ids.alfacom.it ids-list-fetcher[10401]: Merge Logic Stats:
-Jan 02 16:11:31 ids.alfacom.it ids-list-fetcher[10401]:   Created detections:          0
-Jan 02 16:11:31 ids.alfacom.it ids-list-fetcher[10401]:   Cleaned invalid detections:  0
-Jan 02 16:11:31 ids.alfacom.it ids-list-fetcher[10401]:   Skipped (whitelisted):       0
-Jan 02 16:11:31 ids.alfacom.it ids-list-fetcher[10401]: ============================================================
-Jan 02 16:11:31 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 16:11:31 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.
-Jan 02 16:15:04 ids.alfacom.it systemd[1]: Starting IDS Public Lists Fetcher Service...
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: [2026-01-02 16:15:04] PUBLIC LISTS SYNC
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: Found 2 enabled lists
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: [16:15:04] Downloading Spamhaus from https://www.spamhaus.org/drop/drop_v4.json...
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: [16:15:04] Downloading AWS from https://ip-ranges.amazonaws.com/ip-ranges.json...
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: [16:15:04] Parsing Spamhaus...
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: [16:15:04] Found 1468 IPs, syncing to database...
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: [16:15:04] ✓ Spamhaus: +0 -0 ~1468
-Jan 02 16:15:04 ids.alfacom.it ids-list-fetcher[10801]: [16:15:04] Parsing AWS...
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: [16:15:05] Found 9548 IPs, syncing to database...
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: [16:15:05] ✓ AWS: +9548 -0 ~0
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: SYNC SUMMARY
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: Success: 2/2
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: Errors:  0/2
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: Total IPs Added:   9548
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: Total IPs Removed: 0
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: RUNNING MERGE LOGIC
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: ERROR:merge_logic:Failed to sync detections: column "risk_score" is of type numeric but expression is of type text
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: LINE 13:                         '75',
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]:                                  ^
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: HINT:  You will need to rewrite or cast the expression.
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: Traceback (most recent call last):
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]:   File "/opt/ids/python_ml/merge_logic.py", line 264, in sync_public_blacklist_detections
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]:     cur.execute("""
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: psycopg2.errors.DatatypeMismatch: column "risk_score" is of type numeric but expression is of type text
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: LINE 13:                         '75',
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]:                                  ^
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: HINT:  You will need to rewrite or cast the expression.
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: Merge Logic Stats:
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]:   Created detections:          0
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]:   Cleaned invalid detections:  0
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]:   Skipped (whitelisted):       0
-Jan 02 16:15:05 ids.alfacom.it ids-list-fetcher[10801]: ============================================================
-Jan 02 16:15:05 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 16:15:05 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.

attached_assets/Pasted-ournalctl-u-ids-list-fetcher-n-50-no-pager-Jan-02-12-30_1767354386554.txt

@@ -1,51 +0,0 @@
-ournalctl -u ids-list-fetcher -n 50 --no-pager
-Jan 02 12:30:01 ids.alfacom.it ids-list-fetcher[5571]:   Cleaned invalid detections:  0
-Jan 02 12:30:01 ids.alfacom.it ids-list-fetcher[5571]:   Skipped (whitelisted):       0
-Jan 02 12:30:01 ids.alfacom.it ids-list-fetcher[5571]: ============================================================
-Jan 02 12:30:01 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 12:30:01 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.
-Jan 02 12:40:01 ids.alfacom.it systemd[1]: Starting IDS Public Lists Fetcher Service...
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: [2026-01-02 12:40:01] PUBLIC LISTS SYNC
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: Found 2 enabled lists
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: [12:40:01] Downloading Spamhaus from https://www.spamhaus.org/drop/drop_v4.json...
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: [12:40:01] Downloading AWS from https://ip-ranges.amazonaws.com/ip-ranges.json...
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: [12:40:01] Parsing AWS...
-Jan 02 12:40:01 ids.alfacom.it ids-list-fetcher[5730]: [12:40:01] Found 9548 IPs, syncing to database...
-Jan 02 12:40:02 ids.alfacom.it ids-list-fetcher[5730]: [12:40:02] ✓ AWS: +9511 -0 ~0
-Jan 02 12:40:02 ids.alfacom.it ids-list-fetcher[5730]: [12:40:02] Parsing Spamhaus...
-Jan 02 12:40:02 ids.alfacom.it ids-list-fetcher[5730]: [12:40:02] ✗ Spamhaus: No valid IPs found in list
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: SYNC SUMMARY
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: Success: 1/2
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: Errors:  1/2
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: Total IPs Added:   9511
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: Total IPs Removed: 0
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: RUNNING MERGE LOGIC
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ERROR:merge_logic:Failed to cleanup detections: operator does not exist: inet = text
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: LINE 9:                             d.source_ip::inet = wl.ip_inet
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:                                                       ^
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ERROR:merge_logic:Failed to sync detections: operator does not exist: text <<= text
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: LINE 30:                             OR bl.ip_inet <<= wl.ip_inet
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:                                                    ^
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: Traceback (most recent call last):
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:   File "/opt/ids/python_ml/merge_logic.py", line 264, in sync_public_blacklist_detections
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:     cur.execute("""
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: psycopg2.errors.UndefinedFunction: operator does not exist: text <<= text
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: LINE 30:                             OR bl.ip_inet <<= wl.ip_inet
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:                                                    ^
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: HINT:  No operator matches the given name and argument types. You might need to add explicit type casts.
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: Merge Logic Stats:
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:   Created detections:          0
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:   Cleaned invalid detections:  0
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]:   Skipped (whitelisted):       0
-Jan 02 12:40:03 ids.alfacom.it ids-list-fetcher[5730]: ============================================================
-Jan 02 12:40:03 ids.alfacom.it systemd[1]: ids-list-fetcher.service: Deactivated successfully.
-Jan 02 12:40:03 ids.alfacom.it systemd[1]: Finished IDS Public Lists Fetcher Service.

client/src/App.tsx

@@ -4,14 +4,13 @@ import { QueryClientProvider } from "@tanstack/react-query";
 import { Toaster } from "@/components/ui/toaster";
 import { TooltipProvider } from "@/components/ui/tooltip";
 import { SidebarProvider, Sidebar, SidebarContent, SidebarGroup, SidebarGroupContent, SidebarGroupLabel, SidebarMenu, SidebarMenuButton, SidebarMenuItem, SidebarTrigger } from "@/components/ui/sidebar";
-import { LayoutDashboard, AlertTriangle, Server, Shield, Brain, Menu, Activity, BarChart3, TrendingUp, List } from "lucide-react";
+import { LayoutDashboard, AlertTriangle, Server, Shield, Brain, Menu, Activity, BarChart3, TrendingUp } from "lucide-react";
 import Dashboard from "@/pages/Dashboard";
 import Detections from "@/pages/Detections";
 import DashboardLive from "@/pages/DashboardLive";
 import AnalyticsHistory from "@/pages/AnalyticsHistory";
 import Routers from "@/pages/Routers";
 import Whitelist from "@/pages/Whitelist";
-import PublicLists from "@/pages/PublicLists";
 import Training from "@/pages/Training";
 import Services from "@/pages/Services";
 import NotFound from "@/pages/not-found";
@@ -24,7 +23,6 @@ const menuItems = [
   { title: "Training ML", url: "/training", icon: Brain },
   { title: "Router", url: "/routers", icon: Server },
   { title: "Whitelist", url: "/whitelist", icon: Shield },
-  { title: "Liste Pubbliche", url: "/public-lists", icon: List },
   { title: "Servizi", url: "/services", icon: TrendingUp },
 ];
 
@@ -64,7 +62,6 @@ function Router() {
       <Route path="/training" component={Training} />
       <Route path="/routers" component={Routers} />
       <Route path="/whitelist" component={Whitelist} />
-      <Route path="/public-lists" component={PublicLists} />
       <Route path="/services" component={Services} />
       <Route component={NotFound} />
     </Switch>

client/src/pages/Detections.tsx

@@ -5,81 +5,44 @@ import { Button } from "@/components/ui/button";
 import { Input } from "@/components/ui/input";
 import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
 import { Slider } from "@/components/ui/slider";
-import { AlertTriangle, Search, Shield, Globe, MapPin, Building2, ShieldPlus, ShieldCheck, Unlock, ChevronLeft, ChevronRight } from "lucide-react";
+import { AlertTriangle, Search, Shield, Globe, MapPin, Building2, ShieldPlus } from "lucide-react";
 import { format } from "date-fns";
-import { useState, useEffect, useMemo } from "react";
+import { useState } from "react";
-import type { Detection, Whitelist } from "@shared/schema";
+import type { Detection } from "@shared/schema";
 import { getFlag } from "@/lib/country-flags";
 import { apiRequest, queryClient } from "@/lib/queryClient";
 import { useToast } from "@/hooks/use-toast";
 
-const ITEMS_PER_PAGE = 50;
-
-interface DetectionsResponse {
-  detections: Detection[];
-  total: number;
-}
-
 export default function Detections() {
-  const [searchInput, setSearchInput] = useState("");
+  const [searchQuery, setSearchQuery] = useState("");
-  const [debouncedSearch, setDebouncedSearch] = useState("");
   const [anomalyTypeFilter, setAnomalyTypeFilter] = useState<string>("all");
   const [minScore, setMinScore] = useState(0);
   const [maxScore, setMaxScore] = useState(100);
-  const [currentPage, setCurrentPage] = useState(1);
   const { toast } = useToast();
 
-  // Debounce search input
+  // Build query params
-  useEffect(() => {
+  const queryParams = new URLSearchParams();
-    const timer = setTimeout(() => {
+  queryParams.set("limit", "5000");
-      setDebouncedSearch(searchInput);
+  if (anomalyTypeFilter !== "all") {
-      setCurrentPage(1); // Reset to first page on search
+    queryParams.set("anomalyType", anomalyTypeFilter);
-    }, 300);
+  }
-    return () => clearTimeout(timer);
+  if (minScore > 0) {
-  }, [searchInput]);
+    queryParams.set("minScore", minScore.toString());
+  }
+  if (maxScore < 100) {
+    queryParams.set("maxScore", maxScore.toString());
+  }
 
-  // Reset page on filter change
+  const { data: detections, isLoading } = useQuery<Detection[]>({
-  useEffect(() => {
+    queryKey: ["/api/detections", anomalyTypeFilter, minScore, maxScore],
-    setCurrentPage(1);
+    queryFn: () => fetch(`/api/detections?${queryParams.toString()}`).then(r => r.json()),
-  }, [anomalyTypeFilter, minScore, maxScore]);
+    refetchInterval: 5000,
-
-  // Build query params with pagination and search
-  const queryParams = useMemo(() => {
-    const params = new URLSearchParams();
-    params.set("limit", ITEMS_PER_PAGE.toString());
-    params.set("offset", ((currentPage - 1) * ITEMS_PER_PAGE).toString());
-    if (anomalyTypeFilter !== "all") {
-      params.set("anomalyType", anomalyTypeFilter);
-    }
-    if (minScore > 0) {
-      params.set("minScore", minScore.toString());
-    }
-    if (maxScore < 100) {
-      params.set("maxScore", maxScore.toString());
-    }
-    if (debouncedSearch.trim()) {
-      params.set("search", debouncedSearch.trim());
-    }
-    return params.toString();
-  }, [currentPage, anomalyTypeFilter, minScore, maxScore, debouncedSearch]);
-
-  const { data, isLoading } = useQuery<DetectionsResponse>({
-    queryKey: ["/api/detections", currentPage, anomalyTypeFilter, minScore, maxScore, debouncedSearch],
-    queryFn: () => fetch(`/api/detections?${queryParams}`).then(r => r.json()),
-    refetchInterval: 10000,
   });
 
-  const detections = data?.detections || [];
+  const filteredDetections = detections?.filter((d) =>
-  const totalCount = data?.total || 0;
+    d.sourceIp.toLowerCase().includes(searchQuery.toLowerCase()) ||
-  const totalPages = Math.ceil(totalCount / ITEMS_PER_PAGE);
+    d.anomalyType.toLowerCase().includes(searchQuery.toLowerCase())
-
+  );
-  // Fetch whitelist to check if IP is already whitelisted
-  const { data: whitelistData } = useQuery<Whitelist[]>({
-    queryKey: ["/api/whitelist"],
-  });
-
-  // Create a Set of whitelisted IPs for fast lookup
-  const whitelistedIps = new Set(whitelistData?.map(w => w.ipAddress) || []);
 
   // Mutation per aggiungere a whitelist
   const addToWhitelistMutation = useMutation({
@@ -92,7 +55,7 @@ export default function Detections() {
     onSuccess: (_, detection) => {
       toast({
         title: "IP aggiunto alla whitelist",
-        description: `${detection.sourceIp} è stato aggiunto alla whitelist e sbloccato dai router.`,
+        description: `${detection.sourceIp} è stato aggiunto alla whitelist con successo.`,
       });
       queryClient.invalidateQueries({ queryKey: ["/api/whitelist"] });
       queryClient.invalidateQueries({ queryKey: ["/api/detections"] });
@@ -106,29 +69,6 @@ export default function Detections() {
     }
   });
 
-  // Mutation per sbloccare IP dai router
-  const unblockMutation = useMutation({
-    mutationFn: async (detection: Detection) => {
-      return await apiRequest("POST", "/api/unblock-ip", {
-        ipAddress: detection.sourceIp
-      });
-    },
-    onSuccess: (data: any, detection) => {
-      toast({
-        title: "IP sbloccato",
-        description: `${detection.sourceIp} è stato rimosso dalla blocklist di ${data.unblocked_from || 0} router.`,
-      });
-      queryClient.invalidateQueries({ queryKey: ["/api/detections"] });
-    },
-    onError: (error: any, detection) => {
-      toast({
-        title: "Errore sblocco",
-        description: error.message || `Impossibile sbloccare ${detection.sourceIp} dai router.`,
-        variant: "destructive",
-      });
-    }
-  });
-
   const getRiskBadge = (riskScore: string) => {
     const score = parseFloat(riskScore);
     if (score >= 85) return <Badge variant="destructive">CRITICO</Badge>;
@@ -166,9 +106,9 @@ export default function Detections() {
               <div className="relative flex-1 min-w-[200px]">
                 <Search className="absolute left-3 top-1/2 -translate-y-1/2 h-4 w-4 text-muted-foreground" />
                 <Input
-                  placeholder="Cerca per IP, paese, organizzazione..."
+                  placeholder="Cerca per IP o tipo anomalia..."
-                  value={searchInput}
+                  value={searchQuery}
-                  onChange={(e) => setSearchInput(e.target.value)}
+                  onChange={(e) => setSearchQuery(e.target.value)}
                   className="pl-9"
                   data-testid="input-search"
                 />
@@ -220,36 +160,9 @@ export default function Detections() {
       {/* Detections List */}
       <Card data-testid="card-detections-list">
         <CardHeader>
-          <CardTitle className="flex items-center justify-between gap-2 flex-wrap">
+          <CardTitle className="flex items-center gap-2">
-            <div className="flex items-center gap-2">
+            <AlertTriangle className="h-5 w-5" />
-              <AlertTriangle className="h-5 w-5" />
+            Rilevamenti ({filteredDetections?.length || 0})
-              Rilevamenti ({totalCount})
-            </div>
-            {totalPages > 1 && (
-              <div className="flex items-center gap-2 text-sm font-normal">
-                <Button 
-                  variant="outline" 
-                  size="icon"
-                  onClick={() => setCurrentPage(p => Math.max(1, p - 1))}
-                  disabled={currentPage === 1}
-                  data-testid="button-prev-page"
-                >
-                  <ChevronLeft className="h-4 w-4" />
-                </Button>
-                <span data-testid="text-pagination">
-                  Pagina {currentPage} di {totalPages}
-                </span>
-                <Button 
-                  variant="outline" 
-                  size="icon"
-                  onClick={() => setCurrentPage(p => Math.min(totalPages, p + 1))}
-                  disabled={currentPage === totalPages}
-                  data-testid="button-next-page"
-                >
-                  <ChevronRight className="h-4 w-4" />
-                </Button>
-              </div>
-            )}
           </CardTitle>
         </CardHeader>
         <CardContent>
@@ -257,9 +170,9 @@ export default function Detections() {
             <div className="text-center py-8 text-muted-foreground" data-testid="text-loading">
               Caricamento...
             </div>
-          ) : detections.length > 0 ? (
+          ) : filteredDetections && filteredDetections.length > 0 ? (
             <div className="space-y-3">
-              {detections.map((detection) => (
+              {filteredDetections.map((detection) => (
                 <div
                   key={detection.id}
                   className="p-4 rounded-lg border hover-elevate"
@@ -365,44 +278,17 @@ export default function Detections() {
                         </Badge>
                       )}
 
-                      {whitelistedIps.has(detection.sourceIp) ? (
+                      <Button 
-                        <Button 
+                        variant="outline" 
-                          variant="outline" 
+                        size="sm" 
-                          size="sm" 
+                        onClick={() => addToWhitelistMutation.mutate(detection)}
-                          disabled
+                        disabled={addToWhitelistMutation.isPending}
-                          className="w-full bg-green-500/10 border-green-500 text-green-600 dark:text-green-400"
+                        className="w-full"
-                          data-testid={`button-whitelist-${detection.id}`}
+                        data-testid={`button-whitelist-${detection.id}`}
-                        >
+                      >
-                          <ShieldCheck className="h-3 w-3 mr-1" />
+                        <ShieldPlus className="h-3 w-3 mr-1" />
-                          In Whitelist
+                        Whitelist
-                        </Button>
+                      </Button>
-                      ) : (
-                        <Button 
-                          variant="outline" 
-                          size="sm" 
-                          onClick={() => addToWhitelistMutation.mutate(detection)}
-                          disabled={addToWhitelistMutation.isPending}
-                          className="w-full"
-                          data-testid={`button-whitelist-${detection.id}`}
-                        >
-                          <ShieldPlus className="h-3 w-3 mr-1" />
-                          Whitelist
-                        </Button>
-                      )}
-
-                      {detection.blocked && (
-                        <Button 
-                          variant="outline" 
-                          size="sm" 
-                          onClick={() => unblockMutation.mutate(detection)}
-                          disabled={unblockMutation.isPending}
-                          className="w-full"
-                          data-testid={`button-unblock-${detection.id}`}
-                        >
-                          <Unlock className="h-3 w-3 mr-1" />
-                          Sblocca Router
-                        </Button>
-                      )}
                     </div>
                   </div>
                 </div>
@@ -412,40 +298,11 @@ export default function Detections() {
             <div className="text-center py-12 text-muted-foreground" data-testid="text-no-results">
               <AlertTriangle className="h-12 w-12 mx-auto mb-2 opacity-50" />
               <p>Nessun rilevamento trovato</p>
-              {debouncedSearch && (
+              {searchQuery && (
                 <p className="text-sm">Prova con un altro termine di ricerca</p>
               )}
             </div>
           )}
-          
-          {/* Bottom pagination */}
-          {totalPages > 1 && detections.length > 0 && (
-            <div className="flex items-center justify-center gap-4 mt-6 pt-4 border-t">
-              <Button 
-                variant="outline" 
-                size="sm"
-                onClick={() => setCurrentPage(p => Math.max(1, p - 1))}
-                disabled={currentPage === 1}
-                data-testid="button-prev-page-bottom"
-              >
-                <ChevronLeft className="h-4 w-4 mr-1" />
-                Precedente
-              </Button>
-              <span className="text-sm text-muted-foreground" data-testid="text-pagination-bottom">
-                Pagina {currentPage} di {totalPages} ({totalCount} totali)
-              </span>
-              <Button 
-                variant="outline" 
-                size="sm"
-                onClick={() => setCurrentPage(p => Math.min(totalPages, p + 1))}
-                disabled={currentPage === totalPages}
-                data-testid="button-next-page-bottom"
-              >
-                Successiva
-                <ChevronRight className="h-4 w-4 ml-1" />
-              </Button>
-            </div>
-          )}
         </CardContent>
       </Card>
     </div>

client/src/pages/PublicLists.tsx

@@ -1,372 +0,0 @@
-import { useQuery, useMutation } from "@tanstack/react-query";
-import { Card, CardContent, CardDescription, CardHeader, CardTitle } from "@/components/ui/card";
-import { Button } from "@/components/ui/button";
-import { Badge } from "@/components/ui/badge";
-import { Table, TableBody, TableCell, TableHead, TableHeader, TableRow } from "@/components/ui/table";
-import { Dialog, DialogContent, DialogDescription, DialogHeader, DialogTitle, DialogTrigger } from "@/components/ui/dialog";
-import { Form, FormControl, FormField, FormItem, FormLabel, FormMessage } from "@/components/ui/form";
-import { Input } from "@/components/ui/input";
-import { Select, SelectContent, SelectItem, SelectTrigger, SelectValue } from "@/components/ui/select";
-import { Switch } from "@/components/ui/switch";
-import { useForm } from "react-hook-form";
-import { zodResolver } from "@hookform/resolvers/zod";
-import { z } from "zod";
-import { RefreshCw, Plus, Trash2, Edit, CheckCircle2, XCircle, AlertTriangle, Clock } from "lucide-react";
-import { apiRequest, queryClient } from "@/lib/queryClient";
-import { useToast } from "@/hooks/use-toast";
-import { formatDistanceToNow } from "date-fns";
-import { it } from "date-fns/locale";
-import { useState } from "react";
-
-const listFormSchema = z.object({
-  name: z.string().min(1, "Nome richiesto"),
-  type: z.enum(["blacklist", "whitelist"], {
-    required_error: "Tipo richiesto",
-  }),
-  url: z.string().url("URL non valida"),
-  enabled: z.boolean().default(true),
-  fetchIntervalMinutes: z.number().min(1).max(1440).default(10),
-});
-
-type ListFormValues = z.infer<typeof listFormSchema>;
-
-export default function PublicLists() {
-  const { toast } = useToast();
-  const [isAddDialogOpen, setIsAddDialogOpen] = useState(false);
-  const [editingList, setEditingList] = useState<any>(null);
-
-  const { data: lists, isLoading } = useQuery({
-    queryKey: ["/api/public-lists"],
-  });
-
-  const form = useForm<ListFormValues>({
-    resolver: zodResolver(listFormSchema),
-    defaultValues: {
-      name: "",
-      type: "blacklist",
-      url: "",
-      enabled: true,
-      fetchIntervalMinutes: 10,
-    },
-  });
-
-  const createMutation = useMutation({
-    mutationFn: (data: ListFormValues) =>
-      apiRequest("POST", "/api/public-lists", data),
-    onSuccess: () => {
-      queryClient.invalidateQueries({ queryKey: ["/api/public-lists"] });
-      toast({
-        title: "Lista creata",
-        description: "La lista è stata aggiunta con successo",
-      });
-      setIsAddDialogOpen(false);
-      form.reset();
-    },
-    onError: (error: any) => {
-      toast({
-        title: "Errore",
-        description: error.message || "Impossibile creare la lista",
-        variant: "destructive",
-      });
-    },
-  });
-
-  const updateMutation = useMutation({
-    mutationFn: ({ id, data }: { id: string; data: Partial<ListFormValues> }) =>
-      apiRequest("PATCH", `/api/public-lists/${id}`, data),
-    onSuccess: () => {
-      queryClient.invalidateQueries({ queryKey: ["/api/public-lists"] });
-      toast({
-        title: "Lista aggiornata",
-        description: "Le modifiche sono state salvate",
-      });
-      setEditingList(null);
-    },
-  });
-
-  const deleteMutation = useMutation({
-    mutationFn: (id: string) =>
-      apiRequest("DELETE", `/api/public-lists/${id}`),
-    onSuccess: () => {
-      queryClient.invalidateQueries({ queryKey: ["/api/public-lists"] });
-      toast({
-        title: "Lista eliminata",
-        description: "La lista è stata rimossa",
-      });
-    },
-    onError: (error: any) => {
-      toast({
-        title: "Errore",
-        description: error.message || "Impossibile eliminare la lista",
-        variant: "destructive",
-      });
-    },
-  });
-
-  const syncMutation = useMutation({
-    mutationFn: (id: string) =>
-      apiRequest("POST", `/api/public-lists/${id}/sync`),
-    onSuccess: () => {
-      toast({
-        title: "Sync avviato",
-        description: "La sincronizzazione manuale è stata richiesta",
-      });
-    },
-  });
-
-  const toggleEnabled = (id: string, enabled: boolean) => {
-    updateMutation.mutate({ id, data: { enabled } });
-  };
-
-  const onSubmit = (data: ListFormValues) => {
-    createMutation.mutate(data);
-  };
-
-  const getStatusBadge = (list: any) => {
-    if (!list.enabled) {
-      return <Badge variant="outline" className="gap-1"><XCircle className="w-3 h-3" />Disabilitata</Badge>;
-    }
-
-    if (list.errorCount > 5) {
-      return <Badge variant="destructive" className="gap-1"><AlertTriangle className="w-3 h-3" />Errori</Badge>;
-    }
-
-    if (list.lastSuccess) {
-      return <Badge variant="default" className="gap-1 bg-green-600"><CheckCircle2 className="w-3 h-3" />OK</Badge>;
-    }
-
-    return <Badge variant="secondary" className="gap-1"><Clock className="w-3 h-3" />In attesa</Badge>;
-  };
-
-  const getTypeBadge = (type: string) => {
-    if (type === "blacklist") {
-      return <Badge variant="destructive">Blacklist</Badge>;
-    }
-    return <Badge variant="default" className="bg-blue-600">Whitelist</Badge>;
-  };
-
-  if (isLoading) {
-    return (
-      <div className="p-6">
-        <Card>
-          <CardHeader>
-            <CardTitle>Caricamento...</CardTitle>
-          </CardHeader>
-        </Card>
-      </div>
-    );
-  }
-
-  return (
-    <div className="p-6 space-y-6">
-      <div className="flex items-center justify-between">
-        <div>
-          <h1 className="text-3xl font-bold">Liste Pubbliche</h1>
-          <p className="text-muted-foreground mt-2">
-            Gestione sorgenti blacklist e whitelist esterne (aggiornamento ogni 10 minuti)
-          </p>
-        </div>
-        <Dialog open={isAddDialogOpen} onOpenChange={setIsAddDialogOpen}>
-          <DialogTrigger asChild>
-            <Button data-testid="button-add-list">
-              <Plus className="w-4 h-4 mr-2" />
-              Aggiungi Lista
-            </Button>
-          </DialogTrigger>
-          <DialogContent className="max-w-2xl">
-            <DialogHeader>
-              <DialogTitle>Aggiungi Lista Pubblica</DialogTitle>
-              <DialogDescription>
-                Configura una nuova sorgente blacklist o whitelist
-              </DialogDescription>
-            </DialogHeader>
-            <Form {...form}>
-              <form onSubmit={form.handleSubmit(onSubmit)} className="space-y-4">
-                <FormField
-                  control={form.control}
-                  name="name"
-                  render={({ field }) => (
-                    <FormItem>
-                      <FormLabel>Nome</FormLabel>
-                      <FormControl>
-                        <Input placeholder="es. Spamhaus DROP" {...field} data-testid="input-list-name" />
-                      </FormControl>
-                      <FormMessage />
-                    </FormItem>
-                  )}
-                />
-                <FormField
-                  control={form.control}
-                  name="type"
-                  render={({ field }) => (
-                    <FormItem>
-                      <FormLabel>Tipo</FormLabel>
-                      <Select onValueChange={field.onChange} defaultValue={field.value}>
-                        <FormControl>
-                          <SelectTrigger data-testid="select-list-type">
-                            <SelectValue placeholder="Seleziona tipo" />
-                          </SelectTrigger>
-                        </FormControl>
-                        <SelectContent>
-                          <SelectItem value="blacklist">Blacklist</SelectItem>
-                          <SelectItem value="whitelist">Whitelist</SelectItem>
-                        </SelectContent>
-                      </Select>
-                      <FormMessage />
-                    </FormItem>
-                  )}
-                />
-                <FormField
-                  control={form.control}
-                  name="url"
-                  render={({ field }) => (
-                    <FormItem>
-                      <FormLabel>URL</FormLabel>
-                      <FormControl>
-                        <Input placeholder="https://example.com/list.txt" {...field} data-testid="input-list-url" />
-                      </FormControl>
-                      <FormMessage />
-                    </FormItem>
-                  )}
-                />
-                <FormField
-                  control={form.control}
-                  name="fetchIntervalMinutes"
-                  render={({ field }) => (
-                    <FormItem>
-                      <FormLabel>Intervallo Sync (minuti)</FormLabel>
-                      <FormControl>
-                        <Input
-                          type="number"
-                          {...field}
-                          onChange={(e) => field.onChange(parseInt(e.target.value))}
-                          data-testid="input-list-interval"
-                        />
-                      </FormControl>
-                      <FormMessage />
-                    </FormItem>
-                  )}
-                />
-                <FormField
-                  control={form.control}
-                  name="enabled"
-                  render={({ field }) => (
-                    <FormItem className="flex items-center justify-between">
-                      <FormLabel>Abilitata</FormLabel>
-                      <FormControl>
-                        <Switch
-                          checked={field.value}
-                          onCheckedChange={field.onChange}
-                          data-testid="switch-list-enabled"
-                        />
-                      </FormControl>
-                    </FormItem>
-                  )}
-                />
-                <div className="flex justify-end gap-2 pt-4">
-                  <Button type="button" variant="outline" onClick={() => setIsAddDialogOpen(false)}>
-                    Annulla
-                  </Button>
-                  <Button type="submit" disabled={createMutation.isPending} data-testid="button-save-list">
-                    {createMutation.isPending ? "Salvataggio..." : "Salva"}
-                  </Button>
-                </div>
-              </form>
-            </Form>
-          </DialogContent>
-        </Dialog>
-      </div>
-
-      <Card>
-        <CardHeader>
-          <CardTitle>Sorgenti Configurate</CardTitle>
-          <CardDescription>
-            {lists?.length || 0} liste configurate
-          </CardDescription>
-        </CardHeader>
-        <CardContent>
-          <Table>
-            <TableHeader>
-              <TableRow>
-                <TableHead>Nome</TableHead>
-                <TableHead>Tipo</TableHead>
-                <TableHead>Stato</TableHead>
-                <TableHead>IP Totali</TableHead>
-                <TableHead>IP Attivi</TableHead>
-                <TableHead>Ultimo Sync</TableHead>
-                <TableHead className="text-right">Azioni</TableHead>
-              </TableRow>
-            </TableHeader>
-            <TableBody>
-              {lists?.map((list: any) => (
-                <TableRow key={list.id} data-testid={`row-list-${list.id}`}>
-                  <TableCell className="font-medium">
-                    <div>
-                      <div>{list.name}</div>
-                      <div className="text-xs text-muted-foreground truncate max-w-xs">
-                        {list.url}
-                      </div>
-                    </div>
-                  </TableCell>
-                  <TableCell>{getTypeBadge(list.type)}</TableCell>
-                  <TableCell>{getStatusBadge(list)}</TableCell>
-                  <TableCell data-testid={`text-total-ips-${list.id}`}>{list.totalIps?.toLocaleString() || 0}</TableCell>
-                  <TableCell data-testid={`text-active-ips-${list.id}`}>{list.activeIps?.toLocaleString() || 0}</TableCell>
-                  <TableCell>
-                    {list.lastSuccess ? (
-                      <span className="text-sm">
-                        {formatDistanceToNow(new Date(list.lastSuccess), {
-                          addSuffix: true,
-                          locale: it,
-                        })}
-                      </span>
-                    ) : (
-                      <span className="text-sm text-muted-foreground">Mai</span>
-                    )}
-                  </TableCell>
-                  <TableCell className="text-right">
-                    <div className="flex items-center justify-end gap-2">
-                      <Switch
-                        checked={list.enabled}
-                        onCheckedChange={(checked) => toggleEnabled(list.id, checked)}
-                        data-testid={`switch-enable-${list.id}`}
-                      />
-                      <Button
-                        variant="outline"
-                        size="icon"
-                        onClick={() => syncMutation.mutate(list.id)}
-                        disabled={syncMutation.isPending}
-                        data-testid={`button-sync-${list.id}`}
-                      >
-                        <RefreshCw className="w-4 h-4" />
-                      </Button>
-                      <Button
-                        variant="destructive"
-                        size="icon"
-                        onClick={() => {
-                          if (confirm(`Eliminare la lista "${list.name}"?`)) {
-                            deleteMutation.mutate(list.id);
-                          }
-                        }}
-                        data-testid={`button-delete-${list.id}`}
-                      >
-                        <Trash2 className="w-4 h-4" />
-                      </Button>
-                    </div>
-                  </TableCell>
-                </TableRow>
-              ))}
-              {(!lists || lists.length === 0) && (
-                <TableRow>
-                  <TableCell colSpan={7} className="text-center text-muted-foreground py-8">
-                    Nessuna lista configurata. Aggiungi la prima lista.
-                  </TableCell>
-                </TableRow>
-              )}
-            </TableBody>
-          </Table>
-        </CardContent>
-      </Card>
-    </div>
-  );
-}

client/src/pages/Whitelist.tsx

@@ -2,7 +2,7 @@ import { useQuery, useMutation } from "@tanstack/react-query";
 import { queryClient, apiRequest } from "@/lib/queryClient";
 import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card";
 import { Button } from "@/components/ui/button";
-import { Shield, Plus, Trash2, CheckCircle2, XCircle, Search } from "lucide-react";
+import { Shield, Plus, Trash2, CheckCircle2, XCircle } from "lucide-react";
 import { format } from "date-fns";
 import { useState } from "react";
 import { useForm } from "react-hook-form";
@@ -44,7 +44,6 @@ const whitelistFormSchema = insertWhitelistSchema.extend({
 export default function WhitelistPage() {
   const { toast } = useToast();
   const [isAddDialogOpen, setIsAddDialogOpen] = useState(false);
-  const [searchQuery, setSearchQuery] = useState("");
 
   const form = useForm<z.infer<typeof whitelistFormSchema>>({
     resolver: zodResolver(whitelistFormSchema),
@@ -60,13 +59,6 @@ export default function WhitelistPage() {
     queryKey: ["/api/whitelist"],
   });
 
-  // Filter whitelist based on search query
-  const filteredWhitelist = whitelist?.filter((item) =>
-    item.ipAddress.toLowerCase().includes(searchQuery.toLowerCase()) ||
-    item.reason?.toLowerCase().includes(searchQuery.toLowerCase()) ||
-    item.comment?.toLowerCase().includes(searchQuery.toLowerCase())
-  );
-
   const addMutation = useMutation({
     mutationFn: async (data: z.infer<typeof whitelistFormSchema>) => {
       return await apiRequest("POST", "/api/whitelist", data);
@@ -197,27 +189,11 @@ export default function WhitelistPage() {
         </Dialog>
       </div>
 
-      {/* Search Bar */}
-      <Card data-testid="card-search">
-        <CardContent className="pt-6">
-          <div className="relative">
-            <Search className="absolute left-3 top-1/2 -translate-y-1/2 h-4 w-4 text-muted-foreground" />
-            <Input
-              placeholder="Cerca per IP, motivo o note..."
-              value={searchQuery}
-              onChange={(e) => setSearchQuery(e.target.value)}
-              className="pl-9"
-              data-testid="input-search-whitelist"
-            />
-          </div>
-        </CardContent>
-      </Card>
-
       <Card data-testid="card-whitelist">
         <CardHeader>
           <CardTitle className="flex items-center gap-2">
             <Shield className="h-5 w-5" />
-            IP Protetti ({filteredWhitelist?.length || 0}{searchQuery && whitelist ? ` di ${whitelist.length}` : ''})
+            IP Protetti ({whitelist?.length || 0})
           </CardTitle>
         </CardHeader>
         <CardContent>
@@ -225,9 +201,9 @@ export default function WhitelistPage() {
             <div className="text-center py-8 text-muted-foreground" data-testid="text-loading">
               Caricamento...
             </div>
-          ) : filteredWhitelist && filteredWhitelist.length > 0 ? (
+          ) : whitelist && whitelist.length > 0 ? (
             <div className="space-y-3">
-              {filteredWhitelist.map((item) => (
+              {whitelist.map((item) => (
                 <div
                   key={item.id}
                   className="p-4 rounded-lg border hover-elevate"

database-schema/apply_migrations.sh

@@ -13,7 +13,6 @@ set -e
 SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 MIGRATIONS_DIR="$SCRIPT_DIR/migrations"
 IDS_DIR="$(dirname "$SCRIPT_DIR")"
-DEPLOYMENT_MIGRATIONS_DIR="$IDS_DIR/deployment/migrations"
 
 # Carica variabili ambiente ed esportale
 if [ -f "$IDS_DIR/.env" ]; then
@@ -80,25 +79,9 @@ echo -e "${CYAN}📊 Versione database corrente: ${YELLOW}${CURRENT_VERSION}${NC
 # STEP 3: Trova migrazioni da applicare
 # =============================================================================
 # Formato migrazioni: 001_description.sql, 002_another.sql, etc.
-# Cerca in ENTRAMBE le cartelle: database-schema/migrations E deployment/migrations
 MIGRATIONS_TO_APPLY=()
 
-# Combina migrations da entrambe le cartelle e ordina per numero
+for migration_file in $(find "$MIGRATIONS_DIR" -name "[0-9][0-9][0-9]_*.sql" | sort); do
-ALL_MIGRATIONS=""
-if [ -d "$MIGRATIONS_DIR" ]; then
-    ALL_MIGRATIONS+=$(find "$MIGRATIONS_DIR" -name "[0-9][0-9][0-9]_*.sql" 2>/dev/null || true)
-fi
-if [ -d "$DEPLOYMENT_MIGRATIONS_DIR" ]; then
-    if [ -n "$ALL_MIGRATIONS" ]; then
-        ALL_MIGRATIONS+=$'\n'
-    fi
-    ALL_MIGRATIONS+=$(find "$DEPLOYMENT_MIGRATIONS_DIR" -name "[0-9][0-9][0-9]_*.sql" 2>/dev/null || true)
-fi
-
-# Ordina le migrations per nome file (NNN_*.sql) estraendo il basename
-SORTED_MIGRATIONS=$(echo "$ALL_MIGRATIONS" | grep -v '^$' | while read f; do echo "$(basename "$f"):$f"; done | sort | cut -d':' -f2)
-
-for migration_file in $SORTED_MIGRATIONS; do
     MIGRATION_NAME=$(basename "$migration_file")
     
     # Estrai numero versione dal nome file (001, 002, etc.)

database-schema/schema.sql

@@ -2,9 +2,9 @@
 -- PostgreSQL database dump
 --
 
-\restrict Jq3ohS02Qcz3l9bNbeQprTZolEFbFh84eEwk4en2HkAqc2Xojxrd4AFqHJvBETG
+\restrict ANf2BaMgim8MRxq72JYZlQCMz7jS6Eh1bDga3LDLUbhCQLTS4WJKd9DetA35V3Q
 
--- Dumped from database version 16.11 (74c6bb6)
+-- Dumped from database version 16.9 (415ebe8)
 -- Dumped by pg_dump version 16.10
 
 SET statement_timeout = 0;
@@ -45,9 +45,7 @@ CREATE TABLE public.detections (
     organization text,
     as_number text,
     as_name text,
-    isp text,
+    isp text
-    detection_source text DEFAULT 'ml_model'::text,
-    blacklist_id character varying
 );
 
 
@@ -98,44 +96,6 @@ CREATE TABLE public.network_logs (
 );
 
 
---
--- Name: public_blacklist_ips; Type: TABLE; Schema: public; Owner: -
---
-
-CREATE TABLE public.public_blacklist_ips (
-    id character varying DEFAULT (gen_random_uuid())::text NOT NULL,
-    ip_address text NOT NULL,
-    cidr_range text,
-    ip_inet text,
-    cidr_inet text,
-    list_id character varying NOT NULL,
-    first_seen timestamp without time zone DEFAULT now() NOT NULL,
-    last_seen timestamp without time zone DEFAULT now() NOT NULL,
-    is_active boolean DEFAULT true NOT NULL
-);
-
-
---
--- Name: public_lists; Type: TABLE; Schema: public; Owner: -
---
-
-CREATE TABLE public.public_lists (
-    id character varying DEFAULT (gen_random_uuid())::text NOT NULL,
-    name text NOT NULL,
-    type text NOT NULL,
-    url text NOT NULL,
-    enabled boolean DEFAULT true NOT NULL,
-    fetch_interval_minutes integer DEFAULT 10 NOT NULL,
-    last_fetch timestamp without time zone,
-    last_success timestamp without time zone,
-    total_ips integer DEFAULT 0 NOT NULL,
-    active_ips integer DEFAULT 0 NOT NULL,
-    error_count integer DEFAULT 0 NOT NULL,
-    last_error text,
-    created_at timestamp without time zone DEFAULT now() NOT NULL
-);
-
-
 --
 -- Name: routers; Type: TABLE; Schema: public; Owner: -
 --
@@ -193,10 +153,7 @@ CREATE TABLE public.whitelist (
     reason text,
     created_by text,
     active boolean DEFAULT true NOT NULL,
-    created_at timestamp without time zone DEFAULT now() NOT NULL,
+    created_at timestamp without time zone DEFAULT now() NOT NULL
-    source text DEFAULT 'manual'::text,
-    list_id character varying,
-    ip_inet text
 );
 
 
@@ -232,30 +189,6 @@ ALTER TABLE ONLY public.network_logs
     ADD CONSTRAINT network_logs_pkey PRIMARY KEY (id);
 
 
---
--- Name: public_blacklist_ips public_blacklist_ips_ip_address_list_id_key; Type: CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY public.public_blacklist_ips
-    ADD CONSTRAINT public_blacklist_ips_ip_address_list_id_key UNIQUE (ip_address, list_id);
-
-
---
--- Name: public_blacklist_ips public_blacklist_ips_pkey; Type: CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY public.public_blacklist_ips
-    ADD CONSTRAINT public_blacklist_ips_pkey PRIMARY KEY (id);
-
-
---
--- Name: public_lists public_lists_pkey; Type: CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY public.public_lists
-    ADD CONSTRAINT public_lists_pkey PRIMARY KEY (id);
-
-
 --
 -- Name: routers routers_ip_address_unique; Type: CONSTRAINT; Schema: public; Owner: -
 --
@@ -375,17 +308,9 @@ ALTER TABLE ONLY public.network_logs
     ADD CONSTRAINT network_logs_router_id_routers_id_fk FOREIGN KEY (router_id) REFERENCES public.routers(id);
 
 
---
--- Name: public_blacklist_ips public_blacklist_ips_list_id_fkey; Type: FK CONSTRAINT; Schema: public; Owner: -
---
-
-ALTER TABLE ONLY public.public_blacklist_ips
-    ADD CONSTRAINT public_blacklist_ips_list_id_fkey FOREIGN KEY (list_id) REFERENCES public.public_lists(id) ON DELETE CASCADE;
-
-
 --
 -- PostgreSQL database dump complete
 --
 
-\unrestrict Jq3ohS02Qcz3l9bNbeQprTZolEFbFh84eEwk4en2HkAqc2Xojxrd4AFqHJvBETG
+\unrestrict ANf2BaMgim8MRxq72JYZlQCMz7jS6Eh1bDga3LDLUbhCQLTS4WJKd9DetA35V3Q
 

deployment/cron_train.sh

@@ -12,7 +12,7 @@ echo "=========================================" >> "$LOG_FILE"
 
 curl -X POST http://localhost:8000/train \
   -H "Content-Type: application/json" \
-  -d '{"max_records": 1000000, "hours_back": 24}' \
+  -d '{"max_records": 100000, "hours_back": 24}' \
   --max-time 300 >> "$LOG_FILE" 2>&1
 
 EXIT_CODE=$?

deployment/docs/PUBLIC_LISTS_LIMITATIONS.md

@@ -1,48 +0,0 @@
-# Public Lists - Known Limitations (v2.0.0)
-
-## CIDR Range Matching
-
-**Current Status**: MVP with exact IP matching
-**Impact**: CIDR ranges (e.g., Spamhaus /24 blocks) are stored but not yet matched against detections
-
-### Details:
-- `public_blacklist_ips.cidr_range` field exists and is populated by parsers
-- Detections currently use **exact IP matching only**
-- Whitelist entries with CIDR notation not expanded
-
-### Future Iteration:
-Requires PostgreSQL INET/CIDR column types and query optimizations:
-1. Add dedicated `inet` columns to `public_blacklist_ips` and `whitelist`
-2. Rewrite merge logic with CIDR containment operators (`<<=`, `>>=`)
-3. Index optimization for network range queries
-
-### Workaround (Production):
-Most critical single IPs are still caught. For CIDR-heavy feeds, parser can be extended to expand ranges to individual IPs (trade-off: storage vs query performance).
-
----
-
-## Integration Status
-
-✅ **Working**:
-- Fetcher syncs every 10 minutes (systemd timer)
-- Manual whitelist > Public whitelist > Blacklist priority
-- Automatic cleanup of invalid detections
-
-⚠️ **Manual Sync**:
-- UI manual sync triggers by resetting `lastAttempt` timestamp
-- Actual sync occurs on next fetcher cycle (max 10 min delay)
-- For immediate sync: `sudo systemctl start ids-list-fetcher.service`
-
----
-
-## Performance Notes
-
-- Bulk SQL operations avoid O(N) per-IP queries
-- Tested with 186M+ network_logs records
-- Query optimization ongoing for CIDR expansion
-
----
-
-**Version**: 2.0.0 MVP  
-**Date**: 2025-11-26  
-**Next Iteration**: Full CIDR matching support

deployment/docs/PUBLIC_LISTS_V2_CIDR.md

@@ -1,295 +0,0 @@
-# Public Lists v2.0.0 - CIDR Complete Implementation
-
-## Overview
-Sistema completo di integrazione liste pubbliche con supporto CIDR per matching di network ranges tramite operatori PostgreSQL INET.
-
-## Database Schema v7
-
-### Migration 007: CIDR Support
-```sql
--- Aggiunte colonne INET/CIDR
-ALTER TABLE public_blacklist_ips 
-  ADD COLUMN ip_inet inet,
-  ADD COLUMN cidr_inet cidr;
-
-ALTER TABLE whitelist
-  ADD COLUMN ip_inet inet;
-
--- Indexes GiST per operatori di rete
-CREATE INDEX public_blacklist_ip_inet_idx ON public_blacklist_ips USING gist(ip_inet inet_ops);
-CREATE INDEX public_blacklist_cidr_inet_idx ON public_blacklist_ips USING gist(cidr_inet inet_ops);
-CREATE INDEX whitelist_ip_inet_idx ON whitelist USING gist(ip_inet inet_ops);
-```
-
-### Colonne Aggiunte
-| Tabella | Colonna | Tipo | Scopo |
-|---------|---------|------|-------|
-| public_blacklist_ips | ip_inet | inet | IP singolo per matching esatto |
-| public_blacklist_ips | cidr_inet | cidr | Range di rete per containment |
-| whitelist | ip_inet | inet | IP/range per whitelist CIDR-aware |
-
-## CIDR Matching Logic
-
-### Operatori PostgreSQL INET
-```sql
--- Containment: IP è contenuto in CIDR range?
-'192.168.1.50'::inet <<= '192.168.1.0/24'::inet  -- TRUE
-
--- Esempi pratici
-'8.8.8.8'::inet <<= '8.8.8.0/24'::inet           -- TRUE  
-'1.1.1.1'::inet <<= '8.8.8.0/24'::inet           -- FALSE
-'52.94.10.5'::inet <<= '52.94.0.0/16'::inet      -- TRUE (AWS range)
-```
-
-### Priority Logic con CIDR
-```sql
--- Creazione detections con priorità CIDR-aware
-INSERT INTO detections (source_ip, risk_score, ...)
-SELECT bl.ip_address, 75, ...
-FROM public_blacklist_ips bl
-WHERE bl.is_active = true
-AND bl.ip_inet IS NOT NULL
--- Priorità 1: Whitelist manuale (massima)
-AND NOT EXISTS (
-    SELECT 1 FROM whitelist wl
-    WHERE wl.active = true
-    AND wl.source = 'manual'
-    AND (bl.ip_inet = wl.ip_inet OR bl.ip_inet <<= wl.ip_inet)
-)
--- Priorità 2: Whitelist pubblica
-AND NOT EXISTS (
-    SELECT 1 FROM whitelist wl
-    WHERE wl.active = true
-    AND wl.source != 'manual'
-    AND (bl.ip_inet = wl.ip_inet OR bl.ip_inet <<= wl.ip_inet)
-)
-```
-
-### Cleanup CIDR-Aware
-```sql
--- Rimuove detections per IP in whitelist ranges
-DELETE FROM detections d
-WHERE d.detection_source = 'public_blacklist'
-AND EXISTS (
-    SELECT 1 FROM whitelist wl
-    WHERE wl.active = true
-    AND wl.ip_inet IS NOT NULL
-    AND (d.source_ip::inet = wl.ip_inet 
-         OR d.source_ip::inet <<= wl.ip_inet)
-)
-```
-
-## Performance
-
-### Index Strategy
-- **GiST indexes** ottimizzati per operatori `<<=` e `>>=`
-- Query log(n) anche con 186M+ record
-- Bulk operations mantenute per efficienza
-
-### Benchmark
-| Operazione | Complessità | Tempo Medio |
-|------------|-------------|-------------|
-| Exact IP lookup | O(log n) | ~5ms |
-| CIDR containment | O(log n) | ~15ms |
-| Bulk detection (10k IPs) | O(n) | ~2s |
-| Priority filtering (100k) | O(n log m) | ~500ms |
-
-## Testing Matrix
-
-| Scenario | Implementazione | Status |
-|----------|-----------------|--------|
-| Exact IP (8.8.8.8) | inet equality | ✅ Completo |
-| CIDR range (192.168.1.0/24) | `<<=` operator | ✅ Completo |
-| Mixed exact + CIDR | Combined query | ✅ Completo |
-| Manual whitelist priority | Source-based exclusion | ✅ Completo |
-| Public whitelist priority | Nested NOT EXISTS | ✅ Completo |
-| Performance (186M+ rows) | Bulk + indexes | ✅ Completo |
-
-## Deployment su AlmaLinux 9
-
-### Pre-Deployment
-```bash
-# Backup database
-sudo -u postgres pg_dump ids_production > /opt/ids/backups/pre_v2_$(date +%Y%m%d).sql
-
-# Verifica versione schema
-sudo -u postgres psql ids_production -c "SELECT version FROM schema_version;"
-```
-
-### Esecuzione Migration
-```bash
-cd /opt/ids
-sudo -u postgres psql ids_production < deployment/migrations/007_add_cidr_support.sql
-
-# Verifica successo
-sudo -u postgres psql ids_production -c "
-SELECT version, updated_at FROM schema_version WHERE id = 1;
-SELECT COUNT(*) FROM public_blacklist_ips WHERE ip_inet IS NOT NULL;
-SELECT COUNT(*) FROM whitelist WHERE ip_inet IS NOT NULL;
-"
-```
-
-### Update Codice Python
-```bash
-# Pull da GitLab
-./update_from_git.sh
-
-# Restart services
-sudo systemctl restart ids-list-fetcher
-sudo systemctl restart ids-ml-backend
-
-# Verifica logs
-journalctl -u ids-list-fetcher -n 50
-journalctl -u ids-ml-backend -n 50
-```
-
-### Validazione Post-Deploy
-```bash
-# Test CIDR matching
-sudo -u postgres psql ids_production -c "
--- Verifica popolazione INET columns
-SELECT 
-    COUNT(*) as total_blacklist,
-    COUNT(ip_inet) as with_inet,
-    COUNT(cidr_inet) as with_cidr
-FROM public_blacklist_ips;
-
--- Test containment query
-SELECT * FROM whitelist 
-WHERE active = true 
-AND '192.168.1.50'::inet <<= ip_inet
-LIMIT 5;
-
--- Verifica priority logic
-SELECT source, COUNT(*) 
-FROM whitelist 
-WHERE active = true 
-GROUP BY source;
-"
-```
-
-## Monitoring
-
-### Service Health Checks
-```bash
-# Status fetcher
-systemctl status ids-list-fetcher
-systemctl list-timers ids-list-fetcher
-
-# Logs real-time
-journalctl -u ids-list-fetcher -f
-```
-
-### Database Queries
-```sql
--- Sync status liste
-SELECT 
-    name, 
-    type, 
-    last_success, 
-    total_ips, 
-    active_ips,
-    error_count,
-    last_error
-FROM public_lists 
-ORDER BY last_success DESC;
-
--- CIDR coverage
-SELECT 
-    COUNT(*) as total,
-    COUNT(CASE WHEN cidr_range IS NOT NULL THEN 1 END) as with_cidr,
-    COUNT(CASE WHEN ip_inet IS NOT NULL THEN 1 END) as with_inet,
-    COUNT(CASE WHEN cidr_inet IS NOT NULL THEN 1 END) as cidr_inet_populated
-FROM public_blacklist_ips;
-
--- Detection sources
-SELECT 
-    detection_source,
-    COUNT(*) as count,
-    AVG(risk_score) as avg_score
-FROM detections
-GROUP BY detection_source;
-```
-
-## Esempi d'Uso
-
-### Scenario 1: AWS Range Whitelist
-```sql
--- Whitelist AWS range 52.94.0.0/16
-INSERT INTO whitelist (ip_address, ip_inet, source, comment)
-VALUES ('52.94.0.0/16', '52.94.0.0/16'::inet, 'aws', 'AWS us-east-1 range');
-
--- Verifica matching
-SELECT * FROM detections 
-WHERE source_ip::inet <<= '52.94.0.0/16'::inet
-AND detection_source = 'public_blacklist';
--- Queste detections verranno automaticamente cleanup
-```
-
-### Scenario 2: Priority Override
-```sql
--- Blacklist Spamhaus: 1.2.3.4
--- Public whitelist GCP: 1.2.3.0/24
--- Manual whitelist utente: NESSUNA
-
--- Risultato: 1.2.3.4 NON genera detection (public whitelist vince)
-
--- Se aggiungi manual whitelist:
-INSERT INTO whitelist (ip_address, ip_inet, source)
-VALUES ('1.2.3.4', '1.2.3.4'::inet, 'manual');
-
--- Ora 1.2.3.4 è protetto da priorità massima (manual > public > blacklist)
-```
-
-## Troubleshooting
-
-### INET Column Non Populated
-```sql
--- Manually populate se necessario
-UPDATE public_blacklist_ips 
-SET ip_inet = ip_address::inet,
-    cidr_inet = COALESCE(cidr_range::cidr, (ip_address || '/32')::cidr)
-WHERE ip_inet IS NULL;
-
-UPDATE whitelist
-SET ip_inet = CASE
-    WHEN ip_address ~ '/' THEN ip_address::inet
-    ELSE ip_address::inet
-END
-WHERE ip_inet IS NULL;
-```
-
-### Index Missing
-```sql
--- Ricrea indexes se mancanti
-CREATE INDEX IF NOT EXISTS public_blacklist_ip_inet_idx 
-  ON public_blacklist_ips USING gist(ip_inet inet_ops);
-CREATE INDEX IF NOT EXISTS public_blacklist_cidr_inet_idx 
-  ON public_blacklist_ips USING gist(cidr_inet inet_ops);
-CREATE INDEX IF NOT EXISTS whitelist_ip_inet_idx 
-  ON whitelist USING gist(ip_inet inet_ops);
-```
-
-### Performance Degradation
-```bash
-# Reindex GiST
-sudo -u postgres psql ids_production -c "REINDEX INDEX CONCURRENTLY public_blacklist_ip_inet_idx;"
-
-# Vacuum analyze
-sudo -u postgres psql ids_production -c "VACUUM ANALYZE public_blacklist_ips;"
-sudo -u postgres psql ids_production -c "VACUUM ANALYZE whitelist;"
-```
-
-## Known Issues
-Nessuno. Sistema production-ready con CIDR completo.
-
-## Future Enhancements (v2.1+)
-- Incremental sync (delta updates)
-- Redis caching per query frequenti
-- Additional threat feeds (SANS ISC, AbuseIPDB)
-- Table partitioning per scalabilità
-
-## References
-- PostgreSQL INET/CIDR docs: https://www.postgresql.org/docs/current/datatype-net-types.html
-- GiST indexes: https://www.postgresql.org/docs/current/gist.html
-- Network operators: https://www.postgresql.org/docs/current/functions-net.html

deployment/install_list_fetcher.sh

@@ -1,105 +0,0 @@
-#!/bin/bash
-# =============================================================================
-# IDS - Installazione Servizio List Fetcher
-# =============================================================================
-# Installa e configura il servizio systemd per il fetcher delle liste pubbliche
-# Eseguire come ROOT: ./install_list_fetcher.sh
-# =============================================================================
-
-set -e
-
-RED='\033[0;31m'
-GREEN='\033[0;32m'
-YELLOW='\033[1;33m'
-BLUE='\033[0;34m'
-NC='\033[0m'
-
-echo -e "${BLUE}"
-echo "╔═══════════════════════════════════════════════╗"
-echo "║   📋 INSTALLAZIONE IDS LIST FETCHER           ║"
-echo "╚═══════════════════════════════════════════════╝"
-echo -e "${NC}"
-
-IDS_DIR="/opt/ids"
-SYSTEMD_DIR="/etc/systemd/system"
-
-# Verifica di essere root
-if [ "$EUID" -ne 0 ]; then 
-    echo -e "${RED}❌ Questo script deve essere eseguito come root${NC}"
-    echo -e "${YELLOW}   Esegui: sudo ./install_list_fetcher.sh${NC}"
-    exit 1
-fi
-
-# Verifica che i file sorgente esistano
-SERVICE_SRC="$IDS_DIR/deployment/systemd/ids-list-fetcher.service"
-TIMER_SRC="$IDS_DIR/deployment/systemd/ids-list-fetcher.timer"
-
-if [ ! -f "$SERVICE_SRC" ]; then
-    echo -e "${RED}❌ File service non trovato: $SERVICE_SRC${NC}"
-    exit 1
-fi
-
-if [ ! -f "$TIMER_SRC" ]; then
-    echo -e "${RED}❌ File timer non trovato: $TIMER_SRC${NC}"
-    exit 1
-fi
-
-# Verifica che il virtual environment Python esista
-VENV_PYTHON="$IDS_DIR/python_ml/venv/bin/python3"
-if [ ! -f "$VENV_PYTHON" ]; then
-    echo -e "${YELLOW}⚠️  Virtual environment non trovato, creazione...${NC}"
-    cd "$IDS_DIR/python_ml"
-    python3.11 -m venv venv
-    ./venv/bin/pip install --upgrade pip
-    ./venv/bin/pip install -r requirements.txt
-    echo -e "${GREEN}✅ Virtual environment creato${NC}"
-fi
-
-# Verifica che run_fetcher.py esista
-FETCHER_SCRIPT="$IDS_DIR/python_ml/list_fetcher/run_fetcher.py"
-if [ ! -f "$FETCHER_SCRIPT" ]; then
-    echo -e "${RED}❌ Script fetcher non trovato: $FETCHER_SCRIPT${NC}"
-    exit 1
-fi
-
-# Copia file systemd
-echo -e "${BLUE}📦 Installazione file systemd...${NC}"
-
-cp "$SERVICE_SRC" "$SYSTEMD_DIR/ids-list-fetcher.service"
-cp "$TIMER_SRC" "$SYSTEMD_DIR/ids-list-fetcher.timer"
-
-echo -e "${GREEN}   ✅ ids-list-fetcher.service installato${NC}"
-echo -e "${GREEN}   ✅ ids-list-fetcher.timer installato${NC}"
-
-# Ricarica systemd
-echo -e "${BLUE}🔄 Ricarica configurazione systemd...${NC}"
-systemctl daemon-reload
-echo -e "${GREEN}✅ Daemon ricaricato${NC}"
-
-# Abilita e avvia timer
-echo -e "${BLUE}⏱️  Abilitazione timer (ogni 10 minuti)...${NC}"
-systemctl enable ids-list-fetcher.timer
-systemctl start ids-list-fetcher.timer
-echo -e "${GREEN}✅ Timer abilitato e avviato${NC}"
-
-# Test esecuzione manuale
-echo -e "${BLUE}🧪 Test esecuzione fetcher...${NC}"
-if systemctl start ids-list-fetcher.service; then
-    echo -e "${GREEN}✅ Fetcher eseguito con successo${NC}"
-else
-    echo -e "${YELLOW}⚠️  Prima esecuzione potrebbe fallire se liste non configurate${NC}"
-fi
-
-# Mostra stato
-echo ""
-echo -e "${GREEN}╔═══════════════════════════════════════════════╗${NC}"
-echo -e "${GREEN}║   ✅ INSTALLAZIONE COMPLETATA                 ║${NC}"
-echo -e "${GREEN}╚═══════════════════════════════════════════════╝${NC}"
-echo ""
-echo -e "${BLUE}📋 COMANDI UTILI:${NC}"
-echo -e "   • Stato timer:     ${YELLOW}systemctl status ids-list-fetcher.timer${NC}"
-echo -e "   • Stato service:   ${YELLOW}systemctl status ids-list-fetcher.service${NC}"
-echo -e "   • Esegui manuale:  ${YELLOW}systemctl start ids-list-fetcher.service${NC}"
-echo -e "   • Visualizza logs: ${YELLOW}journalctl -u ids-list-fetcher -n 50${NC}"
-echo -e "   • Timer attivi:    ${YELLOW}systemctl list-timers | grep ids${NC}"
-echo ""

deployment/migrations/006_add_public_lists.sql

@@ -1,116 +0,0 @@
--- Migration 006: Add Public Lists Integration
--- Description: Adds blacklist/whitelist public sources with auto-sync support
--- Author: IDS System
--- Date: 2024-11-26
--- NOTE: Fully idempotent - safe to run multiple times
-
-BEGIN;
-
--- ============================================================================
--- 1. CREATE NEW TABLES
--- ============================================================================
-
--- Public threat/whitelist sources configuration
-CREATE TABLE IF NOT EXISTS public_lists (
-    id VARCHAR PRIMARY KEY DEFAULT gen_random_uuid(),
-    name TEXT NOT NULL,
-    type TEXT NOT NULL CHECK (type IN ('blacklist', 'whitelist')),
-    url TEXT NOT NULL,
-    enabled BOOLEAN NOT NULL DEFAULT true,
-    fetch_interval_minutes INTEGER NOT NULL DEFAULT 10,
-    last_fetch TIMESTAMP,
-    last_success TIMESTAMP,
-    total_ips INTEGER NOT NULL DEFAULT 0,
-    active_ips INTEGER NOT NULL DEFAULT 0,
-    error_count INTEGER NOT NULL DEFAULT 0,
-    last_error TEXT,
-    created_at TIMESTAMP NOT NULL DEFAULT NOW()
-);
-
-CREATE INDEX IF NOT EXISTS public_lists_type_idx ON public_lists(type);
-CREATE INDEX IF NOT EXISTS public_lists_enabled_idx ON public_lists(enabled);
-
--- Public blacklist IPs from external sources
-CREATE TABLE IF NOT EXISTS public_blacklist_ips (
-    id VARCHAR PRIMARY KEY DEFAULT gen_random_uuid(),
-    ip_address TEXT NOT NULL,
-    cidr_range TEXT,
-    list_id VARCHAR NOT NULL REFERENCES public_lists(id) ON DELETE CASCADE,
-    first_seen TIMESTAMP NOT NULL DEFAULT NOW(),
-    last_seen TIMESTAMP NOT NULL DEFAULT NOW(),
-    is_active BOOLEAN NOT NULL DEFAULT true
-);
-
-CREATE INDEX IF NOT EXISTS public_blacklist_ip_idx ON public_blacklist_ips(ip_address);
-CREATE INDEX IF NOT EXISTS public_blacklist_list_idx ON public_blacklist_ips(list_id);
-CREATE INDEX IF NOT EXISTS public_blacklist_active_idx ON public_blacklist_ips(is_active);
-
--- Create unique constraint only if not exists
-DO $$ 
-BEGIN
-    IF NOT EXISTS (
-        SELECT 1 FROM pg_indexes 
-        WHERE indexname = 'public_blacklist_ip_list_key'
-    ) THEN
-        CREATE UNIQUE INDEX public_blacklist_ip_list_key ON public_blacklist_ips(ip_address, list_id);
-    END IF;
-END $$;
-
--- ============================================================================
--- 2. ALTER EXISTING TABLES
--- ============================================================================
-
--- Extend detections table with public list source tracking
-ALTER TABLE detections 
-ADD COLUMN IF NOT EXISTS detection_source TEXT NOT NULL DEFAULT 'ml_model',
-ADD COLUMN IF NOT EXISTS blacklist_id VARCHAR;
-
-CREATE INDEX IF NOT EXISTS detection_source_idx ON detections(detection_source);
-
--- Add check constraint for valid detection sources
-DO $$ 
-BEGIN
-    IF NOT EXISTS (
-        SELECT 1 FROM pg_constraint 
-        WHERE conname = 'detections_source_check'
-    ) THEN
-        ALTER TABLE detections 
-        ADD CONSTRAINT detections_source_check 
-        CHECK (detection_source IN ('ml_model', 'public_blacklist', 'hybrid'));
-    END IF;
-END $$;
-
--- Extend whitelist table with source tracking
-ALTER TABLE whitelist
-ADD COLUMN IF NOT EXISTS source TEXT NOT NULL DEFAULT 'manual',
-ADD COLUMN IF NOT EXISTS list_id VARCHAR;
-
-CREATE INDEX IF NOT EXISTS whitelist_source_idx ON whitelist(source);
-
--- Add check constraint for valid whitelist sources
-DO $$ 
-BEGIN
-    IF NOT EXISTS (
-        SELECT 1 FROM pg_constraint 
-        WHERE conname = 'whitelist_source_check'
-    ) THEN
-        ALTER TABLE whitelist 
-        ADD CONSTRAINT whitelist_source_check 
-        CHECK (source IN ('manual', 'aws', 'gcp', 'cloudflare', 'iana', 'ntp', 'other'));
-    END IF;
-END $$;
-
--- ============================================================================
--- 3. UPDATE SCHEMA VERSION
--- ============================================================================
-
-INSERT INTO schema_version (id, version, description)
-VALUES (1, 6, 'Add public lists integration (blacklist/whitelist sources)')
-ON CONFLICT (id) DO UPDATE 
-SET version = 6, 
-    description = 'Add public lists integration (blacklist/whitelist sources)',
-    applied_at = NOW();
-
-COMMIT;
-
-SELECT 'Migration 006 completed successfully' as status;

deployment/migrations/007_add_cidr_support.sql

@@ -1,88 +0,0 @@
--- Migration 007: Add INET/CIDR support for proper network range matching
--- Required for public lists integration (Spamhaus /24, AWS ranges, etc.)
--- Date: 2025-11-26
--- NOTE: Handles case where columns exist as TEXT type (from Drizzle)
-
-BEGIN;
-
--- ============================================================================
--- FIX: Drop TEXT columns and recreate as proper INET/CIDR types
--- ============================================================================
-
--- Check column type and fix if needed for public_blacklist_ips
-DO $$
-DECLARE
-    col_type text;
-BEGIN
-    -- Check ip_inet column type
-    SELECT data_type INTO col_type 
-    FROM information_schema.columns 
-    WHERE table_name = 'public_blacklist_ips' AND column_name = 'ip_inet';
-    
-    IF col_type = 'text' THEN
-        -- Drop the wrong type columns
-        ALTER TABLE public_blacklist_ips DROP COLUMN IF EXISTS ip_inet;
-        ALTER TABLE public_blacklist_ips DROP COLUMN IF EXISTS cidr_inet;
-        RAISE NOTICE 'Dropped TEXT columns, will recreate as INET/CIDR';
-    END IF;
-END $$;
-
--- Add INET/CIDR columns with correct types
-ALTER TABLE public_blacklist_ips 
-  ADD COLUMN IF NOT EXISTS ip_inet inet,
-  ADD COLUMN IF NOT EXISTS cidr_inet cidr;
-
--- Populate new columns from existing text data
-UPDATE public_blacklist_ips 
-SET ip_inet = ip_address::inet,
-    cidr_inet = CASE 
-      WHEN cidr_range IS NOT NULL THEN cidr_range::cidr
-      ELSE (ip_address || '/32')::cidr
-    END
-WHERE ip_inet IS NULL OR cidr_inet IS NULL;
-
--- Create GiST indexes for INET operators
-CREATE INDEX IF NOT EXISTS public_blacklist_ip_inet_idx ON public_blacklist_ips USING gist(ip_inet inet_ops);
-CREATE INDEX IF NOT EXISTS public_blacklist_cidr_inet_idx ON public_blacklist_ips USING gist(cidr_inet inet_ops);
-
--- ============================================================================
--- Fix whitelist table
--- ============================================================================
-
-DO $$
-DECLARE
-    col_type text;
-BEGIN
-    SELECT data_type INTO col_type 
-    FROM information_schema.columns 
-    WHERE table_name = 'whitelist' AND column_name = 'ip_inet';
-    
-    IF col_type = 'text' THEN
-        ALTER TABLE whitelist DROP COLUMN IF EXISTS ip_inet;
-        RAISE NOTICE 'Dropped TEXT column from whitelist, will recreate as INET';
-    END IF;
-END $$;
-
--- Add INET column to whitelist
-ALTER TABLE whitelist
-  ADD COLUMN IF NOT EXISTS ip_inet inet;
-
--- Populate whitelist INET column
-UPDATE whitelist
-SET ip_inet = CASE
-  WHEN ip_address ~ '/' THEN ip_address::inet
-  ELSE ip_address::inet
-END
-WHERE ip_inet IS NULL;
-
--- Create index for whitelist INET matching
-CREATE INDEX IF NOT EXISTS whitelist_ip_inet_idx ON whitelist USING gist(ip_inet inet_ops);
-
--- Update schema version
-UPDATE schema_version SET version = 7, applied_at = NOW() WHERE id = 1;
-
-COMMIT;
-
--- Verification
-SELECT 'Migration 007 completed successfully' as status;
-SELECT version, applied_at FROM schema_version WHERE id = 1;

deployment/migrations/008_force_inet_types.sql

@@ -1,92 +0,0 @@
--- Migration 008: Force INET/CIDR types (unconditional)
--- Fixes issues where columns remained TEXT after conditional migration 007
--- Date: 2026-01-02
-
-BEGIN;
-
--- ============================================================================
--- FORCE DROP AND RECREATE ALL INET COLUMNS
--- This is unconditional - always executes regardless of current state
--- ============================================================================
-
--- Drop indexes first (if exist)
-DROP INDEX IF EXISTS public_blacklist_ip_inet_idx;
-DROP INDEX IF EXISTS public_blacklist_cidr_inet_idx;
-DROP INDEX IF EXISTS whitelist_ip_inet_idx;
-
--- ============================================================================
--- FIX public_blacklist_ips TABLE
--- ============================================================================
-
--- Drop columns unconditionally
-ALTER TABLE public_blacklist_ips DROP COLUMN IF EXISTS ip_inet;
-ALTER TABLE public_blacklist_ips DROP COLUMN IF EXISTS cidr_inet;
-
--- Recreate with correct INET/CIDR types
-ALTER TABLE public_blacklist_ips ADD COLUMN ip_inet inet;
-ALTER TABLE public_blacklist_ips ADD COLUMN cidr_inet cidr;
-
--- Populate from existing text data
-UPDATE public_blacklist_ips 
-SET 
-    ip_inet = CASE 
-        WHEN ip_address ~ '/' THEN ip_address::inet
-        ELSE ip_address::inet
-    END,
-    cidr_inet = CASE 
-        WHEN cidr_range IS NOT NULL AND cidr_range != '' THEN cidr_range::cidr
-        WHEN ip_address ~ '/' THEN ip_address::cidr
-        ELSE (ip_address || '/32')::cidr
-    END
-WHERE ip_inet IS NULL;
-
--- Create GiST indexes for fast INET/CIDR containment operators
-CREATE INDEX public_blacklist_ip_inet_idx ON public_blacklist_ips USING gist(ip_inet inet_ops);
-CREATE INDEX public_blacklist_cidr_inet_idx ON public_blacklist_ips USING gist(cidr_inet inet_ops);
-
--- ============================================================================
--- FIX whitelist TABLE
--- ============================================================================
-
--- Drop column unconditionally
-ALTER TABLE whitelist DROP COLUMN IF EXISTS ip_inet;
-
--- Recreate with correct INET type
-ALTER TABLE whitelist ADD COLUMN ip_inet inet;
-
--- Populate from existing text data
-UPDATE whitelist
-SET ip_inet = CASE
-    WHEN ip_address ~ '/' THEN ip_address::inet
-    ELSE ip_address::inet
-END
-WHERE ip_inet IS NULL;
-
--- Create index for whitelist
-CREATE INDEX whitelist_ip_inet_idx ON whitelist USING gist(ip_inet inet_ops);
-
--- ============================================================================
--- UPDATE SCHEMA VERSION
--- ============================================================================
-
-UPDATE schema_version SET version = 8, applied_at = NOW() WHERE id = 1;
-
-COMMIT;
-
--- ============================================================================
--- VERIFICATION
--- ============================================================================
-
-SELECT 'Migration 008 completed successfully' as status;
-SELECT version, applied_at FROM schema_version WHERE id = 1;
-
--- Verify column types
-SELECT 
-    table_name, 
-    column_name, 
-    data_type
-FROM information_schema.columns 
-WHERE 
-    (table_name = 'public_blacklist_ips' AND column_name IN ('ip_inet', 'cidr_inet'))
-    OR (table_name = 'whitelist' AND column_name = 'ip_inet')
-ORDER BY table_name, column_name;

deployment/migrations/009_add_microsoft_meta_lists.sql

@@ -1,33 +0,0 @@
--- Migration 009: Add Microsoft Azure and Meta/Facebook public lists
--- Date: 2026-01-02
-
--- Microsoft Azure IP ranges (whitelist - cloud provider)
-INSERT INTO public_lists (name, url, type, format, enabled, description, fetch_interval)
-VALUES (
-    'Microsoft Azure',
-    'https://raw.githubusercontent.com/femueller/cloud-ip-ranges/master/microsoft-azure-ip-ranges.json',
-    'whitelist',
-    'json',
-    true,
-    'Microsoft Azure cloud IP ranges - auto-updated from Azure Service Tags',
-    3600
-) ON CONFLICT (name) DO UPDATE SET
-    url = EXCLUDED.url,
-    description = EXCLUDED.description;
-
--- Meta/Facebook IP ranges (whitelist - major service provider)
-INSERT INTO public_lists (name, url, type, format, enabled, description, fetch_interval)
-VALUES (
-    'Meta (Facebook)',
-    'https://raw.githubusercontent.com/parseword/util-misc/master/block-facebook/facebook-ip-ranges.txt',
-    'whitelist',
-    'plain',
-    true,
-    'Meta/Facebook IP ranges (includes Instagram, WhatsApp, Oculus) from BGP AS32934/AS54115/AS63293',
-    3600
-) ON CONFLICT (name) DO UPDATE SET
-    url = EXCLUDED.url,
-    description = EXCLUDED.description;
-
--- Verify insertion
-SELECT id, name, type, enabled, url FROM public_lists WHERE name IN ('Microsoft Azure', 'Meta (Facebook)');

deployment/scripts/deploy_public_lists.sh

@@ -1,50 +0,0 @@
-#!/bin/bash
-# Deploy Public Lists Integration (v2.0.0)
-# Run on AlmaLinux 9 server after git pull
-
-set -e
-
-echo "=================================="
-echo "PUBLIC LISTS DEPLOYMENT - v2.0.0"
-echo "=================================="
-
-# 1. Database Migration
-echo -e "\n[1/5] Running database migration..."
-sudo -u postgres psql -d ids_system -f deployment/migrations/006_add_public_lists.sql
-echo "✓ Migration 006 applied"
-
-# 2. Seed default lists
-echo -e "\n[2/5] Seeding default public lists..."
-cd python_ml/list_fetcher
-DATABASE_URL=$DATABASE_URL python seed_lists.py
-cd ../..
-echo "✓ Default lists seeded"
-
-# 3. Install systemd services
-echo -e "\n[3/5] Installing systemd services..."
-sudo cp deployment/systemd/ids-list-fetcher.service /etc/systemd/system/
-sudo cp deployment/systemd/ids-list-fetcher.timer /etc/systemd/system/
-sudo systemctl daemon-reload
-echo "✓ Systemd services installed"
-
-# 4. Enable and start
-echo -e "\n[4/5] Enabling services..."
-sudo systemctl enable ids-list-fetcher.timer
-sudo systemctl start ids-list-fetcher.timer
-echo "✓ Timer enabled (10-minute intervals)"
-
-# 5. Initial sync
-echo -e "\n[5/5] Running initial sync..."
-sudo systemctl start ids-list-fetcher.service
-echo "✓ Initial sync triggered"
-
-echo -e "\n=================================="
-echo "DEPLOYMENT COMPLETE"
-echo "=================================="
-echo ""
-echo "Verify:"
-echo "  journalctl -u ids-list-fetcher -n 50"
-echo "  systemctl status ids-list-fetcher.timer"
-echo ""
-echo "Check UI: http://your-server/public-lists"
-echo ""

deployment/systemd/ids-list-fetcher.service

@@ -1,29 +0,0 @@
-[Unit]
-Description=IDS Public Lists Fetcher Service
-Documentation=https://github.com/yourorg/ids
-After=network.target postgresql.service
-
-[Service]
-Type=oneshot
-User=root
-WorkingDirectory=/opt/ids/python_ml
-Environment="PYTHONUNBUFFERED=1"
-EnvironmentFile=/opt/ids/.env
-
-# Run list fetcher with virtual environment
-ExecStart=/opt/ids/python_ml/venv/bin/python3 /opt/ids/python_ml/list_fetcher/run_fetcher.py
-
-# Logging
-StandardOutput=journal
-StandardError=journal
-SyslogIdentifier=ids-list-fetcher
-
-# Security settings
-PrivateTmp=true
-NoNewPrivileges=true
-
-# Restart policy
-Restart=no
-
-[Install]
-WantedBy=multi-user.target

deployment/systemd/ids-list-fetcher.timer

@@ -1,13 +0,0 @@
-[Unit]
-Description=IDS Public Lists Fetcher Timer (every 10 minutes)
-Documentation=https://github.com/yourorg/ids
-
-[Timer]
-# Run every 10 minutes
-OnCalendar=*:0/10
-OnBootSec=2min
-AccuracySec=1min
-Persistent=true
-
-[Install]
-WantedBy=timers.target

deployment/update_from_git.sh

@@ -158,20 +158,6 @@ if [ -f "./deployment/setup_rsyslog.sh" ]; then
     fi
 fi
 
-# Verifica e installa servizio list-fetcher se mancante
-echo -e "\n${BLUE}📋 Verifica servizio list-fetcher...${NC}"
-if ! systemctl list-unit-files | grep -q "ids-list-fetcher"; then
-    echo -e "${YELLOW}   Servizio ids-list-fetcher non installato, installazione...${NC}"
-    if [ -f "./deployment/install_list_fetcher.sh" ]; then
-        chmod +x ./deployment/install_list_fetcher.sh
-        ./deployment/install_list_fetcher.sh
-    else
-        echo -e "${RED}   ❌ Script install_list_fetcher.sh non trovato${NC}"
-    fi
-else
-    echo -e "${GREEN}   ✅ Servizio ids-list-fetcher già installato${NC}"
-fi
-
 # Restart servizi
 echo -e "\n${BLUE}🔄 Restart servizi...${NC}"
 if [ -f "./deployment/restart_all.sh" ]; then

main.py

@@ -1,6 +0,0 @@
-def main():
-    print("Hello from repl-nix-workspace!")
-
-
-if __name__ == "__main__":
-    main()

pyproject.toml

@@ -1,8 +0,0 @@
-[project]
-name = "repl-nix-workspace"
-version = "0.1.0"
-description = "Add your description here"
-requires-python = ">=3.11"
-dependencies = [
-    "httpx>=0.28.1",
-]

python_ml/list_fetcher/__init__.py

@@ -1,2 +0,0 @@
-# Public Lists Fetcher Module
-# Handles download, parsing, and sync of public blacklist/whitelist sources

python_ml/list_fetcher/fetcher.py

@@ -1,401 +0,0 @@
-import asyncio
-import httpx
-from datetime import datetime
-from typing import Dict, List, Set, Tuple, Optional
-import psycopg2
-from psycopg2.extras import execute_values
-import os
-import sys
-
-# Add parent directory to path for imports
-sys.path.append(os.path.dirname(os.path.dirname(__file__)))
-
-from list_fetcher.parsers import parse_list
-
-
-class ListFetcher:
-    """Fetches and synchronizes public IP lists"""
-    
-    def __init__(self, database_url: str):
-        self.database_url = database_url
-        self.timeout = 30.0
-        self.max_retries = 3
-    
-    def get_db_connection(self):
-        """Create database connection"""
-        return psycopg2.connect(self.database_url)
-    
-    async def fetch_url(self, url: str) -> Optional[str]:
-        """Download content from URL with retry logic"""
-        async with httpx.AsyncClient(timeout=self.timeout, follow_redirects=True) as client:
-            for attempt in range(self.max_retries):
-                try:
-                    response = await client.get(url)
-                    response.raise_for_status()
-                    return response.text
-                except httpx.HTTPError as e:
-                    if attempt == self.max_retries - 1:
-                        raise Exception(f"HTTP error after {self.max_retries} attempts: {e}")
-                    await asyncio.sleep(2 ** attempt)  # Exponential backoff
-                except Exception as e:
-                    if attempt == self.max_retries - 1:
-                        raise Exception(f"Download failed: {e}")
-                    await asyncio.sleep(2 ** attempt)
-        return None
-    
-    def get_enabled_lists(self) -> List[Dict]:
-        """Get all enabled public lists from database"""
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                cur.execute("""
-                    SELECT id, name, type, url, fetch_interval_minutes
-                    FROM public_lists
-                    WHERE enabled = true
-                    ORDER BY type, name
-                """)
-                if cur.description is None:
-                    return []
-                columns = [desc[0] for desc in cur.description]
-                return [dict(zip(columns, row)) for row in cur.fetchall()]
-        finally:
-            conn.close()
-    
-    def get_existing_ips(self, list_id: str, list_type: str) -> Set[str]:
-        """Get existing IPs for a list from database"""
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                if list_type == 'blacklist':
-                    cur.execute("""
-                        SELECT ip_address 
-                        FROM public_blacklist_ips 
-                        WHERE list_id = %s AND is_active = true
-                    """, (list_id,))
-                else:  # whitelist
-                    cur.execute("""
-                        SELECT ip_address 
-                        FROM whitelist 
-                        WHERE list_id = %s AND active = true
-                    """, (list_id,))
-                
-                return {row[0] for row in cur.fetchall()}
-        finally:
-            conn.close()
-    
-    def sync_blacklist_ips(self, list_id: str, new_ips: Set[Tuple[str, Optional[str]]]):
-        """Sync blacklist IPs: add new, mark inactive old ones"""
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                # Get existing IPs
-                existing = self.get_existing_ips(list_id, 'blacklist')
-                new_ip_addresses = {ip for ip, _ in new_ips}
-                
-                # Calculate diff
-                to_add = new_ip_addresses - existing
-                to_deactivate = existing - new_ip_addresses
-                to_update = existing & new_ip_addresses
-                
-                # Mark old IPs as inactive
-                if to_deactivate:
-                    cur.execute("""
-                        UPDATE public_blacklist_ips
-                        SET is_active = false
-                        WHERE list_id = %s AND ip_address = ANY(%s)
-                    """, (list_id, list(to_deactivate)))
-                
-                # Update last_seen for existing active IPs
-                if to_update:
-                    cur.execute("""
-                        UPDATE public_blacklist_ips
-                        SET last_seen = NOW()
-                        WHERE list_id = %s AND ip_address = ANY(%s)
-                    """, (list_id, list(to_update)))
-                
-                # Add new IPs with INET/CIDR support
-                if to_add:
-                    values = []
-                    for ip, cidr in new_ips:
-                        if ip in to_add:
-                            # Compute INET values for CIDR matching
-                            cidr_inet = cidr if cidr else f"{ip}/32"
-                            values.append((ip, cidr, ip, cidr_inet, list_id))
-                    
-                    execute_values(cur, """
-                        INSERT INTO public_blacklist_ips 
-                        (ip_address, cidr_range, ip_inet, cidr_inet, list_id)
-                        VALUES %s
-                        ON CONFLICT (ip_address, list_id) DO UPDATE
-                        SET is_active = true, last_seen = NOW(),
-                            ip_inet = EXCLUDED.ip_inet,
-                            cidr_inet = EXCLUDED.cidr_inet
-                    """, values)
-                
-                # Update list stats
-                cur.execute("""
-                    UPDATE public_lists
-                    SET total_ips = %s,
-                        active_ips = %s,
-                        last_success = NOW()
-                    WHERE id = %s
-                """, (len(new_ip_addresses), len(new_ip_addresses), list_id))
-                
-                conn.commit()
-                return len(to_add), len(to_deactivate), len(to_update)
-        
-        except Exception as e:
-            conn.rollback()
-            raise e
-        finally:
-            conn.close()
-    
-    def sync_whitelist_ips(self, list_id: str, list_name: str, new_ips: Set[Tuple[str, Optional[str]]]):
-        """Sync whitelist IPs: add new, deactivate old ones"""
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                # Get existing IPs
-                existing = self.get_existing_ips(list_id, 'whitelist')
-                new_ip_addresses = {ip for ip, _ in new_ips}
-                
-                # Calculate diff
-                to_add = new_ip_addresses - existing
-                to_deactivate = existing - new_ip_addresses
-                to_update = existing & new_ip_addresses
-                
-                # Determine source name from list name
-                source = 'other'
-                list_lower = list_name.lower()
-                if 'aws' in list_lower:
-                    source = 'aws'
-                elif 'gcp' in list_lower or 'google' in list_lower:
-                    source = 'gcp'
-                elif 'cloudflare' in list_lower:
-                    source = 'cloudflare'
-                elif 'iana' in list_lower:
-                    source = 'iana'
-                elif 'ntp' in list_lower:
-                    source = 'ntp'
-                
-                # Mark old IPs as inactive
-                if to_deactivate:
-                    cur.execute("""
-                        UPDATE whitelist
-                        SET active = false
-                        WHERE list_id = %s AND ip_address = ANY(%s)
-                    """, (list_id, list(to_deactivate)))
-                
-                # Add new IPs with INET support for CIDR matching
-                if to_add:
-                    values = []
-                    for ip, cidr in new_ips:
-                        if ip in to_add:
-                            comment = f"Auto-imported from {list_name}"
-                            if cidr:
-                                comment += f" (CIDR: {cidr})"
-                            # Compute ip_inet for CIDR-aware whitelisting
-                            ip_inet = cidr if cidr else ip
-                            values.append((ip, ip_inet, comment, source, list_id))
-                    
-                    execute_values(cur, """
-                        INSERT INTO whitelist (ip_address, ip_inet, comment, source, list_id)
-                        VALUES %s
-                        ON CONFLICT (ip_address) DO UPDATE
-                        SET active = true, 
-                            ip_inet = EXCLUDED.ip_inet,
-                            source = EXCLUDED.source, 
-                            list_id = EXCLUDED.list_id
-                    """, values)
-                
-                # Update list stats
-                cur.execute("""
-                    UPDATE public_lists
-                    SET total_ips = %s,
-                        active_ips = %s,
-                        last_success = NOW()
-                    WHERE id = %s
-                """, (len(new_ip_addresses), len(new_ip_addresses), list_id))
-                
-                conn.commit()
-                return len(to_add), len(to_deactivate), len(to_update)
-        
-        except Exception as e:
-            conn.rollback()
-            raise e
-        finally:
-            conn.close()
-    
-    async def fetch_and_sync_list(self, list_config: Dict) -> Dict:
-        """Fetch and sync a single list"""
-        list_id = list_config['id']
-        list_name = list_config['name']
-        list_type = list_config['type']
-        url = list_config['url']
-        
-        result = {
-            'list_id': list_id,
-            'list_name': list_name,
-            'success': False,
-            'added': 0,
-            'removed': 0,
-            'updated': 0,
-            'error': None
-        }
-        
-        conn = self.get_db_connection()
-        
-        try:
-            # Update last_fetch timestamp
-            with conn.cursor() as cur:
-                cur.execute("""
-                    UPDATE public_lists 
-                    SET last_fetch = NOW() 
-                    WHERE id = %s
-                """, (list_id,))
-                conn.commit()
-            
-            # Download content
-            print(f"[{datetime.now().strftime('%H:%M:%S')}] Downloading {list_name} from {url}...")
-            content = await self.fetch_url(url)
-            
-            if not content:
-                raise Exception("Empty response from server")
-            
-            # Parse IPs
-            print(f"[{datetime.now().strftime('%H:%M:%S')}] Parsing {list_name}...")
-            ips = parse_list(list_name, content)
-            
-            if not ips:
-                raise Exception("No valid IPs found in list")
-            
-            print(f"[{datetime.now().strftime('%H:%M:%S')}] Found {len(ips)} IPs, syncing to database...")
-            
-            # Sync to database
-            if list_type == 'blacklist':
-                added, removed, updated = self.sync_blacklist_ips(list_id, ips)
-            else:
-                added, removed, updated = self.sync_whitelist_ips(list_id, list_name, ips)
-            
-            result.update({
-                'success': True,
-                'added': added,
-                'removed': removed,
-                'updated': updated
-            })
-            
-            print(f"[{datetime.now().strftime('%H:%M:%S')}] ✓ {list_name}: +{added} -{removed} ~{updated}")
-            
-            # Reset error count on success
-            with conn.cursor() as cur:
-                cur.execute("""
-                    UPDATE public_lists 
-                    SET error_count = 0, last_error = NULL 
-                    WHERE id = %s
-                """, (list_id,))
-                conn.commit()
-        
-        except Exception as e:
-            error_msg = str(e)
-            result['error'] = error_msg
-            print(f"[{datetime.now().strftime('%H:%M:%S')}] ✗ {list_name}: {error_msg}")
-            
-            # Increment error count
-            with conn.cursor() as cur:
-                cur.execute("""
-                    UPDATE public_lists 
-                    SET error_count = error_count + 1,
-                        last_error = %s
-                    WHERE id = %s
-                """, (error_msg[:500], list_id))
-                conn.commit()
-        
-        finally:
-            conn.close()
-        
-        return result
-    
-    async def fetch_all_lists(self) -> List[Dict]:
-        """Fetch and sync all enabled lists"""
-        print(f"\n{'='*60}")
-        print(f"[{datetime.now().strftime('%Y-%m-%d %H:%M:%S')}] PUBLIC LISTS SYNC")
-        print(f"{'='*60}\n")
-        
-        # Get enabled lists
-        lists = self.get_enabled_lists()
-        
-        if not lists:
-            print("No enabled lists found")
-            return []
-        
-        print(f"Found {len(lists)} enabled lists\n")
-        
-        # Fetch all lists in parallel
-        tasks = [self.fetch_and_sync_list(list_config) for list_config in lists]
-        results = await asyncio.gather(*tasks, return_exceptions=True)
-        
-        # Summary
-        print(f"\n{'='*60}")
-        print("SYNC SUMMARY")
-        print(f"{'='*60}")
-        
-        success_count = sum(1 for r in results if isinstance(r, dict) and r.get('success'))
-        error_count = len(results) - success_count
-        total_added = sum(r.get('added', 0) for r in results if isinstance(r, dict))
-        total_removed = sum(r.get('removed', 0) for r in results if isinstance(r, dict))
-        
-        print(f"Success: {success_count}/{len(results)}")
-        print(f"Errors:  {error_count}/{len(results)}")
-        print(f"Total IPs Added:   {total_added}")
-        print(f"Total IPs Removed: {total_removed}")
-        print(f"{'='*60}\n")
-        
-        return [r for r in results if isinstance(r, dict)]
-
-
-async def main():
-    """Main entry point for list fetcher"""
-    database_url = os.getenv('DATABASE_URL')
-    
-    if not database_url:
-        print("ERROR: DATABASE_URL environment variable not set")
-        return 1
-    
-    fetcher = ListFetcher(database_url)
-    
-    try:
-        # Fetch and sync all lists
-        await fetcher.fetch_all_lists()
-        
-        # Run merge logic to sync detections with blacklist/whitelist priority
-        print("\n" + "="*60)
-        print("RUNNING MERGE LOGIC")
-        print("="*60 + "\n")
-        
-        # Import merge logic (avoid circular imports)
-        import sys
-        from pathlib import Path
-        merge_logic_path = Path(__file__).parent.parent
-        sys.path.insert(0, str(merge_logic_path))
-        from merge_logic import MergeLogic
-        
-        merge = MergeLogic(database_url)
-        stats = merge.sync_public_blacklist_detections()
-        
-        print(f"\nMerge Logic Stats:")
-        print(f"  Created detections:          {stats['created']}")
-        print(f"  Cleaned invalid detections:  {stats['cleaned']}")
-        print(f"  Skipped (whitelisted):       {stats['skipped_whitelisted']}")
-        print("="*60 + "\n")
-        
-        return 0
-    except Exception as e:
-        print(f"FATAL ERROR: {e}")
-        import traceback
-        traceback.print_exc()
-        return 1
-
-
-if __name__ == "__main__":
-    exit_code = asyncio.run(main())
-    sys.exit(exit_code)

python_ml/list_fetcher/parsers.py

@@ -1,362 +0,0 @@
-import re
-import json
-from typing import List, Dict, Set, Optional
-from datetime import datetime
-import ipaddress
-
-
-class ListParser:
-    """Base parser for public IP lists"""
-    
-    @staticmethod
-    def validate_ip(ip_str: str) -> bool:
-        """Validate IP address or CIDR range"""
-        try:
-            ipaddress.ip_network(ip_str, strict=False)
-            return True
-        except ValueError:
-            return False
-    
-    @staticmethod
-    def normalize_cidr(ip_str: str) -> tuple[str, Optional[str]]:
-        """
-        Normalize IP/CIDR to (ip_address, cidr_range)
-        For CIDR ranges, use the full CIDR notation as ip_address to ensure uniqueness
-        Example: '1.2.3.0/24' -> ('1.2.3.0/24', '1.2.3.0/24')
-                 '1.2.3.4' -> ('1.2.3.4', None)
-        """
-        try:
-            network = ipaddress.ip_network(ip_str, strict=False)
-            if '/' in ip_str:
-                normalized_cidr = str(network)
-                return (normalized_cidr, normalized_cidr)
-            else:
-                return (ip_str, None)
-        except ValueError:
-            return (ip_str, None)
-
-
-class SpamhausParser(ListParser):
-    """Parser for Spamhaus DROP list"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse Spamhaus DROP format:
-        - NDJSON (new): {"cidr":"1.2.3.0/24","sblid":"SBL12345","rir":"apnic"}
-        - Text (old): 1.2.3.0/24 ; SBL12345
-        """
-        ips = set()
-        lines = content.strip().split('\n')
-        
-        for line in lines:
-            line = line.strip()
-            
-            # Skip comments and empty lines
-            if not line or line.startswith(';') or line.startswith('#'):
-                continue
-            
-            # Try NDJSON format first (new Spamhaus format)
-            if line.startswith('{'):
-                try:
-                    data = json.loads(line)
-                    cidr = data.get('cidr')
-                    if cidr and ListParser.validate_ip(cidr):
-                        ips.add(ListParser.normalize_cidr(cidr))
-                    continue
-                except json.JSONDecodeError:
-                    pass
-            
-            # Fallback: old text format
-            parts = line.split(';')
-            if parts:
-                ip_part = parts[0].strip()
-                if ip_part and ListParser.validate_ip(ip_part):
-                    ips.add(ListParser.normalize_cidr(ip_part))
-        
-        return ips
-
-
-class TalosParser(ListParser):
-    """Parser for Talos Intelligence blacklist"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse Talos format (plain IP list):
-        1.2.3.4
-        5.6.7.0/24
-        """
-        ips = set()
-        lines = content.strip().split('\n')
-        
-        for line in lines:
-            line = line.strip()
-            
-            # Skip comments and empty lines
-            if not line or line.startswith('#') or line.startswith('//'):
-                continue
-            
-            # Validate and add
-            if ListParser.validate_ip(line):
-                ips.add(ListParser.normalize_cidr(line))
-        
-        return ips
-
-
-class AWSParser(ListParser):
-    """Parser for AWS IP ranges JSON"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse AWS JSON format:
-        {
-          "prefixes": [
-            {"ip_prefix": "1.2.3.0/24", "region": "us-east-1", "service": "EC2"}
-          ]
-        }
-        """
-        ips = set()
-        
-        try:
-            data = json.loads(content)
-            
-            # IPv4 prefixes
-            for prefix in data.get('prefixes', []):
-                ip_prefix = prefix.get('ip_prefix')
-                if ip_prefix and ListParser.validate_ip(ip_prefix):
-                    ips.add(ListParser.normalize_cidr(ip_prefix))
-            
-            # IPv6 prefixes (optional)
-            for prefix in data.get('ipv6_prefixes', []):
-                ipv6_prefix = prefix.get('ipv6_prefix')
-                if ipv6_prefix and ListParser.validate_ip(ipv6_prefix):
-                    ips.add(ListParser.normalize_cidr(ipv6_prefix))
-        
-        except json.JSONDecodeError:
-            pass
-        
-        return ips
-
-
-class GCPParser(ListParser):
-    """Parser for Google Cloud IP ranges JSON"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse GCP JSON format:
-        {
-          "prefixes": [
-            {"ipv4Prefix": "1.2.3.0/24"},
-            {"ipv6Prefix": "2001:db8::/32"}
-          ]
-        }
-        """
-        ips = set()
-        
-        try:
-            data = json.loads(content)
-            
-            for prefix in data.get('prefixes', []):
-                # IPv4
-                ipv4 = prefix.get('ipv4Prefix')
-                if ipv4 and ListParser.validate_ip(ipv4):
-                    ips.add(ListParser.normalize_cidr(ipv4))
-                
-                # IPv6
-                ipv6 = prefix.get('ipv6Prefix')
-                if ipv6 and ListParser.validate_ip(ipv6):
-                    ips.add(ListParser.normalize_cidr(ipv6))
-        
-        except json.JSONDecodeError:
-            pass
-        
-        return ips
-
-
-class AzureParser(ListParser):
-    """Parser for Microsoft Azure IP ranges JSON (Service Tags format)"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse Azure Service Tags JSON format:
-        {
-          "values": [
-            {
-              "name": "ActionGroup",
-              "properties": {
-                "addressPrefixes": ["1.2.3.0/24", "5.6.7.0/24"]
-              }
-            }
-          ]
-        }
-        """
-        ips = set()
-        
-        try:
-            data = json.loads(content)
-            
-            for value in data.get('values', []):
-                properties = value.get('properties', {})
-                prefixes = properties.get('addressPrefixes', [])
-                
-                for prefix in prefixes:
-                    if prefix and ListParser.validate_ip(prefix):
-                        ips.add(ListParser.normalize_cidr(prefix))
-        
-        except json.JSONDecodeError:
-            pass
-        
-        return ips
-
-
-class MetaParser(ListParser):
-    """Parser for Meta/Facebook IP ranges (plain CIDR list from BGP)"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse Meta format (plain CIDR list):
-        31.13.24.0/21
-        31.13.64.0/18
-        157.240.0.0/17
-        """
-        ips = set()
-        lines = content.strip().split('\n')
-        
-        for line in lines:
-            line = line.strip()
-            
-            # Skip empty lines and comments
-            if not line or line.startswith('#') or line.startswith('//'):
-                continue
-            
-            if ListParser.validate_ip(line):
-                ips.add(ListParser.normalize_cidr(line))
-        
-        return ips
-
-
-class CloudflareParser(ListParser):
-    """Parser for Cloudflare IP list"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse Cloudflare format (plain CIDR list):
-        1.2.3.0/24
-        5.6.7.0/24
-        """
-        ips = set()
-        lines = content.strip().split('\n')
-        
-        for line in lines:
-            line = line.strip()
-            
-            # Skip empty lines and comments
-            if not line or line.startswith('#'):
-                continue
-            
-            if ListParser.validate_ip(line):
-                ips.add(ListParser.normalize_cidr(line))
-        
-        return ips
-
-
-class IANAParser(ListParser):
-    """Parser for IANA Root Servers"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse IANA root servers (extract IPs from HTML/text)
-        Look for IPv4 addresses in format XXX.XXX.XXX.XXX
-        """
-        ips = set()
-        
-        # Regex for IPv4 addresses
-        ipv4_pattern = r'\b(?:[0-9]{1,3}\.){3}[0-9]{1,3}\b'
-        matches = re.findall(ipv4_pattern, content)
-        
-        for ip in matches:
-            if ListParser.validate_ip(ip):
-                ips.add(ListParser.normalize_cidr(ip))
-        
-        return ips
-
-
-class NTPPoolParser(ListParser):
-    """Parser for NTP Pool servers"""
-    
-    @staticmethod
-    def parse(content: str) -> Set[tuple[str, Optional[str]]]:
-        """
-        Parse NTP pool format (plain IP list or JSON)
-        Tries multiple formats
-        """
-        ips = set()
-        
-        # Try JSON first
-        try:
-            data = json.loads(content)
-            if isinstance(data, list):
-                for item in data:
-                    if isinstance(item, str) and ListParser.validate_ip(item):
-                        ips.add(ListParser.normalize_cidr(item))
-                    elif isinstance(item, dict):
-                        ip = item.get('ip') or item.get('address')
-                        if ip and ListParser.validate_ip(ip):
-                            ips.add(ListParser.normalize_cidr(ip))
-        except json.JSONDecodeError:
-            # Fallback to plain text parsing
-            lines = content.strip().split('\n')
-            for line in lines:
-                line = line.strip()
-                if line and ListParser.validate_ip(line):
-                    ips.add(ListParser.normalize_cidr(line))
-        
-        return ips
-
-
-# Parser registry
-PARSERS: Dict[str, type[ListParser]] = {
-    'spamhaus': SpamhausParser,
-    'talos': TalosParser,
-    'aws': AWSParser,
-    'gcp': GCPParser,
-    'google': GCPParser,
-    'azure': AzureParser,
-    'microsoft': AzureParser,
-    'meta': MetaParser,
-    'facebook': MetaParser,
-    'cloudflare': CloudflareParser,
-    'iana': IANAParser,
-    'ntp': NTPPoolParser,
-}
-
-
-def get_parser(list_name: str) -> Optional[type[ListParser]]:
-    """Get parser by list name (case-insensitive match)"""
-    list_name_lower = list_name.lower()
-    
-    for key, parser in PARSERS.items():
-        if key in list_name_lower:
-            return parser
-    
-    # Default fallback: try plain text parser
-    return TalosParser
-
-
-def parse_list(list_name: str, content: str) -> Set[tuple[str, Optional[str]]]:
-    """
-    Parse list content using appropriate parser
-    Returns set of (ip_address, cidr_range) tuples
-    """
-    parser_class = get_parser(list_name)
-    if parser_class:
-        parser = parser_class()
-        return parser.parse(content)
-    return set()

python_ml/list_fetcher/run_fetcher.py

@@ -1,17 +0,0 @@
-#!/usr/bin/env python3
-"""
-IDS List Fetcher Runner
-Fetches and syncs public blacklist/whitelist sources every 10 minutes
-"""
-import asyncio
-import sys
-import os
-
-# Add parent directory to path
-sys.path.append(os.path.dirname(os.path.dirname(__file__)))
-
-from list_fetcher.fetcher import main
-
-if __name__ == "__main__":
-    exit_code = asyncio.run(main())
-    sys.exit(exit_code)

python_ml/list_fetcher/seed_lists.py

@@ -1,174 +0,0 @@
-#!/usr/bin/env python3
-"""
-Seed default public lists into database
-Run after migration 006 to populate initial lists
-"""
-import psycopg2
-import os
-import sys
-import argparse
-
-# Add parent directory to path
-sys.path.append(os.path.dirname(os.path.dirname(__file__)))
-
-from list_fetcher.fetcher import ListFetcher
-import asyncio
-
-
-DEFAULT_LISTS = [
-    # Blacklists
-    {
-        'name': 'Spamhaus DROP',
-        'type': 'blacklist',
-        'url': 'https://www.spamhaus.org/drop/drop.txt',
-        'enabled': True,
-        'fetch_interval_minutes': 10
-    },
-    {
-        'name': 'Talos Intelligence IP Blacklist',
-        'type': 'blacklist',
-        'url': 'https://talosintelligence.com/documents/ip-blacklist',
-        'enabled': False,  # Disabled by default - verify URL first
-        'fetch_interval_minutes': 10
-    },
-    
-    # Whitelists
-    {
-        'name': 'AWS IP Ranges',
-        'type': 'whitelist',
-        'url': 'https://ip-ranges.amazonaws.com/ip-ranges.json',
-        'enabled': True,
-        'fetch_interval_minutes': 10
-    },
-    {
-        'name': 'Google Cloud IP Ranges',
-        'type': 'whitelist',
-        'url': 'https://www.gstatic.com/ipranges/cloud.json',
-        'enabled': True,
-        'fetch_interval_minutes': 10
-    },
-    {
-        'name': 'Cloudflare IPv4',
-        'type': 'whitelist',
-        'url': 'https://www.cloudflare.com/ips-v4',
-        'enabled': True,
-        'fetch_interval_minutes': 10
-    },
-    {
-        'name': 'IANA Root Servers',
-        'type': 'whitelist',
-        'url': 'https://www.iana.org/domains/root/servers',
-        'enabled': True,
-        'fetch_interval_minutes': 10
-    },
-    {
-        'name': 'NTP Pool Servers',
-        'type': 'whitelist',
-        'url': 'https://www.ntppool.org/zone/@',
-        'enabled': False,  # Disabled by default - zone parameter needed
-        'fetch_interval_minutes': 10
-    }
-]
-
-
-def seed_lists(database_url: str, dry_run: bool = False):
-    """Insert default lists into database"""
-    conn = psycopg2.connect(database_url)
-    
-    try:
-        with conn.cursor() as cur:
-            # Check if lists already exist
-            cur.execute("SELECT COUNT(*) FROM public_lists")
-            result = cur.fetchone()
-            existing_count = result[0] if result else 0
-            
-            if existing_count > 0 and not dry_run:
-                print(f"⚠️  Warning: {existing_count} lists already exist in database")
-                response = input("Continue and add default lists? (y/n): ")
-                if response.lower() != 'y':
-                    print("Aborted")
-                    return
-            
-            print(f"\n{'='*60}")
-            print("SEEDING DEFAULT PUBLIC LISTS")
-            print(f"{'='*60}\n")
-            
-            for list_config in DEFAULT_LISTS:
-                if dry_run:
-                    status = "✓ ENABLED" if list_config['enabled'] else "○ DISABLED"
-                    print(f"{status} {list_config['type'].upper()}: {list_config['name']}")
-                    print(f"         URL: {list_config['url']}")
-                    print()
-                else:
-                    cur.execute("""
-                        INSERT INTO public_lists (name, type, url, enabled, fetch_interval_minutes)
-                        VALUES (%s, %s, %s, %s, %s)
-                        RETURNING id, name
-                    """, (
-                        list_config['name'],
-                        list_config['type'],
-                        list_config['url'],
-                        list_config['enabled'],
-                        list_config['fetch_interval_minutes']
-                    ))
-                    
-                    result = cur.fetchone()
-                    if result:
-                        list_id, list_name = result
-                        status = "✓" if list_config['enabled'] else "○"
-                        print(f"{status} Added: {list_name} (ID: {list_id})")
-            
-            if not dry_run:
-                conn.commit()
-                print(f"\n✓ Successfully seeded {len(DEFAULT_LISTS)} lists")
-                print(f"{'='*60}\n")
-            else:
-                print(f"\n{'='*60}")
-                print(f"DRY RUN: Would seed {len(DEFAULT_LISTS)} lists")
-                print(f"{'='*60}\n")
-    
-    except Exception as e:
-        conn.rollback()
-        print(f"✗ Error: {e}")
-        import traceback
-        traceback.print_exc()
-        return 1
-    finally:
-        conn.close()
-    
-    return 0
-
-
-async def sync_lists(database_url: str):
-    """Run initial sync of all enabled lists"""
-    print("\nRunning initial sync of enabled lists...\n")
-    fetcher = ListFetcher(database_url)
-    await fetcher.fetch_all_lists()
-
-
-def main():
-    parser = argparse.ArgumentParser(description='Seed default public lists')
-    parser.add_argument('--dry-run', action='store_true', help='Show what would be added without inserting')
-    parser.add_argument('--sync', action='store_true', help='Run initial sync after seeding')
-    args = parser.parse_args()
-    
-    database_url = os.getenv('DATABASE_URL')
-    if not database_url:
-        print("ERROR: DATABASE_URL environment variable not set")
-        return 1
-    
-    # Seed lists
-    exit_code = seed_lists(database_url, dry_run=args.dry_run)
-    
-    if exit_code != 0:
-        return exit_code
-    
-    # Optionally sync
-    if args.sync and not args.dry_run:
-        asyncio.run(sync_lists(database_url))
-    
-    return 0
-
-
-if __name__ == "__main__":
-    sys.exit(main())

python_ml/merge_logic.py

@@ -1,376 +0,0 @@
-#!/usr/bin/env python3
-"""
-Merge Logic for Public Lists Integration
-Implements priority: Manual Whitelist > Public Whitelist > Public Blacklist
-"""
-import os
-import psycopg2
-from typing import Dict, Set, Optional
-from datetime import datetime
-import logging
-import ipaddress
-
-logging.basicConfig(level=logging.INFO)
-logger = logging.getLogger(__name__)
-
-
-def ip_matches_cidr(ip_address: str, cidr_range: Optional[str]) -> bool:
-    """
-    Check if IP address matches CIDR range
-    Returns True if cidr_range is None (exact match) or if IP is in range
-    """
-    if not cidr_range:
-        return True  # Exact match handling
-    
-    try:
-        ip = ipaddress.ip_address(ip_address)
-        network = ipaddress.ip_network(cidr_range, strict=False)
-        return ip in network
-    except (ValueError, TypeError):
-        logger.warning(f"Invalid IP/CIDR: {ip_address}/{cidr_range}")
-        return False
-
-
-class MergeLogic:
-    """
-    Handles merge logic between manual entries and public lists
-    Priority: Manual whitelist > Public whitelist > Public blacklist
-    """
-    
-    def __init__(self, database_url: str):
-        self.database_url = database_url
-    
-    def get_db_connection(self):
-        """Create database connection"""
-        return psycopg2.connect(self.database_url)
-    
-    def get_all_whitelisted_ips(self) -> Set[str]:
-        """
-        Get all whitelisted IPs (manual + public)
-        Manual whitelist has higher priority than public whitelist
-        """
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                cur.execute("""
-                    SELECT DISTINCT ip_address 
-                    FROM whitelist 
-                    WHERE active = true
-                """)
-                return {row[0] for row in cur.fetchall()}
-        finally:
-            conn.close()
-    
-    def get_public_blacklist_ips(self) -> Set[str]:
-        """Get all active public blacklist IPs"""
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                cur.execute("""
-                    SELECT DISTINCT ip_address 
-                    FROM public_blacklist_ips 
-                    WHERE is_active = true
-                """)
-                return {row[0] for row in cur.fetchall()}
-        finally:
-            conn.close()
-    
-    def should_block_ip(self, ip_address: str) -> tuple[bool, str]:
-        """
-        Determine if IP should be blocked based on merge logic
-        Returns: (should_block, reason)
-        
-        Priority:
-        1. Manual whitelist (exact or CIDR) → DON'T block (highest priority)
-        2. Public whitelist (exact or CIDR) → DON'T block
-        3. Public blacklist (exact or CIDR) → DO block
-        4. Not in any list → DON'T block (only ML decides)
-        """
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                # Check manual whitelist (highest priority) - exact + CIDR matching
-                cur.execute("""
-                    SELECT ip_address, list_id FROM whitelist 
-                    WHERE active = true 
-                    AND source = 'manual'
-                """)
-                for row in cur.fetchall():
-                    wl_ip, wl_cidr = row[0], None
-                    # Check if whitelist entry has CIDR notation
-                    if '/' in wl_ip:
-                        wl_cidr = wl_ip
-                    if wl_ip == ip_address or ip_matches_cidr(ip_address, wl_cidr):
-                        return (False, "manual_whitelist")
-                
-                # Check public whitelist (any source except 'manual') - exact + CIDR
-                cur.execute("""
-                    SELECT ip_address, list_id FROM whitelist 
-                    WHERE active = true 
-                    AND source != 'manual'
-                """)
-                for row in cur.fetchall():
-                    wl_ip, wl_cidr = row[0], None
-                    if '/' in wl_ip:
-                        wl_cidr = wl_ip
-                    if wl_ip == ip_address or ip_matches_cidr(ip_address, wl_cidr):
-                        return (False, "public_whitelist")
-                
-                # Check public blacklist - exact + CIDR matching
-                cur.execute("""
-                    SELECT id, ip_address, cidr_range FROM public_blacklist_ips 
-                    WHERE is_active = true
-                """)
-                for row in cur.fetchall():
-                    bl_id, bl_ip, bl_cidr = row
-                    # Match exact IP or check if IP is in CIDR range
-                    if bl_ip == ip_address or ip_matches_cidr(ip_address, bl_cidr):
-                        return (True, f"public_blacklist:{bl_id}")
-                
-                # Not in any list
-                return (False, "not_listed")
-        finally:
-            conn.close()
-    
-    def create_detection_from_blacklist(
-        self, 
-        ip_address: str, 
-        blacklist_id: str,
-        risk_score: int = 75
-    ) -> Optional[str]:
-        """
-        Create detection record for public blacklist IP
-        Only if not whitelisted (priority check)
-        """
-        should_block, reason = self.should_block_ip(ip_address)
-        
-        if not should_block:
-            logger.info(f"IP {ip_address} not blocked - reason: {reason}")
-            return None
-        
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                # Check if detection already exists
-                cur.execute("""
-                    SELECT id FROM detections 
-                    WHERE source_ip = %s 
-                    AND detection_source = 'public_blacklist'
-                    LIMIT 1
-                """, (ip_address,))
-                
-                existing = cur.fetchone()
-                if existing:
-                    logger.info(f"Detection already exists for {ip_address}")
-                    return existing[0]
-                
-                # Create new detection
-                cur.execute("""
-                    INSERT INTO detections (
-                        source_ip,
-                        risk_score,
-                        confidence,
-                        anomaly_type,
-                        reason,
-                        log_count,
-                        first_seen,
-                        last_seen,
-                        detection_source,
-                        blacklist_id,
-                        detected_at,
-                        blocked
-                    ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
-                    RETURNING id
-                """, (
-                    ip_address,
-                    risk_score,  # numeric, not string
-                    100.0,  # confidence
-                    'public_blacklist',
-                    'IP in public blacklist',
-                    1,  # log_count
-                    datetime.utcnow(),  # first_seen
-                    datetime.utcnow(),  # last_seen
-                    'public_blacklist',
-                    blacklist_id,
-                    datetime.utcnow(),
-                    False  # Will be blocked by auto-block service if risk_score >= 80
-                ))
-                
-                result = cur.fetchone()
-                if not result:
-                    logger.error(f"Failed to get detection ID after insert for {ip_address}")
-                    return None
-                
-                detection_id = result[0]
-                conn.commit()
-                
-                logger.info(f"Created detection {detection_id} for blacklisted IP {ip_address}")
-                return detection_id
-        except Exception as e:
-            conn.rollback()
-            logger.error(f"Failed to create detection for {ip_address}: {e}")
-            return None
-        finally:
-            conn.close()
-    
-    def cleanup_invalid_detections(self) -> int:
-        """
-        Remove detections for IPs that are now whitelisted
-        CIDR-aware: checks both exact match and network containment
-        Respects priority: manual/public whitelist overrides blacklist
-        """
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                # Delete detections for IPs in whitelist ranges (CIDR-aware)
-                # Cast both sides to inet explicitly for type safety
-                cur.execute("""
-                    DELETE FROM detections d
-                    WHERE d.detection_source = 'public_blacklist'
-                    AND EXISTS (
-                        SELECT 1 FROM whitelist wl
-                        WHERE wl.active = true
-                        AND wl.ip_inet IS NOT NULL
-                        AND (
-                            d.source_ip::inet = wl.ip_inet::inet
-                            OR d.source_ip::inet <<= wl.ip_inet::inet
-                        )
-                    )
-                """)
-                deleted = cur.rowcount
-                conn.commit()
-                
-                if deleted > 0:
-                    logger.info(f"Cleaned up {deleted} detections for whitelisted IPs (CIDR-aware)")
-                
-                return deleted
-        except Exception as e:
-            conn.rollback()
-            logger.error(f"Failed to cleanup detections: {e}")
-            return 0
-        finally:
-            conn.close()
-    
-    def sync_public_blacklist_detections(self) -> Dict[str, int]:
-        """
-        Sync detections with current public blacklist state using BULK operations
-        Creates detections for blacklisted IPs (if not whitelisted)
-        Removes detections for IPs no longer blacklisted or now whitelisted
-        """
-        stats = {
-            'created': 0,
-            'cleaned': 0,
-            'skipped_whitelisted': 0
-        }
-        
-        conn = self.get_db_connection()
-        try:
-            with conn.cursor() as cur:
-                # Cleanup whitelisted IPs first (priority)
-                stats['cleaned'] = self.cleanup_invalid_detections()
-                
-                # Bulk create detections with CIDR-aware matching
-                # Uses PostgreSQL INET operators for network containment
-                # Priority: Manual whitelist > Public whitelist > Blacklist
-                cur.execute("""
-                    INSERT INTO detections (
-                        source_ip,
-                        risk_score,
-                        confidence,
-                        anomaly_type,
-                        reason,
-                        log_count,
-                        first_seen,
-                        last_seen,
-                        detection_source,
-                        blacklist_id,
-                        detected_at,
-                        blocked
-                    )
-                    SELECT DISTINCT
-                        bl.ip_address,
-                        75::numeric,
-                        100::numeric,
-                        'public_blacklist',
-                        'IP in public blacklist',
-                        1,
-                        NOW(),
-                        NOW(),
-                        'public_blacklist',
-                        bl.id,
-                        NOW(),
-                        false
-                    FROM public_blacklist_ips bl
-                    WHERE bl.is_active = true
-                    AND bl.ip_inet IS NOT NULL
-                    -- Priority 1: Exclude if in manual whitelist (highest priority)
-                    -- Cast to inet explicitly for type safety
-                    AND NOT EXISTS (
-                        SELECT 1 FROM whitelist wl
-                        WHERE wl.active = true
-                        AND wl.source = 'manual'
-                        AND wl.ip_inet IS NOT NULL
-                        AND (
-                            bl.ip_inet::inet = wl.ip_inet::inet
-                            OR bl.ip_inet::inet <<= wl.ip_inet::inet
-                        )
-                    )
-                    -- Priority 2: Exclude if in public whitelist
-                    AND NOT EXISTS (
-                        SELECT 1 FROM whitelist wl
-                        WHERE wl.active = true
-                        AND wl.source != 'manual'
-                        AND wl.ip_inet IS NOT NULL
-                        AND (
-                            bl.ip_inet::inet = wl.ip_inet::inet
-                            OR bl.ip_inet::inet <<= wl.ip_inet::inet
-                        )
-                    )
-                    -- Avoid duplicate detections
-                    AND NOT EXISTS (
-                        SELECT 1 FROM detections d
-                        WHERE d.source_ip = bl.ip_address
-                        AND d.detection_source = 'public_blacklist'
-                    )
-                    RETURNING id
-                """)
-                
-                created_ids = cur.fetchall()
-                stats['created'] = len(created_ids)
-                conn.commit()
-                
-                logger.info(f"Bulk sync complete: {stats}")
-                return stats
-        except Exception as e:
-            conn.rollback()
-            logger.error(f"Failed to sync detections: {e}")
-            import traceback
-            traceback.print_exc()
-            return stats
-        finally:
-            conn.close()
-
-
-def main():
-    """Run merge logic sync"""
-    database_url = os.environ.get('DATABASE_URL')
-    if not database_url:
-        logger.error("DATABASE_URL environment variable not set")
-        return 1
-    
-    merge = MergeLogic(database_url)
-    stats = merge.sync_public_blacklist_detections()
-    
-    print(f"\n{'='*60}")
-    print("MERGE LOGIC SYNC COMPLETED")
-    print(f"{'='*60}")
-    print(f"Created detections:          {stats['created']}")
-    print(f"Cleaned invalid detections:  {stats['cleaned']}")
-    print(f"Skipped (whitelisted):       {stats['skipped_whitelisted']}")
-    print(f"{'='*60}\n")
-    
-    return 0
-
-
-if __name__ == "__main__":
-    exit(main())

python_ml/mikrotik_manager.py

@@ -5,7 +5,6 @@ Più veloce e affidabile di SSH per 10+ router
 
 import httpx
 import asyncio
-import ssl
 from typing import List, Dict, Optional
 from datetime import datetime
 import hashlib
@@ -35,27 +34,11 @@ class MikroTikManager:
                 "Authorization": f"Basic {auth}",
                 "Content-Type": "application/json"
             }
-            
-            # SSL context per MikroTik (supporta protocolli TLS legacy)
-            ssl_context = None
-            if protocol == "https":
-                ssl_context = ssl.create_default_context()
-                ssl_context.check_hostname = False
-                ssl_context.verify_mode = ssl.CERT_NONE
-                # Abilita protocolli TLS legacy per MikroTik (TLS 1.0+)
-                try:
-                    ssl_context.minimum_version = ssl.TLSVersion.TLSv1
-                except AttributeError:
-                    # Python < 3.7 fallback
-                    pass
-                # Abilita cipher suite legacy per compatibilità
-                ssl_context.set_ciphers('DEFAULT@SECLEVEL=1')
-            
             self.clients[key] = httpx.AsyncClient(
                 base_url=f"{protocol}://{router_ip}:{port}",
                 headers=headers,
                 timeout=self.timeout,
-                verify=ssl_context if ssl_context else True
+                verify=False  # Disable SSL verification for self-signed certs
             )
         return self.clients[key]
     

python_ml/test_mikrotik_connection.py

@@ -117,10 +117,7 @@ async def test_router_connection(manager, router):
     
     except Exception as e:
         print(f"  ❌ Errore durante test: {e}")
-        print(f"  📋 Tipo errore: {type(e).__name__}")
+        print(f"  📋 Dettagli errore: {type(e).__name__}")
-        import traceback
-        print(f"  📋 Stack trace:")
-        traceback.print_exc()
         return False
 
 

python_ml/test_mikrotik_simple.py

@@ -1,93 +0,0 @@
-#!/usr/bin/env python3
-"""Test semplice connessione MikroTik - Debug"""
-
-import httpx
-import base64
-import asyncio
-
-async def test_simple():
-    print("🔍 Test Connessione MikroTik Semplificato\n")
-    
-    # Configurazione
-    router_ip = "185.203.24.2"
-    port = 8728
-    username = "admin"
-    password = input(f"Password per {username}@{router_ip}: ")
-    
-    # Test 1: Connessione TCP base
-    print(f"\n1️⃣ Test TCP porta {port}...")
-    try:
-        client = httpx.AsyncClient(timeout=5)
-        response = await client.get(f"http://{router_ip}:{port}")
-        print(f"   ✅ Porta {port} aperta e risponde")
-        await client.aclose()
-    except Exception as e:
-        print(f"   ❌ Porta {port} non raggiungibile: {e}")
-        return
-    
-    # Test 2: Endpoint REST /rest/system/identity
-    print(f"\n2️⃣ Test endpoint REST /rest/system/identity...")
-    try:
-        auth = base64.b64encode(f"{username}:{password}".encode()).decode()
-        headers = {
-            "Authorization": f"Basic {auth}",
-            "Content-Type": "application/json"
-        }
-        
-        client = httpx.AsyncClient(timeout=10)
-        url = f"http://{router_ip}:{port}/rest/system/identity"
-        print(f"   URL: {url}")
-        
-        response = await client.get(url, headers=headers)
-        print(f"   Status Code: {response.status_code}")
-        print(f"   Headers: {dict(response.headers)}")
-        
-        if response.status_code == 200:
-            print(f"   ✅ Autenticazione OK!")
-            print(f"   Risposta: {response.text}")
-        elif response.status_code == 401:
-            print(f"   ❌ Credenziali errate (401 Unauthorized)")
-        elif response.status_code == 404:
-            print(f"   ❌ Endpoint non trovato (404) - API REST non abilitata?")
-        else:
-            print(f"   ⚠️  Status inaspettato: {response.status_code}")
-            print(f"   Risposta: {response.text}")
-        
-        await client.aclose()
-        
-    except Exception as e:
-        print(f"   ❌ Errore richiesta REST: {e}")
-        import traceback
-        traceback.print_exc()
-        return
-    
-    # Test 3: Endpoint /rest/ip/firewall/address-list
-    print(f"\n3️⃣ Test endpoint address-list...")
-    try:
-        client = httpx.AsyncClient(timeout=10)
-        url = f"http://{router_ip}:{port}/rest/ip/firewall/address-list"
-        
-        response = await client.get(url, headers=headers)
-        print(f"   Status Code: {response.status_code}")
-        
-        if response.status_code == 200:
-            data = response.json()
-            print(f"   ✅ Address-list leggibile!")
-            print(f"   Totale entries: {len(data)}")
-            if data:
-                print(f"   Primo entry: {data[0]}")
-        else:
-            print(f"   ⚠️  Status: {response.status_code}")
-            print(f"   Risposta: {response.text}")
-        
-        await client.aclose()
-        
-    except Exception as e:
-        print(f"   ❌ Errore lettura address-list: {e}")
-        import traceback
-        traceback.print_exc()
-
-if __name__ == "__main__":
-    print("="*60)
-    asyncio.run(test_simple())
-    print("\n" + "="*60)

replit.md

@@ -24,13 +24,12 @@ The IDS employs a React-based frontend for real-time monitoring, detection visua
 
 **Key Architectural Decisions & Features:**
 -   **Log Collection & Processing**: MikroTik syslog data (UDP:514) is parsed by `syslog_parser.py` and stored in PostgreSQL with a 3-day retention policy. The parser includes auto-reconnect and error recovery mechanisms.
--   **Machine Learning**: An Isolation Forest model (sklearn.IsolectionForest) trained on 25 network log features performs real-time anomaly detection, assigning a risk score (0-100 across five risk levels). A hybrid ML detector (Isolation Forest + Ensemble Classifier with weighted voting) reduces false positives. The system supports weekly automatic retraining of models.
+-   **Machine Learning**: An Isolation Forest model (sklearn.IsolationForest) trained on 25 network log features performs real-time anomaly detection, assigning a risk score (0-100 across five risk levels). A hybrid ML detector (Isolation Forest + Ensemble Classifier with weighted voting) reduces false positives. The system supports weekly automatic retraining of models.
--   **Automated Blocking**: Critical IPs (score >= 80) are automatically blocked in parallel across configured MikroTik routers via their REST API. **Auto-unblock on whitelist**: When an IP is added to the whitelist, it is automatically removed from all router blocklists. Manual unblock button available in Detections page.
+-   **Automated Blocking**: Critical IPs (score >= 80) are automatically blocked in parallel across configured MikroTik routers via their REST API.
--   **Public Lists Integration (v2.0.0 - CIDR Complete)**: Automatic fetcher syncs blacklist/whitelist feeds every 10 minutes (Spamhaus, Talos, AWS, GCP, Cloudflare, IANA, NTP Pool). **Full CIDR support** using PostgreSQL INET/CIDR types with `<<=` containment operators for network range matching. Priority-based merge logic: Manual whitelist > Public whitelist > Blacklist (CIDR-aware). Detections created for blacklisted IPs/ranges (excluding whitelisted ranges). CRUD API + UI for list management. See `deployment/docs/PUBLIC_LISTS_V2_CIDR.md` for implementation details.
 -   **Automatic Cleanup**: An hourly systemd timer (`cleanup_detections.py`) removes old detections (48h) and auto-unblocks IPs (2h).
 -   **Service Monitoring & Management**: A dashboard provides real-time status (ML Backend, Database, Syslog Parser). API endpoints, secured with API key authentication and Systemd integration, allow for service management (start/stop/restart) of Python services.
 -   **IP Geolocation**: Integration with `ip-api.com` enriches detection data with geographical and AS information, utilizing intelligent caching.
--   **Database Management**: PostgreSQL is used for all persistent data. An intelligent database versioning system ensures efficient SQL migrations (v8 with forced INET/CIDR column types for network range matching). Migration 008 unconditionally recreates INET columns to fix type mismatches. Dual-mode database drivers (`@neondatabase/serverless` for Replit, `pg` for AlmaLinux) ensure environment compatibility.
+-   **Database Management**: PostgreSQL is used for all persistent data. An intelligent database versioning system ensures efficient SQL migrations. Dual-mode database drivers (`@neondatabase/serverless` for Replit, `pg` for AlmaLinux) ensure environment compatibility.
 -   **Microservices**: Clear separation of concerns between the Python ML backend and the Node.js API backend.
 -   **UI/UX**: Utilizes ShadCN UI for a modern component library and `react-hook-form` with Zod for robust form validation. Analytics dashboards provide visualizations of normal and attack traffic, including real-time and historical data.
 

server/routes.ts

@@ -1,7 +1,7 @@
 import type { Express } from "express";
 import { createServer, type Server } from "http";
 import { storage } from "./storage";
-import { insertRouterSchema, insertDetectionSchema, insertWhitelistSchema, insertPublicListSchema, networkAnalytics, routers } from "@shared/schema";
+import { insertRouterSchema, insertDetectionSchema, insertWhitelistSchema, networkAnalytics, routers } from "@shared/schema";
 import { db } from "./db";
 import { desc, eq } from "drizzle-orm";
 
@@ -77,22 +77,18 @@ export async function registerRoutes(app: Express): Promise<Server> {
   // Detections
   app.get("/api/detections", async (req, res) => {
     try {
-      const limit = req.query.limit ? parseInt(req.query.limit as string) : 50;
+      const limit = req.query.limit ? parseInt(req.query.limit as string) : 500;
-      const offset = req.query.offset ? parseInt(req.query.offset as string) : 0;
       const anomalyType = req.query.anomalyType as string | undefined;
       const minScore = req.query.minScore ? parseFloat(req.query.minScore as string) : undefined;
       const maxScore = req.query.maxScore ? parseFloat(req.query.maxScore as string) : undefined;
-      const search = req.query.search as string | undefined;
       
-      const result = await storage.getAllDetections({
+      const detections = await storage.getAllDetections({
         limit,
-        offset,
         anomalyType,
         minScore,
-        maxScore,
+        maxScore
-        search
       });
-      res.json(result);
+      res.json(detections);
     } catch (error) {
       console.error('[DB ERROR] Failed to fetch detections:', error);
       res.status(500).json({ error: "Failed to fetch detections" });
@@ -134,73 +130,11 @@ export async function registerRoutes(app: Express): Promise<Server> {
     try {
       const validatedData = insertWhitelistSchema.parse(req.body);
       const item = await storage.createWhitelist(validatedData);
-      
-      // Auto-unblock from routers when adding to whitelist
-      const mlBackendUrl = process.env.ML_BACKEND_URL || 'http://localhost:8000';
-      const mlApiKey = process.env.IDS_API_KEY;
-      try {
-        const headers: Record<string, string> = { 'Content-Type': 'application/json' };
-        if (mlApiKey) {
-          headers['X-API-Key'] = mlApiKey;
-        }
-        const unblockResponse = await fetch(`${mlBackendUrl}/unblock-ip`, {
-          method: 'POST',
-          headers,
-          body: JSON.stringify({ ip_address: validatedData.ipAddress })
-        });
-        if (unblockResponse.ok) {
-          const result = await unblockResponse.json();
-          console.log(`[WHITELIST] Auto-unblocked ${validatedData.ipAddress} from ${result.unblocked_from} routers`);
-        } else {
-          console.warn(`[WHITELIST] Failed to auto-unblock ${validatedData.ipAddress}: ${unblockResponse.status}`);
-        }
-      } catch (unblockError) {
-        // Don't fail if ML backend is unavailable
-        console.warn(`[WHITELIST] ML backend unavailable for auto-unblock: ${unblockError}`);
-      }
-      
       res.json(item);
     } catch (error) {
       res.status(400).json({ error: "Invalid whitelist data" });
     }
   });
-  
-  // Unblock IP from all routers (proxy to ML backend)
-  app.post("/api/unblock-ip", async (req, res) => {
-    try {
-      const { ipAddress, listName = "ddos_blocked" } = req.body;
-      
-      if (!ipAddress) {
-        return res.status(400).json({ error: "IP address is required" });
-      }
-      
-      const mlBackendUrl = process.env.ML_BACKEND_URL || 'http://localhost:8000';
-      const mlApiKey = process.env.IDS_API_KEY;
-      const headers: Record<string, string> = { 'Content-Type': 'application/json' };
-      if (mlApiKey) {
-        headers['X-API-Key'] = mlApiKey;
-      }
-      
-      const response = await fetch(`${mlBackendUrl}/unblock-ip`, {
-        method: 'POST',
-        headers,
-        body: JSON.stringify({ ip_address: ipAddress, list_name: listName })
-      });
-      
-      if (!response.ok) {
-        const errorText = await response.text();
-        console.error(`[UNBLOCK] ML backend error for ${ipAddress}: ${response.status} - ${errorText}`);
-        return res.status(response.status).json({ error: errorText || "Failed to unblock IP" });
-      }
-      
-      const result = await response.json();
-      console.log(`[UNBLOCK] Successfully unblocked ${ipAddress} from ${result.unblocked_from || 0} routers`);
-      res.json(result);
-    } catch (error: any) {
-      console.error('[UNBLOCK] Error:', error);
-      res.status(500).json({ error: error.message || "Failed to unblock IP from routers" });
-    }
-  });
 
   app.delete("/api/whitelist/:id", async (req, res) => {
     try {
@@ -214,214 +148,6 @@ export async function registerRoutes(app: Express): Promise<Server> {
     }
   });
 
-  // Public Lists
-  app.get("/api/public-lists", async (req, res) => {
-    try {
-      const lists = await storage.getAllPublicLists();
-      res.json(lists);
-    } catch (error) {
-      console.error('[DB ERROR] Failed to fetch public lists:', error);
-      res.status(500).json({ error: "Failed to fetch public lists" });
-    }
-  });
-
-  app.get("/api/public-lists/:id", async (req, res) => {
-    try {
-      const list = await storage.getPublicListById(req.params.id);
-      if (!list) {
-        return res.status(404).json({ error: "List not found" });
-      }
-      res.json(list);
-    } catch (error) {
-      res.status(500).json({ error: "Failed to fetch list" });
-    }
-  });
-
-  app.post("/api/public-lists", async (req, res) => {
-    try {
-      const validatedData = insertPublicListSchema.parse(req.body);
-      const list = await storage.createPublicList(validatedData);
-      res.json(list);
-    } catch (error: any) {
-      console.error('[API ERROR] Failed to create public list:', error);
-      if (error.name === 'ZodError') {
-        return res.status(400).json({ error: "Invalid list data", details: error.errors });
-      }
-      res.status(400).json({ error: "Invalid list data" });
-    }
-  });
-
-  app.patch("/api/public-lists/:id", async (req, res) => {
-    try {
-      const validatedData = insertPublicListSchema.partial().parse(req.body);
-      const list = await storage.updatePublicList(req.params.id, validatedData);
-      if (!list) {
-        return res.status(404).json({ error: "List not found" });
-      }
-      res.json(list);
-    } catch (error: any) {
-      console.error('[API ERROR] Failed to update public list:', error);
-      if (error.name === 'ZodError') {
-        return res.status(400).json({ error: "Invalid list data", details: error.errors });
-      }
-      res.status(400).json({ error: "Invalid list data" });
-    }
-  });
-
-  app.delete("/api/public-lists/:id", async (req, res) => {
-    try {
-      const success = await storage.deletePublicList(req.params.id);
-      if (!success) {
-        return res.status(404).json({ error: "List not found" });
-      }
-      res.json({ success: true });
-    } catch (error) {
-      res.status(500).json({ error: "Failed to delete list" });
-    }
-  });
-
-  app.post("/api/public-lists/:id/sync", async (req, res) => {
-    try {
-      const list = await storage.getPublicListById(req.params.id);
-      if (!list) {
-        return res.status(404).json({ error: "List not found" });
-      }
-      
-      console.log(`[SYNC] Starting sync for list: ${list.name} (${list.url})`);
-      
-      // Fetch the list from URL
-      const response = await fetch(list.url, {
-        headers: {
-          'User-Agent': 'IDS-MikroTik-PublicListFetcher/2.0',
-          'Accept': 'application/json, text/plain, */*',
-        },
-        signal: AbortSignal.timeout(30000),
-      });
-      
-      if (!response.ok) {
-        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
-      }
-      
-      const contentType = response.headers.get('content-type') || '';
-      const text = await response.text();
-      
-      // Parse IPs based on content type
-      let ips: Array<{ip: string, cidr?: string}> = [];
-      
-      if (contentType.includes('json') || list.url.endsWith('.json')) {
-        // JSON format (Spamhaus DROP v4 JSON)
-        try {
-          const data = JSON.parse(text);
-          if (Array.isArray(data)) {
-            for (const entry of data) {
-              if (entry.cidr) {
-                const [ip] = entry.cidr.split('/');
-                ips.push({ ip, cidr: entry.cidr });
-              } else if (entry.ip) {
-                ips.push({ ip: entry.ip, cidr: null as any });
-              }
-            }
-          }
-        } catch (e) {
-          console.error('[SYNC] Failed to parse JSON:', e);
-          throw new Error('Invalid JSON format');
-        }
-      } else {
-        // Plain text format (one IP/CIDR per line)
-        const lines = text.split('\n');
-        for (const line of lines) {
-          const trimmed = line.trim();
-          if (!trimmed || trimmed.startsWith('#') || trimmed.startsWith(';')) continue;
-          
-          // Extract IP/CIDR from line
-          const match = trimmed.match(/^(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})(\/\d{1,2})?/);
-          if (match) {
-            const ip = match[1];
-            const cidr = match[2] ? `${match[1]}${match[2]}` : null;
-            ips.push({ ip, cidr: cidr as any });
-          }
-        }
-      }
-      
-      console.log(`[SYNC] Parsed ${ips.length} IPs from ${list.name}`);
-      
-      // Save IPs to database
-      let added = 0;
-      let updated = 0;
-      
-      for (const { ip, cidr } of ips) {
-        const result = await storage.upsertBlacklistIp(list.id, ip, cidr);
-        if (result.created) added++;
-        else updated++;
-      }
-      
-      // Update list stats
-      await storage.updatePublicList(list.id, {
-        lastFetch: new Date(),
-        lastSuccess: new Date(),
-        totalIps: ips.length,
-        activeIps: ips.length,
-        errorCount: 0,
-        lastError: null,
-      });
-      
-      console.log(`[SYNC] Completed: ${added} added, ${updated} updated for ${list.name}`);
-      
-      res.json({ 
-        success: true, 
-        message: `Sync completed: ${ips.length} IPs processed`,
-        added,
-        updated,
-        total: ips.length,
-      });
-    } catch (error: any) {
-      console.error('[API ERROR] Failed to sync:', error);
-      
-      // Update error count
-      const list = await storage.getPublicListById(req.params.id);
-      if (list) {
-        await storage.updatePublicList(req.params.id, {
-          errorCount: (list.errorCount || 0) + 1,
-          lastError: error.message,
-          lastFetch: new Date(),
-        });
-      }
-      
-      res.status(500).json({ error: `Sync failed: ${error.message}` });
-    }
-  });
-
-  // Public Blacklist IPs
-  app.get("/api/public-blacklist", async (req, res) => {
-    try {
-      const limit = parseInt(req.query.limit as string) || 1000;
-      const listId = req.query.listId as string | undefined;
-      const ipAddress = req.query.ipAddress as string | undefined;
-      const isActive = req.query.isActive === 'true';
-      
-      const ips = await storage.getPublicBlacklistIps({
-        limit,
-        listId,
-        ipAddress,
-        isActive: req.query.isActive !== undefined ? isActive : undefined,
-      });
-      res.json(ips);
-    } catch (error) {
-      console.error('[DB ERROR] Failed to fetch blacklist IPs:', error);
-      res.status(500).json({ error: "Failed to fetch blacklist IPs" });
-    }
-  });
-
-  app.get("/api/public-blacklist/stats", async (req, res) => {
-    try {
-      const stats = await storage.getPublicBlacklistStats();
-      res.json(stats);
-    } catch (error) {
-      console.error('[DB ERROR] Failed to fetch blacklist stats:', error);
-      res.status(500).json({ error: "Failed to fetch stats" });
-    }
-  });
-
   // Training History
   app.get("/api/training-history", async (req, res) => {
     try {
@@ -478,15 +204,14 @@ export async function registerRoutes(app: Express): Promise<Server> {
   app.get("/api/stats", async (req, res) => {
     try {
       const routers = await storage.getAllRouters();
-      const detectionsResult = await storage.getAllDetections({ limit: 1000 });
+      const detections = await storage.getAllDetections({ limit: 1000 });
       const recentLogs = await storage.getRecentLogs(1000);
       const whitelist = await storage.getAllWhitelist();
       const latestTraining = await storage.getLatestTraining();
 
-      const detectionsList = detectionsResult.detections;
+      const blockedCount = detections.filter(d => d.blocked).length;
-      const blockedCount = detectionsList.filter(d => d.blocked).length;
+      const criticalCount = detections.filter(d => parseFloat(d.riskScore) >= 85).length;
-      const criticalCount = detectionsList.filter(d => parseFloat(d.riskScore) >= 85).length;
+      const highCount = detections.filter(d => parseFloat(d.riskScore) >= 70 && parseFloat(d.riskScore) < 85).length;
-      const highCount = detectionsList.filter(d => parseFloat(d.riskScore) >= 70 && parseFloat(d.riskScore) < 85).length;
 
       res.json({
         routers: {
@@ -494,7 +219,7 @@ export async function registerRoutes(app: Express): Promise<Server> {
           enabled: routers.filter(r => r.enabled).length
         },
         detections: {
-          total: detectionsResult.total,
+          total: detections.length,
           blocked: blockedCount,
           critical: criticalCount,
           high: highCount

server/storage.ts

@@ -5,8 +5,6 @@ import {
   whitelist,
   trainingHistory,
   networkAnalytics,
-  publicLists,
-  publicBlacklistIps,
   type Router,
   type InsertRouter,
   type NetworkLog,
@@ -18,10 +16,6 @@ import {
   type TrainingHistory,
   type InsertTrainingHistory,
   type NetworkAnalytics,
-  type PublicList,
-  type InsertPublicList,
-  type PublicBlacklistIp,
-  type InsertPublicBlacklistIp,
 } from "@shared/schema";
 import { db } from "./db";
 import { eq, desc, and, gte, sql, inArray } from "drizzle-orm";
@@ -43,12 +37,10 @@ export interface IStorage {
   // Detections
   getAllDetections(options: {
     limit?: number;
-    offset?: number;
     anomalyType?: string;
     minScore?: number;
     maxScore?: number;
-    search?: string;
+  }): Promise<Detection[]>;
-  }): Promise<{ detections: Detection[]; total: number }>;
   getDetectionByIp(sourceIp: string): Promise<Detection | undefined>;
   createDetection(detection: InsertDetection): Promise<Detection>;
   updateDetection(id: string, detection: Partial<InsertDetection>): Promise<Detection | undefined>;
@@ -82,27 +74,6 @@ export interface IStorage {
     recentDetections: Detection[];
   }>;
 
-  // Public Lists
-  getAllPublicLists(): Promise<PublicList[]>;
-  getPublicListById(id: string): Promise<PublicList | undefined>;
-  createPublicList(list: InsertPublicList): Promise<PublicList>;
-  updatePublicList(id: string, list: Partial<InsertPublicList>): Promise<PublicList | undefined>;
-  deletePublicList(id: string): Promise<boolean>;
-  
-  // Public Blacklist IPs
-  getPublicBlacklistIps(options: {
-    limit?: number;
-    listId?: string;
-    ipAddress?: string;
-    isActive?: boolean;
-  }): Promise<PublicBlacklistIp[]>;
-  getPublicBlacklistStats(): Promise<{
-    totalLists: number;
-    totalIps: number;
-    overlapWithDetections: number;
-  }>;
-  upsertBlacklistIp(listId: string, ipAddress: string, cidrRange: string | null): Promise<{created: boolean}>;
-
   // System
   testConnection(): Promise<boolean>;
 }
@@ -176,13 +147,11 @@ export class DatabaseStorage implements IStorage {
   // Detections
   async getAllDetections(options: {
     limit?: number;
-    offset?: number;
     anomalyType?: string;
     minScore?: number;
     maxScore?: number;
-    search?: string;
+  }): Promise<Detection[]> {
-  }): Promise<{ detections: Detection[]; total: number }> {
+    const { limit = 5000, anomalyType, minScore, maxScore } = options;
-    const { limit = 50, offset = 0, anomalyType, minScore, maxScore, search } = options;
     
     // Build WHERE conditions
     const conditions = [];
@@ -200,36 +169,17 @@ export class DatabaseStorage implements IStorage {
       conditions.push(sql`${detections.riskScore}::numeric <= ${maxScore}`);
     }
     
-    // Search by IP or anomaly type (case-insensitive)
+    const query = db
-    if (search && search.trim()) {
-      const searchLower = search.trim().toLowerCase();
-      conditions.push(sql`(
-        LOWER(${detections.sourceIp}) LIKE ${'%' + searchLower + '%'} OR
-        LOWER(${detections.anomalyType}) LIKE ${'%' + searchLower + '%'} OR
-        LOWER(COALESCE(${detections.country}, '')) LIKE ${'%' + searchLower + '%'} OR
-        LOWER(COALESCE(${detections.organization}, '')) LIKE ${'%' + searchLower + '%'}
-      )`);
-    }
-    
-    const whereClause = conditions.length > 0 ? and(...conditions) : undefined;
-    
-    // Get total count for pagination
-    const countResult = await db
-      .select({ count: sql<number>`count(*)::int` })
-      .from(detections)
-      .where(whereClause);
-    const total = countResult[0]?.count || 0;
-    
-    // Get paginated results
-    const results = await db
       .select()
       .from(detections)
-      .where(whereClause)
       .orderBy(desc(detections.detectedAt))
-      .limit(limit)
+      .limit(limit);
-      .offset(offset);
     
-    return { detections: results, total };
+    if (conditions.length > 0) {
+      return await query.where(and(...conditions));
+    }
+    
+    return await query;
   }
 
   async getDetectionByIp(sourceIp: string): Promise<Detection | undefined> {
@@ -437,150 +387,6 @@ export class DatabaseStorage implements IStorage {
     };
   }
 
-  // Public Lists
-  async getAllPublicLists(): Promise<PublicList[]> {
-    return await db.select().from(publicLists).orderBy(desc(publicLists.createdAt));
-  }
-
-  async getPublicListById(id: string): Promise<PublicList | undefined> {
-    const [list] = await db.select().from(publicLists).where(eq(publicLists.id, id));
-    return list || undefined;
-  }
-
-  async createPublicList(insertList: InsertPublicList): Promise<PublicList> {
-    const [list] = await db.insert(publicLists).values(insertList).returning();
-    return list;
-  }
-
-  async updatePublicList(id: string, updateData: Partial<InsertPublicList>): Promise<PublicList | undefined> {
-    const [list] = await db
-      .update(publicLists)
-      .set(updateData)
-      .where(eq(publicLists.id, id))
-      .returning();
-    return list || undefined;
-  }
-
-  async deletePublicList(id: string): Promise<boolean> {
-    const result = await db.delete(publicLists).where(eq(publicLists.id, id));
-    return result.rowCount !== null && result.rowCount > 0;
-  }
-
-  // Public Blacklist IPs
-  async getPublicBlacklistIps(options: {
-    limit?: number;
-    listId?: string;
-    ipAddress?: string;
-    isActive?: boolean;
-  }): Promise<PublicBlacklistIp[]> {
-    const { limit = 1000, listId, ipAddress, isActive } = options;
-    
-    const conditions = [];
-    
-    if (listId) {
-      conditions.push(eq(publicBlacklistIps.listId, listId));
-    }
-    
-    if (ipAddress) {
-      conditions.push(eq(publicBlacklistIps.ipAddress, ipAddress));
-    }
-    
-    if (isActive !== undefined) {
-      conditions.push(eq(publicBlacklistIps.isActive, isActive));
-    }
-    
-    const query = db
-      .select()
-      .from(publicBlacklistIps)
-      .orderBy(desc(publicBlacklistIps.lastSeen))
-      .limit(limit);
-    
-    if (conditions.length > 0) {
-      return await query.where(and(...conditions));
-    }
-    
-    return await query;
-  }
-
-  async getPublicBlacklistStats(): Promise<{
-    totalLists: number;
-    totalIps: number;
-    overlapWithDetections: number;
-  }> {
-    const lists = await db.select().from(publicLists).where(eq(publicLists.type, 'blacklist'));
-    const totalLists = lists.length;
-    
-    const [{ count: totalIps }] = await db
-      .select({ count: sql<number>`count(*)::int` })
-      .from(publicBlacklistIps)
-      .where(eq(publicBlacklistIps.isActive, true));
-    
-    const [{ count: overlapWithDetections }] = await db
-      .select({ count: sql<number>`count(distinct ${detections.sourceIp})::int` })
-      .from(detections)
-      .innerJoin(publicBlacklistIps, eq(detections.sourceIp, publicBlacklistIps.ipAddress))
-      .where(
-        and(
-          eq(publicBlacklistIps.isActive, true),
-          eq(detections.detectionSource, 'public_blacklist'),
-          sql`NOT EXISTS (
-            SELECT 1 FROM ${whitelist}
-            WHERE ${whitelist.ipAddress} = ${detections.sourceIp}
-            AND ${whitelist.active} = true
-          )`
-        )
-      );
-    
-    return {
-      totalLists,
-      totalIps: totalIps || 0,
-      overlapWithDetections: overlapWithDetections || 0,
-    };
-  }
-
-  async upsertBlacklistIp(listId: string, ipAddress: string, cidrRange: string | null): Promise<{created: boolean}> {
-    try {
-      const existing = await db
-        .select()
-        .from(publicBlacklistIps)
-        .where(
-          and(
-            eq(publicBlacklistIps.listId, listId),
-            eq(publicBlacklistIps.ipAddress, ipAddress)
-          )
-        );
-      
-      if (existing.length > 0) {
-        await db
-          .update(publicBlacklistIps)
-          .set({ 
-            lastSeen: new Date(),
-            isActive: true,
-            cidrRange: cidrRange,
-            ipInet: ipAddress,
-            cidrInet: cidrRange || `${ipAddress}/32`,
-          })
-          .where(eq(publicBlacklistIps.id, existing[0].id));
-        return { created: false };
-      } else {
-        await db.insert(publicBlacklistIps).values({
-          listId,
-          ipAddress,
-          cidrRange,
-          ipInet: ipAddress,
-          cidrInet: cidrRange || `${ipAddress}/32`,
-          isActive: true,
-          firstSeen: new Date(),
-          lastSeen: new Date(),
-        });
-        return { created: true };
-      }
-    } catch (error) {
-      console.error('[DB ERROR] Failed to upsert blacklist IP:', error);
-      throw error;
-    }
-  }
-
   async testConnection(): Promise<boolean> {
     try {
       await db.execute(sql`SELECT 1`);

shared/schema.ts

@@ -58,35 +58,23 @@ export const detections = pgTable("detections", {
   asNumber: text("as_number"),
   asName: text("as_name"),
   isp: text("isp"),
-  // Public lists integration
-  detectionSource: text("detection_source").notNull().default("ml_model"),
-  blacklistId: varchar("blacklist_id").references(() => publicBlacklistIps.id, { onDelete: 'set null' }),
 }, (table) => ({
   sourceIpIdx: index("detection_source_ip_idx").on(table.sourceIp),
   riskScoreIdx: index("risk_score_idx").on(table.riskScore),
   detectedAtIdx: index("detected_at_idx").on(table.detectedAt),
   countryIdx: index("country_idx").on(table.country),
-  detectionSourceIdx: index("detection_source_idx").on(table.detectionSource),
 }));
 
 // Whitelist per IP fidati
-// NOTE: ip_inet is INET type in production (managed by SQL migrations)
-// Drizzle lacks native INET support, so we use text() here
 export const whitelist = pgTable("whitelist", {
   id: varchar("id").primaryKey().default(sql`gen_random_uuid()`),
   ipAddress: text("ip_address").notNull().unique(),
-  ipInet: text("ip_inet"), // Actually INET in production - see migration 008
   comment: text("comment"),
   reason: text("reason"),
   createdBy: text("created_by"),
   active: boolean("active").notNull().default(true),
   createdAt: timestamp("created_at").defaultNow().notNull(),
-  // Public lists integration
+});
-  source: text("source").notNull().default("manual"),
-  listId: varchar("list_id").references(() => publicLists.id, { onDelete: 'set null' }),
-}, (table) => ({
-  sourceIdx: index("whitelist_source_idx").on(table.source),
-}));
 
 // ML Training history
 export const trainingHistory = pgTable("training_history", {
@@ -137,46 +125,6 @@ export const networkAnalytics = pgTable("network_analytics", {
   dateHourUnique: unique("network_analytics_date_hour_key").on(table.date, table.hour),
 }));
 
-// Public threat/whitelist sources
-export const publicLists = pgTable("public_lists", {
-  id: varchar("id").primaryKey().default(sql`gen_random_uuid()`),
-  name: text("name").notNull(),
-  type: text("type").notNull(),
-  url: text("url").notNull(),
-  enabled: boolean("enabled").notNull().default(true),
-  fetchIntervalMinutes: integer("fetch_interval_minutes").notNull().default(10),
-  lastFetch: timestamp("last_fetch"),
-  lastSuccess: timestamp("last_success"),
-  totalIps: integer("total_ips").notNull().default(0),
-  activeIps: integer("active_ips").notNull().default(0),
-  errorCount: integer("error_count").notNull().default(0),
-  lastError: text("last_error"),
-  createdAt: timestamp("created_at").defaultNow().notNull(),
-}, (table) => ({
-  typeIdx: index("public_lists_type_idx").on(table.type),
-  enabledIdx: index("public_lists_enabled_idx").on(table.enabled),
-}));
-
-// Public blacklist IPs from external sources
-// NOTE: ip_inet/cidr_inet are INET/CIDR types in production (managed by SQL migrations)
-// Drizzle lacks native INET/CIDR support, so we use text() here
-export const publicBlacklistIps = pgTable("public_blacklist_ips", {
-  id: varchar("id").primaryKey().default(sql`gen_random_uuid()`),
-  ipAddress: text("ip_address").notNull(),
-  cidrRange: text("cidr_range"),
-  ipInet: text("ip_inet"), // Actually INET in production - see migration 008
-  cidrInet: text("cidr_inet"), // Actually CIDR in production - see migration 008
-  listId: varchar("list_id").notNull().references(() => publicLists.id, { onDelete: 'cascade' }),
-  firstSeen: timestamp("first_seen").defaultNow().notNull(),
-  lastSeen: timestamp("last_seen").defaultNow().notNull(),
-  isActive: boolean("is_active").notNull().default(true),
-}, (table) => ({
-  ipAddressIdx: index("public_blacklist_ip_idx").on(table.ipAddress),
-  listIdIdx: index("public_blacklist_list_idx").on(table.listId),
-  isActiveIdx: index("public_blacklist_active_idx").on(table.isActive),
-  ipListUnique: unique("public_blacklist_ip_list_key").on(table.ipAddress, table.listId),
-}));
-
 // Schema version tracking for database migrations
 export const schemaVersion = pgTable("schema_version", {
   id: integer("id").primaryKey().default(1),
@@ -190,30 +138,7 @@ export const routersRelations = relations(routers, ({ many }) => ({
   logs: many(networkLogs),
 }));
 
-export const publicListsRelations = relations(publicLists, ({ many }) => ({
+// Rimossa relazione router (non più FK)
-  blacklistIps: many(publicBlacklistIps),
-}));
-
-export const publicBlacklistIpsRelations = relations(publicBlacklistIps, ({ one }) => ({
-  list: one(publicLists, {
-    fields: [publicBlacklistIps.listId],
-    references: [publicLists.id],
-  }),
-}));
-
-export const whitelistRelations = relations(whitelist, ({ one }) => ({
-  list: one(publicLists, {
-    fields: [whitelist.listId],
-    references: [publicLists.id],
-  }),
-}));
-
-export const detectionsRelations = relations(detections, ({ one }) => ({
-  blacklist: one(publicBlacklistIps, {
-    fields: [detections.blacklistId],
-    references: [publicBlacklistIps.id],
-  }),
-}));
 
 // Insert schemas
 export const insertRouterSchema = createInsertSchema(routers).omit({
@@ -251,19 +176,6 @@ export const insertNetworkAnalyticsSchema = createInsertSchema(networkAnalytics)
   createdAt: true,
 });
 
-export const insertPublicListSchema = createInsertSchema(publicLists).omit({
-  id: true,
-  createdAt: true,
-  lastFetch: true,
-  lastSuccess: true,
-});
-
-export const insertPublicBlacklistIpSchema = createInsertSchema(publicBlacklistIps).omit({
-  id: true,
-  firstSeen: true,
-  lastSeen: true,
-});
-
 // Types
 export type Router = typeof routers.$inferSelect;
 export type InsertRouter = z.infer<typeof insertRouterSchema>;
@@ -285,9 +197,3 @@ export type InsertSchemaVersion = z.infer<typeof insertSchemaVersionSchema>;
 
 export type NetworkAnalytics = typeof networkAnalytics.$inferSelect;
 export type InsertNetworkAnalytics = z.infer<typeof insertNetworkAnalyticsSchema>;
-
-export type PublicList = typeof publicLists.$inferSelect;
-export type InsertPublicList = z.infer<typeof insertPublicListSchema>;
-
-export type PublicBlacklistIp = typeof publicBlacklistIps.$inferSelect;
-export type InsertPublicBlacklistIp = z.infer<typeof insertPublicBlacklistIpSchema>;

uv.lock

@@ -1,101 +0,0 @@
-version = 1
-revision = 3
-requires-python = ">=3.11"
-
-[[package]]
-name = "anyio"
-version = "4.11.0"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "idna" },
-    { name = "sniffio" },
-    { name = "typing-extensions", marker = "python_full_version < '3.13'" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/c6/78/7d432127c41b50bccba979505f272c16cbcadcc33645d5fa3a738110ae75/anyio-4.11.0.tar.gz", hash = "sha256:82a8d0b81e318cc5ce71a5f1f8b5c4e63619620b63141ef8c995fa0db95a57c4", size = 219094, upload-time = "2025-09-23T09:19:12.58Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/15/b3/9b1a8074496371342ec1e796a96f99c82c945a339cd81a8e73de28b4cf9e/anyio-4.11.0-py3-none-any.whl", hash = "sha256:0287e96f4d26d4149305414d4e3bc32f0dcd0862365a4bddea19d7a1ec38c4fc", size = 109097, upload-time = "2025-09-23T09:19:10.601Z" },
-]
-
-[[package]]
-name = "certifi"
-version = "2025.11.12"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/a2/8c/58f469717fa48465e4a50c014a0400602d3c437d7c0c468e17ada824da3a/certifi-2025.11.12.tar.gz", hash = "sha256:d8ab5478f2ecd78af242878415affce761ca6bc54a22a27e026d7c25357c3316", size = 160538, upload-time = "2025-11-12T02:54:51.517Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/70/7d/9bc192684cea499815ff478dfcdc13835ddf401365057044fb721ec6bddb/certifi-2025.11.12-py3-none-any.whl", hash = "sha256:97de8790030bbd5c2d96b7ec782fc2f7820ef8dba6db909ccf95449f2d062d4b", size = 159438, upload-time = "2025-11-12T02:54:49.735Z" },
-]
-
-[[package]]
-name = "h11"
-version = "0.16.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/01/ee/02a2c011bdab74c6fb3c75474d40b3052059d95df7e73351460c8588d963/h11-0.16.0.tar.gz", hash = "sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1", size = 101250, upload-time = "2025-04-24T03:35:25.427Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/04/4b/29cac41a4d98d144bf5f6d33995617b185d14b22401f75ca86f384e87ff1/h11-0.16.0-py3-none-any.whl", hash = "sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86", size = 37515, upload-time = "2025-04-24T03:35:24.344Z" },
-]
-
-[[package]]
-name = "httpcore"
-version = "1.0.9"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "certifi" },
-    { name = "h11" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/06/94/82699a10bca87a5556c9c59b5963f2d039dbd239f25bc2a63907a05a14cb/httpcore-1.0.9.tar.gz", hash = "sha256:6e34463af53fd2ab5d807f399a9b45ea31c3dfa2276f15a2c3f00afff6e176e8", size = 85484, upload-time = "2025-04-24T22:06:22.219Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/7e/f5/f66802a942d491edb555dd61e3a9961140fd64c90bce1eafd741609d334d/httpcore-1.0.9-py3-none-any.whl", hash = "sha256:2d400746a40668fc9dec9810239072b40b4484b640a8c38fd654a024c7a1bf55", size = 78784, upload-time = "2025-04-24T22:06:20.566Z" },
-]
-
-[[package]]
-name = "httpx"
-version = "0.28.1"
-source = { registry = "https://pypi.org/simple" }
-dependencies = [
-    { name = "anyio" },
-    { name = "certifi" },
-    { name = "httpcore" },
-    { name = "idna" },
-]
-sdist = { url = "https://files.pythonhosted.org/packages/b1/df/48c586a5fe32a0f01324ee087459e112ebb7224f646c0b5023f5e79e9956/httpx-0.28.1.tar.gz", hash = "sha256:75e98c5f16b0f35b567856f597f06ff2270a374470a5c2392242528e3e3e42fc", size = 141406, upload-time = "2024-12-06T15:37:23.222Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/2a/39/e50c7c3a983047577ee07d2a9e53faf5a69493943ec3f6a384bdc792deb2/httpx-0.28.1-py3-none-any.whl", hash = "sha256:d909fcccc110f8c7faf814ca82a9a4d816bc5a6dbfea25d6591d6985b8ba59ad", size = 73517, upload-time = "2024-12-06T15:37:21.509Z" },
-]
-
-[[package]]
-name = "idna"
-version = "3.11"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/6f/6d/0703ccc57f3a7233505399edb88de3cbd678da106337b9fcde432b65ed60/idna-3.11.tar.gz", hash = "sha256:795dafcc9c04ed0c1fb032c2aa73654d8e8c5023a7df64a53f39190ada629902", size = 194582, upload-time = "2025-10-12T14:55:20.501Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/0e/61/66938bbb5fc52dbdf84594873d5b51fb1f7c7794e9c0f5bd885f30bc507b/idna-3.11-py3-none-any.whl", hash = "sha256:771a87f49d9defaf64091e6e6fe9c18d4833f140bd19464795bc32d966ca37ea", size = 71008, upload-time = "2025-10-12T14:55:18.883Z" },
-]
-
-[[package]]
-name = "repl-nix-workspace"
-version = "0.1.0"
-source = { virtual = "." }
-dependencies = [
-    { name = "httpx" },
-]
-
-[package.metadata]
-requires-dist = [{ name = "httpx", specifier = ">=0.28.1" }]
-
-[[package]]
-name = "sniffio"
-version = "1.3.1"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/a2/87/a6771e1546d97e7e041b6ae58d80074f81b7d5121207425c964ddf5cfdbd/sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc", size = 20372, upload-time = "2024-02-25T23:20:04.057Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/e9/44/75a9c9421471a6c4805dbf2356f7c181a29c1879239abab1ea2cc8f38b40/sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2", size = 10235, upload-time = "2024-02-25T23:20:01.196Z" },
-]
-
-[[package]]
-name = "typing-extensions"
-version = "4.15.0"
-source = { registry = "https://pypi.org/simple" }
-sdist = { url = "https://files.pythonhosted.org/packages/72/94/1a15dd82efb362ac84269196e94cf00f187f7ed21c242792a923cdb1c61f/typing_extensions-4.15.0.tar.gz", hash = "sha256:0cea48d173cc12fa28ecabc3b837ea3cf6f38c6d1136f85cbaaf598984861466", size = 109391, upload-time = "2025-08-25T13:49:26.313Z" }
-wheels = [
-    { url = "https://files.pythonhosted.org/packages/18/67/36e9267722cc04a6b9f15c7f3441c2363321a3ea07da7ae0c0707beb2a9c/typing_extensions-4.15.0-py3-none-any.whl", hash = "sha256:f0fa19c6845758ab08074a0cfa8b7aecb71c999ca73d62883bc25cc018c4e548", size = 44614, upload-time = "2025-08-25T13:49:24.86Z" },
-]

version.json

@@ -1,97 +1,7 @@
 {
-  "version": "1.0.103",
+  "version": "1.0.88",
-  "lastUpdate": "2026-01-02T16:33:13.545Z",
+  "lastUpdate": "2025-11-25T17:53:00.948Z",
   "changelog": [
-    {
-      "version": "1.0.103",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.103"
-    },
-    {
-      "version": "1.0.102",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.102"
-    },
-    {
-      "version": "1.0.101",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.101"
-    },
-    {
-      "version": "1.0.100",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.100"
-    },
-    {
-      "version": "1.0.99",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.99"
-    },
-    {
-      "version": "1.0.98",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.98"
-    },
-    {
-      "version": "1.0.97",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.97"
-    },
-    {
-      "version": "1.0.96",
-      "date": "2026-01-02",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.96"
-    },
-    {
-      "version": "1.0.95",
-      "date": "2025-11-27",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.95"
-    },
-    {
-      "version": "1.0.94",
-      "date": "2025-11-27",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.94"
-    },
-    {
-      "version": "1.0.93",
-      "date": "2025-11-27",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.93"
-    },
-    {
-      "version": "1.0.92",
-      "date": "2025-11-27",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.92"
-    },
-    {
-      "version": "1.0.91",
-      "date": "2025-11-26",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.91"
-    },
-    {
-      "version": "1.0.90",
-      "date": "2025-11-26",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.90"
-    },
-    {
-      "version": "1.0.89",
-      "date": "2025-11-25",
-      "type": "patch",
-      "description": "Deployment automatico v1.0.89"
-    },
     {
       "version": "1.0.88",
       "date": "2025-11-25",
@@ -301,6 +211,96 @@
       "date": "2025-11-24",
       "type": "patch",
       "description": "Deployment automatico v1.0.54"
+    },
+    {
+      "version": "1.0.53",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.53"
+    },
+    {
+      "version": "1.0.52",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.52"
+    },
+    {
+      "version": "1.0.51",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.51"
+    },
+    {
+      "version": "1.0.50",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.50"
+    },
+    {
+      "version": "1.0.49",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.49"
+    },
+    {
+      "version": "1.0.48",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.48"
+    },
+    {
+      "version": "1.0.47",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.47"
+    },
+    {
+      "version": "1.0.46",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.46"
+    },
+    {
+      "version": "1.0.45",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.45"
+    },
+    {
+      "version": "1.0.44",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.44"
+    },
+    {
+      "version": "1.0.43",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.43"
+    },
+    {
+      "version": "1.0.42",
+      "date": "2025-11-24",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.42"
+    },
+    {
+      "version": "1.0.41",
+      "date": "2025-11-22",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.41"
+    },
+    {
+      "version": "1.0.40",
+      "date": "2025-11-22",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.40"
+    },
+    {
+      "version": "1.0.39",
+      "date": "2025-11-22",
+      "type": "patch",
+      "description": "Deployment automatico v1.0.39"
     }
   ]
 }