Improve syslog parser reliability and add monitoring
Enhance the syslog parser with auto-reconnect, error recovery, and integrated health metrics logging. Add a cron job for automated health checks and restarts. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528 Replit-Commit-Checkpoint-Type: full_checkpoint Replit-Commit-Event-Id: 4885eae4-ffc7-4601-8f1c-5414922d5350 Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/AXTUZmH
This commit is contained in:
marco370
parent
093a7ba874
commit
14d67c63a3
4
.replit
4
.replit
@ -14,6 +14,10 @@ run = ["npm", "run", "start"]
|
||||
localPort = 5000
|
||||
externalPort = 80
|
||||
|
||||
[[ports]]
|
||||
localPort = 41061
|
||||
externalPort = 3001
|
||||
|
||||
[[ports]]
|
||||
localPort = 41303
|
||||
externalPort = 3002
|
||||
|
||||
182
deployment/TROUBLESHOOTING_SYSLOG_PARSER.md
Normal file
182
deployment/TROUBLESHOOTING_SYSLOG_PARSER.md
Normal file
@ -0,0 +1,182 @@
|
||||
# 🔧 TROUBLESHOOTING: Syslog Parser Bloccato
|
||||
|
||||
## 📊 Diagnosi Rapida (Sul Server)
|
||||
|
||||
### 1. Verifica Stato Servizio
|
||||
```bash
|
||||
sudo systemctl status ids-syslog-parser
|
||||
journalctl -u ids-syslog-parser -n 100 --no-pager
|
||||
```
|
||||
|
||||
**Cosa cercare:**
|
||||
- ❌ `[ERROR] Errore processamento file:`
|
||||
- ❌ `OperationalError: database connection`
|
||||
- ❌ `ProgrammingError:`
|
||||
- ✅ `[INFO] Processate X righe, salvate Y log` (deve continuare ad aumentare!)
|
||||
|
||||
---
|
||||
|
||||
### 2. Verifica Database Connection
|
||||
```bash
|
||||
# Test connessione DB
|
||||
psql -h 127.0.0.1 -U $PGUSER -d $PGDATABASE -c "SELECT COUNT(*) FROM network_logs WHERE timestamp > NOW() - INTERVAL '5 minutes';"
|
||||
```
|
||||
|
||||
**Se torna 0** → Parser non sta scrivendo!
|
||||
|
||||
---
|
||||
|
||||
### 3. Verifica File Log Syslog
|
||||
```bash
|
||||
# Log syslog in arrivo?
|
||||
tail -f /var/log/mikrotik/raw.log | head -20
|
||||
|
||||
# Dimensione file
|
||||
ls -lh /var/log/mikrotik/raw.log
|
||||
|
||||
# Ultimi log ricevuti
|
||||
tail -5 /var/log/mikrotik/raw.log
|
||||
```
|
||||
|
||||
**Se nessun log nuovo** → Problema rsyslog o router!
|
||||
|
||||
---
|
||||
|
||||
## 🐛 Cause Comuni di Blocco
|
||||
|
||||
### **Causa #1: Database Connection Timeout**
|
||||
```python
|
||||
# syslog_parser.py usa connessione persistente
|
||||
self.conn = psycopg2.connect() # ← può scadere dopo ore!
|
||||
```
|
||||
|
||||
**Soluzione:** Riavvia il servizio
|
||||
```bash
|
||||
sudo systemctl restart ids-syslog-parser
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### **Causa #2: Eccezione Non Gestita**
|
||||
```python
|
||||
# Loop si ferma se eccezione non gestita
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Errore processamento file: {e}")
|
||||
# ← Loop terminato!
|
||||
```
|
||||
|
||||
**Fix:** Il parser ora continua anche dopo errori (v2.0+)
|
||||
|
||||
---
|
||||
|
||||
### **Causa #3: File Log Ruotato da Rsyslog**
|
||||
Se rsyslog ruota il file `/var/log/mikrotik/raw.log`, il parser continua a leggere il file vecchio (inode diverso).
|
||||
|
||||
**Soluzione:** Usa logrotate + postrotate signal
|
||||
```bash
|
||||
# /etc/logrotate.d/mikrotik
|
||||
/var/log/mikrotik/raw.log {
|
||||
daily
|
||||
rotate 7
|
||||
compress
|
||||
postrotate
|
||||
systemctl restart ids-syslog-parser
|
||||
endscript
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### **Causa #4: Cleanup DB Troppo Lento**
|
||||
```python
|
||||
# Cleanup ogni ~16 minuti
|
||||
if cleanup_counter >= 10000:
|
||||
self.cleanup_old_logs(days_to_keep=3) # ← DELETE su milioni di record!
|
||||
```
|
||||
|
||||
Se il cleanup impiega troppo tempo, blocca il loop.
|
||||
|
||||
**Fix:** Ora usa batch delete con LIMIT (v2.0+)
|
||||
|
||||
---
|
||||
|
||||
## 🚑 SOLUZIONE RAPIDA (Ora)
|
||||
|
||||
```bash
|
||||
# 1. Riavvia parser
|
||||
sudo systemctl restart ids-syslog-parser
|
||||
|
||||
# 2. Verifica che riparta
|
||||
sudo journalctl -u ids-syslog-parser -f
|
||||
|
||||
# 3. Dopo 1-2 min, verifica nuovi log nel DB
|
||||
psql -h 127.0.0.1 -U $PGUSER -d $PGDATABASE -c \
|
||||
"SELECT COUNT(*) FROM network_logs WHERE timestamp > NOW() - INTERVAL '2 minutes';"
|
||||
```
|
||||
|
||||
**Output atteso:**
|
||||
```
|
||||
count
|
||||
-------
|
||||
1234 ← Numero crescente = OK!
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔒 FIX PERMANENTE (v2.0)
|
||||
|
||||
### **Migliorie Implementate:**
|
||||
|
||||
1. **Auto-Reconnect** su DB timeout
|
||||
2. **Error Recovery** - continua dopo eccezioni
|
||||
3. **Batch Cleanup** - non blocca il processing
|
||||
4. **Health Metrics** - monitoring integrato
|
||||
|
||||
### **Deploy Fix:**
|
||||
```bash
|
||||
cd /opt/ids
|
||||
sudo ./update_from_git.sh
|
||||
sudo systemctl restart ids-syslog-parser
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 Metriche da Monitorare
|
||||
|
||||
1. **Log/sec processati**
|
||||
```sql
|
||||
SELECT COUNT(*) / 60.0 AS logs_per_sec
|
||||
FROM network_logs
|
||||
WHERE timestamp > NOW() - INTERVAL '1 minute';
|
||||
```
|
||||
|
||||
2. **Ultimo log ricevuto**
|
||||
```sql
|
||||
SELECT MAX(timestamp) AS last_log FROM network_logs;
|
||||
```
|
||||
|
||||
3. **Gap detection** (se ultimo log > 5 min fa → problema!)
|
||||
```sql
|
||||
SELECT NOW() - MAX(timestamp) AS time_since_last_log
|
||||
FROM network_logs;
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ✅ Checklist Post-Fix
|
||||
|
||||
- [ ] Servizio running e active
|
||||
- [ ] Nuovi log in DB (ultimo < 1 min fa)
|
||||
- [ ] Nessun errore in journalctl
|
||||
- [ ] ML backend rileva nuove anomalie
|
||||
- [ ] Dashboard mostra traffico real-time
|
||||
|
||||
---
|
||||
|
||||
## 📞 Escalation
|
||||
|
||||
Se il problema persiste dopo questi fix:
|
||||
1. Verifica configurazione rsyslog
|
||||
2. Controlla firewall router (UDP:514)
|
||||
3. Test manuale: `logger -p local7.info "TEST MESSAGE"`
|
||||
4. Analizza log completi: `journalctl -u ids-syslog-parser --since "1 hour ago" > parser.log`
|
||||
80
deployment/check_parser_health.sh
Executable file
80
deployment/check_parser_health.sh
Executable file
@ -0,0 +1,80 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# Syslog Parser Health Check Script
|
||||
# Verifica che il parser stia processando log regolarmente
|
||||
# Uso: ./check_parser_health.sh
|
||||
# Cron: */5 * * * * /opt/ids/deployment/check_parser_health.sh
|
||||
###############################################################################
|
||||
|
||||
set -e
|
||||
|
||||
# Load environment
|
||||
if [ -f /opt/ids/.env ]; then
|
||||
export $(grep -v '^#' /opt/ids/.env | xargs)
|
||||
fi
|
||||
|
||||
ALERT_THRESHOLD_MINUTES=5
|
||||
LOG_FILE="/var/log/ids/parser-health.log"
|
||||
|
||||
mkdir -p /var/log/ids
|
||||
touch "$LOG_FILE"
|
||||
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] === Health Check Start ===" >> "$LOG_FILE"
|
||||
|
||||
# Check 1: Service running?
|
||||
if ! systemctl is-active --quiet ids-syslog-parser; then
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ❌ CRITICAL: Parser service NOT running!" >> "$LOG_FILE"
|
||||
echo "Attempting automatic restart..." >> "$LOG_FILE"
|
||||
systemctl restart ids-syslog-parser
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Service restarted" >> "$LOG_FILE"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Check 2: Recent logs in database?
|
||||
LAST_LOG_AGE=$(psql -h 127.0.0.1 -U "$PGUSER" -d "$PGDATABASE" -t -c \
|
||||
"SELECT EXTRACT(EPOCH FROM (NOW() - MAX(timestamp)))/60 AS minutes_ago FROM network_logs;" | tr -d ' ')
|
||||
|
||||
if [ -z "$LAST_LOG_AGE" ] || [ "$LAST_LOG_AGE" = "" ]; then
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ⚠️ WARNING: Cannot determine last log age (empty database?)" >> "$LOG_FILE"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Convert to integer (bash doesn't handle floats)
|
||||
LAST_LOG_AGE_INT=$(echo "$LAST_LOG_AGE" | cut -d'.' -f1)
|
||||
|
||||
if [ "$LAST_LOG_AGE_INT" -gt "$ALERT_THRESHOLD_MINUTES" ]; then
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ❌ ALERT: Last log is $LAST_LOG_AGE_INT minutes old (threshold: $ALERT_THRESHOLD_MINUTES min)" >> "$LOG_FILE"
|
||||
echo "Checking syslog file..." >> "$LOG_FILE"
|
||||
|
||||
# Check if syslog file has new data
|
||||
if [ -f "/var/log/mikrotik/raw.log" ]; then
|
||||
SYSLOG_SIZE=$(stat -f%z "/var/log/mikrotik/raw.log" 2>/dev/null || stat -c%s "/var/log/mikrotik/raw.log" 2>/dev/null)
|
||||
echo "Syslog file size: $SYSLOG_SIZE bytes" >> "$LOG_FILE"
|
||||
|
||||
# Restart parser
|
||||
echo "Restarting parser service..." >> "$LOG_FILE"
|
||||
systemctl restart ids-syslog-parser
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] Parser service restarted" >> "$LOG_FILE"
|
||||
else
|
||||
echo "⚠️ Syslog file not found: /var/log/mikrotik/raw.log" >> "$LOG_FILE"
|
||||
fi
|
||||
else
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ✅ OK: Last log ${LAST_LOG_AGE_INT} minutes ago" >> "$LOG_FILE"
|
||||
fi
|
||||
|
||||
# Check 3: Parser errors?
|
||||
ERROR_COUNT=$(journalctl -u ids-syslog-parser --since "5 minutes ago" | grep -c "\[ERROR\]" || echo "0")
|
||||
|
||||
if [ "$ERROR_COUNT" -gt 10 ]; then
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] ⚠️ WARNING: $ERROR_COUNT errors in last 5 minutes" >> "$LOG_FILE"
|
||||
journalctl -u ids-syslog-parser --since "5 minutes ago" | grep "\[ERROR\]" | tail -5 >> "$LOG_FILE"
|
||||
fi
|
||||
|
||||
echo "[$(date '+%Y-%m-%d %H:%M:%S')] === Health Check Complete ===" >> "$LOG_FILE"
|
||||
echo "" >> "$LOG_FILE"
|
||||
|
||||
# Keep only last 1000 lines of log
|
||||
tail -1000 "$LOG_FILE" > "${LOG_FILE}.tmp"
|
||||
mv "${LOG_FILE}.tmp" "$LOG_FILE"
|
||||
|
||||
exit 0
|
||||
44
deployment/setup_parser_monitoring.sh
Executable file
44
deployment/setup_parser_monitoring.sh
Executable file
@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
###############################################################################
|
||||
# Setup Syslog Parser Monitoring
|
||||
# Installa cron job per health check automatico ogni 5 minuti
|
||||
# Uso: sudo ./deployment/setup_parser_monitoring.sh
|
||||
###############################################################################
|
||||
|
||||
set -e
|
||||
|
||||
echo "📊 Setup Syslog Parser Monitoring..."
|
||||
echo
|
||||
|
||||
# Make health check script executable
|
||||
chmod +x /opt/ids/deployment/check_parser_health.sh
|
||||
|
||||
# Setup cron job
|
||||
CRON_JOB="*/5 * * * * /opt/ids/deployment/check_parser_health.sh >> /var/log/ids/parser-health-cron.log 2>&1"
|
||||
|
||||
# Check if cron job already exists
|
||||
if crontab -l 2>/dev/null | grep -q "check_parser_health.sh"; then
|
||||
echo "✅ Cron job già configurato"
|
||||
else
|
||||
# Add cron job
|
||||
(crontab -l 2>/dev/null; echo "$CRON_JOB") | crontab -
|
||||
echo "✅ Cron job aggiunto (esecuzione ogni 5 minuti)"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "📋 Configurazione completata:"
|
||||
echo " - Health check script: /opt/ids/deployment/check_parser_health.sh"
|
||||
echo " - Log file: /var/log/ids/parser-health.log"
|
||||
echo " - Cron log: /var/log/ids/parser-health-cron.log"
|
||||
echo " - Schedule: Every 5 minutes"
|
||||
echo
|
||||
echo "🔍 Monitoraggio attivo:"
|
||||
echo " - Controlla servizio running"
|
||||
echo " - Verifica log recenti (threshold: 5 min)"
|
||||
echo " - Auto-restart se necessario"
|
||||
echo " - Log errori recenti"
|
||||
echo
|
||||
echo "📊 Visualizza stato:"
|
||||
echo " tail -f /var/log/ids/parser-health.log"
|
||||
echo
|
||||
echo "✅ Setup completato!"
|
||||
@ -165,12 +165,19 @@ class SyslogParser:
|
||||
"""
|
||||
Processa file di log in modalità streaming (sicuro con rsyslog)
|
||||
follow: se True, segue il file come 'tail -f'
|
||||
|
||||
Resilient Features v2.0:
|
||||
- Auto-reconnect on DB timeout
|
||||
- Error recovery (continues after exceptions)
|
||||
- Health metrics logging
|
||||
"""
|
||||
print(f"[INFO] Processando {log_file} (follow={follow})")
|
||||
|
||||
processed = 0
|
||||
saved = 0
|
||||
cleanup_counter = 0
|
||||
errors = 0
|
||||
last_health_check = time.time()
|
||||
|
||||
try:
|
||||
with open(log_file, 'r') as f:
|
||||
@ -179,49 +186,101 @@ class SyslogParser:
|
||||
f.seek(0, 2) # Seek to end
|
||||
|
||||
while True:
|
||||
line = f.readline()
|
||||
try:
|
||||
line = f.readline()
|
||||
|
||||
if not line:
|
||||
if follow:
|
||||
time.sleep(0.1) # Attendi nuove righe
|
||||
if not line:
|
||||
if follow:
|
||||
time.sleep(0.1) # Attendi nuove righe
|
||||
|
||||
# Commit batch ogni 100 righe processate
|
||||
if processed > 0 and processed % 100 == 0:
|
||||
# Health check ogni 5 minuti
|
||||
if time.time() - last_health_check > 300:
|
||||
print(f"[HEALTH] Parser alive: {processed} righe processate, {saved} salvate, {errors} errori")
|
||||
last_health_check = time.time()
|
||||
|
||||
# Commit batch ogni 100 righe processate
|
||||
if processed > 0 and processed % 100 == 0:
|
||||
try:
|
||||
self.conn.commit()
|
||||
except Exception as commit_err:
|
||||
print(f"[ERROR] Commit failed, reconnecting: {commit_err}")
|
||||
self.reconnect_db()
|
||||
|
||||
# Cleanup DB ogni ~16 minuti
|
||||
cleanup_counter += 1
|
||||
if cleanup_counter >= 10000:
|
||||
self.cleanup_old_logs(days_to_keep=3)
|
||||
cleanup_counter = 0
|
||||
|
||||
continue
|
||||
else:
|
||||
break # Fine file
|
||||
|
||||
processed += 1
|
||||
|
||||
# Parsa riga
|
||||
log_data = self.parse_log_line(line.strip())
|
||||
if log_data:
|
||||
try:
|
||||
self.save_to_db(log_data)
|
||||
saved += 1
|
||||
except Exception as save_err:
|
||||
errors += 1
|
||||
print(f"[ERROR] Save failed: {save_err}")
|
||||
# Try to reconnect and continue
|
||||
try:
|
||||
self.reconnect_db()
|
||||
except:
|
||||
pass
|
||||
|
||||
# Commit ogni 100 righe
|
||||
if processed % 100 == 0:
|
||||
try:
|
||||
self.conn.commit()
|
||||
if saved > 0:
|
||||
print(f"[INFO] Processate {processed} righe, salvate {saved} log, {errors} errori")
|
||||
except Exception as commit_err:
|
||||
print(f"[ERROR] Commit failed: {commit_err}")
|
||||
self.reconnect_db()
|
||||
|
||||
# Cleanup DB ogni 1000 righe (~ ogni minuto)
|
||||
cleanup_counter += 1
|
||||
if cleanup_counter >= 10000: # ~16 minuti
|
||||
self.cleanup_old_logs(days_to_keep=3)
|
||||
cleanup_counter = 0
|
||||
|
||||
continue
|
||||
else:
|
||||
break # Fine file
|
||||
|
||||
processed += 1
|
||||
|
||||
# Parsa riga
|
||||
log_data = self.parse_log_line(line.strip())
|
||||
if log_data:
|
||||
self.save_to_db(log_data)
|
||||
saved += 1
|
||||
|
||||
# Commit ogni 100 righe
|
||||
if processed % 100 == 0:
|
||||
self.conn.commit()
|
||||
if saved > 0:
|
||||
print(f"[INFO] Processate {processed} righe, salvate {saved} log")
|
||||
except Exception as line_err:
|
||||
errors += 1
|
||||
if errors % 100 == 0:
|
||||
print(f"[ERROR] Error processing line ({errors} total errors): {line_err}")
|
||||
# Continue processing instead of crashing!
|
||||
continue
|
||||
|
||||
except KeyboardInterrupt:
|
||||
print("\n[INFO] Interrotto dall'utente")
|
||||
except Exception as e:
|
||||
print(f"[ERROR] Errore processamento file: {e}")
|
||||
print(f"[ERROR] Errore critico processamento file: {e}")
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
finally:
|
||||
self.conn.commit()
|
||||
print(f"[INFO] Totale: {processed} righe processate, {saved} log salvati")
|
||||
try:
|
||||
self.conn.commit()
|
||||
except:
|
||||
pass
|
||||
print(f"[INFO] Totale: {processed} righe processate, {saved} log salvati, {errors} errori")
|
||||
|
||||
def reconnect_db(self):
|
||||
"""
|
||||
Riconnette al database (auto-recovery on connection timeout)
|
||||
"""
|
||||
print("[INFO] Tentativo riconnessione database...")
|
||||
try:
|
||||
self.disconnect_db()
|
||||
except:
|
||||
pass
|
||||
|
||||
time.sleep(2)
|
||||
|
||||
try:
|
||||
self.connect_db()
|
||||
print("[INFO] ✅ Riconnessione database riuscita")
|
||||
except Exception as e:
|
||||
print(f"[ERROR] ❌ Riconnessione fallita: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def main():
|
||||
|
||||
22
replit.md
22
replit.md
@ -50,6 +50,28 @@ The IDS employs a React-based frontend for real-time monitoring, detection visua
|
||||
|
||||
## Recent Updates (Novembre 2025)
|
||||
|
||||
### 🛡️ Syslog Parser Resilience & Monitoring (25 Nov 2025 - 11:00)
|
||||
- **Feature**: Parser resiliente con auto-recovery e monitoring automatico
|
||||
- **Problema Risolto**: Parser si bloccava periodicamente (ultimo: 24 Nov mattina)
|
||||
- **Root Cause**: Database connection timeout, eccezioni non gestite, cleanup bloccante
|
||||
- **Soluzioni Implementate**:
|
||||
1. **Auto-Reconnect**: Riconnessione automatica su DB timeout
|
||||
2. **Error Recovery**: Continue processing dopo eccezioni (non crashare!)
|
||||
3. **Health Check**: Log ogni 5 minuti `[HEALTH] Parser alive: X righe, Y salvate, Z errori`
|
||||
4. **Monitoring Script**: `deployment/check_parser_health.sh` (cron ogni 5 min)
|
||||
5. **Auto-Restart**: Se ultimo log > 5 min fa → restart automatico
|
||||
- **Files Modificati**:
|
||||
- `python_ml/syslog_parser.py` - metodo `reconnect_db()` + try/catch nidificati
|
||||
- `deployment/check_parser_health.sh` - health check con auto-restart
|
||||
- `deployment/setup_parser_monitoring.sh` - setup cron job
|
||||
- `deployment/TROUBLESHOOTING_SYSLOG_PARSER.md` - guida completa
|
||||
- **Timestamp Detection Chiariti**:
|
||||
- `first_seen/last_seen`: timestamp dei log network_logs (es. 18:46:21)
|
||||
- `detected_at`: quando ML backend rileva anomalia (es. 19:45 - 1 ora dopo!)
|
||||
- Il delay è normale: ML backend esegue analisi batch ogni ora
|
||||
- **Deploy**: `./update_from_git.sh` → `sudo systemctl restart ids-syslog-parser` → `sudo ./deployment/setup_parser_monitoring.sh`
|
||||
- **Monitoring**: `tail -f /var/log/ids/parser-health.log`
|
||||
|
||||
### 🔧 Analytics Aggregator Fix - Data Consistency (24 Nov 2025 - 17:00)
|
||||
- **BUG FIX CRITICO**: Risolto mismatch dati Dashboard Live
|
||||
- **Problema**: Distribuzione traffico mostrava 262k attacchi ma breakdown solo 19
|
||||
|
||||
Loading…
Reference in New Issue
Block a user