marco/ids.alfacom.it
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Fix log parsing by adding missing timestamps to incoming data

Browse Source 
The attached log file and agent reasoning indicate that the `ids-syslog-parser` service failed to restart and the `/var/log/mikrotik/raw.log` file lacks timestamps, leading to the parser saving 0 logs. The provided solution involves manually updating the `/etc/rsyslog.d/99-mikrotik.conf` file to include the `%TIMESTAMP%` directive, restarting the `rsyslog` service, clearing and restarting the log file, and then restarting the `ids-syslog-parser` service to process logs correctly.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 9fa8bbb2-1781-4d01-b6d3-3b872fb304a3
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/6ZTQSoP
This commit is contained in:
marco370 2025-11-22 09:02:34 +00:00
parent 9448d54156
commit 24b907e17b
1 changed files with 208 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

208
attached_assets/Pasted-sudo-systemctl-restart-ids-syslog-parser-Failed-to-restart-ids-syslog-parser-service-Unit-ids-syslo-1763802098263_1763802098263.txt Normal file
View File

@ -0,0 +1,208 @@
sudo systemctl restart ids-syslog-parser
Failed to restart ids-syslog-parser.service: Unit ids-syslog-parser.service not found.
[root@ids python_ml]# tail -10 /var/log/mikrotik/raw.log
forward: in:<pppoe-cava.pompe-1> out:sfp-sfpplus2_VS_AS, connection-state:new src-mac 24:5a:4c:3e:a8:2a, proto UDP, 10.0.249.130:44595->165.154.165.238:8800, len 68
forward: in:<pppoe-cava.pompe-1> out:sfp-sfpplus2_VS_AS, connection-state:new src-mac 24:5a:4c:3e:a8:2a, proto UDP, 10.0.249.130:44595->165.154.165.238:8800, len 68
forward: in:<pppoe-cava.pompe-1> out:sfp-sfpplus2_VS_AS, connection-state:new src-mac 24:5a:4c:3e:a8:2a, proto UDP, 10.0.249.130:44594->93.150.220.226:4917, len 72
forward: in:<pppoe-cava.pompe-1> out:sfp-sfpplus2_VS_AS, connection-state:new src-mac 24:5a:4c:3e:a8:2a, proto UDP, 10.0.249.130:44594->93.150.220.226:4917, len 72
forward: in:<pppoe-caronte.hightek_01> out:sfp-sfpplus2_VS_AS, connection-state:new proto TCP (SYN), 185.203.25.233:56352->192.168.25.254:80, len 60
forward: in:<pppoe-caronte.hightek_01> out:sfp-sfpplus2_VS_AS, connection-state:new proto TCP (SYN), 185.203.25.233:56352->192.168.25.254:80, len 60
detected-ddos forward: in:sfp-sfpplus2_VS_AS out:<pppoe-alfabitomega>, connection-state:new src-mac 18:fd:74:7c:aa:85, proto TCP (SYN), 5.99.210.125:23084->185.203.24.2:10204, len 60
detected-ddos forward: in:sfp-sfpplus2_VS_AS out:<pppoe-alfabitomega>, connection-state:new src-mac 18:fd:74:7c:aa:85, proto TCP (SYN), 5.99.210.125:23084->185.203.24.2:10204, len 60
forward: in:<pppoe-1471_1115_nappicarol> out:sfp-sfpplus2_VS_AS, connection-state:new src-mac 84:d8:1b:68:6a:cc, proto UDP, 10.0.254.67:39651->142.250.180.142:443, len 1378
forward: in:<pppoe-1471_1115_nappicarol> out:sfp-sfpplus2_VS_AS, connection-state:new src-mac 84:d8:1b:68:6a:cc, proto UDP, 10.0.254.67:39651->142.250.180.142:443, len 1378
[root@ids python_ml]# nohup sudo -u ids python3 syslog_parser.py > /var/log/ids/syslog_parser.log 2>&1 &
[3] 13114
[root@ids python_ml]# tail -f /var/log/ids/syslog_parser.log
nohup: ignoring input
=== SYSLOG PARSER PER ROUTER MIKROTIK ===
Pressione Ctrl+C per interrompere
[DEBUG] Avvio syslog_parser...
[DEBUG] Caricamento .env da /opt/ids/.env...
[DEBUG] .env caricato
[DEBUG] Configurazione database:
[DEBUG] Host: localhost
[DEBUG] Port: 5432
[DEBUG] Database: ids_database
[DEBUG] User: ids_user
[DEBUG] File log: /var/log/mikrotik/raw.log
[INFO] File log trovato: /var/log/mikrotik/raw.log
[DEBUG] Creazione parser...
[DEBUG] Connessione database...
[INFO] Connesso a PostgreSQL
[INFO] Avvio processamento log (modalità follow)...
[INFO] Processando /var/log/mikrotik/raw.log (follow=True)
[INFO] Processate 100 righe, salvate 0 log
[INFO] Processate 200 righe, salvate 0 log
[INFO] Processate 300 righe, salvate 0 log
[INFO] Processate 400 righe, salvate 0 log
[INFO] Processate 500 righe, salvate 0 log
[INFO] Processate 600 righe, salvate 0 log
[INFO] Processate 700 righe, salvate 0 log
[INFO] Processate 800 righe, salvate 0 log
[INFO] Processate 900 righe, salvate 0 log
[INFO] Processate 1000 righe, salvate 0 log
[INFO] Processate 1100 righe, salvate 0 log
[INFO] Processate 1200 righe, salvate 0 log
[INFO] Processate 1300 righe, salvate 0 log
[INFO] Processate 1400 righe, salvate 0 log
[INFO] Processate 1500 righe, salvate 0 log
[INFO] Processate 1600 righe, salvate 0 log
[INFO] Processate 1700 righe, salvate 0 log
[INFO] Processate 1800 righe, salvate 0 log
[INFO] Processate 1900 righe, salvate 0 log
[INFO] Processate 2000 righe, salvate 0 log
[INFO] Processate 2100 righe, salvate 0 log
[INFO] Processate 2200 righe, salvate 0 log
[INFO] Processate 2300 righe, salvate 0 log
[INFO] Processate 2400 righe, salvate 0 log
[INFO] Processate 2500 righe, salvate 0 log
[INFO] Processate 2600 righe, salvate 0 log
[INFO] Processate 2700 righe, salvate 0 log
[INFO] Processate 2800 righe, salvate 0 log
[INFO] Processate 2900 righe, salvate 0 log
[INFO] Processate 3000 righe, salvate 0 log
[INFO] Processate 3100 righe, salvate 0 log
[INFO] Processate 3200 righe, salvate 0 log
[INFO] Processate 3300 righe, salvate 0 log
[INFO] Processate 3400 righe, salvate 0 log
[INFO] Processate 3500 righe, salvate 0 log
[INFO] Processate 3600 righe, salvate 0 log
[INFO] Processate 3700 righe, salvate 0 log
[INFO] Processate 3800 righe, salvate 0 log
[INFO] Processate 3900 righe, salvate 0 log
[INFO] Processate 4000 righe, salvate 0 log
[INFO] Processate 4100 righe, salvate 0 log
[INFO] Processate 4200 righe, salvate 0 log
[INFO] Processate 4300 righe, salvate 0 log
[INFO] Processate 4400 righe, salvate 0 log
[INFO] Processate 4500 righe, salvate 0 log
[INFO] Processate 4600 righe, salvate 0 log
[INFO] Processate 4700 righe, salvate 0 log
[INFO] Processate 4800 righe, salvate 0 log
[INFO] Processate 4900 righe, salvate 0 log
[INFO] Processate 5000 righe, salvate 0 log
[INFO] Processate 5100 righe, salvate 0 log
[INFO] Processate 5200 righe, salvate 0 log
[INFO] Processate 5300 righe, salvate 0 log
[INFO] Processate 5400 righe, salvate 0 log
[INFO] Processate 5500 righe, salvate 0 log
[INFO] Processate 5600 righe, salvate 0 log
[INFO] Processate 5700 righe, salvate 0 log
[INFO] Processate 5800 righe, salvate 0 log
[INFO] Processate 5900 righe, salvate 0 log
[INFO] Processate 6000 righe, salvate 0 log
[INFO] Processate 6100 righe, salvate 0 log
[INFO] Processate 6200 righe, salvate 0 log
[INFO] Processate 6300 righe, salvate 0 log
[INFO] Processate 6400 righe, salvate 0 log
[INFO] Processate 6500 righe, salvate 0 log
[INFO] Processate 6600 righe, salvate 0 log
[INFO] Processate 6700 righe, salvate 0 log
[INFO] Processate 6800 righe, salvate 0 log
[INFO] Processate 6900 righe, salvate 0 log
[INFO] Processate 7000 righe, salvate 0 log
[INFO] Processate 7100 righe, salvate 0 log
[INFO] Processate 7200 righe, salvate 0 log
[INFO] Processate 7300 righe, salvate 0 log
[INFO] Processate 7400 righe, salvate 0 log
[INFO] Processate 7500 righe, salvate 0 log
[INFO] Processate 7600 righe, salvate 0 log
[INFO] Processate 7700 righe, salvate 0 log
[INFO] Processate 7800 righe, salvate 0 log
[INFO] Processate 7900 righe, salvate 0 log
[INFO] Processate 8000 righe, salvate 0 log
[INFO] Processate 8100 righe, salvate 0 log
[INFO] Processate 8200 righe, salvate 0 log
[INFO] Processate 8300 righe, salvate 0 log
[INFO] Processate 8400 righe, salvate 0 log
[INFO] Processate 8500 righe, salvate 0 log
[INFO] Processate 8600 righe, salvate 0 log
[INFO] Processate 8700 righe, salvate 0 log
[INFO] Processate 8800 righe, salvate 0 log
[INFO] Processate 8900 righe, salvate 0 log
[INFO] Processate 9000 righe, salvate 0 log
[INFO] Processate 9100 righe, salvate 0 log
[INFO] Processate 9200 righe, salvate 0 log
[INFO] Processate 9300 righe, salvate 0 log
[INFO] Processate 9400 righe, salvate 0 log
[INFO] Processate 9500 righe, salvate 0 log
[INFO] Processate 9600 righe, salvate 0 log
[INFO] Processate 9700 righe, salvate 0 log
[INFO] Processate 9800 righe, salvate 0 log
[INFO] Processate 9900 righe, salvate 0 log
[INFO] Processate 10000 righe, salvate 0 log
[INFO] Processate 10100 righe, salvate 0 log
[INFO] Processate 10200 righe, salvate 0 log
[INFO] Processate 10300 righe, salvate 0 log
[INFO] Processate 10400 righe, salvate 0 log
[INFO] Processate 10500 righe, salvate 0 log
[INFO] Processate 10600 righe, salvate 0 log
[INFO] Processate 10700 righe, salvate 0 log
[INFO] Processate 10800 righe, salvate 0 log
[INFO] Processate 10900 righe, salvate 0 log
[INFO] Processate 11000 righe, salvate 0 log
[INFO] Processate 11100 righe, salvate 0 log
[INFO] Processate 11200 righe, salvate 0 log
[INFO] Processate 11300 righe, salvate 0 log
[INFO] Processate 11400 righe, salvate 0 log
[INFO] Processate 11500 righe, salvate 0 log
[INFO] Processate 11600 righe, salvate 0 log
[INFO] Processate 11700 righe, salvate 0 log
[INFO] Processate 11800 righe, salvate 0 log
[INFO] Processate 11900 righe, salvate 0 log
[INFO] Processate 12000 righe, salvate 0 log
[INFO] Processate 12100 righe, salvate 0 log
[INFO] Processate 12200 righe, salvate 0 log
[INFO] Processate 12300 righe, salvate 0 log
[INFO] Processate 12400 righe, salvate 0 log
[INFO] Processate 12500 righe, salvate 0 log
[INFO] Processate 12600 righe, salvate 0 log
[INFO] Processate 12700 righe, salvate 0 log
[INFO] Processate 12800 righe, salvate 0 log
[INFO] Processate 12900 righe, salvate 0 log
[INFO] Processate 13000 righe, salvate 0 log
[INFO] Processate 13100 righe, salvate 0 log
[INFO] Processate 13200 righe, salvate 0 log
[INFO] Processate 13300 righe, salvate 0 log
[INFO] Processate 13400 righe, salvate 0 log
[INFO] Processate 13500 righe, salvate 0 log
[INFO] Processate 13600 righe, salvate 0 log
[INFO] Processate 13700 righe, salvate 0 log
[INFO] Processate 13800 righe, salvate 0 log
[INFO] Processate 13900 righe, salvate 0 log
[INFO] Processate 14000 righe, salvate 0 log
[INFO] Processate 14100 righe, salvate 0 log
[INFO] Processate 14200 righe, salvate 0 log
[INFO] Processate 14300 righe, salvate 0 log
[INFO] Processate 14400 righe, salvate 0 log
[INFO] Processate 14500 righe, salvate 0 log
[INFO] Processate 14600 righe, salvate 0 log
[INFO] Processate 14700 righe, salvate 0 log
[INFO] Processate 14800 righe, salvate 0 log
[INFO] Processate 14900 righe, salvate 0 log
[INFO] Processate 15000 righe, salvate 0 log
[INFO] Processate 15100 righe, salvate 0 log
[INFO] Processate 15200 righe, salvate 0 log
[INFO] Processate 15300 righe, salvate 0 log
[INFO] Processate 15400 righe, salvate 0 log
[INFO] Processate 15500 righe, salvate 0 log
[INFO] Processate 15600 righe, salvate 0 log
[INFO] Processate 15700 righe, salvate 0 log
[INFO] Processate 15800 righe, salvate 0 log
[INFO] Processate 15900 righe, salvate 0 log
[INFO] Processate 16000 righe, salvate 0 log
[INFO] Processate 16100 righe, salvate 0 log
[INFO] Processate 16200 righe, salvate 0 log
[INFO] Processate 16300 righe, salvate 0 log
[INFO] Processate 16400 righe, salvate 0 log
[INFO] Processate 16500 righe, salvate 0 log
[INFO] Processate 16600 righe, salvate 0 log
[INFO] Processate 16700 righe, salvate 0 log
[INFO] Processate 16800 righe, salvate 0 log
[INFO] Processate 16900 righe, salvate 0 log
[INFO] Processate 17000 righe, salvate 0 log
^C
[root@ids python_ml]# grep "TIMESTAMP" /etc/rsyslog.d/99-mikrotik.conf
[root@ids python_ml]# grep "TIMESTAMP" /etc/rsyslog.d/99-mikrotik.conf