From 6f2d0da1c9cb975dd00adbaadf31edb4ad1ec122 Mon Sep 17 00:00:00 2001 From: marco370 <48531002-marco370@users.noreply.replit.com> Date: Fri, 21 Nov 2025 17:06:35 +0000 Subject: [PATCH] Update system with latest code and database changes Applies recent code updates from the Git repository and executes necessary database schema migrations to ensure data integrity and system functionality. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528 Replit-Commit-Checkpoint-Type: full_checkpoint Replit-Commit-Event-Id: 2bb36b2d-b880-414f-ad99-7f18bc18ee9e Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/4LjHWWz --- ...AGGIORNAME-1763744739614_1763744739614.txt | 360 ++++++++++++++++++ 1 file changed, 360 insertions(+) create mode 100644 attached_assets/Pasted--deployment-update-from-git-sh-AGGIORNAME-1763744739614_1763744739614.txt diff --git a/attached_assets/Pasted--deployment-update-from-git-sh-AGGIORNAME-1763744739614_1763744739614.txt b/attached_assets/Pasted--deployment-update-from-git-sh-AGGIORNAME-1763744739614_1763744739614.txt new file mode 100644 index 0000000..a902e96 --- /dev/null +++ b/attached_assets/Pasted--deployment-update-from-git-sh-AGGIORNAME-1763744739614_1763744739614.txt @@ -0,0 +1,360 @@ + ./deployment/update_from_git.sh + +╔═══════════════════════════════════════════════╗ +║  AGGIORNAMENTO SISTEMA IDS DA GIT ║ +╚═══════════════════════════════════════════════╝ + + Verifica configurazione git... + + Backup configurazione locale... +✅ .env salvato in .env.backup + + Verifica modifiche locali... + + Download aggiornamenti da git.alfacom.it... +remote: Enumerating objects: 25, done. +remote: Counting objects: 100% (25/25), done. +remote: Compressing objects: 100% (16/16), done. +remote: Total 16 (delta 13), reused 0 (delta 0), pack-reused 0 (from 0) +Unpacking objects: 100% (16/16), 2.36 KiB | 482.00 KiB/s, done. +From https://git.alfacom.it/marco/ids.alfacom.it + 07f1895..e957556 main -> origin/main + * [new tag] v1.0.27 -> v1.0.27 +From https://git.alfacom.it/marco/ids.alfacom.it + * branch main -> FETCH_HEAD +Updating 07f1895..e957556 +Fast-forward + .replit | 4 ---- + database-schema/apply_migrations.sh | 9 +++++++++ + database-schema/schema.sql | 4 ++-- + deployment/cleanup_database.sh | 4 +++- + deployment/debug_system.sh | 10 +++++++++- + version.json | 10 ++++++++-- + 6 files changed, 31 insertions(+), 10 deletions(-) +✅ Aggiornamenti scaricati con successo + + Ripristino configurazione locale... +✅ .env ripristinato + + Aggiornamento dipendenze Node.js... + +up to date, audited 492 packages in 2s + +65 packages are looking for funding + run `npm fund` for details + +9 vulnerabilities (3 low, 5 moderate, 1 high) + +To address issues that do not require attention, run: + npm audit fix + +To address all issues (including breaking changes), run: + npm audit fix --force + +Run `npm audit` for details. +✅ Dipendenze Node.js aggiornate + + Aggiornamento dipendenze Python... +Defaulting to user installation because normal site-packages is not writeable +Requirement already satisfied: fastapi==0.104.1 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 1)) (0.104.1) +Requirement already satisfied: uvicorn==0.24.0 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 2)) (0.24.0) +Requirement already satisfied: pandas==2.1.3 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 3)) (2.1.3) +Requirement already satisfied: numpy==1.26.2 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 4)) (1.26.2) +Requirement already satisfied: scikit-learn==1.3.2 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 5)) (1.3.2) +Requirement already satisfied: psycopg2-binary==2.9.9 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 6)) (2.9.9) +Requirement already satisfied: python-dotenv==1.0.0 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 7)) (1.0.0) +Requirement already satisfied: pydantic==2.5.0 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 8)) (2.5.0) +Requirement already satisfied: httpx==0.25.1 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 9)) (0.25.1) +Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /home/ids/.local/lib/python3.11/site-packages (from fastapi==0.104.1->-r requirements.txt (line 1)) (3.7.1) +Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /home/ids/.local/lib/python3.11/site-packages (from fastapi==0.104.1->-r requirements.txt (line 1)) (0.27.0) +Requirement already satisfied: typing-extensions>=4.8.0 in /home/ids/.local/lib/python3.11/site-packages (from fastapi==0.104.1->-r requirements.txt (line 1)) (4.15.0) +Requirement already satisfied: click>=7.0 in /home/ids/.local/lib/python3.11/site-packages (from uvicorn==0.24.0->-r requirements.txt (line 2)) (8.3.1) +Requirement already satisfied: h11>=0.8 in /home/ids/.local/lib/python3.11/site-packages (from uvicorn==0.24.0->-r requirements.txt (line 2)) (0.16.0) +Requirement already satisfied: python-dateutil>=2.8.2 in /home/ids/.local/lib/python3.11/site-packages (from pandas==2.1.3->-r requirements.txt (line 3)) (2.9.0.post0) +Requirement already satisfied: pytz>=2020.1 in /home/ids/.local/lib/python3.11/site-packages (from pandas==2.1.3->-r requirements.txt (line 3)) (2025.2) +Requirement already satisfied: tzdata>=2022.1 in /home/ids/.local/lib/python3.11/site-packages (from pandas==2.1.3->-r requirements.txt (line 3)) (2025.2) +Requirement already satisfied: scipy>=1.5.0 in /home/ids/.local/lib/python3.11/site-packages (from scikit-learn==1.3.2->-r requirements.txt (line 5)) (1.16.3) +Requirement already satisfied: joblib>=1.1.1 in /home/ids/.local/lib/python3.11/site-packages (from scikit-learn==1.3.2->-r requirements.txt (line 5)) (1.5.2) +Requirement already satisfied: threadpoolctl>=2.0.0 in /home/ids/.local/lib/python3.11/site-packages (from scikit-learn==1.3.2->-r requirements.txt (line 5)) (3.6.0) +Requirement already satisfied: annotated-types>=0.4.0 in /home/ids/.local/lib/python3.11/site-packages (from pydantic==2.5.0->-r requirements.txt (line 8)) (0.7.0) +Requirement already satisfied: pydantic-core==2.14.1 in /home/ids/.local/lib/python3.11/site-packages (from pydantic==2.5.0->-r requirements.txt (line 8)) (2.14.1) +Requirement already satisfied: certifi in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (2025.11.12) +Requirement already satisfied: httpcore in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (1.0.9) +Requirement already satisfied: idna in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (3.11) +Requirement already satisfied: sniffio in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (1.3.1) +Requirement already satisfied: six>=1.5 in /home/ids/.local/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas==2.1.3->-r requirements.txt (line 3)) (1.17.0) +✅ Dipendenze Python aggiornate + + Aggiornamento schema database... + Applicando migrazioni SQL... + Applicazione migrazioni database... + Trovate 1 migrazioni + Applicando: 001_add_missing_columns.sql + ✅ 001_add_missing_columns.sql applicata +✅ Tutte le migrazioni applicate con successo + ✅ Migrazioni SQL applicate + Sincronizzando schema Drizzle... + +> rest-express@1.0.0 db:push +> drizzle-kit push + +No config path provided, using default 'drizzle.config.ts' +Reading config file '/opt/ids/drizzle.config.ts' +Using 'pg' driver for database querying +[✓] Pulling schema from database... +· You're about to add routers_ip_address_unique unique constraint to the table, which contains 1 items. If this statement fails, you will receive an error from the database. Do you want to truncate routers table? + + Warning Found data-loss statements: +· You're about to delete last_check column in routers table with 1 items +· You're about to delete status column in routers table with 1 items + +THIS ACTION WILL CAUSE DATA LOSS AND CANNOT BE REVERTED + +Do you still want to push changes? +[x] All changes were aborted +✅ Schema database completamente sincronizzato + + Restart servizi... +✅ Servizi riavviati + +╔═══════════════════════════════════════════════╗ +║ ✅ AGGIORNAMENTO COMPLETATO ║ +╚═══════════════════════════════════════════════╝ + + VERIFICA SISTEMA: + • Log backend: tail -f /var/log/ids/backend.log + • Log frontend: tail -f /var/log/ids/frontend.log + • API backend: curl http://localhost:8000/health + • Frontend: curl http://localhost:5000 + + STATO SERVIZI: +ids 5038 0.2 2.0 1894024 331912 ? Sl 09:20 1:17 /usr/bin/python3.11 main.py +root 12022 0.0 0.0 3088 1536 pts/3 S+ 17:51 0:00 tail -f /var/log/ids/syslog_parser.log +root 12832 0.0 0.1 730448 32068 pts/5 Rl+ 18:02 0:00 /usr/bin/node /usr/bin/npm run dev + +[root@ids ids]# sudo -u ids /opt/ids/database-schema/apply_migrations.sh + Applicazione migrazioni database... + Trovate 1 migrazioni + Applicando: 001_add_missing_columns.sql + ✅ 001_add_missing_columns.sql applicata +✅ Tutte le migrazioni applicate con successo +[root@ids ids]# psql postgresql://ids_user:TestPassword123@127.0.0.1:5432/ids_database -c "\d routers" + Table "public.routers" + Column | Type | Collation | Nullable | Default +------------+-----------------------------+-----------+----------+------------------- + id | character varying | | not null | gen_random_uuid() + name | text | | not null | + ip_address | text | | not null | + username | text | | not null | + password | text | | not null | + api_port | integer | | not null | 443 + enabled | boolean | | not null | true + last_check | timestamp without time zone | | | + status | text | | | + created_at | timestamp without time zone | | not null | now() + last_sync | timestamp without time zone | | | +Indexes: + "routers_pkey" PRIMARY KEY, btree (id) + "routers_enabled_idx" btree (enabled) + "routers_ip_address_key" UNIQUE CONSTRAINT, btree (ip_address) + "routers_ip_idx" btree (ip_address) + +[root@ids ids]# psql postgresql://ids_user:TestPassword123@127.0.0.1:5432/ids_database << 'EOF' +-- Conta log da eliminare +SELECT COUNT(*) as logs_da_eliminare FROM network_logs WHERE timestamp < NOW() - INTERVAL '7 days'; + +-- Elimina +DELETE FROM network_logs WHERE timestamp < NOW() - INTERVAL '7 days'; + +-- Libera spazio fisico +VACUUM FULL network_logs; + +-- Verifica risultato +SELECT COUNT(*) as logs_rimasti FROM network_logs; +SELECT pg_size_pretty(pg_database_size(current_database())) as dimensione_db; +EOF + logs_da_eliminare +------------------- + 0 +(1 row) + +DELETE 0 +VACUUM + logs_rimasti +-------------- + 0 +(1 row) + + dimensione_db +--------------- + 8853 kB +(1 row) + +[root@ids ids]# sudo /opt/ids/deployment/setup_cron_cleanup.sh + Configurazione cron job per pulizia database... +⚠ Cron job già configurato + + Cron jobs attuali per utente ids: +# ============================================ +# SISTEMA IDS - CONFIGURAZIONE AUTOMATICA +# ============================================ + +# Training ML ogni 12 ore (alle 00:00 e 12:00) +0 */12 * * * /opt/ids/deployment/cron_train.sh + +# Detection automatica ogni 5 minuti +*/5 * * * * /opt/ids/deployment/cron_detect.sh + +# Verifica processo backend Python ogni 5 minuti (riavvia se non attivo) +*/5 * * * * /opt/ids/deployment/check_backend.sh >> /var/log/ids/cron.log 2>&1 + +# Verifica processo frontend ogni 5 minuti (riavvia se non attivo) +*/5 * * * * /opt/ids/deployment/check_frontend.sh >> /var/log/ids/cron.log 2>&1 + +# Pulizia log settimanale (ogni domenica alle 02:00) +0 2 * * 0 find /var/log/ids -name "*.log" -size +100M -exec truncate -s 50M {} \; >> /var/log/ids/cron.log 2>&1 + +# Restart completo del sistema ogni settimana (domenica alle 03:00) +0 3 * * 0 /opt/ids/deployment/restart_all.sh >> /var/log/ids/cron.log 2>&1 + +# Backup database giornaliero (alle 04:00) +0 4 * * * /opt/ids/deployment/backup_db.sh >> /var/log/ids/cron.log 2>&1 +0 3 * * * /opt/ids/deployment/cleanup_database.sh >> /var/log/ids/cleanup.log 2>&1 + +離 Test manuale pulizia: + sudo -u ids /opt/ids/deployment/cleanup_database.sh +[root@ids ids]# sudo -u ids /opt/ids/deployment/restart_all.sh +pkill: killing pid 12878 failed: Operation not permitted +pkill: killing pid 12832 failed: Operation not permitted +[root@ids ids]# /opt/ids/deployment/debug_system.sh + +╔═══════════════════════════════════════════════╗ +║  DEBUG SISTEMA IDS ║ +╚═══════════════════════════════════════════════╝ + + +═══ 1. VERIFICA DATABASE ═══ + Conta record per tabella: + tabella | record +------------------+-------- + detections | 0 + network_logs | 0 + routers | 1 + training_history | 0 + whitelist | 0 +(5 rows) + + + Schema tabella routers: + Table "public.routers" + Column | Type | Collation | Nullable | Default +------------+-----------------------------+-----------+----------+------------------- + id | character varying | | not null | gen_random_uuid() + name | text | | not null | + ip_address | text | | not null | + username | text | | not null | + password | text | | not null | + api_port | integer | | not null | 443 + enabled | boolean | | not null | true + last_check | timestamp without time zone | | | + status | text | | | + created_at | timestamp without time zone | | not null | now() + last_sync | timestamp without time zone | | | +Indexes: + "routers_pkey" PRIMARY KEY, btree (id) + "routers_enabled_idx" btree (enabled) + "routers_ip_address_key" UNIQUE CONSTRAINT, btree (ip_address) + "routers_ip_idx" btree (ip_address) + + + Ultimi 5 network_logs: + timestamp | router_name | source_ip | destination_ip | protocol | packet_length +-----------+-------------+-----------+----------------+----------+--------------- +(0 rows) + + + Training history: + trained_at | model_version | records_processed | features_count | status | notes +------------+---------------+-------------------+----------------+--------+------- +(0 rows) + + + Detections: + detected_at | source_ip | risk_score | anomaly_type | blocked | log_count +-------------+-----------+------------+--------------+---------+----------- +(0 rows) + + +═══ 2. STATO SERVIZI ═══ + Processi attivi: +ids 5038 0.2 2.0 1894024 331912 ? Sl 09:20 1:17 /usr/bin/python3.11 main.py +root 12022 0.0 0.0 3088 1536 pts/3 S+ 17:51 0:00 tail -f /var/log/ids/syslog_parser.log +root 12832 0.2 0.3 1097848 59768 pts/5 Sl 18:02 0:00 npm run dev + +═══ 3. BACKEND PYTHON ML ═══ +✅ Backend Python attivo + Statistiche ML: +{ + "logs": { + "total": 0, + "last_hour": 0 + }, + "detections": { + "total": 0, + "blocked": 0 + }, + "routers": { + "active": 1 + }, + "latest_training": null +} + +═══ 4. FRONTEND NODE.JS ═══ +✅ Frontend Node attivo + Test API: +{ + "routers": { + "total": 1, + "enabled": 1 + }, + "detections": { + "total": 0, + "blocked": 0, + "critical": 0, + "high": 0 + }, + "logs": { + "recent": 0 + }, + "whitelist": { + "total": 0 + } +} + +═══ 5. SYSLOG PARSER ═══ +❌ Syslog Parser NON attivo + Avvia: cd /opt/ids/python_ml && nohup python syslog_parser.py > /var/log/ids/syslog_parser.log 2>&1 & + +═══ 6. ERRORI RECENTI ═══ +🔴 Errori backend Python: +ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use +ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use +ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use +ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use + +🔴 Errori frontend Node: +[DB ERROR] Failed to fetch routers: error: column "last_sync" does not exist + +╔═══════════════════════════════════════════════╗ +║ 📋 RIEPILOGO ║ +╚═══════════════════════════════════════════════╝ +Database: + • Network logs: 0 + • Detections: 0 + • Training history: 0 + +🔧 COMANDI UTILI: + • Riavvia tutto: sudo -u ids /opt/ids/deployment/restart_all.sh + • Test training: curl -X POST http://localhost:8000/train -H 'Content-Type: application/json' -d '{"max_records": 1000}' + • Log frontend: tail -f /var/log/ids/frontend.log + • Log backend: tail -f /var/log/ids/backend.log