Update system with latest code and database changes
Applies recent code updates from the Git repository and executes necessary database schema migrations to ensure data integrity and system functionality. Replit-Commit-Author: Agent Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528 Replit-Commit-Checkpoint-Type: full_checkpoint Replit-Commit-Event-Id: 2bb36b2d-b880-414f-ad99-7f18bc18ee9e Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/4LjHWWz
This commit is contained in:
marco370
parent
e957556af2
commit
6f2d0da1c9
@ -0,0 +1,360 @@
|
||||
./deployment/update_from_git.sh
|
||||
|
||||
╔═══════════════════════════════════════════════╗
|
||||
║ AGGIORNAMENTO SISTEMA IDS DA GIT ║
|
||||
╚═══════════════════════════════════════════════╝
|
||||
|
||||
Verifica configurazione git...
|
||||
|
||||
Backup configurazione locale...
|
||||
✅ .env salvato in .env.backup
|
||||
|
||||
Verifica modifiche locali...
|
||||
|
||||
Download aggiornamenti da git.alfacom.it...
|
||||
remote: Enumerating objects: 25, done.
|
||||
remote: Counting objects: 100% (25/25), done.
|
||||
remote: Compressing objects: 100% (16/16), done.
|
||||
remote: Total 16 (delta 13), reused 0 (delta 0), pack-reused 0 (from 0)
|
||||
Unpacking objects: 100% (16/16), 2.36 KiB | 482.00 KiB/s, done.
|
||||
From https://git.alfacom.it/marco/ids.alfacom.it
|
||||
07f1895..e957556 main -> origin/main
|
||||
* [new tag] v1.0.27 -> v1.0.27
|
||||
From https://git.alfacom.it/marco/ids.alfacom.it
|
||||
* branch main -> FETCH_HEAD
|
||||
Updating 07f1895..e957556
|
||||
Fast-forward
|
||||
.replit | 4 ----
|
||||
database-schema/apply_migrations.sh | 9 +++++++++
|
||||
database-schema/schema.sql | 4 ++--
|
||||
deployment/cleanup_database.sh | 4 +++-
|
||||
deployment/debug_system.sh | 10 +++++++++-
|
||||
version.json | 10 ++++++++--
|
||||
6 files changed, 31 insertions(+), 10 deletions(-)
|
||||
✅ Aggiornamenti scaricati con successo
|
||||
|
||||
Ripristino configurazione locale...
|
||||
✅ .env ripristinato
|
||||
|
||||
Aggiornamento dipendenze Node.js...
|
||||
|
||||
up to date, audited 492 packages in 2s
|
||||
|
||||
65 packages are looking for funding
|
||||
run `npm fund` for details
|
||||
|
||||
9 vulnerabilities (3 low, 5 moderate, 1 high)
|
||||
|
||||
To address issues that do not require attention, run:
|
||||
npm audit fix
|
||||
|
||||
To address all issues (including breaking changes), run:
|
||||
npm audit fix --force
|
||||
|
||||
Run `npm audit` for details.
|
||||
✅ Dipendenze Node.js aggiornate
|
||||
|
||||
Aggiornamento dipendenze Python...
|
||||
Defaulting to user installation because normal site-packages is not writeable
|
||||
Requirement already satisfied: fastapi==0.104.1 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 1)) (0.104.1)
|
||||
Requirement already satisfied: uvicorn==0.24.0 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 2)) (0.24.0)
|
||||
Requirement already satisfied: pandas==2.1.3 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 3)) (2.1.3)
|
||||
Requirement already satisfied: numpy==1.26.2 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 4)) (1.26.2)
|
||||
Requirement already satisfied: scikit-learn==1.3.2 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 5)) (1.3.2)
|
||||
Requirement already satisfied: psycopg2-binary==2.9.9 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 6)) (2.9.9)
|
||||
Requirement already satisfied: python-dotenv==1.0.0 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 7)) (1.0.0)
|
||||
Requirement already satisfied: pydantic==2.5.0 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 8)) (2.5.0)
|
||||
Requirement already satisfied: httpx==0.25.1 in /home/ids/.local/lib/python3.11/site-packages (from -r requirements.txt (line 9)) (0.25.1)
|
||||
Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /home/ids/.local/lib/python3.11/site-packages (from fastapi==0.104.1->-r requirements.txt (line 1)) (3.7.1)
|
||||
Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /home/ids/.local/lib/python3.11/site-packages (from fastapi==0.104.1->-r requirements.txt (line 1)) (0.27.0)
|
||||
Requirement already satisfied: typing-extensions>=4.8.0 in /home/ids/.local/lib/python3.11/site-packages (from fastapi==0.104.1->-r requirements.txt (line 1)) (4.15.0)
|
||||
Requirement already satisfied: click>=7.0 in /home/ids/.local/lib/python3.11/site-packages (from uvicorn==0.24.0->-r requirements.txt (line 2)) (8.3.1)
|
||||
Requirement already satisfied: h11>=0.8 in /home/ids/.local/lib/python3.11/site-packages (from uvicorn==0.24.0->-r requirements.txt (line 2)) (0.16.0)
|
||||
Requirement already satisfied: python-dateutil>=2.8.2 in /home/ids/.local/lib/python3.11/site-packages (from pandas==2.1.3->-r requirements.txt (line 3)) (2.9.0.post0)
|
||||
Requirement already satisfied: pytz>=2020.1 in /home/ids/.local/lib/python3.11/site-packages (from pandas==2.1.3->-r requirements.txt (line 3)) (2025.2)
|
||||
Requirement already satisfied: tzdata>=2022.1 in /home/ids/.local/lib/python3.11/site-packages (from pandas==2.1.3->-r requirements.txt (line 3)) (2025.2)
|
||||
Requirement already satisfied: scipy>=1.5.0 in /home/ids/.local/lib/python3.11/site-packages (from scikit-learn==1.3.2->-r requirements.txt (line 5)) (1.16.3)
|
||||
Requirement already satisfied: joblib>=1.1.1 in /home/ids/.local/lib/python3.11/site-packages (from scikit-learn==1.3.2->-r requirements.txt (line 5)) (1.5.2)
|
||||
Requirement already satisfied: threadpoolctl>=2.0.0 in /home/ids/.local/lib/python3.11/site-packages (from scikit-learn==1.3.2->-r requirements.txt (line 5)) (3.6.0)
|
||||
Requirement already satisfied: annotated-types>=0.4.0 in /home/ids/.local/lib/python3.11/site-packages (from pydantic==2.5.0->-r requirements.txt (line 8)) (0.7.0)
|
||||
Requirement already satisfied: pydantic-core==2.14.1 in /home/ids/.local/lib/python3.11/site-packages (from pydantic==2.5.0->-r requirements.txt (line 8)) (2.14.1)
|
||||
Requirement already satisfied: certifi in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (2025.11.12)
|
||||
Requirement already satisfied: httpcore in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (1.0.9)
|
||||
Requirement already satisfied: idna in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (3.11)
|
||||
Requirement already satisfied: sniffio in /home/ids/.local/lib/python3.11/site-packages (from httpx==0.25.1->-r requirements.txt (line 9)) (1.3.1)
|
||||
Requirement already satisfied: six>=1.5 in /home/ids/.local/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas==2.1.3->-r requirements.txt (line 3)) (1.17.0)
|
||||
✅ Dipendenze Python aggiornate
|
||||
|
||||
Aggiornamento schema database...
|
||||
Applicando migrazioni SQL...
|
||||
Applicazione migrazioni database...
|
||||
Trovate 1 migrazioni
|
||||
Applicando: 001_add_missing_columns.sql
|
||||
✅ 001_add_missing_columns.sql applicata
|
||||
✅ Tutte le migrazioni applicate con successo
|
||||
✅ Migrazioni SQL applicate
|
||||
Sincronizzando schema Drizzle...
|
||||
|
||||
> rest-express@1.0.0 db:push
|
||||
> drizzle-kit push
|
||||
|
||||
No config path provided, using default 'drizzle.config.ts'
|
||||
Reading config file '/opt/ids/drizzle.config.ts'
|
||||
Using 'pg' driver for database querying
|
||||
[✓] Pulling schema from database...
|
||||
· You're about to add routers_ip_address_unique unique constraint to the table, which contains 1 items. If this statement fails, you will receive an error from the database. Do you want to truncate routers table?
|
||||
|
||||
Warning Found data-loss statements:
|
||||
· You're about to delete last_check column in routers table with 1 items
|
||||
· You're about to delete status column in routers table with 1 items
|
||||
|
||||
THIS ACTION WILL CAUSE DATA LOSS AND CANNOT BE REVERTED
|
||||
|
||||
Do you still want to push changes?
|
||||
[x] All changes were aborted
|
||||
✅ Schema database completamente sincronizzato
|
||||
|
||||
Restart servizi...
|
||||
✅ Servizi riavviati
|
||||
|
||||
╔═══════════════════════════════════════════════╗
|
||||
║ ✅ AGGIORNAMENTO COMPLETATO ║
|
||||
╚═══════════════════════════════════════════════╝
|
||||
|
||||
VERIFICA SISTEMA:
|
||||
• Log backend: tail -f /var/log/ids/backend.log
|
||||
• Log frontend: tail -f /var/log/ids/frontend.log
|
||||
• API backend: curl http://localhost:8000/health
|
||||
• Frontend: curl http://localhost:5000
|
||||
|
||||
STATO SERVIZI:
|
||||
ids 5038 0.2 2.0 1894024 331912 ? Sl 09:20 1:17 /usr/bin/python3.11 main.py
|
||||
root 12022 0.0 0.0 3088 1536 pts/3 S+ 17:51 0:00 tail -f /var/log/ids/syslog_parser.log
|
||||
root 12832 0.0 0.1 730448 32068 pts/5 Rl+ 18:02 0:00 /usr/bin/node /usr/bin/npm run dev
|
||||
|
||||
[root@ids ids]# sudo -u ids /opt/ids/database-schema/apply_migrations.sh
|
||||
Applicazione migrazioni database...
|
||||
Trovate 1 migrazioni
|
||||
Applicando: 001_add_missing_columns.sql
|
||||
✅ 001_add_missing_columns.sql applicata
|
||||
✅ Tutte le migrazioni applicate con successo
|
||||
[root@ids ids]# psql postgresql://ids_user:TestPassword123@127.0.0.1:5432/ids_database -c "\d routers"
|
||||
Table "public.routers"
|
||||
Column | Type | Collation | Nullable | Default
|
||||
------------+-----------------------------+-----------+----------+-------------------
|
||||
id | character varying | | not null | gen_random_uuid()
|
||||
name | text | | not null |
|
||||
ip_address | text | | not null |
|
||||
username | text | | not null |
|
||||
password | text | | not null |
|
||||
api_port | integer | | not null | 443
|
||||
enabled | boolean | | not null | true
|
||||
last_check | timestamp without time zone | | |
|
||||
status | text | | |
|
||||
created_at | timestamp without time zone | | not null | now()
|
||||
last_sync | timestamp without time zone | | |
|
||||
Indexes:
|
||||
"routers_pkey" PRIMARY KEY, btree (id)
|
||||
"routers_enabled_idx" btree (enabled)
|
||||
"routers_ip_address_key" UNIQUE CONSTRAINT, btree (ip_address)
|
||||
"routers_ip_idx" btree (ip_address)
|
||||
|
||||
[root@ids ids]# psql postgresql://ids_user:TestPassword123@127.0.0.1:5432/ids_database << 'EOF'
|
||||
-- Conta log da eliminare
|
||||
SELECT COUNT(*) as logs_da_eliminare FROM network_logs WHERE timestamp < NOW() - INTERVAL '7 days';
|
||||
|
||||
-- Elimina
|
||||
DELETE FROM network_logs WHERE timestamp < NOW() - INTERVAL '7 days';
|
||||
|
||||
-- Libera spazio fisico
|
||||
VACUUM FULL network_logs;
|
||||
|
||||
-- Verifica risultato
|
||||
SELECT COUNT(*) as logs_rimasti FROM network_logs;
|
||||
SELECT pg_size_pretty(pg_database_size(current_database())) as dimensione_db;
|
||||
EOF
|
||||
logs_da_eliminare
|
||||
-------------------
|
||||
0
|
||||
(1 row)
|
||||
|
||||
DELETE 0
|
||||
VACUUM
|
||||
logs_rimasti
|
||||
--------------
|
||||
0
|
||||
(1 row)
|
||||
|
||||
dimensione_db
|
||||
---------------
|
||||
8853 kB
|
||||
(1 row)
|
||||
|
||||
[root@ids ids]# sudo /opt/ids/deployment/setup_cron_cleanup.sh
|
||||
Configurazione cron job per pulizia database...
|
||||
⚠ Cron job già configurato
|
||||
|
||||
Cron jobs attuali per utente ids:
|
||||
# ============================================
|
||||
# SISTEMA IDS - CONFIGURAZIONE AUTOMATICA
|
||||
# ============================================
|
||||
|
||||
# Training ML ogni 12 ore (alle 00:00 e 12:00)
|
||||
0 */12 * * * /opt/ids/deployment/cron_train.sh
|
||||
|
||||
# Detection automatica ogni 5 minuti
|
||||
*/5 * * * * /opt/ids/deployment/cron_detect.sh
|
||||
|
||||
# Verifica processo backend Python ogni 5 minuti (riavvia se non attivo)
|
||||
*/5 * * * * /opt/ids/deployment/check_backend.sh >> /var/log/ids/cron.log 2>&1
|
||||
|
||||
# Verifica processo frontend ogni 5 minuti (riavvia se non attivo)
|
||||
*/5 * * * * /opt/ids/deployment/check_frontend.sh >> /var/log/ids/cron.log 2>&1
|
||||
|
||||
# Pulizia log settimanale (ogni domenica alle 02:00)
|
||||
0 2 * * 0 find /var/log/ids -name "*.log" -size +100M -exec truncate -s 50M {} \; >> /var/log/ids/cron.log 2>&1
|
||||
|
||||
# Restart completo del sistema ogni settimana (domenica alle 03:00)
|
||||
0 3 * * 0 /opt/ids/deployment/restart_all.sh >> /var/log/ids/cron.log 2>&1
|
||||
|
||||
# Backup database giornaliero (alle 04:00)
|
||||
0 4 * * * /opt/ids/deployment/backup_db.sh >> /var/log/ids/cron.log 2>&1
|
||||
0 3 * * * /opt/ids/deployment/cleanup_database.sh >> /var/log/ids/cleanup.log 2>&1
|
||||
|
||||
離 Test manuale pulizia:
|
||||
sudo -u ids /opt/ids/deployment/cleanup_database.sh
|
||||
[root@ids ids]# sudo -u ids /opt/ids/deployment/restart_all.sh
|
||||
pkill: killing pid 12878 failed: Operation not permitted
|
||||
pkill: killing pid 12832 failed: Operation not permitted
|
||||
[root@ids ids]# /opt/ids/deployment/debug_system.sh
|
||||
|
||||
╔═══════════════════════════════════════════════╗
|
||||
║ DEBUG SISTEMA IDS ║
|
||||
╚═══════════════════════════════════════════════╝
|
||||
|
||||
|
||||
═══ 1. VERIFICA DATABASE ═══
|
||||
Conta record per tabella:
|
||||
tabella | record
|
||||
------------------+--------
|
||||
detections | 0
|
||||
network_logs | 0
|
||||
routers | 1
|
||||
training_history | 0
|
||||
whitelist | 0
|
||||
(5 rows)
|
||||
|
||||
|
||||
Schema tabella routers:
|
||||
Table "public.routers"
|
||||
Column | Type | Collation | Nullable | Default
|
||||
------------+-----------------------------+-----------+----------+-------------------
|
||||
id | character varying | | not null | gen_random_uuid()
|
||||
name | text | | not null |
|
||||
ip_address | text | | not null |
|
||||
username | text | | not null |
|
||||
password | text | | not null |
|
||||
api_port | integer | | not null | 443
|
||||
enabled | boolean | | not null | true
|
||||
last_check | timestamp without time zone | | |
|
||||
status | text | | |
|
||||
created_at | timestamp without time zone | | not null | now()
|
||||
last_sync | timestamp without time zone | | |
|
||||
Indexes:
|
||||
"routers_pkey" PRIMARY KEY, btree (id)
|
||||
"routers_enabled_idx" btree (enabled)
|
||||
"routers_ip_address_key" UNIQUE CONSTRAINT, btree (ip_address)
|
||||
"routers_ip_idx" btree (ip_address)
|
||||
|
||||
|
||||
Ultimi 5 network_logs:
|
||||
timestamp | router_name | source_ip | destination_ip | protocol | packet_length
|
||||
-----------+-------------+-----------+----------------+----------+---------------
|
||||
(0 rows)
|
||||
|
||||
|
||||
Training history:
|
||||
trained_at | model_version | records_processed | features_count | status | notes
|
||||
------------+---------------+-------------------+----------------+--------+-------
|
||||
(0 rows)
|
||||
|
||||
|
||||
Detections:
|
||||
detected_at | source_ip | risk_score | anomaly_type | blocked | log_count
|
||||
-------------+-----------+------------+--------------+---------+-----------
|
||||
(0 rows)
|
||||
|
||||
|
||||
═══ 2. STATO SERVIZI ═══
|
||||
Processi attivi:
|
||||
ids 5038 0.2 2.0 1894024 331912 ? Sl 09:20 1:17 /usr/bin/python3.11 main.py
|
||||
root 12022 0.0 0.0 3088 1536 pts/3 S+ 17:51 0:00 tail -f /var/log/ids/syslog_parser.log
|
||||
root 12832 0.2 0.3 1097848 59768 pts/5 Sl 18:02 0:00 npm run dev
|
||||
|
||||
═══ 3. BACKEND PYTHON ML ═══
|
||||
✅ Backend Python attivo
|
||||
Statistiche ML:
|
||||
{
|
||||
"logs": {
|
||||
"total": 0,
|
||||
"last_hour": 0
|
||||
},
|
||||
"detections": {
|
||||
"total": 0,
|
||||
"blocked": 0
|
||||
},
|
||||
"routers": {
|
||||
"active": 1
|
||||
},
|
||||
"latest_training": null
|
||||
}
|
||||
|
||||
═══ 4. FRONTEND NODE.JS ═══
|
||||
✅ Frontend Node attivo
|
||||
Test API:
|
||||
{
|
||||
"routers": {
|
||||
"total": 1,
|
||||
"enabled": 1
|
||||
},
|
||||
"detections": {
|
||||
"total": 0,
|
||||
"blocked": 0,
|
||||
"critical": 0,
|
||||
"high": 0
|
||||
},
|
||||
"logs": {
|
||||
"recent": 0
|
||||
},
|
||||
"whitelist": {
|
||||
"total": 0
|
||||
}
|
||||
}
|
||||
|
||||
═══ 5. SYSLOG PARSER ═══
|
||||
❌ Syslog Parser NON attivo
|
||||
Avvia: cd /opt/ids/python_ml && nohup python syslog_parser.py > /var/log/ids/syslog_parser.log 2>&1 &
|
||||
|
||||
═══ 6. ERRORI RECENTI ═══
|
||||
🔴 Errori backend Python:
|
||||
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use
|
||||
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use
|
||||
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use
|
||||
ERROR: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8000): address already in use
|
||||
|
||||
🔴 Errori frontend Node:
|
||||
[DB ERROR] Failed to fetch routers: error: column "last_sync" does not exist
|
||||
|
||||
╔═══════════════════════════════════════════════╗
|
||||
║ 📋 RIEPILOGO ║
|
||||
╚═══════════════════════════════════════════════╝
|
||||
Database:
|
||||
• Network logs: 0
|
||||
• Detections: 0
|
||||
• Training history: 0
|
||||
|
||||
🔧 COMANDI UTILI:
|
||||
• Riavvia tutto: sudo -u ids /opt/ids/deployment/restart_all.sh
|
||||
• Test training: curl -X POST http://localhost:8000/train -H 'Content-Type: application/json' -d '{"max_records": 1000}'
|
||||
• Log frontend: tail -f /var/log/ids/frontend.log
|
||||
• Log backend: tail -f /var/log/ids/backend.log
|
||||
Loading…
Reference in New Issue
Block a user