Commit Graph

39 Commits

Author SHA1 Message Date
marco370
f0d391b2a1 Map confidence level strings to numeric values for detection results
Converts 'high', 'medium', and 'low' confidence levels to their corresponding numeric values (95.0, 75.0, 50.0) before saving detection results to the database, resolving an invalid input syntax error for type numeric.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: fd44e6f4-fc55-4636-aa7a-f4f462ac978a
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/AXTUZmH
2025-11-25 09:34:44 +00:00
marco370
14d67c63a3 Improve syslog parser reliability and add monitoring
Enhance the syslog parser with auto-reconnect, error recovery, and integrated health metrics logging. Add a cron job for automated health checks and restarts.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 4885eae4-ffc7-4601-8f1c-5414922d5350
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/AXTUZmH
2025-11-25 09:09:21 +00:00
marco370
837f7d4c08 Update detection results to use correct key names for scores
Corrects the key names used to retrieve detection results in `compare_models.py` to match the output format of the hybrid detector.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 600ade79-ad9b-4993-b968-e6466b703598
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 08:51:27 +00:00
marco370
2d7185cdbc Adjust model comparison script to correctly process network logs
Correct logic in `compare_models.py` to pass raw network logs to the detection method, ensuring correct feature extraction and preventing a 'timestamp' KeyError.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: ecdb452a-13bf-4c0b-8da9-eebbafd63834
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 08:43:55 +00:00
marco370
cf3223b247 Update model comparison script to use current database detections
Adjusted script to query existing database detections instead of a specific model version, updating column names to match the actual database schema (source_ip, risk_score, anomaly_type, log_count, last_seen, detected_at).

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 62d703c2-4658-4280-aec5-f5e7c090b266
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 08:42:06 +00:00
marco370
a32700c149 Add script to compare old and new detection models
Creates a Python script that loads old detection data, reanalyzes IPs with the new hybrid detector, and compares the results to identify differences and improvements.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: fe294b77-4492-471d-9d6e-9c924153f4d8
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 08:36:32 +00:00
marco370
a47079c97c Add historical training data logging for hybrid models
Integrate saving of training history to the database within `train_hybrid.py`, ensuring that model versioning is correctly applied for hybrid detector runs.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 9f8d0aa1-70ec-4271-b143-5f66d1d3756b
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 08:08:45 +00:00
marco370
cf094bf750 Update model version tracking for training history
Dynamically set the model version to "2.0.0" for hybrid detectors and "1.0.0" for legacy detectors, and update the database insertion logic in `main.py` to use this dynamic version.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 25db5356-3182-4db3-be10-c524c0561b39
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 08:01:03 +00:00
marco370
5feb691122 Fix error when hybrid detector models are not loaded
Correctly check if the hybrid detector models are loaded by verifying the presence of `isolation_forest` instead of a non-existent `is_trained` attribute in `python_ml/main.py`.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: c8073c39-409d-45f4-a3e8-e48ce4d71e32
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 07:56:15 +00:00
marco370
08af108cfb Fix backend crash when initializing hybrid ML detector
Corrected `main.py` to handle the `ml_analyzer` being `None` when `USE_HYBRID_DETECTOR` is true, preventing an `AttributeError` during startup.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 27f5de5e-5ed6-4ee6-9cc2-a7c448ad2334
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/XSkkaPM
2025-11-25 07:53:47 +00:00
marco370
f6e222d473 Correct SQL query for retrieving network traffic data
Fixes a critical SQL syntax error in `train_hybrid.py` preventing data loading by adjusting the `INTERVAL` calculation for the `timestamp` WHERE clause.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: fd776d7a-7ad0-46a7-9500-792cb8944915
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/XSkkaPM
2025-11-24 18:16:21 +00:00
marco370
b88377e2d5 Adapt ML model to new database schema and automate training
Adjusts SQL queries and feature extraction to accommodate changes in the network_logs database schema, enabling automatic weekly retraining of the ML hybrid detector.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: f4fdd53b-f433-44d9-9f0f-63616a9eeec1
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 18:14:43 +00:00
marco370
7ba039a547 Fix index out of bounds error during synthetic data testing
Corrected an indexing error in `train_hybrid.py` by using `enumerate` to ensure accurate mapping of detections to the test dataset, resolving an `IndexError` when processing synthetic data.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: d05c3dd2-6349-426d-be9c-ec80a07ea78f
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:57:22 +00:00
marco370
9fe2532217 Add timestamp to synthetic data for accurate model testing
Add a 'timestamp' column to the synthetic dataset generation in `python_ml/dataset_loader.py` to resolve a `KeyError` during model training and testing.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 276a3bd4-aaee-40c9-acb7-027f23274a9f
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:52:16 +00:00
marco370
75d3bd56a1 Simplify ML dependency to use standard Isolation Forest
Remove problematic Extended Isolation Forest dependency and leverage existing scikit-learn fallback for Python 3.11 compatibility.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 89ea874d-b572-40ad-9ac7-0c77d2b7d08d
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:44:11 +00:00
marco370
b4aaa5456f Add Cython to Python dependencies for model compilation
Add Cython==3.0.5 to python_ml/requirements.txt and update replit.md to reflect this change, resolving a compilation issue with the eif library.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: f24578fc-6be7-42c0-9a9c-5ffe13dacdbe
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 16:35:15 +00:00
marco370
043690f829 Update dependency version for improved compatibility
Update `eif` dependency from version 2.0.0 to 2.0.2 in `requirements.txt` and documentation to resolve a deployment issue, as version 2.0.0 is not available.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 104a0ac7-f020-4d00-9e3e-5a37b74bbc93
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 16:32:31 +00:00
marco370
16617aa0fa Improve model training by adding robust error handling and logging
Add exception handling to the model training process to log failures and improve robustness.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 9c7ad6b8-3e9d-41fe-83f7-6b2a48f8ff44
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 16:25:40 +00:00
marco370
8b16800bb6 Update system to use hybrid detector and improve validation accuracy
Update main.py endpoints to use the hybrid detector and improve validation logic in train_hybrid.py by mapping detections using source_ip. Also, add synthetic source_ip to dataset_loader.py for both CICIDS2017 and synthetic datasets.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 5c4982f1-3d37-47da-9253-c04888f5ff64
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 16:02:49 +00:00
marco370
4bc4bc5a31 Update backend API to support new hybrid ML detection system
Introduce MLHybridDetector and update FastAPI app configuration to prioritize it, along with a new training script `train_hybrid.py`.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 462e355b-1642-45af-be7c-e04efa9dee67
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/F6DiMv4
2025-11-24 15:57:23 +00:00
marco370
350a0994bd Add dataset loader and validation metrics modules
Introduces `CICIDS2017Loader` for dataset handling and `ValidationMetrics` class for calculating performance metrics in Python.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: ad530f16-3a16-44a3-8fed-6c5d56775c77
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/F6DiMv4
2025-11-24 15:55:30 +00:00
marco370
932931457e Add a hybrid machine learning detection system
Add a new ML hybrid detector module with Extended Isolation Forest, feature selection, and an ensemble classifier, along with updated Python dependencies.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 8b74011c-0e9a-4433-b9a1-896e65cb4ae1
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/F6DiMv4
2025-11-24 15:53:05 +00:00
marco370
3de433f278 Fix analytics data inconsistency on live dashboard
Update analytics aggregator to correctly count attack occurrences and fix type hinting for daily aggregation.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: f54e3953-68c3-42e1-be9d-1d1db98db671
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/F6DiMv4
2025-11-24 15:28:22 +00:00
marco370
3d7a0ce424 Improve attack data accuracy and add validation checks
Update analytics aggregation logic to accurately count attack packets by type and country, including fallbacks for missing data, and add validation to ensure breakdown totals match reported attack packets.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: cafbc828-3e12-4d4f-8a02-5127b485612d
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/F6DiMv4
2025-11-24 15:27:17 +00:00
marco370
2b802397d1 Improve analytics data aggregation security and reliability
Enhance security by removing hardcoded paths, implementing a wrapper script for manual execution, and adding robust credential validation in the analytics aggregator.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 47661cde-4285-4ce6-8d00-fb236a5a01b7
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/oGXAoP7
2025-11-24 08:35:46 +00:00
marco370
cbd03d9e64 Add network analytics and live dashboard features
Introduce new network analytics capabilities with persistent storage, hourly and daily aggregations, and enhanced frontend visualizations. This includes API endpoints for retrieving analytics data, systemd services for automated aggregation, and UI updates for live and historical dashboards. Additionally, country flag emojis are now displayed on the detections page.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 3c14f651-7633-4128-8526-314b4942b3a0
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/oGXAoP7
2025-11-22 11:34:36 +00:00
marco370
1b9df79d56 Add IP geolocation and AS information to detection records
Integrates IP geolocation and Autonomous System (AS) information into detection records by modifying the frontend to display this data and updating the backend to perform asynchronous batch lookups for efficiency. This enhancement includes database schema updates and the creation of a new IP geolocation service.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: e81fd4a1-b7b0-48d2-ae38-f5905e278343
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/SXFWABi
2025-11-22 10:59:50 +00:00
marco370
07c7e02770 Add automatic database cleanup for older log entries
Implement automatic cleanup of log entries older than 3 days in the database and switch to a safer streaming mode for processing log files.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 70142a7e-f1e2-4668-9fee-f1ff7d8615ae
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/SXFWABi
2025-11-22 10:37:12 +00:00
marco370
c31e1ca838 Improve training history logging and file management
Enhance error handling in Python ML backend for training and update script location.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: b7099249-7827-46da-bdf9-2ff1d9c07b6c
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/VDRknFA
2025-11-22 10:21:29 +00:00
marco370
7ec5ff553b Add systemd service management with API key security
Implement systemd service management for ML backend and Syslog parser with API key authentication and robust error handling across frontend and backend.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: e0ddd146-1e7d-40e4-8607-ef8d247a1f49
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/n4Q2eeE
2025-11-22 09:33:30 +00:00
marco370
4a2d7f9c5c Add service monitoring and status indicators to the dashboard
Introduce a new services page, integrate real-time status monitoring for ML backend, database, and syslog parser, and update the dashboard to display service health indicators.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: cde95c60-908b-48a0-b7b9-38e5e924b3b3
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/n4Q2eeE
2025-11-22 09:24:10 +00:00
marco370
7ba65c9d96 Fix errors when retrieving statistics by handling empty results
Update the get_stats function in main.py to safely fetch and process counts from the database, preventing potential errors when no records are found for total logs, recent logs, detections, blocked IPs, and active routers.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 853b2085-c74d-4b3d-adeb-9db4276a24aa
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/1P26v7M
2025-11-17 18:18:30 +00:00
marco370
dc91096d9d Improve log processing and add automated tasks
Update log processing to use the correct timestamp field and introduce cron jobs for automated model training and anomaly detection.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: f0653fd5-fc94-4fcb-8d7e-2a0e90fc81bf
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 18:11:49 +00:00
marco370
46ab780e60 Add detailed logging to track the syslog parser's execution and state
Enhance syslog_parser.py with debug, info, and error logging statements, including checks for log file existence and robust error handling for database connections and parsing.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 1663254c-86fe-4c1e-966a-168eb8cd8f97
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 17:46:37 +00:00
marco370
0d34bf7d3c Update log parsing to better identify network traffic and DDoS events
Refactors the `SyslogParser` class in `python_ml/syslog_parser.py` to use a new, more comprehensive regex pattern (`main_pattern`) for parsing MikroTik logs. This includes improved identification of 'forward' and 'detected-ddos forward' actions, protocol details (UDP, TCP, ICMP), and associated IP addresses, ports, and lengths. The changes aim to accurately capture network traffic and potential DDoS events from MikroTik logs.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: b7377ada-e722-475a-86d2-07f21299ec70
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 17:35:37 +00:00
marco370
e201c0e770 Add ability to load environment variables from a file
Imported the load_dotenv function from the dotenv library in syslog_parser.py.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 58d752f9-b57e-4e4c-be73-8208077248c4
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 17:31:57 +00:00
marco370
62c3a2b238 Update application to use environment variables and force IPv4
Modify syslog_parser.py to load environment variables from .env file and force IPv4 connection to the database. Update replit.md to reflect recent fixes and workflow changes. Increment version in version.json.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 727221f9-ad54-4498-b2e4-e87a951b4308
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/c9ITWqD
2025-11-17 16:48:22 +00:00
marco370
fcd4bbf2b2 Add comprehensive deployment and configuration guides for the IDS system
This commit introduces detailed documentation for deploying the Intrusion Detection System (IDS) on AlmaLinux 9, including setup scripts, MikroTik router configuration, and update procedures via git. It also includes the syslog parser script for processing router logs and saving them to PostgreSQL.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: b2b01a4a-55da-4f33-9143-6bf0399e0a03
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/c9ITWqD
2025-11-15 11:30:55 +00:00
marco370
ac9c35b61f Add database storage for network data and router management
Refactors storage to use a database backend, introducing schemas and functions for routers, network logs, detections, whitelist, and training history. Integrates Drizzle ORM with Neon Postgres for data persistence.

Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 4e9219bb-e0f1-4799-bb3f-6c759dc16069
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/c9ITWqD
2025-11-15 11:12:44 +00:00