marco370
a4bf75394a
Add ability to trigger manual IP blocking and detection
...
Add a curl command to manually trigger IP detection and blocking with specific parameters.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: c0150b70-3a40-4b91-ad03-5beebb46ed63
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/8i4FqXF
2026-01-02 15:44:20 +00:00
marco370
a311573d0c
Fix errors in IP detection and merge logic by correcting data types
...
Addresses type mismatches in `risk_score` handling and INET comparisons within `merge_logic.py`, ensuring correct data insertion and IP range analysis.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: e1f9b236-1e9e-4ac6-a8f7-8ca066dc8467
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/zqNbsxW
2026-01-02 15:19:26 +00:00
marco370
3574ff0274
Update database schema and migrations to correctly handle IP address data types
...
Introduce migration 008 to force INET and CIDR types for IP-related columns in `whitelist` and `public_blacklist_ips` tables, and update `shared/schema.ts` with comments clarifying production type handling.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 1d0f629d-65cf-420d-86d9-a51b24caffa4
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/rDib6Pq
2026-01-02 14:44:54 +00:00
marco370
0301a42825
Update IP address parsing to ensure uniqueness and fix duplicates
...
Update `normalize_cidr` function in `parsers.py` to use the full CIDR notation as the IP address for uniqueness, addressing duplicate entry errors during Spamhaus IP sync and resolving the `operator does not exist: inet = text` error related to the `whitelist` table by ensuring proper IP type handling.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 478f21ca-de02-4a5b-9eec-f73a3e16d0f0
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/rDib6Pq
2026-01-02 11:56:47 +00:00
marco370
3425521215
Update list fetching to handle new Spamhaus format and IP matching
...
Update Spamhaus parser to support NDJSON format and fix IP matching errors by ensuring database migrations are applied.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 11e93061-1fe5-4624-8362-9202aff893d7
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/rDib6Pq
2026-01-02 11:48:33 +00:00
marco370
c3a6f28434
Add idempotency to database migrations and fix data type issues
...
Modify database migrations to use `IF NOT EXISTS` for index creation and adjust column types from TEXT to INET to resolve data type conflicts.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 7b4fcf5a-6a83-4f13-ba5e-c95f24a8825a
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/zauptjn
2026-01-02 11:38:49 +00:00
marco370
c4546f843f
Fix permission errors to allow saving machine learning models
...
Correct ownership of the models directory to allow the ML training process to save generated models.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 51de2a29-c1c5-4d67-b236-7a1824b5b0d1
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/jFtLBWL
2025-11-25 17:40:13 +00:00
marco370
42541724cf
Fix issue where the application fails to start due to port conflicts
...
Resolve "address already in use" error by resetting systemd, killing all Python processes, and ensuring the port is free before restarting the ML backend service.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: fe8f5eaa-c00f-4120-8b35-be03ff3fca3f
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/jFtLBWL
2025-11-25 17:35:58 +00:00
marco370
955a2ee125
Fix backend startup issue by resolving port conflict
...
Resolves an "address already in use" error by killing existing processes on port 8000 before restarting the ML backend service.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 2c691790-1a58-44ba-94dd-f03a528d1174
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/jFtLBWL
2025-11-25 17:33:54 +00:00
marco370
2d7185cdbc
Adjust model comparison script to correctly process network logs
...
Correct logic in `compare_models.py` to pass raw network logs to the detection method, ensuring correct feature extraction and preventing a 'timestamp' KeyError.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: ecdb452a-13bf-4c0b-8da9-eebbafd63834
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/RJGlbTt
2025-11-25 08:43:55 +00:00
marco370
d086b00092
Fix system service to prevent continuous restart failures
...
The systemd service for the ML backend is repeatedly failing and restarting due to an exit-code failure.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: d83d5831-e125-4886-bdea-1bb0aba2d63b
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/XSkkaPM
2025-11-25 07:51:16 +00:00
marco370
b88377e2d5
Adapt ML model to new database schema and automate training
...
Adjusts SQL queries and feature extraction to accommodate changes in the network_logs database schema, enabling automatic weekly retraining of the ML hybrid detector.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: f4fdd53b-f433-44d9-9f0f-63616a9eeec1
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 18:14:43 +00:00
marco370
7ba039a547
Fix index out of bounds error during synthetic data testing
...
Corrected an indexing error in `train_hybrid.py` by using `enumerate` to ensure accurate mapping of detections to the test dataset, resolving an `IndexError` when processing synthetic data.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: d05c3dd2-6349-426d-be9c-ec80a07ea78f
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:57:22 +00:00
marco370
9fe2532217
Add timestamp to synthetic data for accurate model testing
...
Add a 'timestamp' column to the synthetic dataset generation in `python_ml/dataset_loader.py` to resolve a `KeyError` during model training and testing.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 276a3bd4-aaee-40c9-acb7-027f23274a9f
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:52:16 +00:00
marco370
75d3bd56a1
Simplify ML dependency to use standard Isolation Forest
...
Remove problematic Extended Isolation Forest dependency and leverage existing scikit-learn fallback for Python 3.11 compatibility.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: 89ea874d-b572-40ad-9ac7-0c77d2b7d08d
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:44:11 +00:00
marco370
051c838840
Add ability to install ML dependencies and resolve build issues
...
Update install_ml_deps.sh to use --no-build-isolation when installing eif to resolve ModuleNotFoundError during build.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 219383e3-8935-415d-8c84-77e7d6f76af8
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:06:43 +00:00
marco370
102113e950
Improve ML dependency installation script for robust deployment
...
Update deployment script to correctly activate virtual environment, install Cython and numpy as build dependencies before eif, and ensure sequential installation for the ML hybrid detector.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 8b4c76c7-3a42-4713-8396-40f5db530225
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 17:02:15 +00:00
marco370
b78f03392a
Update deployment process to handle machine learning dependencies
...
Create a dedicated script to install machine learning dependencies in the correct order, ensuring Cython is installed before packages that require it for compilation, and update documentation accordingly.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: aa7dc534-7330-4bd4-b726-d6eeb29008af
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 16:38:41 +00:00
marco370
b4aaa5456f
Add Cython to Python dependencies for model compilation
...
Add Cython==3.0.5 to python_ml/requirements.txt and update replit.md to reflect this change, resolving a compilation issue with the eif library.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: f24578fc-6be7-42c0-9a9c-5ffe13dacdbe
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 16:35:15 +00:00
marco370
043690f829
Update dependency version for improved compatibility
...
Update `eif` dependency from version 2.0.0 to 2.0.2 in `requirements.txt` and documentation to resolve a deployment issue, as version 2.0.0 is not available.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 104a0ac7-f020-4d00-9e3e-5a37b74bbc93
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/2lUhxO2
2025-11-24 16:32:31 +00:00
marco370
3d7a0ce424
Improve attack data accuracy and add validation checks
...
Update analytics aggregation logic to accurately count attack packets by type and country, including fallbacks for missing data, and add validation to ensure breakdown totals match reported attack packets.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: cafbc828-3e12-4d4f-8a02-5127b485612d
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/F6DiMv4
2025-11-24 15:27:17 +00:00
marco370
e629bf4ed3
Improve analytics data fetching and server restart process
...
Fixes the analytics API call by correctly formatting the query parameters in the `AnalyticsHistory.tsx` component. Enhances the `restart_frontend.sh` script for more aggressive process killing and port cleanup to prevent 'address already in use' errors. Also, adds a check for the `country` column existence in the database schema, addressing a potential mismatch between development and production environments.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 23dd17a9-47b9-4533-bf4c-8b5cfdb426b4
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/x5P9dcJ
2025-11-24 09:59:06 +00:00
marco370
0bf61dc69d
Improve model training and file saving capabilities
...
Fixes permission errors for model saving and enhances training logging, ensuring proper storage of ML models and historical data.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 2afb7ddf-484b-4d07-8d99-8c1ca39c0be5
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/VDRknFA
2025-11-22 10:27:30 +00:00
marco370
9d5ecf99c4
Add branding and content assets for the application interface
...
Add various branding assets, content files, and screenshots to enhance the application's user interface and visual presentation.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 5db5df9a-80de-4fbe-96fb-67a573fc567e
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/VDRknFA
2025-11-22 10:12:00 +00:00
marco370
26f3589a7e
Improve systemd service setup to aggressively kill manual processes
...
Updates `setup_systemd_services.sh` to forcefully stop existing systemd services, kill all manual Python processes owned by the `ids` user, and verify that port 8000 is free before starting the services.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: ae567421-923d-4371-a127-7bdeca91b824
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/VDRknFA
2025-11-22 10:03:21 +00:00
marco370
e9e74f9944
Add missing Python libraries for backend functionality
...
Update the Python dependency installation script to include `httpx` and `joblib`, and enhance verification checks for installed modules.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: intermediate_checkpoint
Replit-Commit-Event-Id: f0a704c9-cac4-4144-8f26-c6066459f615
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/n4Q2eeE
2025-11-22 09:57:47 +00:00
marco370
5b350ff95f
Add Python dependency installation and virtual environment support
...
Introduce a new script to install Python dependencies in a virtual environment, update systemd services to utilize this environment, and modify the setup script to automate dependency checks and installation.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: ea2a08f4-46e1-463d-9c58-16219914ad23
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/n4Q2eeE
2025-11-22 09:40:57 +00:00
marco370
d187aa533a
Fix ML backend service startup failures
...
Update the systemd service unit for the ML backend to correctly log errors to journalctl, enabling easier debugging of startup failures.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: adfa3c2c-2d2f-40c7-8113-83a526fb3a96
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/n4Q2eeE
2025-11-22 09:38:27 +00:00
marco370
015770609a
Update database schema and restart services
...
Apply SQL migrations and synchronize the database schema using Drizzle Kit, then restart systemd services.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 5dd94895-686f-45b1-9689-fcec435e180e
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/n4Q2eeE
2025-11-22 09:37:13 +00:00
marco370
4a2d7f9c5c
Add service monitoring and status indicators to the dashboard
...
Introduce a new services page, integrate real-time status monitoring for ML backend, database, and syslog parser, and update the dashboard to display service health indicators.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: cde95c60-908b-48a0-b7b9-38e5e924b3b3
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/n4Q2eeE
2025-11-22 09:24:10 +00:00
marco370
24b907e17b
Fix log parsing by adding missing timestamps to incoming data
...
The attached log file and agent reasoning indicate that the `ids-syslog-parser` service failed to restart and the `/var/log/mikrotik/raw.log` file lacks timestamps, leading to the parser saving 0 logs. The provided solution involves manually updating the `/etc/rsyslog.d/99-mikrotik.conf` file to include the `%TIMESTAMP%` directive, restarting the `rsyslog` service, clearing and restarting the log file, and then restarting the `ids-syslog-parser` service to process logs correctly.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 9fa8bbb2-1781-4d01-b6d3-3b872fb304a3
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/6ZTQSoP
2025-11-22 09:02:34 +00:00
marco370
9448d54156
Add timestamp to log entries to ensure proper parsing
...
Fixes an issue where log entries were missing timestamps, preventing the Python parser from correctly processing the data. This change modifies the rsyslog template to include the timestamp.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: d449ad65-7340-48be-804d-a473dce5d0d8
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/6ZTQSoP
2025-11-22 08:57:19 +00:00
marco370
495e845a79
Update log format to include timestamps and filter incoming connections
...
Correct the rsyslog template to include timestamps in logs, ensuring compatibility with the Python parser. This change also refactors the log filtering to capture only incoming connections, significantly reducing log volume.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: c2f849f9-105f-452a-bdc3-a956d102c54b
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/6ZTQSoP
2025-11-22 08:54:21 +00:00
marco370
c9a0ba1b66
Fix issue where logs are not saved to the database
...
Resolve a critical bug in the log processing system where millions of log entries were processed but not committed to the database, leading to zero records in the `network_logs` table. This involved stopping the parser, truncating/rotating the log file, and restarting the parser to process only new entries.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 6976842b-2cfa-4e4f-aea4-7e7206988693
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/DR50xVM
2025-11-21 17:37:31 +00:00
marco370
b31b0ec932
Add log file detailing network traffic and detected DDOS activity
...
Added a log file containing network traffic data, including packet forwarding information, connection states, and detected denial-of-service (DDOS) activities.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 43992744-515b-49bf-8459-fe8c14ec1bde
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/4LjHWWz
2025-11-21 17:10:36 +00:00
marco370
6f2d0da1c9
Update system with latest code and database changes
...
Applies recent code updates from the Git repository and executes necessary database schema migrations to ensure data integrity and system functionality.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 2bb36b2d-b880-414f-ad99-7f18bc18ee9e
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/4LjHWWz
2025-11-21 17:06:35 +00:00
marco370
661e945f57
Implement automatic database cleanup and schema updates
...
Adds scripts for automatic database log cleanup, schema migration application, and cron job setup. Modifies the update script to apply SQL migrations before pushing Drizzle schema.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 9a659f15-d68a-4b7d-99f8-3eccc59afebe
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/4LjHWWz
2025-11-21 16:49:13 +00:00
marco370
d10b470793
Fix database connection errors and schema issues
...
Resolve 500 errors across database API endpoints by implementing a dual-mode database driver and adding missing columns to the routers table.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: c333ed12-07c5-412a-aff0-524321acc652
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/C4ZJnmQ
2025-11-21 16:41:55 +00:00
marco370
f4803a7451
Fix error when fetching router and stats information
...
Add missing `api_port` and `last_sync` columns to the `routers` table in the database to resolve `column "last_sync" does not exist` errors.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: ed8286cc-22d7-40bd-ae28-6bae4dd7f5ea
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/C4ZJnmQ
2025-11-21 16:35:52 +00:00
marco370
e8270da285
Fix database connection errors and improve logging
...
Update database connection handling to resolve errors and log database type instead of connection strings.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 81036932-56b6-46dc-a2eb-153586dfd77d
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/C4ZJnmQ
2025-11-21 16:25:32 +00:00
marco370
42354d5087
Adapt database connection for local and cloud environments
...
Update `server/db.ts` to support both Neon serverless and standard PostgreSQL drivers, add database health checks, and improve error logging for database operations.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 55ee188b-6bb4-49b0-8966-1795106363b1
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/C4ZJnmQ
2025-11-21 16:09:48 +00:00
marco370
e5d038a2b4
Improve how the application starts to ensure environment variables are loaded correctly
...
Modify deployment/check_frontend.sh to use `env $(cat .env | grep -v '^#' | xargs) npm run dev` to inject environment variables into the npm process, resolving issues where `nohup` did not inherit exported variables. Update replit.md to reflect the corrected fix details.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 602cd9af-422f-4b1d-ab37-5fd3e0bd016f
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/dI1I79r
2025-11-21 15:58:17 +00:00
marco370
aec912cf16
Fix frontend startup to correctly load environment variables
...
Update `deployment/check_frontend.sh` to load `.env` variables before running `npm run dev`, resolving the `DATABASE_URL must be set` error and improving deployment on non-Replit environments.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 8cf499b5-f15d-4b0d-b459-f48510ac6539
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/dI1I79r
2025-11-21 15:52:15 +00:00
marco370
b3bbcb69d3
Update system to apply database changes and synchronize dependencies
...
Apply SQL scripts to update database schema and install/update Node.js and Python dependencies.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: bf319927-85dd-43f9-9df2-348d41742705
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/Aqah4U9
2025-11-21 14:31:40 +00:00
marco370
203eaa9a54
Improve system stability by fixing cron job and process monitoring
...
Refactor cron job execution to use separate shell scripts and update process monitoring scripts to correctly handle PID files and log directories.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: f6d4ec61-7130-41dd-aef9-87b4bc73d0e8
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/3R8dKMX
2025-11-18 08:33:05 +00:00
marco370
62ffbe499a
Add automated training and detection jobs to the system
...
Adds new cron jobs and shell scripts for automated ML model training and anomaly detection, along with configurations for logging and system checks.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 534244cd-b293-403e-a6d6-43cd9fbb30fb
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/1P26v7M
2025-11-18 08:24:09 +00:00
marco370
94724ff580
Add ability to detect and block malicious network traffic based on risk
...
Adds functionality to detect network anomalies and block suspicious IP addresses.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: e23fa694-b094-4b45-9ce0-0dbf8fd7a9e2
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/1P26v7M
2025-11-18 07:31:39 +00:00
marco370
dc91096d9d
Improve log processing and add automated tasks
...
Update log processing to use the correct timestamp field and introduce cron jobs for automated model training and anomaly detection.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: f0653fd5-fc94-4fcb-8d7e-2a0e90fc81bf
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 18:11:49 +00:00
marco370
d345a24572
Improve intrusion detection system with functional updates and database fixes
...
Update `replit.md` to reflect recent system improvements including a fully functional syslog parser, PostgreSQL database, updated regex patterns, DDoS detection, and automated deployment workflows. Addresses issues with the `network_logs` table schema and incorrect regex matching. Includes SQL query outputs for verification.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: 70827608-8ca8-471f-a794-336056b4ce88
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 17:54:18 +00:00
marco370
e5f307af27
Fix error when saving network logs to the database
...
Correct SQL query to insert data into the network_logs table, resolving "relation 'network_logs' does not exist" errors.
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 7a657272-55ba-4a79-9a2e-f1ed9bc7a528
Replit-Commit-Checkpoint-Type: full_checkpoint
Replit-Commit-Event-Id: e8e7edaf-8ac8-493c-b890-465ff617b0ce
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/449cf7c4-c97a-45ae-8234-e5c5b8d6a84f/7a657272-55ba-4a79-9a2e-f1ed9bc7a528/MkBJZ0L
2025-11-17 17:40:22 +00:00
