Hybrid Model Approach for Real-Time Detection of Anomalies in Cloud Virtual Private Network Traffic

Abstract

The rising use of cloud services such as OwnCloud has led organizations to rely heavily on Virtual Private Networks (VPNs) for secure remote access. While VPNs encrypt communication channels, traditional anomaly-detection methods are increasingly inadequate, despite the evolving threats. This paper aimed to design and evaluate a hybrid AI-based model for real-time anomaly detection in cloud VPN traffic to improve detection accuracy. A simulated cloud environment was created using three virtual machines, a client, a VPN server, and an OwnCloud server generating both normal and anomalous traffic. The hybrid model combined Isolation Forest, for detecting outliers, with Long Short-Term Memory (LSTM) networks, for analyzing time-dependent patterns. Real-time data streaming and processing were handled using Apache Kafka and Spark. The results showed that Isolation Forest achieved a precision of 0.86, recall of 0.79, and F1-score of 0.82, while LSTM scored 0.88, 0.83, and 0.85, respectively. The hybrid approach outperformed both models, achieving a precision of 0.93, recall of 0.89, F1-score of 0.91, and the highest AUC at 0.97. It is recommended that organizations using cloud-based VPNs integrate this hybrid AI anomaly detection system. The IT security teams, working with network infrastructure providers, should deploy the model for real-time encrypted traffic monitoring, improving accuracy and reducing false positives. From a policy perspective, regulatory bodies and cybersecurity standards organizations should revise compliance frameworks to encourage the usage of hybrid AI-driven detection methods in encrypted environments, ensuring both performance and privacy compliance. Practically, IT managers and IT leads should prioritize hybrid model deployment and continuous retraining using traffic patterns to boost resilience, reduce detection latency, and enhance incident response. Although validated in a simulated environment, future research should evaluate the hybrid model using real-world VPN traffic to confirm its robustness under diverse operational conditions.