AI Ensemble Framework for OSINT-Based Social Media Crime Prevention in Kenya

Abstract

Criminal networks increasingly exploit social media to organize, recruit, incite, and coordinate unlawful activities, whereas law-enforcement monitoring in Kenya remains largely manual and English-centric. This study aimed to design, implement, and rigorously evaluate an artificial intelligence (AI) ensemble framework for open-source intelligence (OSINT) classification of crime- and threat-relevant content in Kenya's trilingual (English, Kiswahili, and Sheng) and code-switched digital environment. A corpus of 6,212 documents was assembled from six sources: Twitter/X, Facebook, Instagram, TikTok, and two Sheng-language corpora. The corpus was weakly labelled using a 214-term trilingual crime-and-threat lexicon. Label noise was quantified against a manually annotated stratified validation sample (n = 320), yielding Cohen's κ = 0.350, silver-label precision of 0.525, and recall of 0.750. Combined word- and character-level term frequency-inverse document frequency (TF-IDF) features comprising 25,831 dimensions were used to train a hyperparameter-tuned, weighted soft-voting ensemble consisting of multinomial naive Bayes, a calibrated linear support vector machine, random forest, logistic regression, and XGBoost. The decision threshold was calibrated using cross-validated out-of-fold probabilities. The optimized ensemble achieved 98.3% accuracy, 80.6% precision, 62.5% recall, 70.4% F1-score, 0.907 ROC-AUC, 0.724 PR-AUC, 0.702 Matthews correlation coefficient (MCC), and a 0.50% false-positive rate. The ensemble outperformed the best single learner and a frozen multilingual transformer baseline (F1 = 26.2%). Ablation analysis demonstrated the contribution of the Sheng-normalization layer, while comparison of class-imbalance strategies showed that cost-sensitive learning combined with threshold calibration produced the strongest overall performance compared with SMOTE-based resampling. The findings demonstrate that linguistically inclusive classical machine-learning ensembles can provide an accurate, auditable, and computationally deployable approach to proactive social-media crime prevention in low-resource, code-switched settings. The study also derives policy implications concerning threshold-calibrated triage, expert annotation, data-protection impact assessments, and safeguards against dual-use misuse.