Abstract:
Malware analysis demands rapid interpretation of complex detonation reports spanning filesystem, network, and process behaviours. While large language models (LLMs) demonstrate impressive capabilities for technical artifact interpretation, the opacity and escalating API costs of closed-weight frontier models motivate exploration of open-weight alternatives. However, many open-weight models are large, demanding significant compute resources and incurring non-trivial hosting costs that place them beyond reach for resource-constrained deployments. This paper investigates whether orchestrated ensembles of small language models (SLMs) can match or exceed single LLM performance on structured questions about malware detonation reports. We established baselines by testing eleven open-weight SLMs, three cyber security pre-trained models, and six frontier LLMs on Meta's CyberSecEval Malware Analysis benchmark. We then designed and evaluated four orchestration architectures: (i) a multi-agent pipeline that decomposes analysis into structured evidence-collection and reasoning stages, (ii) an adversarial debate framework in which two agents iteratively critique each other's reasoning, (iii) a hierarchical consultation system that pairs a general-purpose SLM with a cyber-specialised expert model, and (iv) a hybrid architecture that combines evidence-grounded pipelines with adversarial debate reasoning. The hybrid system (Qwen3-4B with Foundation-Sec-8B) achieved 35.30% overall accuracy, exceeding the strongest cyber-specialised baseline (22.54%) and the strongest ungrounded frontier baseline (34.77%); when given the same evidence pipeline, grounded Gemini remained the strongest configuration at 38.22%. These findings show that evidence-grounded orchestration can substantially improve the performance of collaborative SLMs for supporting interpretation of malware detonation reports.
This talk is based on the following research paper:
Adel ElZemity, Shujun Li and Budi Arief (2026) Small, Free, and Effective: Orchestrating Open-Weight Small Language Models to Outperform Single LLM for Malware Analysis. Accepted to RAID 2026 (29th International Symposium on Research in Attacks, Intrusions and Defenses) in July 2026, to be published in a volume of Lecture Notes in Computer Science (LNCS) by Springer. Preprint available at https://arxiv.org/abs/2607.20216
Bio:
Adel ElZemity is a PhD researcher in Artificial Intelligence and Cyber Security at the University of Kent’s School of Computing, under supervision of Dr Budi Arief and Professor Shujun Li. From October 2026, he is also starting a new Research Associate post at the School of Electronic Engineering and Computer Science, Queen Mary University of London. His work spans the safety and security of large language models for cyber security — from evaluating risks in fine-tuned models and instruction datasets, to designing safer alignment approaches, to studying AI agents that can discover, exploit, and mitigate vulnerabilities on real systems, including industrial OT and IoT environments. His research has been accepted at venues including RAID 2026, AISec (co-located with ACM CCS), and SECAI (co-located with ESORICS), and he has a paper under review at Computers & Security. More details: https://adelsamir.com
Please use the link below to join this event online:
Meeting ID: 319 712 936 207 504
Passcode: f6YF6ab6