A team of researchers has developed CyberSecQwen-4B, a 4 billion parameter cybersecurity model that matches the performance of larger models while running on a single consumer GPU.

The model achieves 97.3% of the accuracy of Cisco's Foundation-Sec-Instruct-8B on CVE-to-CWE mapping tasks, while exceeding it by 8.7 percentage points on cybersecurity multiple-choice questions. The researchers trained the model on a single AMD Instinct MI300X GPU through the AMD Developer Cloud.

Why smaller models matter for cybersecurity

The team argues that frontier models create unacceptable tradeoffs for defensive cybersecurity work. SOC analysts handling leaked credential dumps or malware researchers dissecting samples cannot send sensitive data to external APIs without risking breaches.

Per-call API costs also compound quickly when processing thousands of low-confidence alerts daily. Air-gapped environments in critical infrastructure and government work require models that run entirely on-premises.

The researchers built on Hugging Face's Qwen3-4B-Instruct-2507, an Apache 2.0-licensed instruction-tuned model. They fine-tuned it using LoRA with rank 64 and a learning rate of 5e-5 over 10 epochs.

Training data included 2021 CVE-to-CWE mappings from MITRE and NVD public records, with all overlap with evaluation benchmarks removed to prevent contamination. The team also generated synthetic defensive analyst Q&A grounded in deduplicated CVE descriptions.

The model runs in full bf16 precision with FlashAttention-2 on the MI300X's 192GB of HBM3 memory. The researchers report the training recipe is hardware-agnostic and can run on other 40GB+ datacenter GPUs.

Benchmark results show CyberSecQwen-4B scored 0.5868 on CTI-MCQ compared to Foundation-Sec-Instruct-8B's 0.4996, while achieving 0.6664 on CTI-RCM versus 0.6850 for the larger model.

The model and training code are available under Apache 2.0 licensing. The researchers plan to release additional specialized models for other cybersecurity tasks.