Researchers from King's College London and other institutions have identified fundamental flaws in self-play red teaming, a widely used method for training AI models to resist harmful prompts.

The study, published on arXiv, demonstrates that when the same model plays both attacker and defender roles in safety training, the system collapses to "self-consistency" rather than creating genuine adversarial pressure. This means the model essentially agrees with itself, undermining the core purpose of red team training.

"The set of Nash equilibria that can be reached corresponds to a broad class of behaviours that includes trivial always refuse strategies and oracle-like defenders," the researchers wrote. This limits the practical applicability of current safety training approaches used by major AI labs.

Why Current Methods Fail

Traditional self-play safety training uses identical model instances as both attacker and defender in a zero-sum game. While parameter sharing improves stability, it creates theoretical limitations that prevent effective adversarial training.

The research team tested their findings on Qwen2.5 models ranging from 3 billion to 14 billion parameters across established safety benchmarks. Results consistently showed that shared-parameter approaches fail to maintain the adversarial dynamics necessary for robust safety training.

Anchored Bipolicy Solution

To address these limitations, the researchers propose "Anchored Bipolicy Self-Play," which trains distinct LoRA adapters for attacker and defender roles on top of a frozen base model. This approach maintains stable optimization while preserving adversarial pressure through explicit role separation.

The new method achieved up to 100x greater parameter efficiency compared to traditional fine-tuning approaches. Cross-play experiments demonstrated that models trained with the bipolicy method outperformed standard self-play models in both adversarial defense and safety metrics.

Testing showed improved robustness without compromising reasoning ability, addressing a key concern in AI safety research where security measures often degrade model performance.

The findings have significant implications for AI safety practices at companies like Anthropic and OpenAI, which rely heavily on red teaming for model safety. The research suggests current industry-standard approaches may be fundamentally insufficient for training truly robust AI systems.

The team plans to release implementation details and benchmarking code to enable broader adoption of the bipolicy training methodology.