Researchers from multiple institutions have developed a novel approach to address cross-task interference in large language model training, publishing their findings in a paper accepted at ICML 2026.

The method, called Basic Abilities Decomposition for multi-task Instruct-Tuning (BADIT), tackles a fundamental problem in current LLM training where different tasks create conflicting gradients over shared parameters.

Multi-task instruct-tuning has driven much of the recent performance gains in models from companies like OpenAI and Anthropic. However, this training approach suffers from cross-task interference when tasks compete for the same model parameters.

Previous solutions attempted to isolate task-specific parameters through methods like task-specific neuron selection and mixture-of-experts architectures. The research team, led by Bing Wang, found these approaches still leave many parameters shared across tasks.

Novel Decomposition Approach

BADIT operates on the principle that LLMs encode several orthogonal basic abilities, with any task representable as a linear combination of these abilities. The method decomposes model parameters into orthogonal high-singular-value LoRA experts representing these basic abilities.

The approach dynamically enforces orthogonality during training through spherical clustering of rank-1 components. This prevents different tasks from interfering with each other's parameter updates.

The researchers tested BADIT across six different LLMs using the SuperNI benchmark. Results showed the method outperformed existing state-of-the-art approaches while reducing cross-task interference.

The team empirically discovered that certain parameters consistently activate together, naturally organizing into base groups. This observation led to their analogy of LLMs encoding orthogonal basic abilities.

The research provides both theoretical insights into how LLMs organize knowledge and a practical solution for improving multi-task training efficiency. The code for BADIT has been made available on GitHub for further research and implementation.