Researchers have identified a critical flaw in Activation Oracles, AI systems designed to interpret the internal workings of other language models by answering questions about their hidden states.

The study, published on arXiv by Tobias Bersia and Tatiana Gaintseva, examined how these interpretability tools perform when trained on models that internally use concepts while avoiding direct disclosure.

Activation Oracles represent a promising approach to AI interpretability. They function as specialized language models trained to read and report on the internal activations of other AI systems, potentially revealing information that models represent internally but don't express in their outputs.

Unexpected Anti-Reading Behavior

The researchers tested Activation Oracles in a controlled "Taboo Word Guessing" scenario. Subject models were fine-tuned to use hidden concepts internally while avoiding direct mention of them in their responses.

Contrary to expectations, the Activation Oracles didn't become better at reading these hidden concepts. Instead, they developed what the researchers term "concept-specific anti-reading" behavior — selectively failing to detect concepts that were persistently present during their training.

This failure wasn't due to the absence of the target concept from the model representations. The researchers confirmed the information remained decodable within the oracle itself using LogitLens and layer-ablation analyses.

The problem appears to arise in the oracle's readout pathway rather than in the underlying representations. This suggests the interpretability tool learns to ignore certain information rather than failing to access it.

Implications for AI Safety

The findings raise significant concerns about the reliability of learned interpretability interfaces. The research demonstrates that behavioral outputs, representation-level information, and oracle verbalization can diverge in unexpected ways.

This disconnect could prove problematic for AI safety applications, where interpretability tools are increasingly relied upon to understand model behavior and detect potential risks.

The study highlights the need for more robust evaluation methods for AI interpretability systems, particularly as these tools become more sophisticated and widely deployed in safety-critical applications.