A 146-million-parameter behavioral auditor trained from scratch can detect AI compliance gaps that human raters consistently miss, according to new research from arXiv.

The compact model achieved 90.7% binary compliance accuracy on tasks where trained human raters showed poor agreement, with a Fleiss kappa score of just 0.074. The study compared the small auditor against a frontier judge across 720 identical replies.

Researchers found that measurement targets significantly impact evaluator performance. The auditor led on "exposure" detection — identifying whether a reply was produced under behavior-inducing conditions — scoring 0.804 AUROC against the frontier judge's 0.718.

However, the ranking reversed for "manifestation" detection, where the auditor trailed at 0.690 AUROC compared to the frontier judge's 0.811. Manifestation measures whether problematic behavior actually surfaced in the response.

Output resolution drives performance gaps

The performance gap between instruments shifted by roughly 0.2 AUROC when researchers changed the measurement target. Under single-verdict interfaces, the auditor and frontier judge switched rankings depending on the specific detection task.

Matching output resolution from either direction — asking target-specific questions with continuous confidence scores or thresholding the auditor's read-out — removed the ranking reversal but preserved the interaction effect.

The study tested three different output resolutions, finding consistent interactions that excluded zero at all levels (0.207, 0.237, and 0.169). The target determines how far apart the instruments perform, while the interface governs whether that distance changes their relative order.

Contrary to expectations, the auditor's hyperbolic geometry provided no measurable advantage in these behavioral detection tasks.

The research highlights that single behavioral-detection AUROC scores are under-specified without stating the estimand, evaluator type, and output interface. This makes direct comparisons between different AI safety evaluation methods unreliable without proper context.

The findings suggest that compact, specialized models can match or exceed larger systems on specific AI safety tasks, potentially offering more efficient approaches to behavioral auditing in production systems.