Google Research has developed a new statistical framework to verify whether AI systems have successfully "forgotten" specific parts of their training data through machine unlearning.

The Regularized f-Divergence Kernel Tests, presented at AISTATS 2026, addresses a critical challenge as AI models process increasingly sensitive datasets. Current verification methods often fail to detect subtle differences or produce false positives, making it difficult to prove compliance with regulations like GDPR's "Right to be Forgotten".

Machine unlearning allows AI systems to remove specific data without the massive computational cost of retraining from scratch. However, auditing these systems has proven challenging because traditional two-sample testing methods lack the statistical power needed for large-scale models.

Why existing tools fall short

Standard verification tools like maximum mean discrepancy excel at detecting broad shifts across data but miss localized anomalies. For instance, if adding a specific person's data causes a model to generate outlier outputs only under exact prompts, traditional tests might overlook this localized shift entirely.

Existing frameworks also force researchers to make error-prone manual choices, such as selecting statistics suited for global versus local shifts or tuning complex parameters like kernel bandwidths.

Google's research team, led by Mónica Ribero, found that traditional two-sample testing produces false positives when comparing models trained on identical data but with different batch sizes. This fundamental flaw means auditors cannot reliably distinguish between successfully unlearned models and those with different training configurations.

The new framework resolves this by measuring whether an unlearned model is distributionally closer to a safely retrained model or to the original compromised version. This relative distance approach provides more reliable verification.

The framework leverages f-divergences to detect specific types of data shifts, including Chi-squared and Kullback-Leibler divergences for smooth differences, and Hockey-stick divergence specifically designed for privacy and unlearning definitions.

Google's researchers theoretically prove their tests control false positives for any sample size, with false negative risks converging to zero as available data samples increase. This makes real-world auditing computationally feasible while maintaining statistical rigor.

The framework addresses a growing industry need as AI systems handle more sensitive data and face stricter regulatory requirements for demonstrable privacy compliance.