Databricks has published results from an internal benchmark testing coding agents on actual engineering tasks performed against its multi-million line codebase.

The data infrastructure company evaluated models from OpenAI, Anthropic, and open-source alternatives on real coding tasks spanning Python, Go, TypeScript, and Scala. The benchmark measured both performance and cost across different complexity levels.

Three capability tiers emerged

The results clustered models into distinct performance groups. Premium models like GPT-4 and Claude Opus delivered the highest quality but at significant cost. Mid-tier models proved highly effective for common tasks while being substantially cheaper.

Open-source GLM 5.2 landed in the top capability tier, statistically tied with Opus 4.8 on quality while costing $1.28 per task versus Opus's $1.94.

"GLM quality scores are consistent with qualitative feedback we've gotten from internal developers who have been piloting GLM for daily development," the company said.

Databricks found that token pricing poorly predicts actual task costs. Sonnet 5 costs 1.7x less per token than Opus 4.8, but ended up costing $2.09 per task versus Opus's $1.94 while scoring six points lower on completion rates.

This occurred because Sonnet consumed 1.9x more tokens to complete tasks, working longer and reading more content.

Harness choice matters significantly

The platform used to access models dramatically affected both cost and quality. Simple harnesses like Pi often outperformed more complex alternatives on Databricks' workloads.

The benchmark covered tasks ranging from basic configuration updates to complex design explorations. Common operational tasks like flag changes didn't require the most intelligent models, while deeper architectural work benefited from premium capabilities.

Based on these findings, Databricks plans to shift more routine work to cheaper models like Claude Haiku and GPT-4 Mini, reserving premium models for complex tasks.

The company said the analysis has already made its engineering team "meaningfully more efficient with coding agents" by matching model capability to task complexity.