Voice AI startup HOAi discovered that sending duplicate requests to standard-tier LLMs delivers better tail latency than paying double for priority processing.

The company, which builds voice agents for phone calls, faced a critical problem: occasional LLM responses taking 10 to 20 seconds created awkward silences that caused callers to hang up. With 20 to 30 turns per typical call, even a 1% rate of slow requests meant a 22% chance of hitting catastrophic delays.

HOAi tested two solutions against 50 production requests. The first option was upgrading to OpenAI's priority tier at 2x cost per token. The second was staying on standard tier but sending every request twice and taking whichever response arrived first.

Duplicate requests beat priority processing

The results favored the duplicate approach across all metrics. For time to first token — when the agent starts speaking — the median improved from 0.61 seconds on priority tier to 0.58 seconds with duplicates. The 99th percentile dropped dramatically from 4.2 seconds to 1.2 seconds.

Time to complete response showed even starker improvements. While median performance matched at 1.35 seconds, the worst-case scenario improved from 9.8 seconds on priority tier to 3.5 seconds with duplicate requests.

The technique works because slow responses are typically rare and independent events. Sending two requests makes it statistically unlikely that both copies will be slow simultaneously.

"This significantly reduced the 10-second silences our callers were experiencing," HOAi's engineering team wrote in their analysis.

The approach delivers better latency at the same cost as priority tiers. While it doubles the number of API calls, standard-tier pricing means total costs match priority-tier single requests.

HOAi recommends that developers building real-time LLM products benchmark duplicate requests against premium service tiers before upgrading. The simple technique may deliver superior performance without the premium pricing.