A new open-source project called LoRA Speedrun has created the first public leaderboard for fine-tuning speed records, challenging developers to optimize Low-Rank Adaptation techniques on standardized hardware.

The competition uses frozen tasks and identical Modal L40S sandboxes to ensure fair comparisons. Participants must fine-tune Qwen 2.5-1.5B to achieve at least 57% accuracy on GSM8K math problems, with timing measured from start to finish.

The current record stands at 22 seconds, set by contributor @slippylolo using an aggressive optimization strategy. The technique loads the model directly from safetensors with a warmed page cache, then performs just 12 optimizer updates: eight full-network passes followed by four updates through only the top decoder layer.

Two-Track System Tests Generalization

The project runs two parallel tracks to prevent overfitting to specific setups. Track 1 uses Qwen2.5-1.5B on GSM8K math problems, while Track 2 employs SmolLM2-1.7B on SQuAD question-answering tasks.

Both tracks enforce a 30-million parameter limit for adapter-only training. The dual-track structure ensures techniques must transfer across different model families and task types to prove their general effectiveness.

Every submitted record undergoes independent verification with three fresh runs on identical hardware. Modal's free compute credits cover the verification process, making participation accessible to any developer.

The leaderboard has already documented significant speedups. The baseline LoRA implementation took nearly 12 minutes, while recent optimizations include sequence packing, completion-only loss masking, and custom GPU-resident loops.

Creator Saivineeth147 designed the competition to address unfalsifiable speedup claims in the fine-tuning community. "Every technique reports numbers on different models, data, and hardware," the project documentation states. "In practice the claims are unfalsifiable."

The project builds on nanoGPT's pretraining speedrun format, which previously helped validate optimizer improvements like Muon. Each record submission must include detailed technical explanations, creating a public lab notebook of verified optimization techniques.

The competition remains open with Modal providing free sandbox access for official timing runs.