SolanaSOL, coin
DevelopingResearch

AssemblyAI benchmark shows small LLMs match large models on short transcripts but fall behind on longer meetings

AssemblyAI's Voice Standard Bench finds small language models compete closely with larger ones on brief voice transcripts but lose advantage as length grows past 15 minutes.

Why it matters

The benchmark suggests deployment decisions for voice AI should weigh transcript length against cost, since smaller models are competitive on brief interactions but larger models become cost-effective for extended meetings. Significance is limited to organizations evaluating speech-to-text pipelines.

Key facts

AssemblyAI ran a Voice Standard Bench comparing models on speech-to-text output at four transcript lengths.

Reported by Blockchain.News

GPT-6 Luna scored 96, 94, 89, and 88 at 30 seconds, 2 minutes, 15 minutes, and 60 minutes respectively.

Reported by Blockchain.News

gemma-4-31b scored 95, 92, 87, and 85 across the same four lengths.

Reported by Blockchain.News

Qwen3.5 4B Fast scored 93, 89, 64, and 63 across the four lengths.

Reported by Blockchain.News

An unnamed model scored 74, 64, 47, and 31 across the four lengths.

Reported by Blockchain.News

AssemblyAI considers scores within 2 to 3 points effectively tied.

Reported by Blockchain.News

The top-scoring model costs 15 to 77 times more per audio hour than a model within two quality points on short tasks.

Reported by Blockchain.News

On hour-long transcripts the cost premium narrows to about four times.

Reported by Blockchain.News

What happened

AssemblyAI published results from its Voice Standard Bench run dated October 13, 2026, comparing language models on real speech-to-text output at durations of roughly 30 seconds, two minutes, 15 minutes, and 60 minutes. The company said small models such as gemma-4-31b and Qwen3.5 4B Fast stay within two to three points of the top scorer GPT-6 Luna on short inputs, but the gap widens sharply by the 15-minute mark. AssemblyAI also reported that the cost premium for the highest-scoring model narrows from 15 to 77 times more per audio hour on short tasks to about four times on hour-long transcripts, where larger models begin to justify their expense. The post frames transcript length, not model class alone, as the key variable. Only AssemblyAI's own benchmark data is cited; no independent replication is mentioned.

How the story developed

  1. Update

    AssemblyAI publishes Voice Standard Bench results

    Benchmark run dated October 13, 2026; post published October 20, 2026

    Blockchain.News

  2. First report

    First report by Blockchain.News

    Small LLMs stay competitive on short transcripts, then fade at meeting length

    Blockchain.News

Affected entities

Original sources

PublisherReportRolePublished
Blockchain.News
Crypto media
Small LLMs stay competitive on short transcripts, then fade at meeting lengthOriginal