# KolWrite's Oracle Max scores the lowest Hebrew error rate on a public test of 22 transcription engines

**From the publisher · ITANU**

On all 792 Hebrew clips of the FLEURS test, Oracle Max got 12.6% of the words wrong. The 21 engines on the public ivrit.ai leaderboard range from 17.4% to 46.0% on the same clips.

Software · By ITANU Magazine newsroom · Published 2026-11-21T08:59:00.000Z · 3 min read

## Key points

- We ran Oracle Max on all 792 Hebrew clips of FLEURS on 21 November 2026 and measured a word error rate of 12.6%.
- The ivrit.ai Hebrew leaderboard lists 21 engines on the same clips, and the lowest published figure is 17.4%.
- Compared clip by clip, Oracle Max made 29% fewer word errors than Soniox and 31% fewer than ElevenLabs Scribe v1.

On 21 November 2026 we ran Oracle Max, the top cloud tier of KolWrite, our transcription app, on every Hebrew clip of FLEURS, a public speech test set: 792 recorded sentences, 13,751 words [1]. It got 12.6% of the words wrong [1]. The ivrit.ai Hebrew transcription leaderboard publishes results for 21 engines on the same clips, and they range from 17.4% to 46.0% [2].

## The published figures

A word error rate counts the words an engine got wrong, left out or added, as a share of the words that were spoken [1]. These are the error rates on the Hebrew FLEURS clips, ours and six from the leaderboard's table [1][2].

- KolWrite Oracle Max: 12.6% [1]
- Soniox: 17.7% [2]
- ElevenLabs Scribe v1: 18.1% [2]
- OpenAI gpt-4o-transcribe: 21.0% [2]
- Amazon Transcribe: 23.0% [2]
- Deepgram Nova-3: 23.3% [2]
- Google Speech: 38.5% [2]

The lowest figure in the leaderboard's table, 17.4%, belongs to a Hebrew model published by ivrit.ai itself [2]. Oracle Max's 12.6% is below all 21 [1]. At that rate it gets seven of every eight words exactly right, and the count is strict: a name spelled another way is an error [1].

## Same clips, same scoring

FLEURS is a set of sentences read aloud and recorded in 102 languages, with about 12 hours of speech for each language [3]. Its Hebrew test part has 792 clips [1]. We scored Oracle Max with the leaderboard's own published method, one error rate over all the words of all the clips, after removing vowel points, punctuation and letter case [1]. Before the run, our copy of the scoring code reproduced the figures in the leaderboard's table to within 0.17 percentage points [1].

We sent the clips through the public KolWrite API, the same service a customer uses, with the language set to Hebrew [1]. All 792 clips are counted [1]. One returned no text and is scored as all errors [1].

## Ahead in 10,000 of 10,000 resamples

A gap between two error rates can depend on which sentences happen to be in a test [1]. To check ours, we took the transcripts the leaderboard publishes for 16 of its 21 engines and compared them with Oracle Max's clip by clip [1]. We drew 10,000 random resamples of the 792 clips and scored both sides on each one [1]. Oracle Max had the lower error rate than every one of the 16 in all 10,000 [1].

Soniox is the closest of them [1]. Oracle Max's lead over it is 5.1 percentage points, with a 95% interval of 4.3 to 5.9 points [1]. Counted in words, that is 1,734 errors against 2,435, or 29% fewer [1]. Against ElevenLabs Scribe v1 it made 31% fewer word errors, against OpenAI's gpt-4o-transcribe 40% fewer, and against Google Speech 67% fewer [1].

Oracle Max transcribed 309 of the 792 sentences with no error at all [1]. The highest count among the 16 is 218 [1]. The other five engines publish no transcripts for this set, and their published figures, from 17.4% to 24.5%, are all above the top of Oracle Max's 95% interval, 14.1% [1].

## What the test covers

The test is Hebrew, read sentences, one speaker in each clip, and a single run on 21 November 2026 [1]. The other engines' figures are the leaderboard's own runs, in the versions it names [1]. We are running the same comparison in other languages and will publish each one with its files [1].

## Check it yourself

Every one of the 792 transcripts is in a public file, with its reference sentence and its score, at itanu.org/data/oracle-max-hebrew-fleurs-792-2026-11-21.csv [1]. The note beside it gives the method, the table of all the comparisons and the random seed [1]. The leaderboard's transcripts are public as well, so anyone can repeat the comparison [1].

## Sources

1. [KolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026](https://itanu.org/magazine/standards/), ITANU Studios Ltd (Supplied)
2. [benchmark.csv · ivrit-ai/hebrew-transcription-leaderboard](https://huggingface.co/spaces/ivrit-ai/hebrew-transcription-leaderboard/blob/main/benchmark.csv), ivrit.ai (Independent)
3. [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://arxiv.org/abs/2205.12446), arXiv (Primary)

ITANU’s newsroom system produced this article with AI and checked it against the sources listed above.

Canonical: https://itanu.org/magazine/kolwrite-s-oracle-max-scores-the-lowest-hebrew-error-rate-on-a-zzmjkzdz/
