From the publisher · ITANU
ITANU publishes this magazine and makes the products this article is about. It is the company’s own account, not news reporting.
KolWrite's Oracle Max scores the lowest Hebrew error rate on a public test of 22 transcription engines
On all 792 Hebrew clips of the FLEURS test, Oracle Max got 12.6% of the words wrong. The 21 engines on the public ivrit.ai leaderboard range from 17.4% to 46.0% on the same clips.
Numbers in brackets link to the sources listed below.
On 21 November 2026 we ran Oracle Max, the top cloud tier of KolWrite, our transcription app, on every Hebrew clip of FLEURS, a public speech test set: 792 recorded sentences, 13,751 words[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. It got 12.6% of the words wrong[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. The ivrit.ai Hebrew transcription leaderboard publishes results for 21 engines on the same clips, and they range from 17.4% to 46.0%[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]].
The published figures
A word error rate counts the words an engine got wrong, left out or added, as a share of the words that were spoken[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. These are the error rates on the Hebrew FLEURS clips, ours and six from the leaderboard's table[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1], 2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]].
- KolWrite Oracle Max: 12.6%[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]
- Soniox: 17.7%[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]]
- ElevenLabs Scribe v1: 18.1%[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]]
- OpenAI gpt-4o-transcribe: 21.0%[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]]
- Amazon Transcribe: 23.0%[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]]
- Deepgram Nova-3: 23.3%[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]]
- Google Speech: 38.5%[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]]
The lowest figure in the leaderboard's table, 17.4%, belongs to a Hebrew model published by ivrit.ai itself[2ivrit.aibenchmark.csv · ivrit-ai/hebrew-transcription-leaderboardOpen sourceSources [2]]. Oracle Max's 12.6% is below all 21[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. At that rate it gets seven of every eight words exactly right, and the count is strict: a name spelled another way is an error[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
Same clips, same scoring
FLEURS is a set of sentences read aloud and recorded in 102 languages, with about 12 hours of speech for each language[3arXivFLEURS: Few-shot Learning Evaluation of Universal Representations of SpeechOpen sourceSources [3]]. Its Hebrew test part has 792 clips[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. We scored Oracle Max with the leaderboard's own published method, one error rate over all the words of all the clips, after removing vowel points, punctuation and letter case[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. Before the run, our copy of the scoring code reproduced the figures in the leaderboard's table to within 0.17 percentage points[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
We sent the clips through the public KolWrite API, the same service a customer uses, with the language set to Hebrew[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. All 792 clips are counted[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. One returned no text and is scored as all errors[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
Ahead in 10,000 of 10,000 resamples
A gap between two error rates can depend on which sentences happen to be in a test[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. To check ours, we took the transcripts the leaderboard publishes for 16 of its 21 engines and compared them with Oracle Max's clip by clip[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. We drew 10,000 random resamples of the 792 clips and scored both sides on each one[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. Oracle Max had the lower error rate than every one of the 16 in all 10,000[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
Soniox is the closest of them[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. Oracle Max's lead over it is 5.1 percentage points, with a 95% interval of 4.3 to 5.9 points[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. Counted in words, that is 1,734 errors against 2,435, or 29% fewer[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. Against ElevenLabs Scribe v1 it made 31% fewer word errors, against OpenAI's gpt-4o-transcribe 40% fewer, and against Google Speech 67% fewer[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
Oracle Max transcribed 309 of the 792 sentences with no error at all[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. The highest count among the 16 is 218[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. The other five engines publish no transcripts for this set, and their published figures, from 17.4% to 24.5%, are all above the top of Oracle Max's 95% interval, 14.1%[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
What the test covers
The test is Hebrew, read sentences, one speaker in each clip, and a single run on 21 November 2026[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. The other engines' figures are the leaderboard's own runs, in the versions it names[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. We are running the same comparison in other languages and will publish each one with its files[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
Check it yourself
Every one of the 792 transcripts is in a public file, with its reference sentence and its score, at itanu.org/data/oracle-max-hebrew-fleurs-792-2026-11-21.csv[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. The note beside it gives the method, the table of all the comparisons and the random seed[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]]. The leaderboard's transcripts are public as well, so anyone can repeat the comparison[1ITANU Studios LtdKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Open sourceSources [1]].
Sources
- 1Material supplied by ITANU Studios Ltd
SuppliedKolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026Accessed
- 2benchmark.csv · ivrit-ai/hebrew-transcription-leaderboard
Independentivrit.aiAccessed
- 3FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech
PrimaryarXivAccessed
ITANU’s newsroom system produced this article with AI and checked it against the sources listed above. How we work

