# KolWrite Oracle Max on the full Hebrew FLEURS test: 792 clips, measured 21 November 2026

Published by ITANU Studios Ltd, the maker of KolWrite. This is our own measurement of our own product, set beside the
published results of the ivrit.ai Hebrew transcription leaderboard on the same clips. Every transcript and score is in
the file next to this note: https://itanu.org/data/oracle-max-hebrew-fleurs-792-2026-11-21.csv

## Result

Word error rate 12.6%: 1,734 word errors (967 substitutions, 510 deletions, 257 insertions) over 13,751 reference
words in 792 clips, 2.05 hours of speech. Resampling the 792 clips 10,000 times gives a 95% interval of 11.2% to 14.1%.
309 of the 792 sentences were transcribed with no error.

The leaderboard lists 21 engines. Their published error rates on these 792 clips run from 17.4% to 46.0%. Oracle
Max's figure, and the top of its interval, are below all 21.

## Clip by clip against each engine

For 16 of the 21 engines the leaderboard publishes the transcript of every clip, and one further file for a timing
run of an ivrit.ai model. We scored those 17 files and ours with the same code, then resampled the 792 clips 10,000
times, using the same resampled clips for both sides of each comparison. Oracle Max had the lower error rate than
every one of the 17 in all 10,000 resamples.

| Engine, as the leaderboard names it | Word error rate | 95% interval | Sentences with no error | Oracle Max's lead, in points (95% interval) | Fewer word errors |
| --- | --- | --- | --- | --- | --- |
| **KolWrite Oracle Max** | **12.6%** | 11.2% to 14.1% | 309 | | |
| Soniox (stt) | 17.7% | 16.2% to 19.3% | 192 | 5.1 (4.3 to 5.9) | 29% |
| ElevenLabs Scribe v1 | 18.1% | 16.6% to 19.7% | 218 | 5.5 (4.7 to 6.4) | 31% |
| ivrit.ai whisper-large-v3-turbo (the leaderboard's timing run) | 18.3% | 16.8% to 19.8% | 166 | 5.7 (5.0 to 6.4) | 31% |
| ivrit.ai whisper-large-v3-turbo-ct2-20250403 | 20.9% | 19.4% to 22.5% | 117 | 8.3 (7.5 to 9.2) | 40% |
| OpenAI gpt-4o-transcribe | 21.1% | 19.5% to 22.8% | 166 | 8.5 (7.6 to 9.4) | 40% |
| Amazon Transcribe (batch) | 23.0% | 21.4% to 24.6% | 101 | 10.4 (9.5 to 11.3) | 45% |
| Deepgram Nova-3 | 23.4% | 21.8% to 24.9% | 105 | 10.7 (9.8 to 11.7) | 46% |
| zipformer2-66m-streaming-he | 23.4% | 21.9% to 25.1% | 108 | 10.8 (9.9 to 11.7) | 46% |
| ivrit.ai faster-whisper-v2-d4 | 24.2% | 22.7% to 25.8% | 93 | 11.6 (10.7 to 12.6) | 48% |
| ivrit.ai faster-whisper-v2-d3-e3 | 25.6% | 24.0% to 27.2% | 76 | 13.0 (12.0 to 14.0) | 51% |
| Whisper large-v3 (faster-whisper) | 26.2% | 24.6% to 27.8% | 65 | 13.6 (12.6 to 14.6) | 52% |
| Whisper large-v2 (faster-whisper) | 26.7% | 25.1% to 28.3% | 73 | 14.0 (13.1 to 15.1) | 53% |
| Amazon Transcribe (streaming) | 28.7% | 27.0% to 30.4% | 62 | 16.1 (15.0 to 17.1) | 56% |
| Whisper large-v3-turbo (faster-whisper) | 28.9% | 27.2% to 30.6% | 65 | 16.3 (15.1 to 17.5) | 56% |
| OpenAI gpt-4o-mini-transcribe | 30.1% | 28.3% to 31.9% | 82 | 17.5 (16.2 to 18.8) | 58% |
| Google Speech | 38.5% | 36.8% to 40.4% | 32 | 25.9 (24.6 to 27.3) | 67% |
| nemotron-3.5-asr-streaming-0.6b | 46.0% | 43.7% to 48.4% | 32 | 33.4 (31.2 to 35.6) | 73% |

The other five engines publish no per-clip file for this set, so they can be compared only by their published
figures: ivrit.ai whisper-large-v3-ct2-20250513 17.4%, whisper-large-v3-turbo-ct2-20250513 18.1%,
whisper-large-v3-ct2-20250403 20.0%, whisper-large-v3-ct2-20250209 20.8% and whisper-large-v3-turbo-ct2-20250209
24.5%. All five are above the top of Oracle Max's interval, 14.1%.

Our scoring of the 16 engines' files differs from the leaderboard's published figures by at most 0.14 percentage
points on this set; the published table is at https://huggingface.co/spaces/ivrit-ai/hebrew-transcription-leaderboard

## What was measured

- Product: KolWrite cloud transcription, tier Oracle Max, called through the public KolWrite API on the production
  service on 21 November 2026, with the language set to Hebrew and speaker separation off. These are the calls a
  customer makes.
- Data: the Hebrew (he_il) test split of FLEURS (Google; licence CC BY 4.0), all 792 clips. Nothing was left out.
  One clip returned no text; it is scored as all errors.
- Scoring: the method of the ivrit.ai Hebrew transcription leaderboard (code `ivrit-ai/asr-training`): one error rate
  over all the words of all clips, after removing vowel points, punctuation, quote marks and letter case. Before the
  run our scoring code reproduced the figures in the leaderboard's table to within 0.17 percentage points.
- Statistics: percentile bootstrap over clips, 10,000 resamples, random seed 20261121 (numpy `default_rng`).
- The other engines: the leaderboard's own runs, in the versions it names, as published on 14 October 2026. We did
  not run them ourselves.

## Scope

One tier, one language, read sentences with one speaker per clip, one run. A transcription service does not return
identical text on every run, so a repeat will land near 12.6%, inside or close to the interval, and not exactly on
it. FLEURS is a public test set, for Oracle Max as for every engine in the table. The leaderboard also reports four
other Hebrew sets; their licences or access terms do not let us publish a run on them. We do not publish which models
or processing steps make up Oracle Max.

## The file

Columns: `id` (FLEURS row number), `reference_text` (the FLEURS transcript), `predicted_text` (what Oracle Max
returned), `norm_reference_text` and `norm_predicted_text` (both after the leaderboard's text cleaning), `wer` (that
clip's error rate), `substitutions`, `deletions`, `insertions`, `hits`, `audio_duration` (seconds).

The reference sentences are from FLEURS (Conneau et al., 2022), © Google, used under CC BY 4.0
(https://creativecommons.org/licenses/by/4.0/). Our transcripts in the file may be reused under the same licence.

Questions or a correction: https://itanu.org/magazine/contact/
