Swiss German voicemails to structured orders — a three-model ensemble that killed Whisper's looping hallucination problem
A Swiss B2B food wholesaler had 796 voicemails from restaurant clients that a single operator was retyping by hand every day. The first speech-to-text run — on a model specifically fine-tuned for Swiss German — produced high-confidence garbage: "drei Pommes drei Pommes drei Pommes…" for 50 lines. We diagnosed why, rebuilt the pipeline as a three-model ensemble with a Claude 4.7 Opus arbiter, and recovered 99 of the 100 previously-unusable files.
Client
Swiss wholesale distributor of foodservice goods (frozen products, sauces, packaging) supplying restaurants, cafés, kebab shops, and bars across German-speaking Switzerland. Clients leave orders on an answering machine — 20–40 voicemails per day, longer on Mondays.
Engagement
Two-phase build: Phase 1 — batch pipeline over the 796-voicemail archive to prove viability. Phase 2 — always-on IMAP watcher that transcribes new Combox voicemails and drafts a reply email with a structured order to the operator.
Swiss German is not German — and the phone line makes it harder.
Every day, restaurant owners leave orders on the wholesaler's answering machine: "Hallo, hier ist Restaurant Rathausbrauerei Luzern, bestellen Sie zehn Karton Pommes, sieben Karton Spicy Veggies, Vanillesauce…". An operator listens to every message and retypes the order into the ERP. Slow, expensive, error-prone.
The brief was simple on paper: transcribe the voicemail so the operator confirms a ready-made order instead of typing it from scratch.
Why it's hard in practice:
- Schwyzerdütsch is a family of Alemannic dialects, not a variant of German. A speaker from Berlin often can't understand a speaker from Lucerne. There is no unified written form — Zürich sounds different from Bern, Bern different from Luzern.
- The audio profile is brutal: telephone-grade compressed audio, kitchen noise in the background, callers speaking fast, swallowing endings, code-switching between dialect and Hochdeutsch.
- The content is domain-specific: restaurant names, customer numbers, product SKUs, quantities. One wrong digit and the wrong quantity ships to the wrong client.
Specialized Swiss-German model. High confidence. Complete garbage.
The logical first step was the model built for the job. We picked whisper-large-v3-turbo-swiss-german — a Swiss university fine-tune trained on 300 hours of Swiss audio. All 796 files processed cleanly. Zero errors. Mean confidence 75%. We sent the results to the client.
The client sorted the Excel by confidence, descending. In files with confidence 0.95 she found this:
"7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7, 7 …" (endless digit stream)
"Grossrat, Grossrat, Grossrat, Grossrat …" (60 times in a row)
"ä ä ä ä ä ä ä ä ä ä ä ä …" (just sounds)
Confidence was inverted. The higher the model's certainty, the worse the output. Files in the 0.70–0.80 confidence band contained perfect order transcriptions — customer number, restaurant, quantities, product names.
The client's verdict, direct quote: "This is garbage."
Root cause: looping hallucination on unclear audio
Whisper is autoregressive. It generates text token by token, each conditioned on the previous. When the audio is unclear — as telephone voicemail routinely is — the model latches onto the last recognized token and starts repeating it. This is a known failure mode: looping hallucination. The paradox: repeating a single token is easy for the model — every next repetition has high probability — so the reported confidence skyrockets on exactly the outputs that are broken.
Whisper has a full suite of guards against this: repetition penalty, compression-ratio filter, cascading temperature fallback for hard segments, context isolation. All of them have to be explicitly turned on. In the first run they were off — we had used the parameters from the spec "as is", without tuning for the client's actual audio profile.
Same model, correct parameters, three models compared head-to-head.
We turned on the full anti-hallucination stack and re-ran an A/B on the five worst files, feeding each through three model families:
- Model A — the same Swiss German fine-tune, with correct anti-loop parameters.
- Model B — vanilla OpenAI Whisper Large V3 (full).
- Model C — OpenAI Whisper Large V3 Turbo (fast).
All three produced readable transcripts. Zero loops. The file that had been an endless "ä ä ä ä ä…" became:
The winner was Whisper Large V3 Turbo — the vanilla OpenAI model, not the Swiss-specialised fine-tune. It gave the most complete transcripts (including phone numbers that other models dropped), ran 3× faster than the Swiss fine-tune, and 7× faster than full Large V3.
Counter-intuitive result. Worth stating plainly: a specialised model is not always the best choice. The Swiss fine-tune saw 300 hours of dialect. Vanilla Turbo saw 680 000 hours across languages. Breadth beat specialisation on this audio profile.
Three-model ensemble with a Claude 4.7 Opus arbiter.
One model is one failure mode. The M2 pipeline runs three independent transcribers and lets a reasoning model pick or merge the best output based on deterministic trust signals.
Why an ensemble? In M1, we re-ran the Swiss fine-tune and Whisper Turbo on 100 worst-case files. Turbo won 74, Swiss won 16, the rest tied. The wins are uncorrelated — Swiss saved Turbo on 10 files where Turbo looped. Adding Gemini 2.5 Pro (a multimodal LLM, not autoregressive over audio tokens) brings a third architecture that does not share Whisper's looping failure mode, and can use customer context inside its prompt.
The arbiter. Claude 4.7 Opus sees all three transcripts alongside deterministic trust signals (loop-detector output, compression ratio, agreement metrics) and picks the best version — or merges them — then extracts a structured order (customer number, restaurant name, line items with quantity + SKU match against the Vermo master file).
Two-tier arbitration for cost. When all three transcribers agree, a lightweight prompt on Claude Haiku confirms and extracts. When they disagree, the full Opus prompt runs. This saves roughly 60% versus calling Opus on every file.
Architecture (data flow)
From "this is garbage" to production-ready in three weeks.
Every number below is measured on the client's real voicemail archive, not on synthetic test audio.
Files recovered from garbage
The 100 worst files (confidence 0.85–0.96, all previously looping garbage) re-processed on the fixed pipeline. 99 came back readable. One truly unrecoverable due to audio quality.
Looping hallucinations
After enabling repetition penalty, compression-ratio filter, and cascading temperature, the loop failure mode disappeared across the entire 796-file archive.
Mean calibrated confidence
The confidence number now correlates with actual transcript quality. Files below 0.60 are correctly flagged for human review; files above 0.85 are correctly trusted.
All-in cost, 796 files
Whisper ~$2.40 + Gemini ~$1.00 + two-tier Claude ~$10. Two-tier arbitration saves ~60% versus calling Opus on every file.
Speedup
Whisper Turbo runs 3× faster than the Swiss fine-tune and 7× faster than full Large V3 — while producing the most complete transcripts on this audio profile.
Quality distribution (of 99)
37 fully readable orders with names, customer numbers, and quantities. 47 good quality with minor SKU spelling drift. 15 partial due to bad recording — but no garbage.
Before, the operator was retyping every voicemail from scratch. Now she confirms a ready order and moves on. The garbage rows we saw in the first run are gone.
Four things worth stating out loud.
1. A specialised model is not always the best model. The Swiss fine-tune saw 300 hours of dialect. Vanilla Whisper Turbo saw 680 000 hours across languages — and won the head-to-head on this specific audio profile. Breadth compensated for specialisation. Test both, don't assume.
2. Parameters matter more than the model. The same Swiss fine-tune with wrong parameters produced 50-line garbage. With the correct anti-hallucination stack turned on, it produced readable text. The gap between "project failed" and "ready for production" was six config values, not a different model.
3. Confidence lies. Autoregressive LLM confidence measures how sure the model is of each next token — not whether the sequence describes reality. It can be 95% sure of complete nonsense. You need external checks: a deterministic loop detector, compression-ratio filter, cross-model agreement. We now treat confidence as one signal among several, not the truth.
4. Dialect is effectively a separate language. Schwyzerdütsch has no unified written form. A pipeline that works for Hochdeutsch may fail entirely on the same audio spoken in dialect — and vice-versa. Any Swiss German deployment should be tested on the client's actual audio, not on public benchmarks.
From "this is garbage" to a production pipeline in three weeks.
- Week 1
M1 batch on Swiss German fine-tune
All 796 files processed with the client-specified model. Reported success — until the client sorted by confidence and found looping garbage in the top rows.
- Week 1
Diagnosis of looping hallucination
Identified the failure mode. Enabled repetition penalty, compression-ratio filter, cascading temperature fallback, and context isolation. Ran an A/B on the five worst files across three model families.
- Week 2
M2 ensemble design + implementation
Built the three-transcriber pipeline (Whisper Turbo + Gemini 2.5 Pro + FHNW Swiss German) with a Claude 4.7 Opus arbiter. Trust-signal extraction (loop detector, compression ratio, 3-way agreement) for arbiter grounding.
- Week 2
Validation on 100 previously-garbage files
99/100 recovered. Mean calibrated confidence 77.4% (and this time it correlates with reality). Human-review queue built for the 15% partial-quality band.
- Week 3
Customer DB enrichment + article matcher
Caller-number → restaurant name via customer master. SKU matcher against the Vermo article catalogue with fuzzy match + confidence scoring per line item.
- Week 3
Widget + API + IMAP watcher
Self-hostable HTTP API with a drag-and-drop widget for manual review. Docker Compose deployment behind Caddy with auto-HTTPS. IMAP watcher for always-on Combox voicemail intake.
Other production voice and audio work.
Dialect, phone-line audio, or domain-specific vocabulary — we can tell you in 20 minutes whether it will ship.
Send us five representative audio files and the outcome you need (structured order, appointment, ticket, whatever). We come back with a yes / no / not-yet based on WER on your actual audio — not on a public benchmark.