How we got on-device dictation under a second on an M2
Push-to-talk only feels magical if text appears the moment you let go. Here’s what we measured while building Kitflo on a MacBook Air M2 (8-core GPU, 16 GB).
Speech to text
| Engine | Time after key release | Notes |
|---|---|---|
| Whisper large-v3-turbo (whisper.cpp, Metal, q5) | ~2.4 s for a 9 s clip | The encoder alone took ~2.3 s |
| Apple SpeechAnalyzer (streaming) | ~0.15–0.25 s | Built into macOS 26, no download, ignores custom vocabulary |
| NVIDIA Parakeet TDT 0.6B v3 (Neural Engine) | ~0.3 s for 5 s, ~0.65 s for 20 s | Best accuracy on names and tech words (18/20 vs 14/20) |
The lesson: Whisper is excellent, but its 30-second encoder window is the wrong shape for short bursts of speech on an M2 GPU. Streaming recognition and the Neural Engine win for push-to-talk. Kitflo uses Parakeet by default, with Apple’s engine as a zero-download fallback.
Cleanup: smaller model, better prompt
Raw transcripts need tidying: filler words out, self-corrections applied, punctuation fixed. We ran local models through llama.cpp on Metal.
- Qwen2.5-3B, rules only: ~1–2 s, and it sometimes answered questions in the transcript instead of cleaning them.
- Qwen2.5-1.5B with eight worked examples: ~0.2–0.5 s, and it handled every test case, including “5, no wait, 6” corrections.
Two small tricks mattered most. Worked examples teach a small model the task far better than a list of rules. And keeping the start of the prompt identical between calls let the server reuse its cache, which roughly halved the time.
The result
From key release to pasted text: about 0.6–1.0 seconds, fully offline. Kitflo is free and open source, so every number here can be checked against the code on GitHub.