← Benchmarks 🕐 3 min read
Benchmarks

Local Podcast Transcription MLX Models (May 2026)

Evidence tier · March 2026

task-relevant, and includes finance-like `earnings21` / `earnings22` subsets,

MEDIUM-HIGH for model ordering: the HF Open ASR long-form CSV is dated, task-relevant, and includes finance-like earnings21 / earnings22 subsets, but it is still a public leaderboard rather than our exact podcast corpus.

MEDIUM for runtime fit: Parakeet MLX is active, Apache-2.0, and exposes a usable CLI/API for Apple Silicon, but it is not yet wired into scripts/podcast_mine.py and has not been validated on our speaker-heavy episodes.

Decision

Keep whisperx-mlx as the default scripted backend for now because our wiki needs quote attribution, timestamps, and optional diarization more than it needs the absolute fastest transcript.

Add Parakeet MLX to the next ASR evaluation pass. It is the strongest local candidate for high-throughput long-form transcript generation because nvidia/parakeet-tdt-0.6b-v3 beats openai/whisper-large-v3-turbo on the 2026-03-26 HF Open ASR long-form average WER: 10.72 vs. 11.01, with reported RTFx 1002.91 vs. 148.26.

Add Qwen3-ASR to the same evaluation pass. Its official Apache-2.0 model family now includes Qwen/Qwen3-ASR-0.6B, Qwen/Qwen3-ASR-1.7B, and Qwen/Qwen3-ForcedAligner-0.6B; the model card and technical report emphasize multilingual/accented/noisy speech, streaming/offline inference, vLLM serving, and forced alignment. A community mlx-qwen3-asr implementation gives an Apple Silicon CLI with SRT/VTT/JSON outputs and optional diarization. That makes it a serious candidate for finance podcast ingestion, but not the default until it beats WhisperX-MLX on quote-level timestamp review and speaker attribution effort for our archived episodes.

Do not use commercial APIs as default source ingestion for proprietary-finance work. They lead the public WER table, but the privacy and cost posture is wrong for this repo’s local-source-material goal unless the episode is public, non-sensitive, and explicitly marked as an external transcription exception.

Candidate Ranking For This Repo

Rank Candidate Why it matters Current limitation
1 whisperx-mlx + large-v3 Alignment and optional diarization support make quote review practical. Slower and not best WER/throughput among 2026 local ASR candidates.
2 Parakeet MLX + mlx-community/parakeet-tdt-0.6b-v3 Best local/open long-form WER/RTFx signal found in this pass; CLI emits SRT/VTT/JSON. Needs pipeline integration and corpus validation; no built-in diarization path documented in the README.
3 mlx_whisper + mlx-community/whisper-large-v3-turbo Stable fallback already present; faster than large-v3. No diarization; weaker than Parakeet v3 in the long-form CSV.
4 Qwen3-ASR + mlx-qwen3-asr Current Apache-2.0 Qwen-family ASR with 0.6B/1.7B sizes, forced aligner, vLLM route, MLX CLI, SRT/VTT/JSON output, and optional diarization. Needs local install/smoke and corpus validation; official speed/quality claims are not our podcast corpus; diarization uses pyannote terms/token.
5 Canary-Qwen / older Qwen ASR controls Relevant as historical Qwen-audio lineage and a CSV comparison point. Lower priority now that Qwen3-ASR has official ASR artifacts and MLX routes.
6 Commercial APIs ElevenLabs, AssemblyAI, Speechmatics, Rev, and Cohere beat local models on WER in the CSV. External-data exposure and recurring cost conflict with the local ingestion posture.

Implementation Implications

scripts/podcast_mine.py now exposes ASR knobs:

PODCAST_ASR_BACKEND=whisperx-mlx
WHISPERX_MLX_MODEL=large-v3
MLX_WHISPER_MODEL=mlx-community/whisper-large-v3-turbo
MLX_QWEN3_ASR_MODEL=Qwen/Qwen3-ASR-0.6B

The script now accepts PODCAST_ASR_BACKEND=mlx-qwen3-asr when the mlx-qwen3-asr CLI is installed in scripts/.venv-tts. It is intentionally not installed or promoted by default in this pass.

The next ASR code change should add PODCAST_ASR_BACKEND=parakeet-mlx only after a local smoke test confirms:

  1. parakeet-mlx can be installed in scripts/.venv-tts.
  2. It can produce SRT or JSON for one archived finance/quant episode.
  3. Its timestamps survive _flatten_srt() or a new JSON flattener.
  4. Quote extraction quality on the same episode matches or beats WhisperX.
  5. Human review can still identify speaker attribution with acceptable effort.

The next Qwen3-ASR smoke should confirm:

  1. mlx-qwen3-asr can be installed in scripts/.venv-tts.
  2. PODCAST_ASR_BACKEND=mlx-qwen3-asr produces SRT for one archived finance or quant episode.
  3. _flatten_srt() preserves enough timestamp structure for quote review.
  4. Qwen/Qwen3-ASR-0.6B and Qwen/Qwen3-ASR-1.7B are compared separately.
  5. Optional diarization is evaluated only after pyannote terms and token handling are explicit.

Extractor Model Policy Tie-In

The transcription model and transcript-extraction model are different choices. Independent ASR benchmarks should drive ASR. Independent SLM benchmarks and our corpus eval should drive extraction.

For extraction, the default is now mlx-community/Qwen3.5-9B-OptiQ-4bit because repo SLM research identifies Qwen3.5-9B as the strongest sub-10B instruction-following / structured-output starting point, and the HF MLX API snapshot shows a text-generation OptiQ port. Qwen3.6-27B should be evaluated in the repo-building/code-reasoning benchmark lane first; its MLX community port is currently mlx-vlm-oriented, so do not assume compatibility with mlx_lm.load().

Sources