mlx-serve

Summary

Native LLM inference server for Apple Silicon. OpenAI + Anthropic API compatible. No Python. Includes MLX Core macOS app with chat, agent mode, and tool calling.

Command line

Values are placeholders taken from each option’s declared default. Nothing is executed here — the output shown is a recording of a run that already happened.

Release history

1. v26.8.11 Aug 29, 2026 · issue 012

   mlx-serve v26.8.11 adds Qwen 3.8 Flash Next, multimodal video input, sparse attention kernels, and batch-concurrent inference.

   └──▷ GET THIS VERSION
   copy

   $ git clone --branch v26.8.11 https://github.com/ddalcu/mlx-serve.git
   # already have the repo? check out this version:
   $ git checkout v26.8.11

   └──▷ TRY IT
   Enable speculative decoding on Flash Next to accelerate code generation by ~41% on an M4 Max.
   copy

   $ mlx-serve --model ddalcu/Qwen3.8-Flash-Next-MLX-Serve-4bit --mtp

   Enable the MoE/speculative decoding toggle for Flash Next through the UI when running without CLI access.
   📍In the mlx-serve Settings panel, locate the MoE toggle and enable it to activate speculative decoding for Flash Next.
   • ›Adds --mtp flag (and MoE toggle in Settings) to opt into speculative decoding on Flash Next: +41% on code generation, with speculative decoding now also applied to image questions.
   • ›Adds native support for Qwen 3.8 Flash Next (125B), delivering ~60 tok/s on M4 Max, ~78 tok/s with speculative decoding, using ~70 GB RAM with the 4-bit pack (ddalcu/Qwen3.8-Flash-Next-MLX-Serve-4bit).
   • ›Adds image and video understanding with follow-up questions on the same image answered instantly by reusing the cached visual context.
   • ›Adds sparse attention past 2k tokens via custom kernels, keeping 8k-token prompt throughput competitive even under speculative decoding.
   • ›Adds concurrent request batching on Flash Next: 2 parallel streams yield 1.3x total throughput, 4 streams yield 1.8x, with no regression on single-stream latency.
   +2 moreshow less

   • ›Adds native support for gpt-oss 20B and 120B (OpenAI MoE, harmony format).
   • ›Updates to MLX 0.32.2, delivering up to +3% speed improvement on MoE models at long context.