vMLX - JANGTQ Uber Compressed MLX Models - L2 Disk Cache (survives restart) + L1 Paged (super fast ttft) + Hybrid SSM Scheduler + Cont Batching + etc!
vMLX v1.3.6 adds disk streaming for RAM-oversize models, Mistral Small 4 (119B) support, and universal MoE JANG gate dequant.
$ git clone --branch v1.3.6 https://github.com/jjang-ai/vmlx.git # already have the repo? check out this version: $ git checkout v1.3.6
model_type: mistral4
- ›Adds
--stream-from-diskflag to load models larger than available RAM by leveraging macOS SSD paging (~7.4 GB/s); automatically disables all caching, and exposes a virtual memory budget slider in the session settings UI. - ›Adds
model_type: mistral4config value to support Mistral Small 4 (119B) with MLA attention, 128-expert MoE, and YaRN interleaved RoPE, routed through the mlx-lm text path and mistral3 VLM wrapper. - ›Extends JANG gate dequantization to any model with
n_routed_experts > 0(Mistral 4, DeepSeek V3, etc.), previously limited to Nemotron only.