llama.cpp — The AI Toolchain

llama.cpp v0.3.0

Tue, 25 Aug 2026 10:22:58 GMT

llama.cpp v0.3.0 adds dots3-note multimodal model, DeepSeek 4 tensor-split via `-sm tensor`, WebP support, and a new `LLAMA_SERVER_SLOTS_N_DIFF` debug env var. • Adds `-sm tensor` flag to enable tensor-split mode for DeepSeek 4 multi-GPU inference. • Adds `LLAMA_SERVER_SLOTS_N_DIFF` environment variable to widen the slot debug diff window in the llama-server. • Adds the dots3-note multimodal model with a new DSA-ISWA KV cache type, including vision and audio support. • Adds MTP (multi-token prediction) support for the GLM-4.5-Air model. • Adds WebP image decoding support in the multimodal subsystem via ffmpeg. • Adds DSpark architecture support for the bailingmoe3 model. • Adds tabbed navigation for chat conversations in the built-in web UI. • Upgrades ggml to v0.22.0, bringing meta-backend tensor split, per-op Metal kernels with parallel compilation, and non-in-place `ggml_clamp`. • Improves mamba2 throughput by flattening in/out projections to dispatch GEMM instead of GEMV. Breaking changes: • The `-no-cnv` CLI option has been removed.

llama.cpp v0.2.0

Fri, 21 Aug 2026 18:32:48 GMT

llama.cpp v0.2.0 introduces stable semantic versioning, new model support, backend kernels, and server/multimodal improvements. • Adds `--mmproj-device` argument to specify the device for multimodal projector inference. • Adds `dedup-cache-models` preset option to the server. • Adds `mtmd_bitmap_set_mergeable` API to the multimodal subsystem. • Makes `/metrics` endpoint accessible during server sleep state. • Makes models endpoints private when authentication is enabled on the server. • Adds SME2 F32 GEMV kernel support via KleidiAI on ARM. • Ports fused `ssm_scan` kernel (Mamba-2, d_state 128/256) to GPU via OpenCL. • Enables tensor split support for LFM2 and LFM2MOE models. • Adds support for GraniteSWAForCausalLM and GraniteMoeSWAForCausalLM model architectures. • Adds support for DSpark architecture for LFM2 models. • Adds tiled transpose for 0<->2 permuted CONT on the Vulkan backend. • Adds mulmat with overlapping src0/src1 (e.g. for MiniMax-01) on the WebGPU backend. • Adds RPC use_count population to enable operation fusion inside backends. • Optimises quantisation memory usage by evicting weights after processing each layer. • Adds CUDA switch points per hardware and quantisation type to tune the MVQ-to-MMQ decode crossover. • Introduces stable semantic versioning (`vX.Y.Z` tags) alongside existing nightly `b[NUM]` tags for easier downstream distribution. • Adds graceful fallback for unsupported regex patterns in JSON schema on the common layer. • Adds size guards to `GGUFReader` in the `gguf-py` library. • Adds `GGML_OPENCL_A7X_LMHEAD_CPU` environment variable to override automatic CPU offload of vocab-scale K-quant lm_head on Adreno A7X GPUs. • Adds Windows ARM64 CUDA support to the manual CI workflow. • Updates OpenVINO backend to version 2026.3.

llama.cpp b10549

Fri, 21 Aug 2026 09:23:07 GMT

llama.cpp b10549 adds `--split-mode tensor` support for LFM2 and LFM2MOE model families • Adds `--split-mode tensor` support for LFM2 and LFM2MOE model families, enabling multi-GPU tensor parallelism for these architectures. • Extends `ggml_rope_set_offset` support to the OpenCL, SYCL, WebGPU, and Hexagon backends.

llama.cpp b10541

Fri, 21 Aug 2026 03:26:59 GMT

llama.cpp b10541 adds `--mmproj-device` / `-mmdev` flag and `MTMD_BACKEND_DEVICE` env var for per-device multimodal projector offloading, plus DSpark speculative decoding support for LFM2 models. • Adds `--mmproj-device` (short flag: `-mmdev`) CLI argument and `MTMD_BACKEND_DEVICE` environment variable to select which compute device handles the multimodal projector (mmproj), enabling scenarios like offloading mmproj to an iGPU while the main model runs on a dGPU; accepts `none` to disable GPU offload entirely. • Adds DSpark speculative decoding support for LFM2 models, including partial rollback for LFM2 recurrent state, enabling draft-model-accelerated inference for LFM2.5 series models.

llama.cpp b10534

Fri, 21 Aug 2026 01:37:40 GMT

llama.cpp b10534 adds per-hardware CUDA MMVQ→MMQ switch points, yielding up to 41% faster K-quant decode on Blackwell/Ada GPUs. • Adds per-hardware and per-quantization-type batch crossover thresholds for the CUDA `mul_mat_vec_q` → MMQ (int8 tensor-core) dispatch path, with tuned switch points for Blackwell (RTX 5090, DGX Spark) and Ada (RTX 4090) GPU families, delivering +23–41% decode throughput at batch size 8 for K-quants (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K) on dense models with no low-batch regression.

llama.cpp b10532

Fri, 21 Aug 2026 00:42:58 GMT

Metal backend now dequantizes Q4_0/Q4_1/Q5_0/Q5_1/Q8_0 KV caches to F16 before flash attention, improving accuracy and performance on Apple Silicon. • Adds a preprocessing pass on the Metal backend for `GGML_OP_FLASH_ATTN_EXT` that dequantizes quantized KV caches (Q4_0, Q4_1, Q5_0, Q5_1, Q8_0) into contiguous F16 scratch buffers before running the existing F16 flash-attention kernels, instead of performing in-kernel dequantization. • Adds a new Metal kernel `kernel_flash_attn_ext_dequant_to_f16` dispatched separately for K and V into per-op scratch memory, with stride-aware dequantization supporting permuted KV layouts. • Skips the redundant V dequantization when V is a view of K (as in MLA-based models), reading dequantized V from the K F16 buffer with K's strides instead. • Adds new quantized-KV flash-attention test cases in `test-backend-ops`, including the MLA shape (576/512, V is a view of K), pad path (kv=113), and large context (kv=16384).

llama.cpp b10519

Thu, 20 Aug 2026 20:31:11 GMT

llama-server now enforces auth on `/v1/models` and exposes `/metrics` during sleep mode • The `/v1/models` endpoint is now private when server authentication is enabled, blocking unauthenticated access to model file paths (which may contain PII such as home directory names). • The `/metrics` endpoint is now accessible while the server is in sleep/idle state, allowing monitoring systems to scrape metrics without waking the server. Breaking changes: • When authentication is enabled, unauthenticated clients that previously could query `/v1/models` without a key will now receive an auth error — any tooling or scripts relying on unauthenticated access to `/v1/models` will break.

llama.cpp b10514

Thu, 20 Aug 2026 11:17:28 GMT

Adds support for GraniteSWAForCausalLM and GraniteMoeSWAForCausalLM models with interleaved Sliding Window Attention • Adds conversion and inference support for `GraniteSWAForCausalLM` and `GraniteMoeSWAForCausalLM` model architectures, enabling IBM Granite models with interleaved Sliding Window Attention (SWA) and Attention Sinks. • Adds per-layer RoPE/NoPE determination via `llama_hparams::has_rope`, allowing models with mixed attention layers (some with rotary embeddings, some without) to load and run correctly. • SYCL backend now reports zero devices instead of aborting when no SYCL-capable hardware is present, allowing tools like `llama-quantize` to run on non-SYCL hosts without crashing.

llama.cpp b10509

Thu, 20 Aug 2026 06:52:43 GMT

Adds `ggml_rope_set_offset` operation with CPU, Metal, CUDA, and Vulkan backend support for DSA and 2D RoPE use cases. • Adds `ggml_rope_set_offset` ggml operation enabling DSA-indexer RoPE ('nope | rope' style with `offset = n_dims_nope`) and 2D RoPE for multimodal models, with support across CPU, Metal, CUDA, and Vulkan backends.

llama.cpp b10505

Thu, 20 Aug 2026 02:36:34 GMT

llama.cpp b10505 adds `dedup-cache-models` preset option to hide duplicate cached models from the server model list. • Adds `dedup-cache-models` (values `0|1`) as a server router preset option in `preset.ini`: when set in global scope, hides cached models from `GET /models` when a preset resolves to the same model file, deduplicating the model list.

llama.cpp b10502

Wed, 19 Aug 2026 13:26:32 GMT

llama.cpp b10502 adds signed release artifact attestations and reduces quantization memory use via layer-weight eviction. • Adds GitHub Actions attestation for signed release artifacts, enabling cryptographic verification of official binaries via the attestations page. • Reduces peak memory usage during quantization by evicting processed layer weights from memory pages once each layer is done (benefits `llama-quantize` and similar workflows). • SYCL backend now supports `GGML_HINT_SRC0_IS_HADAMARD`, porting the Fast Walsh-Hadamard Transform kernel from CUDA to SYCL for accelerated Hadamard matrix-multiply on Intel GPUs. • Optimizes thread pool usage when `-t` and `-tb` differ: tools now share a single pool sized to the larger count rather than allocating two separate pools.

llama.cpp b10470

Mon, 17 Aug 2026 13:59:43 GMT

Adds BailingMoE3/Ling 3.0 Flash model support and removes `get_datetime` from built-in agent tools • Adds support for BailingMoE3 model architecture (Ling 3.0 Flash) with MTP support, enabling inference on the new Ling 3.0 Flash model. • Removes `get_datetime` from the list of available `--tools` built-in agent tools; the remaining available tools are `read_file`, `file_glob_search`, `grep_search`, `exec_shell_command`, `write_file`, `edit_file`, and `get_info`. Breaking changes: • The `get_datetime` tool is no longer listed as an available option for the `--tools` flag; any agent configuration specifying `get_datetime` will silently receive no matching tool.

llama.cpp b10455

Mon, 17 Aug 2026 05:47:28 GMT

SYCL backend gains AdamW and SGD optimizer step support for on-device training workloads. • Adds `OPT_STEP_ADAMW` and `OPT_STEP_SGD` operation support to the SYCL backend, enabling GPU-accelerated optimizer steps for AdamW and SGD on Intel GPUs.

llama.cpp b10448

Sat, 15 Aug 2026 20:48:35 GMT

llama.cpp b10448 adds Kimi-K3 model support with hybrid KDA+MLA attention, latent MoE, and a dedicated chat format. • Adds `common_chat_params_init_kimi_k3` (PEG_NATIVE) chat format for Kimi-K3 with reasoning extraction, response unwrapping, and typed tool-call parsing using the `tools/call/argument` tag structure. • Adds `kda_gate_lower_bound` GGUF key (emitted by the model saver) so save/load roundtrips correctly preserve Kimi-K3's KDA gate activation; K3 sets this to `-5.0`. • Increases `LLAMA_MAX_EXPERTS` from 512 to 1024 to accommodate Kimi-K3's expert count. • Adds Kimi-K3 text model support with hybrid KDA (linear) + MLA (full) attention, cross-layer residual attention (`attn_res_block_size`), latent MoE (`n_expert_latent`), situ activation, MLA output gate, and full-rank KDA gate (`ssm_g`). • Adds lossless MXFP4 repack for Kimi-K3's compressed-tensors 'mxfp4-pack-quantized' routed experts — bit-compatible with ggml MXFP4, repacked rather than dequantized, avoiding a ~5.5 TB bf16 round-trip; the `repack_mxfp4_blocks` helper is now shared with DeepSeek-V4. • Adds per-role message-start delimiters (`message_delimiters`) for Kimi-K3, enabling token-level span splitting for user, assistant, system, and tool messages. • Routes Kimi-K3 model architecture on the top-level model name (not `text_config`'s `KimiLinearForCausalLM`) via `get_model_architecture`. • CPU and CUDA backends support Kimi-K3's cross-layer residuals via `ggml_dsv4_hc_pre`; Metal and Vulkan fall back per-node until dedicated kernels are added.

llama.cpp b10448

Sat, 15 Aug 2026 20:48:35 GMT

llama.cpp b10448 adds Kimi-K3 model support with hybrid KDA+MLA attention, latent MoE, and MXFP4 lossless weight repack • Adds `common_chat_params_init_kimi_k3` chat format (`PEG_NATIVE`) with `thinking_forced_open` handling, reasoning extraction, response unwrapping, and typed tool-call parsing for Kimi-K3's XTML-ish tagged output format. • Increases `LLAMA_MAX_EXPERTS` from 512 to 1024 to accommodate Kimi-K3's latent MoE expert count. • Adds support for the Kimi-K3 text model architecture, featuring hybrid KDA (linear) + MLA (full) attention, cross-layer residual attention (`attn_res_block_size`), latent MoE (`n_expert_latent`), situ activation (replacing SwiGLU), MLA output gate, and full-rank KDA gate (`ssm_g`). • Adds `kda_gate_lower_bound` config key (set to -5.0 for Kimi-K3) controlling KDA gate activation swap — from -exp(A_log)*softplus(x) to lower_bound*sigmoid(exp(A_log)*x) — now correctly emitted by the model saver to survive save/load roundtrips. • Adds lossless MXFP4 weight repack for Kimi-K3's compressed-tensors 'mxfp4-pack-quantized' routed experts, repacking nibble positions without a bf16 dequantization round-trip; shared repack helper with DeepSeek-V4. • Adds per-role message delimiters for Kimi-K3 chat format, enabling token-level span splitting for user, assistant, system, and tool messages using the K3 tiktoken vocabulary. • Routes Kimi-K3 model architecture detection on the top-level model name rather than `text_config`'s `KimiLinearForCausalLM` to correctly distinguish K3 from the older Kimi-Linear-48B architecture.

llama.cpp b10444

Sat, 15 Aug 2026 18:42:10 GMT

llama.cpp b10444 adds `--models-dir` support for automatically loading MTP assistant (draft) models. • Extends `--models-dir` to automatically discover and load MTP assistant models using strict prefix matching, enabling speculative decoding setups without manually specifying draft model paths.

llama.cpp b10444

Sat, 15 Aug 2026 18:42:10 GMT

llama.cpp b10444 adds automatic MTP assistant model discovery via `--models-dir` • Extends `--models-dir` to automatically detect and load MTP assistant models (files matching the `mtp-*.gguf` prefix) as companion models, similar to existing mmproj auto-discovery, preventing them from being incorrectly loaded as the main model.

llama.cpp b10441

Sat, 15 Aug 2026 11:24:40 GMT

llama.cpp b10441 unifies --mmap, --no-mmap, --mlock, and --direct-io into a single --load-mode flag. • Replaces `--mmap`, `--no-mmap`, `--mlock`, and `--direct-io` flags with the unified `--load-mode` argument across scripts, examples, and documentation. Breaking changes: • The `--mmap`, `--no-mmap`, `--mlock`, and `--direct-io` flags are deprecated and replaced by `--load-mode`; existing scripts using those flags will trigger a warning and may break in future releases.

llama.cpp b10441

Sat, 15 Aug 2026 11:24:40 GMT

llama.cpp b10441 replaces deprecated `--mmap`/`--no-mmap`/`--mlock`/`--direct-io` flags with unified `--load-mode` argument • Replaces deprecated `--mmap`, `--no-mmap`, `--mlock`, and `--direct-io` CLI flags with the unified `--load-mode` argument across scripts, examples, and documentation.

llama.cpp b10437

Sat, 15 Aug 2026 05:24:03 GMT

llama.cpp b10437 adds support for MiniMaxText01ForCausalLM and MiniMaxM1ForCausalLM models with Jinja chat templates. • Adds support for `MiniMaxText01ForCausalLM` and `MiniMaxM1ForCausalLM` architectures, enabling local inference of MiniMax-Text-01 and MiniMax-M1 models. • Adds a Jinja chat template for MiniMax-M1 to enable structured conversation formatting with the new model family. • Adds converter support for `MiniMaxM1ForCausalLM` during GGUF conversion, treating it as equivalent to `MiniMaxText01ForCausalLM`.

llama.cpp b10437

Sat, 15 Aug 2026 05:24:03 GMT

llama.cpp b10437 adds inference support for MiniMax-Text-01 and MiniMax-M1 lightning-attention models • Adds support for `MiniMaxText01ForCausalLM` and `MiniMaxM1ForCausalLM` model architectures, enabling inference for MiniMax-Text-01 and MiniMax-M1 lightning-attention hybrid recurrent models. • Adds Jinja chat template for MiniMax-M1, automatically applied during conversion (stored in `models/templates/MiniMax-M1.jinja`). • Adds conversion support for `MiniMaxM1ForCausalLM` in the GGUF converter, treating it as equivalent to `MiniMaxText01ForCausalLM`. Breaking changes: • GGUF files for MiniMax-Text-01 created with the previous (January 2025) implementation are incompatible with b10437 due to a model architecture rename.

llama.cpp b10434

Fri, 14 Aug 2026 19:22:14 GMT

llama.cpp b10434 adds `reasoning_effort` to chat templates and generation params, enabling model-specific thinking control via the OpenAI-compatible API. • Adds `reasoning_effort` field to `common_chat_templates_inputs`, making OpenAI Chat Completions `reasoning_effort` available to Jinja templates with model-specific translations. • Adds `reasoning_effort` to generation params, positioned alongside `enable_thinking` for consistent reasoning control. • Server now reads `reasoning_effort` from the request body, with `server_chat_convert_responses_to_chatcmpl` handling conversion of Responses API `reasoning.effort` to `reasoning_effort`.

llama.cpp b10434

Fri, 14 Aug 2026 19:22:14 GMT

Adds `--reasoning-effort` CLI flag and `LLAMA_ARG_REASONING_EFFORT` env var to pass reasoning effort levels to chat templates • Adds `--reasoning-effort` flag (with `LLAMA_ARG_REASONING_EFFORT` env var) accepting levels `'minimal'`, `'low'`, `'medium'`, `'high'`, `'xhigh'`, `'max'`, or `'default'` to control reasoning effort passed to the chat template; available in server, completion, and CLI examples. • Exposes `reasoning_effort` as a field in `common_chat_templates_inputs` so Jinja chat templates (including model-specific translations) can consume it during generation.

llama.cpp b10431

Fri, 14 Aug 2026 17:42:44 GMT

Adds recurrent state rollback support for `ggml_ssm_scan` on CPU and CUDA, enabling speculative decoding with Nemotron/SSM models. • Adds recurrent state rollback to `ggml_ssm_scan` for CPU and CUDA backends, enabling speculative decoding on SSM-based models (e.g. Nemotron) by saving per-draft-token state copies and rolling back to the last accepted token's state. • Extends `ggml_ssm_scan` to support K > 1 across all backends, with automatic CPU fallback when the fused SSM op is not supported by the backend.

llama.cpp b10430

Fri, 14 Aug 2026 12:58:04 GMT

llama.cpp b10430 adds support for virtual iGPU devices. • Adds support for virtual iGPU devices, enabling selection and use of integrated GPU virtual device surfaces in the llama runtime.

llama.cpp b10430

Fri, 14 Aug 2026 12:58:04 GMT

llama.cpp b10430 enables virtual iGPU device support for multi-GPU setups like DGX Spark. • Supports multiple virtual iGPU devices of the same backend (e.g. CUDA) via the `GGML_CUDA_DEVICES` environment variable, enabling workloads across virtual CUDA devices on hardware like DGX Spark.

llama.cpp b10429

Fri, 14 Aug 2026 12:13:37 GMT

llama.cpp server now serves /metrics and /slots endpoints without blocking during active llama_decode() calls • Enables concurrent access to `/metrics` and `/slots` endpoints while llama_decode() is executing, eliminating observability blind spots during active inference.

llama.cpp b10429

Fri, 14 Aug 2026 12:13:37 GMT

llama.cpp server now serves /metrics and /slots endpoints concurrently during active llama_decode() calls • The `llama-server` `/metrics` and `/slots` endpoints are now accessible during active llama_decode() calls, eliminating request blocking while inference is running.

llama.cpp b10427

Fri, 14 Aug 2026 08:14:14 GMT

SYCL backend gains fused dense-FFN kernel for Q4_K models, boosting batched throughput up to +12% on Intel Arc GPUs. • Fuses mul_mat(gate) + mul_mat(up) + GLU into a single `q4_K` reorder mat-vec kernel on the SYCL backend, delivering up to +12.4% token generation throughput at batch size 8 on Intel Arc Pro B70 (Battlemage) hardware.

llama.cpp b10423

Thu, 13 Aug 2026 22:44:06 GMT

llama.cpp b10423 applies CPU parameters consistently across all tools in the suite. • Applies CPU parameters (such as thread counts and affinity settings) consistently across all llama.cpp tools, not just the main inference binary.

llama.cpp b10423

Thu, 13 Aug 2026 22:44:06 GMT

CPU parameters `--cpu-mask`, `--cpu-range`, and `--prio` now apply consistently across all llama.cpp tools • Extends `--cpu-mask`, `--cpu-range`, and `--prio` CPU parameters to `llama-cli`, `llama-server`, and other tools — previously only available in `llama-completion`.

llama.cpp b10419

Thu, 13 Aug 2026 22:11:34 GMT

OpenVINO backend gains Qwen3.5 support, new env-var memory controls, and a frontend model cache cutting compile time ~2.4x. • Adds `GGML_OPENVINO_RELEASE_WEIGHTS` env var to madvise(MADV_DONTNEED) host weight buffers after GPU compile, dropping steady-state RSS from ~1555 MB to ~710 MB for a 1B Q4_K_M model on Arc iGPU. • Adds `GGML_OPENVINO_REDUCE_COMPILE_MEM` env var to enable streaming weight requantization and cross-compile weight-node caching, cutting compile-time peak RSS by ~1 GB (1B model) to ~2 GB (8B model). • Adds `GGML_OPENVINO_MODEL_CACHE_DIR` env var to enable a frontend model cache that imports a previously compiled `CompiledModel`, reducing compile phase from 15.3 s to 6.3 s (~2.4x) on an 8B Q4_K_M GPU run. • Adds cache-key coverage for `GGML_OPENVINO_DISABLE_KV_SLICE` and `GGML_OPENVINO_MANUAL_GQA_ATTN` to prevent the frontend model cache from reusing blobs compiled under different attention-graph configurations. • Enables Qwen3.5 (including dense multi-sequence and MoE variants) on the OpenVINO backend. • Enables OpenVINO fallback to CPU backend when GPU ops are unsupported. • Adds `GGML_OPENVINO_RELEASE_WEIGHTS` hard-abort (`GGML_ABORT`) on cache-miss recompile or second model load to prevent silent wrong-model or zeroed-weight inference. • Optimizes MoE expert aggregation in the OpenVINO backend using `ReduceSum`. • Enables `GGML_UNARY_OP_SIGMOID`, `SQR`, `SQRT`, `FILL`, and `GET_ROWS` with multi-dimensional support in the OpenVINO backend. • Adds `gpt-oss` MoE and `mxfp4` support to the OpenVINO backend. • Enables zero-size copy for view ops in the OpenVINO backend. • Adds per-node runtime offsets (`rs_slot_begin_*`, `rs_src_begin_*`) to the OpenVINO backend for correct recurrent state rollback CPY writeback across any kv head, sequence count, and snapshot slot.

llama.cpp b10419

Thu, 13 Aug 2026 22:11:34 GMT

OpenVINO backend gains Qwen3.5 Dense+MoE support, GPU memory optimizations, and three new opt-in env flags • Adds `GGML_OPENVINO_RELEASE_WEIGHTS` environment variable to reclaim host RSS after GPU model compilation — measured ~845 MB reduction on Llama-3.2-1B-Q4_K_M on Arc iGPU. • Adds `GGML_OPENVINO_REDUCE_COMPILE_MEM` environment variable to enable streaming weight requantization and a cross-compile weight-node cache, cutting compile-time peak RSS by ~1 GB (1B model) to ~2 GB (8B model). • Adds `GGML_OPENVINO_MODEL_CACHE_DIR` environment variable for a frontend model cache that skips requantization, graph conversion, and compilation on warm starts — measured ~2.4x faster compile phase on an 8B Q4_K_M model on GPU. • Adds `GGML_OPENVINO_DISABLE_KV_SLICE` and `GGML_OPENVINO_MANUAL_GQA_ATTN` environment variables included in the frontend model cache key for correctness across different OpenVINO backend configurations. • Enables Qwen3.5 Dense and MoE model inference on the OpenVINO CPU and GPU backends. • Adds support for the `GGML_UNARY_OP_SIGMOID`, `SQR`, `SQRT`, and `FILL` ops in the OpenVINO backend. • Enables CPU backend fallback in the OpenVINO backend when an op is unsupported on the primary device.

llama.cpp b10418

Thu, 13 Aug 2026 21:36:06 GMT

llama.cpp b10418 adds SYCL host pinned memory support via `ggml_backend_sycl_host_buffer_type_get_max_size` to improve Host-to-Device memory access. • Adds `ggml_backend_sycl_host_buffer_type_get_max_size` to support host pinned memory on SYCL backends, improving Host-to-Device memory access performance.

llama.cpp b10418

Thu, 13 Aug 2026 21:36:06 GMT

SYCL backend gains host pinned memory support via `GGML_SYCL_ENABLE_HOST_PINNED_MEM`, boosting Host-to-Device throughput by up to 13.5%. • Adds `GGML_SYCL_ENABLE_HOST_PINNED_MEM` environment variable to enable host pinned memory for the SYCL backend, improving Host-to-Device memory access throughput — benchmarks show up to +13.5% prompt-processing speed on Intel Arc B70 with IQ2_XXS quantized models.

llama.cpp b10415

Thu, 13 Aug 2026 19:29:10 GMT

llama.cpp b10415 adds auto-detection of MTP draft model type for speculative decoding. • Adds auto-detection of the MTP draft model type in the speculative decoding spec, removing the need to manually identify the draft model.

llama.cpp b10415

Thu, 13 Aug 2026 19:29:10 GMT

llama.cpp b10415 adds auto-detection of MTP draft model type by checking for the NextN tensor. • Adds auto-detection of the `draft-mtp` model type by inspecting the presence of the `NextN` tensor in the draft model, eliminating the need to manually specify the speculative decoding mode.

llama.cpp b10414

Thu, 13 Aug 2026 18:40:34 GMT

llama.cpp b10414 adds TQ2_0 ternary quantization support to the Metal backend for Apple Silicon. • Adds `GGML_TYPE_TQ2_0` (ternary, 2 bits per element) quantization support to the Metal backend, enabling GPU-accelerated inference with TQ2_0 models on Apple Silicon.

llama.cpp b10414

Thu, 13 Aug 2026 18:40:34 GMT

llama.cpp b10414 adds TQ2_0 (ternary, 2-bit) quantization support to the Metal backend for Apple Silicon • Adds `GGML_TYPE_TQ2_0` (ternary, 2 bits per element) support to the Metal backend, enabling GPU-accelerated inference of TQ2_0-quantized models on Apple Silicon.

llama.cpp b10413

Thu, 13 Aug 2026 17:58:50 GMT

llama.cpp b10413 auto-detects speculative decoding type from local draft GGUF metadata, fixing silent no-op when using -md without --spec-type. • Adds common_speculative_types_from_gguf() in `speculative.cpp`/`.h` to auto-detect `--spec-type` from a local draft model's `general.architecture` GGUF header when `-md` is used without `--spec-type`, mapping `dflash` + `markov_w1.weight` tensor to `draft-dspark` and `dflash` alone to `draft-dflash`. • Emits a `LOG_INF` message when spec-type auto-detection fires, giving users visibility into why speculative decoding was enabled.

llama.cpp b10413

Thu, 13 Aug 2026 17:58:50 GMT

llama.cpp b10413 auto-detects speculative decoding type from local draft GGUF metadata, eliminating the need for explicit --spec-type. • Adds common_speculative_types_from_gguf() in `speculative.cpp`/`.h` to read `general.architecture` from a local draft GGUF header and automatically map it to `draft-dspark` (dflash + `markov_w1.weight` tensor) or `draft-dflash` (dflash without markov head), activating speculative decoding when `-md` loads a local draft model without `--spec-type`.

llama.cpp b10408

Thu, 13 Aug 2026 14:30:05 GMT

llama.cpp b10408 adds SYCL ESIMD kernels for Q3_K, Q4_K, and Q6_K quantized inference on Intel GPUs • Adds DMMV ESIMD kernels for `Q3_K`, `Q4_K`, and `Q6_K` quantization formats under the SYCL backend, enabled at build time with `-DGGML_SYCL_ESIMD=ON` and used by default at runtime when available on the device.

llama.cpp b10408

Thu, 13 Aug 2026 14:30:05 GMT

SYCL backend gains ESIMD kernels for Q3_K, Q4_K, Q6_K quantizations and concat support for Q4_0–Q8_0. • Adds SYCL ESIMD DMMV kernels for `Q3_K`, `Q4_K`, and `Q6_K` quantization types, enabled automatically at runtime when available on Intel GPU hardware. • Extends SYCL `OP_CONCAT` to support `Q4_0`, `Q4_1`, `Q5_0`, `Q5_1`, and `Q8_0` quantized tensor concatenation.

llama.cpp b10400

Thu, 13 Aug 2026 06:03:55 GMT

llama.cpp b10400 adds system-level config files, a UI read_media tool, slot save/restore with media, and Vulkan TQ2_0 support. • Adds system-level config file support at `/etc/llama.cpp/config.ini` (Linux/macOS), `%PROGRAMDATA%\llama.cpp\config.ini` (Windows), and `${XDG_CONFIG_HOME:-~/.config}/llama.cpp/config.ini` (user-level) — later files override earlier ones, enabling org-wide or per-user llama.cpp defaults without touching command-line flags. • Adds `read_media` tool to the built-in UI, enabling multimodal media input directly from the web interface. • Adds slot save/restore support for sessions that include media inputs in `llama-server`. • Adds TQ2_0 (ternary quantization) support to the Vulkan backend. • Adds a warp-per-row WKV7 CUDA kernel for single-token decode on head-size-64 RWKV7 models, improving throughput at batch size 1. • Adds KleidiAI runtime feature detection for aarch64 — enables dynamic dispatch to optimized routines without a separate build. • Adds `per_layer_config` handling in the Gemma4 model converter for compatibility with transformers 5.15. • Hardens the GGUF loader against malformed tensor dimensions and metadata types. • Refines CUDA graph disabling logic so graphs remain active for quantized and bf16/f16 MoE models that use the sync-free MMQ/MMF path, improving parallel decode performance.

llama.cpp b10369

Wed, 12 Aug 2026 04:52:57 GMT

llama.cpp b10369 adds pocket-tts support with 80% faster CUDA decoding and new mmproj audio config keys • Adds `clip.gen.audio.frames_after_eos` and `clip.gen.audio.pad_short_text` keys to mmproj files, letting language packs tune end-of-speech padding and short-prompt padding per pack (e.g. `french_24l` requests 8 tail frames, `english_2026-01` pads short prompts with spaces). • Adds `clip.gen.audio.model_variant` key to mmproj files for per-pack model variant selection. • Adds `mtmd_gen_inp_default` API function for pocket-tts input generation. • Adds pocket-tts (PocketTTS) model support to the `mtmd` multimodal subsystem, enabling on-device TTS inference. • Replaces grouped transposed convolutions in the Seanet decoder with a GEMM + `col2im_1d` formulation, cutting generation time per frame by 80% on CUDA and 50% on CPU. Breaking changes: • Existing mmproj files must be re-converted to carry the new `clip.gen.audio.frames_after_eos` and `clip.gen.audio.pad_short_text` keys; mmproj files without them fall back to previous defaults.

llama.cpp b10369

Wed, 12 Aug 2026 04:52:57 GMT

llama.cpp b10369 adds pocket-tts TTS support and a new `auto` default for `--load-mode` that avoids mmap on iGPUs. • Adds `auto` value to the `--load-mode` / `-lm` flag (now the default), which automatically disables mmap on iGPUs (CUDA/ROCm, Vulkan) to avoid doubling RAM usage during model load. • Adds pocket-tts support to the multimodal (`mtmd`) subsystem, enabling local text-to-speech generation via llama.cpp. • Adds `clip.gen.audio.frames_after_eos` and `clip.gen.audio.pad_short_text` keys to mmproj files, carrying per-language-pack TTS tuning (tail-frame padding and short-prompt padding); existing mmproj files must be reconverted to carry these keys. • Adds `clip.gen.audio.model_variant` key to mmproj files to identify the pocket-tts model variant. • Adds `mtmd_gen_inp_default` to the mtmd API for default TTS input generation. • Implements pocket-tts transposed convolutions as GEMM + `col2im_1d`, reducing TTS generation time per frame by 80% on CUDA and 50% on CPU. Breaking changes: • The default value of `--load-mode` changes from `mmap` to `auto`; setups relying on mmap behavior by default on iGPU systems will now skip mmap automatically. • Existing mmproj files for pocket-tts must be reconverted to include the new `clip.gen.audio.frames_after_eos` and `clip.gen.audio.pad_short_text` keys.

llama.cpp b10355

Mon, 10 Aug 2026 23:15:19 GMT

llama.cpp b10355 enables multi-output backend sampling with token speculation support across CPU and GPU backends. • Adds multi-output backend sampling, enabling GPU-accelerated sampling across multiple output sequences simultaneously, including support for token speculation workflows. • Introduces a numeric context parameter declaring the maximum outputs per sequence, allowing fine-grained control over multi-output sampling capacity.

llama.cpp b10355

Mon, 10 Aug 2026 23:15:19 GMT

llama.cpp b10355 adds multi-output backend sampling, enabling GPU-accelerated speculative decoding with ~8% throughput gain. • Adds `n_sampling_outputs_per_seq_max` to context parameters, separating the total output budget from the per-sequence backend-sampling limit to support multiple sampler graph rows per output. • Enables backend sampling during speculative decoding (token speculation), extending GPU-accelerated sampling from one output per sequence to multiple outputs. • Makes distribution sampling deterministic between CPU and backend paths by using a temporary RNG for multi-output sampling so rejected speculative tokens do not advance the RNG state.

llama.cpp b10353

Mon, 10 Aug 2026 22:00:50 GMT

Adds `podman` and `ssh` tool isolation runtimes to `--tools-runtime`, plus Muse Glimmer model support and mmproj quantization restored • Adds `podman:` and `podman-container:` options to the `--tools-runtime` flag, enabling tool isolation in rootless Podman containers alongside the existing Docker options • Adds `ssh:` option to `--tools-runtime` for running tools on a remote POSIX host over SSH (requires key-based auth and a trusted host key) • Adds support for the Muse Glimmer model architecture • Restores quantization support for mmproj (multimodal projector) GGUF files, which was inadvertently broken in a prior refactor • Adds Windows ARM64 CUDA 13.4 pre-built binary packages, enabling GPU-accelerated inference on ARM64 Windows devices (e.g. RTX Spark)

llama.cpp b10344

Mon, 10 Aug 2026 16:24:12 GMT

llama.cpp b10344 adds Multi-Token Prediction (MTP) support for the Nemotron Nano model. • Adds `mtp_flags` configuration and Multi-Token Prediction (MTP) support for the Nemotron Nano model, enabling speculative decoding workflows with this architecture.

llama.cpp b10344

Mon, 10 Aug 2026 16:24:12 GMT

llama.cpp b10344 adds Multi-Token Prediction (MTP) support for the Nemotron Nano model. • Adds Multi-Token Prediction (MTP) support for the Nemotron Nano model, including `mtp_flags` configuration for Nemotron-specific MTP behaviour.

llama.cpp b10342

Mon, 10 Aug 2026 12:47:43 GMT

llama.cpp b10342 adds the Granite-Switch architecture: a dense Granite-4.1 model with per-token LoRA adapter switching via in-graph router attention. • Adds `graniteswitch` arch registration (in `llama-arch.h/cpp`, `llama-model.h/cpp`) and `src/models/granite_switch.cpp` implementing per-token switched-LoRA inference via `ggml_mul_mat_id` over stacked `lora_a`/`lora_b` tensors. • Adds `conversion/granite.py` `GraniteSwitchModel` converter that stacks N LoRA adapters plus a zero base slot into per-projection A/B tensors and emits switch metadata into GGUF. • Introduces `llm_graph_input_switch` in `granite_switch.cpp` providing stateless per-token adapter-index resolution via a single-head causal router attention layer (`hparams.router_layer`) stored in the KV cache, isolating concurrent requests automatically. • Adds hparams.has_rope(il) predicate to skip k-shift RoPE on the router layer, preventing corruption of the literal-magnitude router K signal. • Granite-Switch conversion honors `--outtype` for LoRA tensor precision instead of forcing F16. • Validates substitute (adapter) token IDs against `n_vocab` and bounds adapter count and LoRA rank from GGUF at load time, throwing on bad metadata.

llama.cpp b10342

Mon, 10 Aug 2026 12:47:43 GMT

Adds `graniteswitch` architecture support for IBM Granite-Switch 4.1 models with per-token switched LoRA adapters • Adds `graniteswitch` architecture registration, enabling llama.cpp to load and run IBM Granite-Switch 4.1 models (dense all-attention models with N embedded LoRA adapters selected per-token via control tokens); runs on CPU and Metal (Apple Silicon). • Adds `conversion/granite.py` `GraniteSwitchModel` converter that stacks N adapters plus a zero base slot into per-projection `.lora_a`/`.lora_b` tensors and emits switch metadata into GGUF. • Adds GGUF schema extensions for the `graniteswitch` arch including KV keys under the `{arch}.adapters.*` namespace and `router_gain` metadata key (read from GGUF instead of hardcoded), with stacked LoRA tensor names using `.lora_a`/`.lora_b` suffixes. • Implements an in-graph router attention mechanism (stored in the model KV cache at layer index `router_layer`) that isolates adapter selection per sequence, enabling safe concurrent multi-request inference without adapter-index leakage. • Adds `llm_graph_input_switch` input class in `src/models/granite_switch.cpp` providing stateless per-token adapter-index mapping with `adapter_token_*` control token substitution and validation of substitute token IDs against `n_vocab`. • Adds hparams.has_rope(il) predicate to skip KV-shift RoPE for the router layer, preventing corruption of the literal magnitude stored in router K dim-0. • Supports all three preview Granite-Switch model sizes in the converter and architecture handler.

llama.cpp b10329

Sat, 08 Aug 2026 16:00:17 GMT

llama.cpp server tools now declare working-directory support, so the WebUI `/cwd` control only appears when a relevant tool is active. • Built-in tools now publish a working-directory flag in the `/tools` listing, indicating whether they resolve paths against the working directory — the WebUI shows the working-directory chip and enables the `/cwd` command only when at least one such tool is both served and enabled.

llama.cpp b10328

Sat, 08 Aug 2026 15:22:20 GMT

llama.cpp server gains Docker-backed tool isolation via `x-tool-runtime` for sandboxed LLM tool execution. • Adds initial tool isolation support to the `server` via Docker, separating tool execution into `tools_io_sandbox` and `tools_io_docker` modes, exposed through the `x-tool-runtime` request header.

llama.cpp b10328

Sat, 08 Aug 2026 15:22:20 GMT

llama-server gains `--tools-runtime` flag to sandbox LLM tool calls inside Docker containers • Adds `--tools-runtime` flag (env: `LLAMA_ARG_TOOLS_RUNTIME`) to `llama-server` to run tool calls in an isolated runtime environment; supports `docker:` (spins up a fresh container, cleaned up on server exit) and `docker-container:` (reuses an existing container by ID, not stopped on exit) as values.

llama.cpp b10318

Fri, 07 Aug 2026 17:35:40 GMT

WebUI gains `?model=xxx&load=true` URL parameter to pre-load models on page open; npm supply-chain hardening added. • Adds `?load=true` URL parameter to the WebUI so that a model specified via `?model=` is loaded immediately when the page opens, rather than waiting for the first message. • Sets npm `min-release-age` in the WebUI build to reduce exposure to supply-chain attacks by avoiding newly-published packages.

llama.cpp b10313

Fri, 07 Aug 2026 16:58:40 GMT

llama.cpp server gains an LRU scheduler for smarter request routing across slots • Adds `lru_sched` LRU (Least Recently Used) scheduler to the server router, improving slot assignment for concurrent inference requests.

llama.cpp b10313

Fri, 07 Aug 2026 16:58:40 GMT

llama.cpp server gains an LRU scheduler for smarter multi-model request queuing and eviction. • Adds an LRU scheduler to the `llama-server` router, queuing eviction requests when server capacity is exhausted and unblocking all waiting requests for the same model simultaneously once it is loaded. • Disconnected requests now automatically remove themselves from the LRU queue, preventing stale waiters from holding slots.

llama.cpp b10305

Fri, 07 Aug 2026 12:03:40 GMT

SYCL backend gains DSv4 operator support: LIGHTNING_INDEXER, DSV4_HC_COMB, DSV4_HC_POST, DSV4_HC_PRE • Adds SYCL backend support for four DSv4 operations: `LIGHTNING_INDEXER`, `DSV4_HC_COMB`, `DSV4_HC_POST`, and `DSV4_HC_PRE`, expanding hardware-accelerated inference coverage on Intel SYCL devices.

llama.cpp b10305

Fri, 07 Aug 2026 12:03:40 GMT

SYCL backend gains four DSv4 ops — LIGHTNING_INDEXER, DSV4_HC_COMB, DSV4_HC_POST, DSV4_HC_PRE — paving the way for DeepSeek-V4-Flash inference on Intel GPUs. • Adds SYCL backend support for four DSv4 compute ops: `LIGHTNING_INDEXER`, `DSV4_HC_COMB`, `DSV4_HC_POST`, and `DSV4_HC_PRE`, enabling models that depend on these ops (including DeepSeek-V4-Flash) to run on Intel GPUs via the SYCL path.

llama.cpp b10298

Thu, 06 Aug 2026 22:10:39 GMT

llama.cpp b10298 adds save/load API for multimodal input chunks in the mtmd subsystem. • Adds chunk save/load functions to the `mtmd` multimodal subsystem, enabling serialization and deserialization of `mtmd_input_chunk` objects.

llama.cpp b10293

Thu, 06 Aug 2026 09:12:47 GMT

Adds `--model-name` flag to model conversion scripts and onboards AMD ROCm CI with gfx1151 (Strix Halo) support. • Adds `--model-name` flag to the causal and embedding model conversion scripts to explicitly set the `general.name` metadata field, overriding the default (basename of the model path). • Onboards AMD ROCm CI pipeline with gfx1151 (Strix Halo / integrated RDNA3.5) support, enabling automated testing of the HIP backend on that architecture.

llama.cpp b10285

Wed, 05 Aug 2026 15:20:58 GMT

llama.cpp b10285 adds multi-row batching support for DeepSeek-OCR in the multimodal subsystem. • Adds multi-row batching for DeepSeek-OCR in the `mtmd` multimodal subsystem, grouping same-size rows into a single batch pass to improve throughput (overview images with differing sizes are excluded from batching).

llama.cpp b10282

Wed, 05 Aug 2026 13:50:09 GMT

llama.cpp b10282 adds speculative-decoding counters to the `/metrics` endpoint, aligned with vLLM naming. • Adds spec-decode counters to the `/metrics` endpoint on the llama.cpp server, with parameter names aligned to vLLM conventions for drop-in compatibility.

llama.cpp b10282

Wed, 05 Aug 2026 13:50:09 GMT

llama.cpp server gains speculative-decoding Prometheus counters on /metrics, aligned with vLLM schema • Adds four speculative-decoding counters to the `/metrics` Prometheus endpoint: `llamacpp:spec_decode_num_draft_tokens`, `llamacpp:spec_decode_num_accepted_tokens`, `llamacpp:spec_decode_num_drafts`, and `llamacpp:spec_decode_num_accepted_tokens_per_pos{position="N"}` (per-draft-position labeled counter, absent until the first completed speculative request), aligned with the vLLM counter schema. • Extends `gguf_convert_endian.py` to support endianness conversion for Q1 and TQ2 quantization formats (swapping only the fp16 values, since sub-byte quantized values require no byte-swap).

llama.cpp b10280

Wed, 05 Aug 2026 11:35:24 GMT

llama.cpp b10280 enables generation statistics by default in the built-in chat UI • Built-in web UI now shows generation statistics by default in chat settings.

llama.cpp b10271

Tue, 04 Aug 2026 18:56:20 GMT

llama.cpp b10271 adds per-conversation working directory picker with path navigation to the server UI agent. • Adds per-conversation working directory picker to the server UI, supporting path navigation via POSIX (`/`, `~`), Windows drive roots (`C:`), and UNC shares (`//host/share`) in addition to glob-matching. • Extends `file_glob_search` on the server with hardened directory listing including kind enum, timeout, symlink guard, and absolute base enforcement. • Persists per-call tool `cwd` on tool result messages in the database, so working directory context survives across conversation turns. • Adds `get_info` tool call block rendering in the UI.

llama.cpp b10270

Tue, 04 Aug 2026 18:03:46 GMT

llama.cpp b10270 adds Qwen3-TTS support via a revamped `llama-tts` binary and new `mtmd_helper_gen_audio` API. • Adds `mtmd_helper_gen_audio` API for multimodal audio generation, enabling programmatic TTS workflows. • Adds `llama_model_get_tok_embd` function to the llama library API. • Adds `mtmd_helper_model_can_chat` helper to query whether a loaded model supports chat-style interaction. • Adds step_prompt() method to the mtmd generation API for incremental prompt stepping. • Supports Qwen3-TTS model in the `mtmd` multimodal framework, including text model, speaker encoder, code predictor, and `GEN_WAV` (audio generation) subgraph. • Revamps `llama-tts` binary with updated CLI interface and Qwen3-TTS support, including voice clone demo capability. • Adds `suppress_tokens` support to the TTS backbone vocabulary pipeline. Breaking changes: • The `llama-tts` binary interface has breaking changes as part of the Qwen3-TTS integration revamp — existing invocations will need to be updated.

llama.cpp b10255

Tue, 04 Aug 2026 05:39:04 GMT

llama.cpp b10255 extends SYCL oneDNN SDPA to Q4_0–Q8_0 and FP32 KV caches on Intel hardware. • Extends the oneDNN Scaled Dot-Product Attention (SDPA) path on SYCL to support quantized KV cache types `Q4_0`, `Q4_1`, `Q5_0`, `Q5_1`, `Q8_0`, and `F32` in addition to the existing `F16` path, by dequantizing K/V to dense FP16 on-device before the fused systolic kernel runs. • Non-F16 KV cache SDPA is gated to prefill scenarios only: requires sequence length K >= 1024 and query count Q >= 32; F16 KV continues to run at any length.

llama.cpp b10254

Tue, 04 Aug 2026 04:53:21 GMT

llama.cpp b10254 adds a DeepSeek V4 Flash 0731 chat template with reasoning-effort mapping and structured output support. • Adds a separate chat template for DeepSeek-V4-Flash-0731 with updated `high` and `max` reasoning-effort mappings. • Adds structured output response-format instructions to DeepSeek V4 templates, passing the schema into template rendering. • Defaults `drop_thinking` for DeepSeek V4 conversation history so prior reasoning is omitted unless `preserve_reasoning` is requested or tools are present.

llama.cpp b10251

Tue, 04 Aug 2026 03:06:31 GMT

llama.cpp b10251 adds Multi-Token Prediction (MTP) support for the GLM-4.7-Flash model. • Adds Multi-Token Prediction (MTP) support for the GLM-4.7-Flash model.

llama.cpp b10249

Tue, 04 Aug 2026 01:36:29 GMT

llama.cpp server gains a `get_info` tool for probing host OS and environment diagnostics. • Adds `get_info` tool to the server, exposing host OS identity and environment diagnostics via a structured probe mechanism.

llama.cpp b10242

Mon, 03 Aug 2026 17:22:43 GMT

llama.cpp b10242 adds GPU-accelerated CUDA backend sampler for penalty handling including frequency, presence, and top-k penalties. • Adds CUDA backend sampler for `llama_sampler_penalties`, offloading frequency, presence, and repeat penalty computation to GPU via the `llama_sampler_backend` infrastructure. • Adds `llama_n_ctx` parameter to `common_sampler_init` so `penalty_last_n` defaults to the model context length when not explicitly set. • Adds support for top-k penalties in backend sampling, with configurable positions in the sampler chain. • Adds vocabulary-sized count tensor to replace the per-candidate penalty comparison matrix, improving scalability for large history windows. • Adds validation that repeat penalty is finite and greater than 0, and preserves masked logits as `-Inf` to eliminate NaN generation. Breaking changes: • The signature of `common_sampler_init` now requires an additional `llama_n_ctx` parameter; callers that do not pass it will fail to compile.

llama.cpp b10238

Mon, 03 Aug 2026 08:56:47 GMT

llama.cpp b10238 adds Multi-Token Prediction (MTP) support for Qwen3-Next models. • Adds Multi-Token Prediction (MTP) support for Qwen3-Next, including `load_mtp` flags and `opt_num_mtp_layers` defined in the model mixin.

llama.cpp b10237

Mon, 03 Aug 2026 07:50:08 GMT

llama.cpp b10237 adds Multi-Token Prediction (MTP) support for DeepSeek V3.2 models. • Adds Multi-Token Prediction (MTP) support for DeepSeek V3.2, enabling faster speculative decoding with that model family.

llama.cpp b10232

Sun, 02 Aug 2026 18:57:43 GMT

llama.cpp b10232 adds Metal GPU kernels for DeepSeek V4 hyper-connections via three new GGML ops. • Implements `GGML_OP_DSV4_HC_COMB`, `GGML_OP_DSV4_HC_PRE`, and `GGML_OP_DSV4_HC_POST` Metal kernels for DeepSeek V4 hyper-connection layers, using SIMDgroup register and shuffle optimizations for GPU-accelerated inference on Apple Silicon.

llama.cpp b10231

Sun, 02 Aug 2026 18:14:07 GMT

llama.cpp b10231 adds DSpark speculative sidecar support with Markov-head auto-selection priority over DFlash. • Adds DSpark speculative sidecar resolution to `common`, matching the existing sidecar discovery rules: the `-hfd` tag applies, a requested sidecar resolves without a full model at the tag, and an explicit `-md` selection disables discovery. • DSpark auto-ranks above DFlash in sidecar auto-selection when no type is explicitly requested, because the DSpark sidecar carries an extra Markov head.

llama.cpp b10227

Sun, 02 Aug 2026 09:43:15 GMT

llama.cpp b10227 adds a specialized Qwen3 tool-call chat parser with tagged thinking and `` omission support. • Adds a specialized Qwen3 chat parser that handles tagged thinking blocks and supports optional `` omission, enabling cleaner agentic tool-use workflows with Qwen3 and Qwen3-Coder models.

llama.cpp b10219

Sat, 01 Aug 2026 16:46:07 GMT

llama-cli now persists reasoning_content in chat history so --reasoning-preserve can re-inject prior thoughts on later turns. • Stores `reasoning_content` in the chat message history within `llama-cli`, enabling `--reasoning-preserve` to re-inject prior model thoughts on subsequent conversation turns.

llama.cpp b10218

Sat, 01 Aug 2026 12:46:12 GMT

llama.cpp b10218 adds MiniCPM-V 4.6 multimodal downsample support via the mtmd image preprocessor. • Adds MiniCPM-V 4.6 downsample mode to the `mtmd` multimodal image preprocessor, enabling inference with the MiniCPM-V 4.6 vision model via `mtmd_image_preprocessor_llava_uhd`. • Stores the downsample mode inside the GGUF file metadata, allowing model files to self-describe their preprocessing requirements. • Adds a 4x ignore ViT merger path for MiniCPM-V 4.6 image tiling.

llama.cpp b10217

Sat, 01 Aug 2026 06:50:00 GMT

llama.cpp b10217 enables tool call support during thinking mode for DeepSeek v4 (DS4) models. • Enables tool call execution within the thinking/reasoning phase for DeepSeek v4 (DS4) models in chat mode.

llama.cpp b10216

Fri, 31 Jul 2026 22:04:52 GMT

llama.cpp b10216 adds full GGML_OP_POOL_1D support to the Vulkan backend via a new compute shader. • Adds `GGML_OP_POOL_1D` support to the Vulkan backend, including `vk_op_pool1d_push_constants`, `pipeline_pool1d_f32`, and the `pool1d.comp` compute shader, enabling 1D pooling operations on Vulkan-accelerated hardware.

llama.cpp b10208

Fri, 31 Jul 2026 16:02:42 GMT

SYCL backend gains oneMKL GEMM flash attention for XMX-accelerated prompt processing, delivering up to 1.97x throughput on Intel Battlemage GPUs. • Adds `GGML_SYCL_ENABLE_MKL_FA` environment variable to enable/disable oneMKL GEMM flash attention on the SYCL backend (set to 0 to disable); activates automatically when flash-attn is on and Q tokens >= 32. • Adds `GGML_SYCL_MKL_FA_DEBUG` environment variable to enable an FA watchdog with timing diagnostics for the MKL flash attention path. • Adds `GGML_SYCL_MKL_FA_DIAG` environment variable to emit FA-DIAG output fingerprints for A/B testing the MKL vs. TILE attention paths. • New oneMKL GEMM flash attention kernel accelerates multi-token prompt processing via XMX hardware on Intel Arc/Battlemage GPUs, supporting all KV cache types (F16, BF16, F32, and quantized) with head dimensions 64–512 (multiples of 64) and GQA >= 2. • XMX-accelerated path delivers up to 1.97x prompt throughput over the TILE path: Gemma-4-26B reaches ~1473 t/s vs. ~746 t/s, Qwen3.6-27B reaches ~609 t/s vs. ~330 t/s, on Intel B70/Battlemage at 32K context with q8_0 KV cache.

llama.cpp b10204

Fri, 31 Jul 2026 12:45:21 GMT

SYCL backend gains device-to-device memcpy support via DEV2DEV_MEMCPY_FORWARD for Intel GPU acceleration. • Adds `DEV2DEV_MEMCPY_FORWARD` support to the SYCL backend, enabling direct device-to-device memory copies between Intel GPU devices.

llama.cpp b10203

Fri, 31 Jul 2026 12:14:34 GMT

llama.cpp b10203 adds SYCL support for Q2_0 quantized matrix multiplication • Adds `q2_0` quantization support to `mul_mat` on the SYCL backend, enabling Q2 models to run on Intel GPUs and other SYCL-capable accelerators.

llama.cpp b10199

Thu, 30 Jul 2026 20:33:03 GMT

llama.cpp server gains support for input embeddings to drive next-token sampling • Adds `inp embd` (input embedding) support to the server, enabling callers to supply raw embedding vectors directly to drive sampled token generation instead of text prompts.

llama.cpp b10198

Thu, 30 Jul 2026 19:37:33 GMT

llama.cpp b10198 adds quantized concat support for the Vulkan backend. • Adds quantized concat operation support to the Vulkan backend, enabling GPU-accelerated concatenation of quantized tensors on Vulkan devices.

llama.cpp b10174

Wed, 29 Jul 2026 07:14:20 GMT

llama.cpp b10174 adds NextN/MTP speculative decoding support for GLM-5.2 (GLM_DSA) with `--spec-type draft-mtp` and `--mtp`/`--no-mtp` convert flags. • Adds `--spec-type draft-mtp` support for GLM_DSA (GLM-5.2) models, enabling NextN/MTP speculative decoding with a dense MLA draft head, sigmoid-gated MoE with shared expert, and NVFP4 `_s` scale tensor support. • Adds `--mtp` and `--no-mtp` export flags to the converter for `GlmMoeDsaForCausalLM` (GLM-5.2): `--no-mtp` drops the NextN block (`blk.78`) and its `nextn_predict_layers` KV; `--mtp` keeps only the NextN block plus shared embeddings, norm, and lm_head.

llama.cpp b10164

Tue, 28 Jul 2026 14:28:22 GMT

llama.cpp b10164 adds chunked SSD matrix multiplication on CUDA/HIP/MUSA to accelerate Mamba-2 prefill. • Adds chunked SSD (State Space Duality) matmul CUDA kernel to accelerate Mamba-2 prefill, with support across CUDA, HIP, and MUSA backends.

llama.cpp b10159

Tue, 28 Jul 2026 11:43:46 GMT

llama.cpp b10159 adds a Fast Walsh-Hadamard Transform (FWHT) kernel for the Metal backend on Apple Silicon. • Adds a Fast Walsh-Hadamard Transform (FWHT) compute kernel to the Metal backend, enabling GPU-accelerated FWHT operations on Apple Silicon devices.

llama.cpp b10155

Mon, 27 Jul 2026 21:59:35 GMT

llama.cpp b10155 adds MiMo-V2.5 audio input support via RVQ-based model in the multimodal backend. • Adds MiMo-V2.5 audio input support (RVQ-based model) to the `mtmd` multimodal backend, including a GGUF converter for MiMo audio.

llama.cpp b10154

Mon, 27 Jul 2026 16:59:26 GMT

llama.cpp b10154 adds common_print_available_devices() to list usable compute devices at runtime. • Adds common_print_available_devices() function to the common library for enumerating available compute devices at runtime.

llama.cpp b10153

Mon, 27 Jul 2026 15:56:00 GMT

llama.cpp b10153 adds support for the Nanbeige4.2 model architecture. • Adds support for the Nanbeige4.2 model architecture.

llama.cpp b10145

Mon, 27 Jul 2026 09:28:29 GMT

llama.cpp b10145 adds `-lm mlock` mode to lock model weights in RAM without mmap. • Adds `-lm mlock` argument to lock model weights in RAM via mlock without using mmap, preventing swap-out of model data. Breaking changes: • The meaning of `--mlock` is reverted to non-mmap'ed mlock behavior, which may change behavior for setups that relied on its previous semantics.

llama.cpp b10142

Mon, 27 Jul 2026 00:20:27 GMT

llama.cpp b10142 adds multimodal vision support for MiniMax-M3, including sparse attention and GPU-accelerated inference. • Adds vision support for MiniMax-M3 via `mtmd` (multimodal) framework, including mmproj and clip graph construction for the vision tower. • Implements MiniMax-M3 sparse attention (MSA) with CUDA-native indexer op and flash attention for sparse layers, with dense fallback for multi-sequence `--kv-unified` mode. • Sets default KV cache type to F32 for MiniMax-M3 and allocates in-cache `k_idx` only for architectures that opt in, reducing compute buffer from ~6.8 GiB to ~4.2 GiB at ub2048/62k context. • Supports prompt caching and multi-stream inference (`-np N` when `kv_unified=false`) for MiniMax-M3, with context shift explicitly disallowed for this architecture. • Adds MiniMax-M3 end-of-sequence token to the llama vocabulary. • Changes MiniMax-M3 image resize padding to 'none' and resize algorithm to Bicubic (Pillow) during conversion. Breaking changes: • All MiniMax-M3 GGUFs generated before this release must be regenerated due to the renaming of MSA index tensors to the indexer naming convention.

llama.cpp b10105

Fri, 24 Jul 2026 09:34:02 GMT

llama.cpp b10105 consolidates `--mmap`, `--mlock`, and `--direct-io` into a single `--load-mode` argument • Replaces the separate `--mmap`, `--mlock`, and `--no-mmap` flags with a unified `--load-mode` argument that accepts modifier-based values (e.g. `mmap`, `mlock`, `dio`) controlling how model weights are loaded from disk. Breaking changes: • The previous `--mmap`, `--no-mmap`, and `--mlock` flags are now deprecated; combining them with the new `--load-mode` flag triggers a deprecation warning and may produce undefined behaviour — users should migrate to `--load-mode`.

llama.cpp b10099

Thu, 23 Jul 2026 23:34:13 GMT

CUDA NVFP4 W4A4 activation quantization improved with fused kernels and intrinsics for faster inference on NVIDIA GPUs. • Improves NVFP4 W4A4 activation quantization on CUDA with fused per-channel amax and quantization kernels, 32-byte loads, and `nvfp4x4` intrinsic support where available, reducing overhead during quantized matrix multiplication.

llama.cpp b10094

Thu, 23 Jul 2026 18:28:00 GMT

llama-server now auto-detects mtp/dflash/eagle3 speculative draft sidecars from `-hfd` repos without requiring `--spec-type`. • Adds automatic speculative decoding type inference when using `-hfd` with a repo that ships `mtp-`, `dflash-`, or `eagle3-` sidecars — selects the first available following priority order `mtp > dflash > eagle3` and sets `--spec-type` automatically, so `llama-server -hf repo:Q3_K_M -hfd repo:Q8_0` works without any extra flag. An explicit `--spec-type` disables the inference.

llama.cpp b10090

Wed, 22 Jul 2026 20:23:31 GMT

llama.cpp b10090 adds GGML_OP_CONV_2D_DW (depthwise conv2d) support to the WebGPU backend. • Implements `GGML_OP_CONV_2D_DW` (depthwise convolution 2D) kernel for the WebGPU backend, ported from the Vulkan backend's `conv2d_dw.comp` shader.

llama.cpp b10089

Wed, 22 Jul 2026 19:45:27 GMT

CUDA GET_ROWS now handles all quantized GGML types on-device, eliminating host fallback for k-quants, i-quants, and mxfp4. • Adds full k-quant support (`q2_K` through `q6_K`) to the CUDA `GET_ROWS` kernel via a new `k_get_rows_kq` kernel, keeping embedding lookups on-device instead of falling back to the host and copying the full embedding matrix per token. • Extends CUDA `GET_ROWS` device-side coverage to all nine i-quants, completing quantized type support alongside k-quants. • Adds `mxfp4` support to CUDA `GET_ROWS`, closing the final gap so every quantized GGML type now takes the direct device path in single-device graphs.

llama.cpp b10087

Wed, 22 Jul 2026 11:03:57 GMT

llama.cpp b10087 adds support for Laguna XS.2 & M.1 hardware. • Adds support for Laguna XS.2 & M.1 accelerator hardware.

llama.cpp b10069

Mon, 20 Jul 2026 06:29:53 GMT

OpenCL Adreno backend gains broadcast support for MUL_MAT and correct view-offset handling for Q8_0 in multi-stream llama-server • Adds broadcast support for Adreno GEMM and GEMV (noshuffle) kernels in the OpenCL backend, enabling multi-stream `llama-server` workloads that require tensor broadcasting. • Adds correct `view_offs` handling for Adreno Q8_0 `MUL_MAT` (noshuffle GEMM/GEMV) in the OpenCL backend, fixing correctness for non-zero-offset tensor views in multi-stream inference.

llama.cpp b10058

Fri, 17 Jul 2026 11:44:06 GMT

llama.cpp b10058 adds Q2_0 quantization support to the Vulkan backend for GPU-accelerated mat-vec-mul. • Adds `Q2_0` quantization format support to the Vulkan backend, enabling GPU-accelerated matrix-vector multiplication for Q2_0 models.

llama.cpp b10045

Thu, 16 Jul 2026 16:09:34 GMT

llama.cpp server gains text-only slot save/restore support for multimodal (mtmd) sessions • Adds text-only slot save/restore capability to the server when running with multimodal (`mtmd`) support, enabling KV-cache slot persistence for text portions of multimodal sessions.

llama.cpp b10043

Thu, 16 Jul 2026 14:02:33 GMT

llama.cpp b10043 adds CUDA Virtual Device support, enabling multi-GPU splits across virtual device boundaries. • Adds CUDA Virtual Device support, allowing llama.cpp to treat virtual CUDA devices as distinct GPU backends (NCCL path is automatically disabled when virtual devices are in use).

llama.cpp b10042

Thu, 16 Jul 2026 11:23:24 GMT

llama.cpp b10042 enables CUDA graphs on Volta and Turing GPUs for improved inference throughput. • Enables CUDA graphs support on Volta and Turing GPU architectures, unlocking lower kernel-launch overhead and higher inference throughput on older NVIDIA hardware.

llama.cpp b10037

Thu, 16 Jul 2026 06:15:08 GMT

llama.cpp b10037 lets `--pure` quantization mode coexist with manual per-tensor type overrides. • Allows manual tensor type overrides to be combined with `--pure` during quantization, so practitioners can pin specific tensors to a chosen quant type while still enforcing a pure quantization pass for the rest of the model.

llama.cpp b10032

Wed, 15 Jul 2026 19:52:26 GMT

llama.cpp b10032 adds CUDA implementation of GGML_OP_LIGHTNING_INDEXER with generic vector and WMMA kernels (Turing+). • Adds CUDA implementation of `GGML_OP_LIGHTNING_INDEXER` with a generic vector kernel and a WMMA (Tensor Core) kernel, requiring Turing-class GPUs or newer.

llama.cpp b10016

Wed, 15 Jul 2026 09:10:19 GMT

llama.cpp b10016 adds SYCL Flash Attention via oneDNN XMX engine on Intel Battlemage (Xe2), delivering up to 4.26× prefill speedup at 80k context. • Adds F16 Flash Attention for SYCL backends using the oneDNN graph API's XMX engine, scoped to Intel Battlemage (Xe2/BMG) GPUs; other architectures fall back to the existing FA kernel. • Multi-GPU synchronization for the oneDNN FA path via stream wait_and_throw(), enabling validated multi-device inference on Battlemage. • Delivers up to 4.26× prefill throughput increase at 80k context (pp512: ~32% gain) for Q8_0 models on supported hardware, unlocking long-context use cases that were previously throughput-constrained.

llama.cpp b10010

Tue, 14 Jul 2026 20:26:04 GMT

llama.cpp b10010 adds `--cors-*` CLI options to the server for fine-grained CORS control. • Adds `--cors-*` options to the `server` command, including a special `localhost` value, enabling fine-grained cross-origin resource sharing control.

llama.cpp b10003

Tue, 14 Jul 2026 17:42:45 GMT

llama.cpp b10003: tokenize tool gains standard model-sourcing flags including HuggingFace and offline support • Exposes `-m`, `-mu`, `-dr`, `-hf`, `-hff`, `--offline`, and `HF_TOKEN` model-sourcing flags to the `tokenize` tool via `LLAMA_EXAMPLE_TOKENIZE`, enabling HuggingFace-hosted and offline model use directly from the tokenizer. • Registers `--ids`, `--stdin`, `--no-bos`, `--no-parse-special`, and `--show-count` as standard common args in the `tokenize` tool, replacing its hand-rolled argument parser.

llama.cpp b9994

Tue, 14 Jul 2026 05:29:11 GMT

llama.cpp b9994 adds Q2_0 quantization support for Metal (Apple GPU) inference. • Adds `Q2_0` quantization format support to the Metal backend, enabling Apple Silicon GPU-accelerated inference with Q2_0 models.

llama.cpp b9993

Mon, 13 Jul 2026 23:09:16 GMT

llama.cpp b9993 adds Tencent Hunyuan 3 (hy_v3) architecture support with MTP speculative decoding. • Adds `hy_v3` (HunyuanV3 / `HYV3ForCausalLM`) architecture support: a MoE decoder with per-head Q/K RMSNorm, sigmoid router with expert selection bias, an always-active ungated shared expert, and leading dense blocks (`first_k_dense_replace`), enabling MTP speculative decoding for Tencent Hunyuan 3 models in GGUF format.

llama.cpp b9990

Mon, 13 Jul 2026 19:51:41 GMT

llama.cpp b9990 adds Minimax2 EAGLE3 speculative decoding support. • Adds EAGLE3 speculative decoding support for Minimax2 models.

llama.cpp b9987

Mon, 13 Jul 2026 15:40:41 GMT

llama.cpp b9987 adds `gguf_get_tensor_ne` accessor for retrieving tensor shapes from GGUF files. • Adds `gguf_get_tensor_ne` to the GGUF API, returning tensor shape as `const int64_t *` for programmatic inspection of model tensor dimensions.

llama.cpp b9985

Mon, 13 Jul 2026 08:32:45 GMT

llama.cpp b9985 adds fused top-k MoE kernel for SYCL backends, controllable via `GGML_SYCL_ENABLE_FUSION`. • Adds `GGML_SYCL_ENABLE_FUSION` environment variable to enable fused top-k Mixture-of-Experts (MoE) kernel dispatch on SYCL backends, printed at startup alongside other env vars.

llama.cpp b9972

Sun, 12 Jul 2026 13:07:51 GMT

llama.cpp server gains streaming support for the exec_shell_command tool • Adds streaming output support for `exec_shell_command` in the llama.cpp server, enabling real-time token delivery when the shell tool is invoked.

llama.cpp b9970

Sun, 12 Jul 2026 12:03:02 GMT

llama.cpp b9970 adds GGML_OP_LIGHTNING_INDEXER for native DeepSeek V3.2 and V4 model support. • Adds `GGML_OP_LIGHTNING_INDEXER` ggml operation implementing the DeepSeek V3.2 and V4 lightning indexer, with f16 mask parameter and mask broadcasting support. • Enables use of `GGML_OP_LIGHTNING_INDEXER` in DeepSeek V3.2 and DeepSeek V4 model inference paths. • Bumps the RPC version to reflect the new `GGML_OP_LIGHTNING_INDEXER` operation.

llama.cpp b9968

Sun, 12 Jul 2026 01:12:46 GMT

llama.cpp b9968 adds int8 DP4 dense and MoE prefill optimizations for Adreno GPUs via OpenCL • Adds int8 DP4 dense GEMM and MoE GEMM prefill optimizations for Qualcomm Adreno GPUs via OpenCL, accelerating both dense and Mixture-of-Experts model inference on mobile hardware.

llama.cpp b9967

Sun, 12 Jul 2026 00:43:27 GMT

llama.cpp server now accepts null sampling params (temperature, top_p, …) to request server defaults, matching the OpenAI spec. • Server API now treats `null` values for nullable sampling parameters (e.g., `temperature`, `top_p`) as absent, falling back to server defaults — matching the OpenAI spec behavior.

llama.cpp b9963

Sat, 11 Jul 2026 08:38:28 GMT

llama.cpp b9963 adds DeepSeek-OCR v1 multi-tile dynamic resolution support with unified image preprocessors. • Adds DeepSeek-OCR v1 multi-tile dynamic resolution support in `mtmd`, unifying image preprocessors across DeepSeek-OCR v1 and v2.

llama.cpp b9957

Fri, 10 Jul 2026 20:17:38 GMT

llama.cpp b9957 improves server tool-use with a new `tools_io` abstraction and enhanced edit tool. • Adds `tools_io` and `tools_io_basic` abstraction layer to the server, giving tool-use I/O a dedicated interface. • Improves the server edit tool with enhanced functionality. • Removes `apply_diff` from the server tool set. Breaking changes: • The `apply_diff` tool has been removed from the server; any workflow relying on it will break on upgrade.

llama.cpp b9951

Fri, 10 Jul 2026 16:15:00 GMT

llama.cpp b9951 adds an initial ExecuTorch (ET) backend with hardware-accelerated kernels for inference on ET-SoC silicon. • Adds new `GGML_ET` backend (ExecuTorch) enabled at build time with `-DGGML_ET=ON`, supporting MUL_MAT, ROPE, RMS_NORM, GLU, SOFT_MAX, GET_ROWS, SET_ROWS, MUL_MAT_ID, FlashAttention, and many more operators offloaded to ET-SoC hardware. • Adds `-DGGML_ET_SYSEMU=ON` compile-time flag to enable ET-SoC emulator (sysemu) support for development and testing without physical hardware. • Supports Q4_0, Q4_K, Q8_0, F16, and F32 quantization formats in the ET backend, including matrix-engine (TensorFMA) accelerated paths for Q4_0 and Q8_0 MUL_MAT. • Supports kernel fusion in the ET backend, initially fusing RMS_NORM + MUL into a single accelerated pass. • Adds FlashAttention support (F32 and F16 via matrix engine) in the ET backend, enabling full offload of attention for models such as LLaMA 3.1 and Gemma 4. • Adds MROPE (multi-rope) operator support in the ET backend, broadening compatible model architectures. • Adds uberkernel support in the ET backend, batching multiple operator kernels into a single dispatch for reduced launch overhead. • Adds WKV*, SSM_CONV, SSM_SCAN, GATED_DELTA_NET, and GROUP_NORM operators to the ET backend, enabling inference for Mamba/RWKV-style and hybrid SSM architectures. • Embeds ET kernels directly into the build via CMake (with filesystem fallback), eliminating the need for runtime kernel import hacks. • Adds DeepSeek-V2-Lite RoPE test coverage to `test-backend-ops`. • Supports Gemma 4 model family via the ET backend.

llama.cpp b9949

Fri, 10 Jul 2026 10:37:26 GMT

llama.cpp b9949 adds cluster-parallel Flash Attention decode for Adreno GPUs via OpenCL. • Adds cluster-parallel decode Flash Attention for Adreno GPUs via the OpenCL backend, improving inference throughput on Qualcomm Adreno hardware.

llama.cpp b9947

Fri, 10 Jul 2026 04:46:59 GMT

llama.cpp b9947 adds `--output` option to the CLI. • Adds `--output` flag to the `llama-cli` command for directing output to a specified destination.

llama.cpp b9931

Wed, 08 Jul 2026 21:11:39 GMT

llama.cpp b9931 adds OpenCL ragged-tile MoE prefill GEMM optimizations with new `GGML_OPENCL_MOE_RAGGED_GRAN` env var control. • Adds `GGML_OPENCL_MOE_RAGGED_GRAN={8,16,32}` environment variable to control OpenCL MoE prefill GEMM tile-skip granularity (quarter/half/off), defaulting to quarter (8-column skip-groups) across all eight `*_f32_ns` MoE GEMMs. • Adds `GGML_OPENCL_MOE_RAGGED_FP16=0` opt-out environment variable to disable ragged-tile half-tile skipping for OpenCL MoE FP16 prefill GEMM. • New OpenCL ragged-tile MoE prefill optimization skips fully-padded expert tiles during GEMM, reducing wasted compute on low tokens-per-expert workloads while remaining numerically identical to the non-skipped path.

llama.cpp b9927

Wed, 08 Jul 2026 18:28:45 GMT

llama.cpp b9927 moves the CLI to an HTTP-based implementation with router mode support. • Migrates the `cli` to an HTTP-based implementation, enabling it to connect to a remote server and operate in router mode (multi-model routing). • Adds router mode support to the `cli`, including handling the single-model router edge case and displaying model aliases.

llama.cpp b9913

Wed, 08 Jul 2026 10:33:24 GMT

llama.cpp b9913 adds Q2_0 quantization type with CPU backend support. • Adds `Q2_0` quantization type definition and CPU backend, enabling a new 2-bit quantization format for model compression.

llama.cpp b9909

Wed, 08 Jul 2026 03:31:58 GMT

llama.cpp b9909 adds timings and progress fields to the /responses API stream endpoint. • Adds timings and progress data to the `/responses` API streaming endpoint, enabling clients to monitor inference performance and completion progress in real time.

llama.cpp b9902

Tue, 07 Jul 2026 17:15:46 GMT

SYCL backend gains support for cross_entropy_loss and cross_entropy_loss_back operations in llama.cpp b9902. • Adds `cross_entropy_loss` and `cross_entropy_loss_back` operator support to the SYCL backend, enabling GPU-accelerated training-related operations on Intel SYCL devices.

llama.cpp b9885

Mon, 06 Jul 2026 11:04:44 GMT

llama.cpp b9885 enables tiled matmul on AIX, delivering ~2x prompt-processing speed gains for FP32, Q4_0, and Q8_0 models. • Enables the `matmul_tiled` path on AIX by reducing stack buffer footprint for `A_pack` and `B_pack`, unlocking ~2x gains in prompt-processing speed for FP32, Q4_0, and Q8_0 models on that platform.

llama.cpp b9882

Mon, 06 Jul 2026 09:01:29 GMT

Download scripts now use `HF_TOKEN` to authenticate when fetching UI assets from Hugging Face. • Adds `HF_TOKEN` environment variable support to the UI asset download scripts, enabling authenticated downloads from Hugging Face.

llama.cpp b9874

Sun, 05 Jul 2026 15:53:37 GMT

llama.cpp b9874 adds CUDA concat support for quantized tensor types. • Adds CUDA `concat` kernel implementation for quantized types, enabling GPU-accelerated concatenation operations on quantized tensors without falling back to CPU.

llama.cpp b9867

Fri, 03 Jul 2026 14:56:03 GMT

llama.cpp b9867 adds `spec-draft-p-min` support to DFlash speculative decoding. • Adds `spec-draft-p-min` parameter support to DFlash speculative decoding, with `n_min` and `n_max` guards to bound draft token counts.

llama.cpp b9864

Fri, 03 Jul 2026 11:28:58 GMT

Adds `sse_ping_interval` as a per-request body field so slow prefill never drops healthy SSE connections • Adds `sse_ping_interval` as a typed per-request body field in the server's request schema, accepting positive integers or `-1` (disabled), with hard limits of `-1` to `INT32_MAX`; the global CLI default remains `30` so existing API clients see no behavior change. • Server now pings silent SSE streams every 1 second and kicks connections only after 3 seconds, preventing slow prefill from dropping healthy long-running connections. • WebUI automatically sends `sse_ping_interval: 1` in each request body to enforce the 3-second visibility-kick contract without requiring server-side reconfiguration.

llama.cpp b9860

Thu, 02 Jul 2026 16:23:41 GMT

llama.cpp b9860 adds llama_ftype_name() to the public C API for querying model quantization type names. • Adds llama_ftype_name() to the public C API, returning a human-readable quantization name (e.g. `Q8_0`, `Q4_K - Medium`) for a given `llama_ftype` enum value; the returned pointer is valid for the lifetime of the model and allocation-free.

llama.cpp b9859

Wed, 01 Jul 2026 18:10:10 GMT

OpenCL backend gains support for loading precompiled binary kernels from an external library, including q8_0, q4_0, q4_1, q4_k, and MoE MX-FP4 GEMM kernels. • Adds OpenCL support for loading precompiled binary kernels (including `gemm_moe_mxfp4_f32_ns`, q8_0 GEMM, q4_0/q4_1/q4_k MoE GEMM) from an external kernel library at runtime, enabling vendor-optimized kernel delivery without recompilation.

llama.cpp b9844

Tue, 30 Jun 2026 08:56:41 GMT

llama.cpp b9844 adds NVFP4 support to the WebGPU backend. • Adds NVFP4 quantization support to the `ggml-webgpu` backend, enabling NVIDIA FP4 model inference via WebGPU.

llama.cpp b9840

Mon, 29 Jun 2026 10:25:16 GMT

llama.cpp b9840 adds DeepSeek V4 model support including conversion, multi-sequence inference, and Flash Attention. • Adds `dsv4` architecture support to the `convert` tooling, enabling conversion of DeepSeek V4 (and Pro) models to GGUF format. • Supports DeepSeek V4 inference with Flash Attention (FA) enabled, graph reuse, multi-sequence support, and save/load state. • Adds compatibility with antirez-format DeepSeek V4 GGUFs, broadening which existing GGUF files can be loaded. • Uses `expert_gating_func` in place of `moe.score_func` for DeepSeek V4 MoE gating configuration.

llama.cpp b9837

Mon, 29 Jun 2026 00:05:43 GMT

llama.cpp b9837 adds `--reasoning-preserve` flag for controlling reasoning content in chat output. • Adds `--reasoning-preserve` flag to control whether reasoning/chain-of-thought content is preserved in chat and Jinja template output.

llama.cpp b9833

Sun, 28 Jun 2026 15:32:30 GMT

llama.cpp b9833 adds a dedicated MiniCPM5 tool-call parser with XML tool-call support via autoparser. • Adds a dedicated MiniCPM5 tool-call parser using the `openbmb-MiniCPM5-1B.jinja` template, with XML tool-call handling via autoparser and strict JSON argument parsing.

llama.cpp b9831

Sun, 28 Jun 2026 13:38:20 GMT

llama.cpp b9831 adds DFlash v2 support with per-layer sliding window attention. • Adds DFlash v2 support, including per-layer sliding window attention via `layer_types` configuration.

llama.cpp b9830

Sun, 28 Jun 2026 11:03:50 GMT

llama.cpp b9830 exposes `--offline` to `llama download` for air-gapped and cache-check workflows. • Adds `--offline` flag support to `llama download`, letting scripts verify whether a model is already cached locally without making any network requests.

llama.cpp b9828

Sat, 27 Jun 2026 23:15:52 GMT

OpenCL Flash Attention gains f16/f32/q4_0/q8_0 support with prefill prepass kernels and tile-skip optimizations. • Adds OpenCL Flash Attention prefill prepass kernels (`flash_attn_kv_pad_f16`, `flash_attn_mask_pad_f16`, `flash_attn_blk_f16`) that classify KV tiles as fully masked, mixed, or fully unmasked — enabling the main kernel to skip fully-masked tiles and bypass mask lookups for fully-unmasked ones. • Adds OpenCL FA kernels for `q4_0` and `q8_0` quantization formats, extending Flash Attention beyond f16/f32 on Adreno and compatible OpenCL backends. • Adds OpenCL `set_rows` operation supporting f32-to-`q8_0`/`q4_0` conversion. • Adds OpenCL dequantization kernels for `q4_0` and `q8_0`. • Adds an OpenCL FA tile tuning table with a per-device override mechanism.

llama.cpp b9821

Fri, 26 Jun 2026 21:55:22 GMT

llama.cpp b9821 adds `--version`, `--licenses`, and `--help` flags to app binaries. • Adds `--version`, `--licenses`, and `--help` flags to llama.cpp app binaries.

llama.cpp b9817

Fri, 26 Jun 2026 17:27:10 GMT

OpenVINO backend updated to OV 2026.2.1 with self-contained release packages and expanded operator support. • Updates OpenVINO backend to OV 2026.2.1 with self-contained release packages (no external dependencies required at runtime). • Adds GLU/SwiGLU OAI operator support (`glu_swiglu_oai`) to the OpenVINO backend. • Enables softmax with sink input in the OpenVINO backend. • Expands `add_id` in the OpenVINO backend to support 2D and 4D tensor shapes. • Optimizes `mul_mat_id` conversion processing in the OpenVINO backend for large sizes. • Removes hardcoded `compute_op_type` sets in the OpenVINO backend, allowing dynamic operator type resolution.

llama.cpp b9813

Fri, 26 Jun 2026 15:38:01 GMT

llama.cpp b9813 adds Vulkan cooperative matrix support for Intel Xe-LPG Plus (Xe1-ARLH) GPUs • Adds `INTEL_XE1` architecture enum to the Vulkan backend and enables `coopmat1` (cooperative matrix) acceleration on Intel Xe-LPG Plus (Xe1-ARLH) hardware, including a Windows driver compatibility check.

llama.cpp b9788

Thu, 25 Jun 2026 08:45:39 GMT

SYCL backend gains `--split-mode tensor` for dual-GPU tensor parallelism with BF16-compressed all-reduce • Adds `--split-mode tensor` support to the SYCL backend, enabling tensor parallelism across dual Intel Arc GPUs without OneCCL or new build dependencies. • Implements a two-path all-reduce: FP32 direct memcpy for small tensors (nelem < 32768) and BF16-compressed cross-device transfer for large tensors (nelem >= 32768), halving PCIe bandwidth for the common large-tensor case. • Delivers up to +78.6% token-generation throughput on Llama-3.3-70B Q4_K_M versus layer-split mode on dual Intel Arc Pro B70 hardware.

llama.cpp b9784

Wed, 24 Jun 2026 19:55:59 GMT

Hexagon NPU backend gains 32×32 tiled weight repack, kernel-param caching, and `GGML_HEXAGON_MM_SELECT` for matmul dispatch control. • Adds `GGML_HEXAGON_MM_SELECT` environment variable to control matmul kernel selection on Hexagon NPU, selectable from the host. • Introduces simple graph caching on the Hexagon backend to avoid recomputing kernel-params on repeated graph executions. • Implements 32×32 tiled weight repack layout for HMX matmul kernels, aligning tiles with DMA for higher throughput. • Adds non-tiled MUL_MAT fallback path on Hexagon, falling back to HVX flat kernels when HMX tiling is not viable. • Drops support for Hexagon arch < v73; HMX is now required for the backend — non-v73+ devices will no longer work. Breaking changes: • Support for Hexagon architecture versions below v73 has been removed; HMX is now required, so devices running arch < v73 will no longer function with the Hexagon backend.

llama.cpp b9777

Wed, 24 Jun 2026 07:43:17 GMT

llama.cpp b9777 adds support for LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M models. • Adds support for the LFM2.5-ColBERT-350M and LFM2.5-Embedding-350M model architectures.

llama.cpp b9768

Tue, 23 Jun 2026 10:48:33 GMT

llama.cpp b9768 adds conversion and inference support for Granite Speech Plus with multi-layer audio concatenation. • Adds conversion support for IBM Granite Speech Plus models, extending the `granite_speech` architecture to handle multi-layer (`feature_layers`) audio concatenation for richer speech understanding.

llama.cpp b9763

Mon, 22 Jun 2026 21:32:31 GMT

llama.cpp server now includes an `id` field in tool call responses. • Adds `id` field to tool call responses in the server API, enabling callers to correlate tool call requests and responses.

llama.cpp b9761

Mon, 22 Jun 2026 17:03:12 GMT

llama.cpp server gains real-time model download progress tracking via a new `/models/sse` SSE endpoint. • Adds `/models/sse` Server-Sent Events endpoint to the llama.cpp server for real-time model load and download progress tracking. • Moves model downloading to a dedicated child process in the server router, isolating download work from the main server process.

llama.cpp b9760

Mon, 22 Jun 2026 15:22:57 GMT

llama.cpp b9760 generalizes the server input file schema to support `input_video` and raw base64 intake. • Adds `input_video` field to the server input file schema, enabling video input alongside existing file types. • Server now accepts raw base64-encoded input directly in the input file schema.

llama.cpp b9748

Sun, 21 Jun 2026 17:09:23 GMT

llama.cpp b9748 adds a `verbose` field to the server schema for richer response metadata. • Adds `verbose` field to the llama.cpp server response schema, enabling clients to request detailed inference metadata.

llama.cpp b9747

Sun, 21 Jun 2026 13:51:11 GMT

llama.cpp b9747 adds real-time model load progress tracking via a new `/models/sse` SSE endpoint. • Adds `GET /models/sse` Server-Sent Events endpoint to the llama.cpp server for real-time model load progress tracking.

llama.cpp b9745

Sun, 21 Jun 2026 11:38:54 GMT

llama.cpp b9745 adds speculative decoding support for Step3.5/3.7 flash MTP via new `llama_set_mtp_layer_offset` and `llama_model_n_nextn_layer` APIs. • Adds `llama_set_mtp_layer_offset` and `llama_model_n_nextn_layer` API functions to support speculative multi-token prediction (MTP) for Step3.5 and Step3.7 flash models. • Adds `mtp_layer_offset` configuration and `nextn` flags with graph reuse support, enabling speculative multi-head draft and process pipelines for MTP blocks.

llama.cpp b9728

Fri, 19 Jun 2026 16:19:15 GMT

llama.cpp b9728 adds comment line support to `--api-key-file` for cleaner API key management. • Adds comment line support to `--api-key-file`, allowing inline documentation inside API key files.

llama.cpp b9726

Fri, 19 Jun 2026 15:15:50 GMT

llama.cpp server gains `--agent` argument for agentic inference modes. • Adds `--agent` argument to the `llama-server` CLI to enable agent-mode operation.

llama.cpp b9716

Fri, 19 Jun 2026 03:55:10 GMT

llama.cpp b9716 adds batching support for InternVL in the multimodal (mtmd) subsystem. • Adds batching support for InternVL models in the `mtmd` (multimodal) subsystem, enabling more efficient multi-image and batch inference with InternVL.

llama.cpp b9714

Fri, 19 Jun 2026 02:57:24 GMT

llama.cpp server now emits `X-Accel-Buffering: no` on streaming endpoints to prevent Nginx reverse-proxy buffering. • Adds `X-Accel-Buffering: no` response header to all streaming endpoints in `llama-server`, preventing Nginx from buffering streamed output when deployed behind a reverse proxy.

llama.cpp b9713

Fri, 19 Jun 2026 02:26:10 GMT

llama.cpp b9713 adds batching support and video tests to the mtmd-cli multimodal tool. • Adds batching support to `mtmd-cli`, enabling more efficient multimodal (image/video) inference throughput. • Adds video tests to the `mtmd` multimodal framework.

llama.cpp b9707

Fri, 19 Jun 2026 00:26:37 GMT

llama.cpp server gains a `schema` field with input validation for structured output enforcement. • Adds `schema` field to the llama.cpp server API for structured output, with validation that includes field-level error messages when constraints are violated.

llama.cpp b9699

Thu, 18 Jun 2026 09:53:46 GMT

SYCL backend gains MUL_MAT and OUT_PROD support for Q1_0 quantization format • Adds `MUL_MAT` and `OUT_PROD` operation support for the `Q1_0` quantization format in the SYCL backend.

llama.cpp b9689

Wed, 17 Jun 2026 19:31:23 GMT

Metal backend concat operator gains f16, bf16, i8, i16, and i64 tensor type support on Apple Silicon. • Extends the Metal backend `kernel_concat` operator to support `f16`, `bf16`, `i8`, `i16`, and `i64` tensor types in addition to the existing `f32` and `i32`, enabling broader model architectures to run GPU-accelerated on Apple Silicon.

llama.cpp b9688

Wed, 17 Jun 2026 18:55:06 GMT

llama.cpp b9688 adds a server-side model management API with download, delete, and SSE real-time update endpoints. • Adds a model management API to the llama.cpp server router, including endpoints to download models, delete models, and stream real-time status updates via SSE (Server-Sent Events).

llama.cpp b9675

Wed, 17 Jun 2026 11:05:39 GMT

llama.cpp b9675 extends SYCL FP16 support to SQR, SQRT, LOG, SIN, COS, and CLAMP operations. • Enables FP16 support in SYCL backend for the `SQR`, `SQRT`, `LOG`, `SIN`, `COS`, and `CLAMP` operations, broadening half-precision GPU acceleration on Intel SYCL devices.

llama.cpp b9673

Wed, 17 Jun 2026 07:32:41 GMT

llama.cpp b9673 adds `GGML_SYCL_USM_SYSTEM` env var to enable VRAM overcommit via USM system allocations on SYCL devices. • Adds `GGML_SYCL_USM_SYSTEM` environment variable to enable optional USM system allocations for large GPU buffers (≥ 1 GB) on SYCL devices, allowing VRAM overcommit by letting the OS manage host/device memory migration; falls back to regular allocations if unsupported.

llama.cpp b9669

Tue, 16 Jun 2026 12:25:36 GMT

llama.cpp b9669 adds backend sampling support for Eagle3 speculative decoding. • Adds backend sampling support for Eagle3, enabling hardware-accelerated speculative decoding with the Eagle3 draft model architecture.

llama.cpp b9665

Tue, 16 Jun 2026 10:11:28 GMT

llama.cpp b9665 adds `--offline` flag to the bench command. • Adds `--offline` flag to `bench` to run benchmarks without network access, with a default value included.

llama.cpp b9664

Tue, 16 Jun 2026 08:09:19 GMT

SYCL backend gains reordered Q4_K, Q5_K, and Q6_K MoE MUL_MAT_ID support for fused expert inference • Extends SYCL reordered-weight handling to fused MoE `MUL_MAT_ID` operations for `Q4_K`, `Q5_K`, and `Q6_K` expert tensors, enabling GPU-accelerated MoE inference with these quantization types on SYCL backends. • Adds `Q5_K` reordered DMMV (dequantize matrix-vector multiply) coverage on SYCL. • Unsupported 3D reorder cases now fall back gracefully instead of aborting.

llama.cpp b9654

Mon, 15 Jun 2026 19:24:18 GMT

llama.cpp b9654 adds a post-decode callback to the multimodal (mtmd) subsystem. • Adds a post-decode callback to the `mtmd` (multimodal) subsystem, enabling callers to hook into the decode pipeline after each multimodal token batch is processed.

llama.cpp b9647

Mon, 15 Jun 2026 15:50:05 GMT

SYCL backend gains `pool_1d` support in llama.cpp b9647 • Adds `pool_1d` operation support to the SYCL backend, with `pool_1d` and `pool_2d` code consolidated into `pool.cpp`/`pool.hpp`.

llama.cpp b9637

Sun, 14 Jun 2026 18:50:23 GMT

llama.cpp b9637 adds a dedicated chat parser for the Cohere2MoE (North Code) model. • Adds a dedicated Cohere2MoE (North Code) chat template parser for correct prompt formatting with that model family.

llama.cpp b9626

Sat, 13 Jun 2026 18:19:28 GMT

llama.cpp b9626 adds architecture support for the cohere2-MoE (cohere2moe) model family. • Adds `cohere2moe` architecture support, enabling inference for Cohere2 Mixture-of-Experts models (including North-Mini-Code-1.0) with iSWA-based sliding window attention, sigmoid expert selection, and LayerNorm support for Command variants. • Adds `cohere2moe` to the Llama Model Saver supported architecture list, enabling GGUF conversion of cohere2moe models.

llama.cpp b9624

Sat, 13 Jun 2026 16:06:09 GMT

llama.cpp b9624 adds build-time gzip compression to the web UI for faster asset delivery. • Adds build-time gzip compression for web UI assets, reducing payload size for browser clients.

llama.cpp b9606

Fri, 12 Jun 2026 08:45:47 GMT

llama.cpp b9606 adds EAGLE3 speculative decoding support, including Gemma4 EAGLE3 models from RedHatAI. • Adds EAGLE3 speculative decoding support, enabling faster inference via a third-generation draft model architecture, with support for Gemma4 EAGLE3 models from RedHatAI. Breaking changes: • The `output_layer_inp` cparams field is renamed to `embeddings_layer_inp`.

llama.cpp b9603

Fri, 12 Jun 2026 05:16:00 GMT

llama.cpp b9603 adds q5_0 and q5_1 GEMM/GEMV OpenCL kernels for Qualcomm Adreno GPUs. • Adds `q5_0` and `q5_1` GEMM and GEMV OpenCL kernels for Qualcomm Adreno GPUs, enabling accelerated inference with 5-bit quantized models on Adreno hardware.

llama.cpp b9577

Tue, 09 Jun 2026 12:09:50 GMT

llama.cpp b9577 adds `--log-prompts-dir` to capture every server prompt to disk for auditing. • Adds `--log-prompts-dir` flag to the server to write each incoming prompt to a separate text file in a specified directory, enabling prompt logging and audit trails.

llama.cpp b9575

Tue, 09 Jun 2026 11:42:17 GMT

llama.cpp b9575 adds GGML_OP_COL2IM_1D, enabling efficient 1D transposed convolution for vocoder-style audio models. • Adds `GGML_OP_COL2IM_1D` CPU operator (F32, F16, BF16 with F32 accumulator) implementing the overlap-add scatter step of a 1D transposed convolution, allowing ConvTranspose1d to be factorized as a plain `mul_mat` followed by `col2im_1d` so the heavy work stays on optimized, quantizable matmul kernels. • Bumps `RPC_PROTO_PATCH_VERSION` to cover the new op (`GGML_OP_COUNT` rises from 96 to 97); the op is appended so no existing op codes shift.

llama.cpp b9568

Mon, 08 Jun 2026 21:10:42 GMT

llama.cpp b9568 adds MTP support for Gemma-4 E2B and E4B assistant models. • Adds multi-token prediction (MTP) support for Gemma-4 E2B and E4B assistant model variants, including `masked_embd` tensors in the `gemma4-assist` architecture and converter updates to handle `masked_embedding` tensor filtering during conversion.

llama.cpp b9562

Mon, 08 Jun 2026 16:41:04 GMT

llama.cpp b9562 adds video input support via `--video` CLI arg, base64 server input, and `MTMD_VIDEO` config. • Adds `--video` CLI argument to enable video input for multimodal inference from the command line, with auto-completion support. • Adds `MTMD_VIDEO` configuration option to control video input support. • Supports video input on the server via base64-encoded input payloads. • Introduces `mtmd_helper_video` helper for video handling in the multimodal (`mtmd`) subsystem.

llama.cpp b9556

Mon, 08 Jun 2026 13:48:00 GMT

llama.cpp b9556 adds HIP support for AMD RDNA3.5 GPUs gfx1152 and gfx1153. • Adds `gfx1152` and `gfx1153` GPU targets to the HIP backend under the RDNA3.5 architecture class, enabling GPU-accelerated inference on those AMD chips.

llama.cpp b9549

Sun, 07 Jun 2026 13:38:17 GMT

llama.cpp b9549 adds Gemma4 MTP (multi-token prediction) support. • Adds Gemma4 MTP (multi-token prediction) support to the llama model backend.

llama.cpp b9547

Sun, 07 Jun 2026 09:38:51 GMT

llama.cpp b9547 skips redundant mmproj downloads when a user-supplied mmproj is already provided. • Skips automatic `mmproj` download when the user has already supplied an `mmproj` via argument, avoiding redundant network fetches.

llama.cpp b9543

Sat, 06 Jun 2026 19:48:45 GMT

llama.cpp b9543 adds video support for Qwen3.5 via 'frame merge' for Qwen-VL-based models. • Adds 'frame merge' multimodal video support for Qwen-VL-based models, enabling video input for Qwen3.5 via the `mtmd` layer.

llama.cpp b9522

Fri, 05 Jun 2026 07:44:17 GMT

llama.cpp b9522 adds dynamic chunk-based scheduling for hybrid CPU/GPU execution via KleidiAI. • Adds dynamic chunk-based scheduling for hybrid execution in the KleidiAI backend, enabling more efficient mixed CPU/GPU inference workloads.

llama.cpp b9521

Fri, 05 Jun 2026 06:54:55 GMT

CUDA PDL enrollment for MoE matrix-vector ops boosts MTP speculative decoding throughput by ~5–6% on Blackwell GPUs. • Enrolls `mul_mat_vec_q_moe` into CUDA PDL (Programmatic Dependent Launch), improving Multi-Token Prediction (MTP) speculative decoding throughput by roughly 5–6% on Blackwell-architecture GPUs (e.g. B4500), with tok/s gains observed across code generation, summarization, QA, translation, and math workloads.

llama.cpp b9499

Thu, 04 Jun 2026 06:08:27 GMT

llama.cpp b9499 refactors WebGPU FlashAttention and standardizes quantization support for k/v tensors on the tile path. • Refactors the WebGPU FlashAttention implementation with split k/v quantization and abstracted quantization logic shared between `flash_attn` and `mul_mat` operations. • Adds quantization support to the WebGPU tile path for FlashAttention, broadening the model formats that can run accelerated attention on WebGPU backends.

llama.cpp b9494

Wed, 03 Jun 2026 17:35:34 GMT

llama.cpp b9494 enables non-causal vision support for Gemma 4 unified multimodal inference. • Enables non-causal vision attention in the `mtmd` (multimodal) subsystem for Gemma 4 unified model inference.

llama.cpp b9489

Wed, 03 Jun 2026 11:22:25 GMT

llama.cpp b9489 reserves CUDA memory for quantized KV-cache at startup to avoid runtime allocation failures. • CUDA backend now reserves GPU memory for quantized KV-cache at startup, preventing runtime out-of-memory errors when using KV-cache quantization.

llama.cpp b9488

Wed, 03 Jun 2026 07:47:42 GMT

llama.cpp b9488 adds Qwen3 SSM architecture support and a new `LLM_KV_ATTENTION_RECURRENT_LAYERS` key. • Adds `LLM_KV_ATTENTION_RECURRENT_LAYERS` architecture key to support recurrent layer configuration in GGUF models. • Extends test coverage to include Qwen3 SSM (state-space model) architectures.

llama.cpp b9482

Tue, 02 Jun 2026 19:43:12 GMT

llama.cpp b9482 adds support for the Mellum model architecture. • Adds support for the Mellum model architecture, enabling inference on Mellum-family models.

llama.cpp b9481

Tue, 02 Jun 2026 16:57:07 GMT

llama.cpp b9481 adds support for IBM Granite multilingual embedding R2 models and a new `.hidden_activation` GGUF key. • Adds new GGUF key `.hidden_activation` (internal key `LLM_KV_HIDDEN_ACT`) and a writer, enabling models to declare their hidden-layer activation function (e.g. SwiGLU) in the GGUF metadata. • Adds llm_ffn_op_type_from_string() helper and a new `LLM_FFN_NONE = 0` sentinel to `llm_ffn_op_type`, centralising hidden-activation mapping in `llama-model.cpp`. • Adds support for `ibm-granite/granite-embedding-97m-multilingual-r2` and `ibm-granite/granite-embedding-311m-multilingual-r2` embedding models, including SwiGLU FFN, a fixed-regex GPT-4o tokenizer variant (97m), and the Gemma4 tokenizer (311m). • Extends the GGUF converter to emit `hidden_activation` from `config.json` and adds model hashes for the Granite Embedding Multilingual R2 models.

llama.cpp b9480

Tue, 02 Jun 2026 16:23:58 GMT

llama.cpp b9480 adds support for the StepFun 3.5 MTP model architecture. • Adds support for the StepFun 3.5 MTP model architecture.

llama.cpp b9478

Tue, 02 Jun 2026 13:01:31 GMT

llama.cpp server gains configurable SSE ping interval to keep streaming connections alive. • Adds SSE ping interval configuration to the `server`, allowing clients to maintain long-lived streaming connections without timing out.

llama.cpp b9474

Tue, 02 Jun 2026 11:21:34 GMT

llama.cpp b9474 adds a Thinking mode toggle with reasoning effort levels to the chat UI. • Adds a 'Thinking' toggle and status icon to the chat UI, enabling chain-of-thought / reasoning mode for supported models. • Adds a reasoning effort levels selector in the model dropdown, letting users tune how much reasoning the model applies per request. • Adds a chat template thinking detector utility and thinking support detection to the models store, so the UI automatically surfaces thinking controls only for capable models. • Makes the server origin configurable via an environment variable for the UI build.

llama.cpp b9468

Tue, 02 Jun 2026 05:53:36 GMT

llama.cpp b9468 adds a `POST /v1/chat/completions/control` endpoint to interrupt model reasoning mid-generation in real time. • Adds `reasoning_control` opt-in flag on completion slots that arms the budget sampler on demand, enabling the new control endpoint to call `common_sampler_reasoning_budget_force` and halt thinking at any point during streaming. • Web UI gains a 'skip' button that appears only during the active reasoning phase (tracked via `isReasoning` streaming state), posts the streamed completion id back to the control endpoint, and is wired through the agentic flow.

llama.cpp b9459

Mon, 01 Jun 2026 18:58:46 GMT

Metal GLU kernels now support f16 inputs, reducing memory bandwidth on Apple Silicon. • Metal GLU kernels (geglu/swiglu) now accept f16 tensor inputs in addition to f32, loading and storing in the native tensor type to reduce memory bandwidth while keeping ALU compute in float for numerical stability.

llama.cpp b9455

Mon, 01 Jun 2026 17:20:46 GMT

llama.cpp b9455 adds quantized KV cache support for tensor parallelism (TP) deployments. • Adds quantized KV cache support for tensor parallelism (TP) configurations, enabling reduced memory footprint in multi-GPU inference setups.

llama.cpp b9453

Mon, 01 Jun 2026 14:56:10 GMT

llama.cpp b9453 adds EXAONE 4.5 model support including multimodal vision capabilities. • Adds EXAONE 4.5 model support with GQA and multimodal vision via `` and `` image boundary markers, routed through the Qwen2.5-VL-style encode path with window attention and optional mmproj input norm. • Adds `convert_hf_to_gguf` support for EXAONE 4.5 mmproj export, enabling conversion of EXAONE 4.5 vision projector weights to GGUF format.

llama.cpp b9442

Sun, 31 May 2026 11:07:43 GMT

llama.cpp b9442 adds tokenizer support for jina-embeddings-v2-base-zh with a whitespace tokenizer. • Adds whitespace tokenizer support for `jina-embeddings-v2-base-zh`, with `lowercase` defaulting to `true`.

llama.cpp b9437

Sat, 30 May 2026 20:56:44 GMT

llama-bench gains `-fa auto` support and `-ngl` now defaults to -1. • Adds `-fa auto` option to `llama-bench`, enabling automatic flash-attention mode selection during benchmarking. • Changes the default value of `-ngl` in `llama-bench` to `-1`, aligning it with the behavior of other llama.cpp tools.

llama.cpp b9436

Sat, 30 May 2026 17:43:37 GMT

llama.cpp b9436 adds bf16 support for OpenCL backend via f16 conversion • Adds bf16 support to the OpenCL backend by converting to f16 at runtime, enabling bf16 model inference on OpenCL-capable GPUs.

llama.cpp b9415

Fri, 29 May 2026 23:36:59 GMT

llama.cpp b9415 adds a `skip_download` option to prevent automatic model downloads. • Adds `skip_download` flag to the download subsystem, allowing operators to prevent automatic model fetching — including when the target file does not exist locally.

llama.cpp b9414

Fri, 29 May 2026 21:01:24 GMT

llama.cpp b9414 adds DeepSeek-OCR 2 support with multi-tile dynamic resolution in the mtmd subsystem. • Adds DeepSeek-OCR 2 model support in `mtmd` with multi-tile dynamic resolution via `tools/mtmd/models/deepseekocr2.cpp`.

llama.cpp b9411

Fri, 29 May 2026 15:30:44 GMT

llama.cpp b9411 adds DeepSeek V3.2 model support with sparse attention and NVFP4 quantization. • Adds support for the `DeepseekV32ForCausalLM` architecture (`LLM_ARCH_DEEPSEEK32`) with a generic DeepSeek Sparse Attention (DSA) implementation including a lightning indexer cache. • Adds `convert` tooling support for the `DeepseekV32ForCausalLM` architecture to enable GGUF conversion of DeepSeek V3.2 model weights. • Adds NVFP4 quantization support for DeepSeek V3.2 models. • Adds `llama_kv_cache_dsa` memory type combining KV cache with a lightning indexer cache to accelerate DSA inference. • Adds f16 support for the `GGML_OP_FILL` operation in the ggml backend.

llama.cpp b9391

Fri, 29 May 2026 02:52:29 GMT

llama.cpp b9391 adds `LLAMA_ARG_API_KEY_FILE` env var to supply API keys via file instead of inline flag. • Adds `LLAMA_ARG_API_KEY_FILE` environment variable as an alternative to passing `--api-key-file` directly, letting operators supply API keys via file path through the environment.

llama.cpp b9389

Fri, 29 May 2026 02:17:02 GMT

llama.cpp b9389 automatically applies the iGPU flag for integrated CUDA/HIP devices • Automatically applies the integrated GPU (`iGPU`) flag for CUDA and HIP backends when an integrated device is detected, removing the need for manual configuration.

llama.cpp b9387

Fri, 29 May 2026 01:04:49 GMT

llama.cpp b9387 adds per-quantization MMQ dispatch thresholds on AMD MFMA hardware, yielding up to +76% prompt throughput. • Introduces ggml_cuda_should_use_mmvq(type, cc, ne11) to apply per-quantization MMVQ/MMQ batch thresholds on AMD CDNA (MFMA) hardware, replacing the single global `MMVQ_MAX_BATCH_SIZE = 8` cutoff: K-quants `Q3_K`/`Q4_K`/`Q5_K` switch to MMQ at batch≥4, `Q2_K`/`Q6_K` at batch≥6, legacy and IQ quants retain the original threshold — delivering up to +76% pp512 throughput on MI250X. • Non-AMD-MFMA paths (NVIDIA, RDNA, CDNA1 without MFMA) are unchanged by this dispatch logic update.

llama.cpp b9383

Thu, 28 May 2026 19:56:56 GMT

llama.cpp b9383 adds the Granite 4.1 chat template for IBM Granite model support. • Adds Granite 4.1 chat template, enabling correct prompt formatting for IBM Granite 4.1 models.

llama.cpp b9380

Thu, 28 May 2026 17:42:35 GMT

llama-server gains HTTP ETag support for UI asset caching in b9380. • Adds HTTP ETag support to `llama-server`, enabling browser-side caching of UI elements via FNV hash-based ETags set on every response.

llama.cpp b9370

Wed, 27 May 2026 18:23:57 GMT

llama.cpp b9370 adds Q4_1 quantization support for Hexagon MUL_MAT and MUL_MAT_ID operations via HVX and HMX. • Adds Q4_1 quantization support to Hexagon `MUL_MAT` and `MUL_MAT_ID` operations, with both HVX-only and HMX paths, enabling the Hexagon backend to offload nearly the entire compute graph. • Adds early-wake support and op-batch completion polling on the Hexagon backend, reducing inference latency when the backend claims full graph ownership.

llama.cpp b9366

Wed, 27 May 2026 17:32:08 GMT

llama.cpp b9366 adds Vulkan REPEAT op support for f16-to-f16 tensors. • Adds Vulkan GPU pipeline support for the REPEAT operation on f16-to-f16 tensor types, enabling models that use this op to run fully on Vulkan backends.

llama.cpp b9354

Wed, 27 May 2026 06:00:03 GMT

llama.cpp b9354 adds MiniCPM5 tokenizer support for GGUF conversion • Adds MiniCPM5 pre-tokenizer support to `convert_hf_to_gguf_update.py`, enabling conversion of MiniCPM5 models to GGUF format with correct BPE tokenization via hardcoded regex handling in `llama-vocab.cpp`.

llama.cpp b9319

Mon, 25 May 2026 23:07:38 GMT

llama.cpp b9319 adds `gguf_init_from_callback` and `gguf_init_from_buffer` for flexible GGUF loading from memory and streaming sources. • Adds `gguf_init_from_buffer` to initialize a GGUF context directly from an in-memory buffer, enabling model loading without file I/O. • Adds `gguf_init_from_callback` with a `gguf_reader_callback_t` callback type to stream GGUF data chunk-by-chunk, with `max_chunk_read == 0` meaning `SIZE_MAX` (unlimited chunk size) and `max_expected_size` and offsets typed as `uint64_t` for large-model safety.

llama.cpp b9310

Mon, 25 May 2026 11:50:27 GMT

llama.cpp b9310 adds `--checkpoint-min-step` flag and autoparser message-barrier detection for smarter server checkpointing. • Adds `--checkpoint-min-step` CLI flag to control minimum spacing between server context checkpoints. • Adds `common_chat_split_by_role` to split chat output by role, enabling message-span extraction from chat templates. • Supports autoparser detection for message barriers, allowing the server to automatically identify prompt split points without manual configuration. • Enables multimodal prompt support when mapping text/template positions to server prompt tokens during checkpoint creation.

llama.cpp b9291

Fri, 22 May 2026 22:17:36 GMT

llama.cpp b9291 improves SYCL MoE prefill throughput via counting sort-based expert routing. • Improves SYCL MoE prefill throughput by replacing an O(n_as * n_routed_rows) expert-routing loop in `k_copy_src1_to_contiguous` with a counting sort-based procedure of O(n_as + n_routed_rows) complexity, enabling faster batch inference on SYCL backends.

llama.cpp b9279

Fri, 22 May 2026 01:51:39 GMT

Vulkan backend gains fused snake activation kernel for audio decoder models (BigVGAN, Vocos) across F32/F16/BF16. • Adds a fused Vulkan kernel (`snake.comp`) that collapses the 5-op snake activation decomposition (mul, sin, sqr, mul, add) into a single elementwise dispatch via `ggml_vk_snake_dispatch_fused`, accelerating audio decoder models such as BigVGAN and Vocos on Vulkan backends across F32, F16, and BF16 precisions.

llama.cpp b9276

Fri, 22 May 2026 00:48:34 GMT

llama.cpp server now exposes prompt token counts in the /slots endpoint for client-side progress monitoring. • Adds `n_prompt_tokens`, `n_prompt_tokens_processed`, and `n_prompt_tokens_cache` fields to the `GET /slots` JSON response, enabling clients to monitor prompt evaluation progress during processing.

llama.cpp b9272

Thu, 21 May 2026 20:55:33 GMT

llama.cpp b9272 adds batched-bench, fit-params, quantize, and perplexity as bundled app commands. • Adds `batched-bench`, `fit-params`, `quantize`, and `perplexity` as built-in app subcommands.

llama.cpp b9270

Thu, 21 May 2026 19:52:00 GMT

llama.cpp b9270 adds native HybridDNATokenizer support for HuggingFaceBio Carbon models via a new `LLAMA_VOCAB_TYPE_HYBRIDDNA` vocab type. • Adds `LLAMA_VOCAB_TYPE_HYBRIDDNA = 7` in `include/llama.h` as a first-class vocab type (peer to SPM/WPM/UGM/RWKV/PLAMO2) for the HybridDNATokenizer used by `HuggingFaceBio/Carbon-{500M,3B,8B}`. • Adds `llm_tokenizer_hybriddna` and its session in `src/llama-vocab.cpp`, routing text through a DNA-aware splitter that chunks sequences inside `...` regions into fixed 6-mers (right-padded with `A`), maps non-ACGT bases to ``, and delegates non-DNA text to an embedded BPE tokenizer. • Adds `_set_vocab_hybriddna` in `conversion/base.py` that writes `tokenizer.ggml.model = 'hybriddna'` during GGUF conversion, dispatched from `conversion/llama.py` when `tokenizer_config.json["tokenizer_class"] == "HybridDNATokenizer"`. Breaking changes: • The short-lived `LLAMA_VOCAB_PRE_TYPE_CARBON` (introduced during this same development branch) is removed from `src/llama-vocab.h` in favour of `LLAMA_VOCAB_TYPE_HYBRIDDNA`; any code referencing `LLAMA_VOCAB_PRE_TYPE_CARBON` will fail to compile.

llama.cpp b9263

Thu, 21 May 2026 04:14:01 GMT

llama.cpp b9263 merges HunyuanOCR into the HunyuanVL projector path, restoring correct vision precision. • Merges HunyuanOCR into the `HUNYUANVL` projector and `HUNYUAN_VL` text architecture, consolidating OCR and vision-language into a single code path with the +0.1 bilinear sampler applied correctly.

llama.cpp b9254

Wed, 20 May 2026 21:38:57 GMT

llama.cpp b9254 adds Programmatic Dependent Launch (PDL) for NVIDIA Hopper+ GPUs, toggled via `GGML_CUDA_PDL`. • Adds `GGML_CUDA_PDL` environment variable to enable or disable Programmatic Dependent Launch (PDL) on NVIDIA Hopper+ GPUs; set `GGML_CUDA_PDL=0` to disable. • Introduces PDL support for NVIDIA Hopper+ (SM90+) GPUs, overlapping CUDA kernel execution to improve inference throughput; enabled by default on qualifying devices, excluded on Ada and below.

llama.cpp b9253

Wed, 20 May 2026 16:35:15 GMT

llama.cpp b9253 introduces a unified `llama` executable with a `serve` subcommand for the server. • Introduces the `llama` unified executable, consolidating multiple tools under a single binary with subcommands including `serve` (for the server) and `help`.

llama.cpp b9244

Wed, 20 May 2026 03:39:52 GMT

llama.cpp b9244 adds OpenCL MoE support for q4_k, q5_k, and q6_k quantizations on Adreno GPUs. • Adds OpenCL Mixture-of-Experts (MoE) inference support for `q4_k`, `q5_k`, and `q6_k` quantization formats on Adreno GPUs.

llama.cpp b9180

Sat, 16 May 2026 16:48:07 GMT

llama.cpp b9180 adds Multi-Token Prediction (MTP) speculative decoding support across CPU, Metal, and Vulkan backends. • Adds MTP (Multi-Token Prediction) support for speculative decoding, with draft models identified by the `draft-mtp` prefix in model naming. • Adds partial sequence rollback for GDN (Gated Delta Net) recurrent models during speculative decoding, storing intermediates up to `draft_max` tokens to avoid full restart on draft rejection. • Extends Metal backend with GDN partial rollback support via a new kernel function constant `K` (snapshot slot count) controlling the 3D state tensor slot behavior. • Extends Vulkan backend with GDN partial rollback support for speculative decoding. • Enables checkpointing with partial rollback in `llama-memory` for recurrent model contexts. • Server disables RS-based MTP when other speculative decoding types are active, preventing incompatible combinations.

llama.cpp b9174

Sat, 16 May 2026 02:21:24 GMT

llama.cpp b9174 renames WebUI surfaces to UI: new `--ui`, `--ui-config`, `--ui-config-file`, and `--ui-mcp-proxy` flags plus matching env vars. • Adds `--ui` / `--no-ui` CLI flags (replacing deprecated `--webui` / `--no-webui`) to control the built-in web interface. • Adds `--ui-config` CLI flag (replacing deprecated `--webui-config`) for passing UI configuration. • Adds `--ui-config-file` CLI flag (replacing deprecated `--webui-config-file`) for specifying the UI config file path. • Adds `--ui-mcp-proxy` / `--no-ui-mcp-proxy` CLI flags (replacing deprecated `--webui-mcp-proxy`) to toggle MCP proxy support in the UI. • Adds environment variables `LLAMA_ARG_UI`, `LLAMA_ARG_UI_CONFIG`, `LLAMA_ARG_UI_CONFIG_FILE`, and `LLAMA_ARG_UI_MCP_PROXY` as alternatives to the corresponding CLI flags. • Renames CMake build variable `LLAMA_BUILD_WEBUI` to `LLAMA_BUILD_UI` and `LLAMA_USE_PREBUILT_WEBUI` to `LLAMA_USE_PREBUILT_UI` (old names kept as deprecated aliases). • Moves UI source tree from `tools/server/public` / `tools/server/webui/` to `tools/ui/` and build output to `build/tools/ui/dist`. • JSON API now emits both `ui` / `ui_settings` and legacy `webui` / `webui_settings` keys for backward compatibility.

llama.cpp b9161

Fri, 15 May 2026 11:50:35 GMT

llama.cpp b9161 adds Codex CLI compatibility by gracefully skipping unsupported Responses API tools. • Adds support for Codex CLI by skipping unsupported Responses tools instead of failing, with a warning emitted for each skipped tool.

llama.cpp b9158

Thu, 14 May 2026 23:24:14 GMT

llama.cpp b9158 adds RDNA3 tensor core support for flash attention and extends CDNA head size support to 256. • Adds RDNA3 tensor core (MMA) support to the CUDA flash attention kernel, using FP16 accumulation with 32-unit tiles for head sizes evenly divisible by 32. • Extends CDNA1 flash attention support to attention head sizes up to 256. • Improves AMD matrix transposition efficiency for RDNA3 and RDNA4 via wider 32-unit tiles that align with the warp size of 32. • Adds a new `ggml_cuda_mma::data_layout` entry to guard against accumulator data layout misuse introduced by the new RDNA3/4 tiling scheme. • Tunes flash attention kernel parameters for RDNA3, RDNA4, and CDNA1 architectures.

llama.cpp b9145

Thu, 14 May 2026 08:13:42 GMT

SYCL backend gains Level Zero memory allocation to eliminate multi-GPU host RAM mirroring on Intel Arc systems. • Adds `GGML_SYCL_ENABLE_LEVEL_ZERO` runtime environment variable (default `1`) to switch the SYCL backend between Level Zero (`zeMemAllocDevice`) and standard SYCL (`sycl::malloc_device`) memory APIs, eliminating the 1:1 VRAM-to-host-RAM mirroring that caused OOM crashes on multi-GPU Intel Arc systems. • Adds `GGML_SYCL_SUPPORT_LEVEL_ZERO` CMake build option (default `ON`) to compile in Level Zero support when the SDK headers and loader are present, with automatic fallback to the SYCL allocation path when Level Zero is unavailable. • Adds Windows Level Zero SDK path detection via `LEVEL_ZERO_V1_SDK_PATH` in `CMakeLists.txt` for building with Level Zero on Windows. • Level Zero allocation path automatically disables itself with a warning at runtime if the detected devices do not support the Level Zero backend, preserving compatibility on non-dGPU or CPU-only systems.

llama.cpp b9142

Thu, 14 May 2026 01:34:43 GMT

OpenCL backend gains q5_0 and q5_1 MoE quantization support for Adreno GPUs. • Adds `q5_0` and `q5_1` MoE (Mixture-of-Experts) kernel support to the OpenCL backend, enabling quantized MoE inference on Adreno GPUs.

llama.cpp b9141

Thu, 14 May 2026 01:12:22 GMT

llama.cpp b9141 adds `continue_final_message` body flag for vLLM/transformers API compatibility in the server and WebUI. • Adds `continue_final_message` request body flag to the server API; when set with `add_generation_prompt: false`, it triggers the assistant prefill code path — aliasing the existing `prefill_assistant` heuristic for vLLM/transformers API compatibility. • Enforces mutual exclusion between `continue_final_message: true` and `add_generation_prompt: true`, returning HTTP 400 on conflict, matching vLLM/transformers behavior. • WebUI 'Continue' button now sends `continue_final_message` and `add_generation_prompt: false` automatically, with a matching opt-in option on the chat service.

llama.cpp b9133

Wed, 13 May 2026 13:49:25 GMT

llama.cpp server and WebUI now support continuing generation on reasoning models with chain-of-thought preservation. • Server and WebUI support continued generation on reasoning models by orchestrating `thinking_start_tag` / `thinking_end_tag` pairs around prefilled messages, so the stream parser routes chunks correctly after a resume. • Server rejects reasoning prefill on channel-based templates (e.g. GPT-OSS) with an explicit error, limiting scope to simple thinking-tag templates.

llama.cpp b9129

Wed, 13 May 2026 11:12:27 GMT

llama.cpp b9129 adds `GGML_ZENDNN_ADAPTIVE_FALLBACK` env var to control ZenDNN-to-CPU fallback for small batch sizes. • Adds `GGML_ZENDNN_ADAPTIVE_FALLBACK` environment variable (default: enabled) to control whether the ZenDNN backend adaptively falls back to the CPU backend for small batch sizes.

llama.cpp b9127

Tue, 12 May 2026 22:29:55 GMT

llama.cpp b9127 adds opt-in Adreno cross-memory F16xF32 GEMM kernel for faster prefill on OpenCL. • Adds opt-in Adreno cross-memory (xmem) F16xF32 GEMM kernel for the OpenCL backend, accelerating prefill on Adreno GPUs.

llama.cpp b9124

Tue, 12 May 2026 19:18:32 GMT

llama.cpp b9124 exposes model modalities via `mtmd_caps` on the `/v1/models` API endpoint. • Adds `mtmd_caps` field to the `/v1/models` API response, exposing the modalities (e.g. text, vision) supported by a loaded model.

llama.cpp b9123

Tue, 12 May 2026 19:09:12 GMT

WebGPU backend gains support for running the gpt-oss-20b model via a refactored mulmat-q kernel. • Enables running the `gpt-oss-20b` model on the WebGPU backend (`ggml-webgpu`), expanding large-model support without CUDA or Vulkan.

llama.cpp b9116

Tue, 12 May 2026 12:46:31 GMT

llama.cpp b9116 adds multimodal vision support for MiMo v2.5 models. • Adds vision (multimodal) support for MiMo v2.5 models via the `mtmd` multimodal subsystem, including fused QKV for the vision encoder.

llama.cpp b9109

Mon, 11 May 2026 21:12:24 GMT

llama.cpp b9109 adds parallel speculative drafting with support for multiple chained speculators in the server and CLI. • Introduces common_speculative_type_from_names() to parse user-provided speculator types as a vector, enabling multiple speculative decoding strategies to be specified at once. • Introduces common_get_enabled_speculative_impls() to determine which speculative decoding implementations are active based on the provided type vector. • Introduces common_speculative_process() as a unified entry point for running multiple speculators sequentially, selecting the best draft by maximizing expected accepted tokens. • Replaces the single `type` field in `common_params_speculative` with a vector of `common_speculative_type` values, allowing multiple speculator types to be configured in one pass. • Adds parallel drafting support to the server, enabling concurrent speculative decoding across multiple request slots via a shared draft context. • Adds draft prompt cache and checkpoints to the server for more efficient speculative decoding across slots. • Supports chaining multiple speculators where all run sequentially and the best-performing draft (by expected accepted tokens) is verified and committed. • Reuses device buffers across draft and main contexts when possible, reducing memory overhead during parallel speculative decoding. Breaking changes: • The `type` field of type `common_speculative_type` in `common_params_speculative` is replaced with a vector of `common_speculative_type`; any code or config directly setting a single `type` value will need to be updated to use a vector.

llama.cpp b9095

Sun, 10 May 2026 09:43:20 GMT

llama.cpp b9095 adds a NCCL-free internal AllReduce CUDA kernel for 2-GPU tensor parallelism, selectable via `GGML_CUDA_ALLREDUCE` or `--reduction-provider`. • Adds `GGML_CUDA_ALLREDUCE` environment variable accepting `'nccl'` or `'internal'` to select the AllReduce provider for `LLAMA_SPLIT_MODE_TENSOR` multi-GPU inference. • Adds `--reduction-provider` / `-rp` flag to `llama-bench` accepting `auto`, `nccl`, or `internal` to control the AllReduce provider per benchmark run. • Adds `GGML_CUDA_NCCL_STATIC` CMake option (default `OFF`) to switch NCCL linking between static and dynamic, with updated `FindNCCL.cmake` searching Windows source-build layouts including `cmake/lib/Release`, `cmake/src/Release`, and `src/include`. • Introduces a NCCL-free internal AllReduce CUDA kernel for 2-GPU FP32 tensor parallelism (tensors up to 256 KB) that pipelines D2H copy, cross-GPU handshake via pinned-memory volatile flags, and reduction in a single kernel launch per GPU — no NCCL dependency required. • Adds an optional AllReduce hang watchdog (enabled by setting `#define GGML_CUDA_AR_WATCHDOG 1` in `allreduce.cu`) with runtime tuning via `GGML_CUDA_AR_WATCHDOG` (ms timeout) and `GGML_CUDA_AR_MAX_SPIN` (kernel bailout) environment variables; zero overhead when disabled.

llama.cpp b9093

Sat, 09 May 2026 21:02:12 GMT

llama.cpp b9093 adds support for the Sarvam MoE model architecture. • Adds `sarvam_moe` architecture support, enabling inference on Sarvam mixture-of-experts models.

llama.cpp b9082

Fri, 08 May 2026 22:21:07 GMT

llama.cpp b9082 adds an L2_NORM HVX kernel for the Hexagon backend. • Adds an `L2_NORM` HVX kernel for the Hexagon backend, enabling hardware-accelerated L2 normalization on Qualcomm Hexagon DSPs.

llama.cpp b9080

Fri, 08 May 2026 21:05:08 GMT

llama.cpp b9080 adds GGUF conversion support for the Gemma4 26B A4B NVFP4 (NVIDIA FP4) checkpoint format. • Adds GGUF conversion support for the Gemma4_26B_A4B_NVFP4 HuggingFace checkpoint format, enabling the Gemma 4 26B mixture-of-experts model with NVIDIA FP4 quantization to be used with llama.cpp.

llama.cpp b9077

Fri, 08 May 2026 19:29:16 GMT

llama.cpp server gains Vertex AI compatible API mode, activated via the `AIP_MODE` environment variable. • Adds Vertex AI compatible API support to the server, enabled when the `AIP_MODE` environment variable is set (along with other `AIP_*` env vars).

llama.cpp b9076

Fri, 08 May 2026 18:53:58 GMT

llama.cpp b9076: router server now exposes child model info via `GET /v1/models` • The router server's `GET /v1/models` endpoint now exposes metadata for child models, enabling clients to discover all models behind a router instance.

llama.cpp b9063

Thu, 07 May 2026 22:23:17 GMT

Adds OpenCL operation-filter regex for targeted GPU kernel debugging in llama.cpp b9063. • Adds an `opfilter` regex to the OpenCL backend to selectively enable or disable specific GPU operations during debugging.

llama.cpp b9060

Thu, 07 May 2026 18:35:29 GMT

llama.cpp b9060 adds FILL, CUMSUM, DIAG, SOLVE_TRI, SSM_SCAN, and GATED_DELTA_NET ops to the SYCL backend. • Adds `FILL`, `CUMSUM`, `DIAG`, `SOLVE_TRI`, `SSM_SCAN`, and `GATED_DELTA_NET` operations to the SYCL backend, expanding GPU-accelerated model coverage on Intel hardware.

llama.cpp b9055

Thu, 07 May 2026 13:29:23 GMT

llama.cpp b9055 adds Mimo v2.5 model support including MTP weights in GGUF • Adds support for the Mimo v2.5 model architecture, including tensor-parallel (TP) handling and MTP weights embedded in GGUF output.

llama.cpp b9049

Wed, 06 May 2026 21:42:52 GMT

llama.cpp b9049 adds multimodal support for MiniCPM-V 4.6 with flash attention compatibility. • Adds support for the MiniCPM-V 4.6 multimodal model via the `TYPE_MINICPMV4_6` model type, including flash attention support through `build_attn`.

llama.cpp b9045

Wed, 06 May 2026 13:33:51 GMT

llama.cpp b9045 adds multimodal speech support for IBM Granite 4.0 1B Speech via the mtmd layer. • Adds `ibm-granite/granite-4.0-1b-speech` model support in `mtmd`, enabling Conformer-encoder speech-to-text with QFormer projector, 80-bin log-mel spectrogram, and 2x frame stacking (80→160 mel dims). • GGUF converter (`convert`) now handles batch-norm folding, fused K/V split, and Conv1d weight reshaping for Granite Speech models at export time, and registers `GraniteModel` for `GraniteSpeechForConditionalGeneration`.

llama.cpp b9038

Wed, 06 May 2026 05:58:02 GMT

llama.cpp b9038 improves OpenCL memory estimation for `--fit` using `CL_DEVICE_GLOBAL_MEM_SIZE`. • Uses `CL_DEVICE_GLOBAL_MEM_SIZE` to report estimated OpenCL device memory when using `--fit`, enabling more accurate model-to-device fitting decisions on OpenCL backends.

llama.cpp b9023

Mon, 04 May 2026 21:12:42 GMT

llama.cpp server gains `GET /models?reload=1` endpoint to reload the model list at runtime. • Adds `reload=1` query parameter to the `GET /models` server endpoint, enabling on-demand model list reloads without restarting the server.

llama.cpp b9018

Mon, 04 May 2026 16:07:19 GMT

llama.cpp b9018 adds a built-in `get_datetime` server tool for time-aware LLM interactions. • Adds `get_datetime` as a built-in server tool, giving the llama.cpp server a native way to expose current date/time to the model without custom tool implementations.

llama.cpp b9014

Mon, 04 May 2026 05:25:36 GMT

llama.cpp b9014 adds layer norm ops to the WebGPU backend shader. • Adds layer norm ops to the `ggml-webgpu` shader backend, enabling WebGPU-accelerated normalization layers.

llama.cpp b9000

Sat, 02 May 2026 05:07:44 GMT

llama.cpp b9000 adds HMX-accelerated flash attention for Hexagon HTP prefill, with multi-threaded Q/O phases and FP16 softmax. • Adds HMX-accelerated flash attention (`op_hmx_flash_attn_ext`) for the Qualcomm Hexagon HTP backend, enabling hardware-accelerated prefill across GQA, ALiBi, softcap, and masked attention shapes. • Enables multi-threaded Q-load and O-store phases via `fa_phase_q_load` and `fa_phase_o_store` worker-pool helpers, parallelising F32↔F16 conversion across row ranges on HVX threads. • Adds experimental FP16 softmax path (`FA_EXP_F16`, controlled via CMake host-side option) using `hvx_exp2_hf` directly — ~22 ALU ops per 64 lanes versus ~44 for the F32 round-trip path. • Relaxes the HMX matmul pipeline gate to cover `k > n` shapes (e.g. FFN_down layers), selecting pipeline vs. sequential layout based on actual chunker output rather than a static `m>=128 && n>=256` threshold. • Adds HMX flash-attention test coverage in `test-backend-ops` over the full `{mask, ALiBi, logit_softcap}` cross-product for prefill — head dims 64/128, GQA 4×4, kv=512/nb=64 and kv=113/nb=32 non-aligned cases, F16 KV.

llama.cpp b8995

Fri, 01 May 2026 16:10:07 GMT

Vulkan coopmat2 path gains asymmetric Flash Attention support, including Q1_0 quantization type. • Adds asymmetric Flash Attention support to the Vulkan `coopmat2` shader path, enabling mixed quantization types for FA workloads. • Adds `Q1_0` quantization type support in the Vulkan `coopmat2` Flash Attention shader path.

llama.cpp b8980

Wed, 29 Apr 2026 22:31:06 GMT

llama.cpp b8980 adds configurable vmem size and buffer counts for Hexagon/Snapdragon backends. • Adds host-side control over maximum vmem size for the Hexagon backend, with a new default of 3.2 GB and support for autodetection of available vmem space. • Increases Hexagon op buffer count to 16 (matching the maximum number of mmaps) and moves pinned mmap management to the host.

llama.cpp b8966

Tue, 28 Apr 2026 20:35:52 GMT

llama.cpp b8966 adds CUDA flash-attention support for Mistral Small 4's non-standard head sizes (DKQ=320/DV=256) with GQA=32. • Adds CUDA flash-attention (MMA-f16) kernel support for head sizes DKQ=320/DV=256 with `ncols2=32`, enabling GPU-accelerated inference for Mistral Small 4's GQA ratio of 32.

llama.cpp b8956

Tue, 28 Apr 2026 13:48:59 GMT

llama.cpp b8956 adds seven new CANN ops and fused GLU/cross-entropy kernels for Ascend NPU backends. • Adds `GGML_OP_SET` to the CANN backend, implemented via `aclnnInplaceCopy` on the target region. • Adds `GGML_OP_CUMSUM` to the CANN backend, implemented via `aclnnCumsum`. • Adds `GGML_OP_FILL` to the CANN backend, implemented via `aclnnInplaceFillScalar`. • Adds `GGML_OP_DIAG` to the CANN backend, implemented via `aclnnInplaceCopy` on diagonal strides. • Adds `GGML_OP_TRI` (lower/lower_diag/upper_diag/upper modes) to the CANN backend via `aclnnTril` and `aclnnTriu` with appropriate diagonal offsets. • Adds `GGML_OP_SOLVE_TRI` to the CANN backend via `aclnnTriangularSolve`. • Adds `GGML_UNARY_OP_SOFTPLUS` to the CANN backend via `aclnnSoftplus`. • Fuses GLU variants (SwiGLU, GeGLU, GeGLU_ERF, GeGLU_QUICK) into single `aclnnSwiGlu`/`aclnnGeGluV3` calls on the CANN backend when applicable. • Replaces the five-kernel cross-entropy sequence (LogSoftmax→Mul→ReduceSum×2→Muls) with a single `aclnnSoftmaxCrossEntropyWithLogits` call on the CANN backend. • Implements CANN backend `memset_tensor` via `aclrtMemset`, replacing a NULL stub.

llama.cpp b8952

Mon, 27 Apr 2026 22:50:35 GMT

llama.cpp server router now forwards form-data, enabling `/v1/audio/transcriptions` in router mode. • Enables the server router to forward `multipart/form-data` requests to model servers, unlocking use of the `/v1/audio/transcriptions` endpoint in router mode.

llama.cpp b8935

Sun, 26 Apr 2026 06:18:01 GMT

llama.cpp b8935 adds IQ4_NL quantization support for OpenCL, including optimized GEMM/GEMV kernels for Adreno GPUs. • Adds `iq4_nl` quantization format support to the OpenCL backend, enabling inference with IQ4_NL-quantized models on OpenCL-capable devices. • Adds optimized `iq4_nl` GEMM and GEMV kernels specifically for Adreno GPUs via OpenCL, improving throughput for quantized model workloads on Qualcomm mobile hardware.

llama.cpp b8929

Sat, 25 Apr 2026 09:31:09 GMT

llama.cpp b8929 changes the default quantization type in `llama_model_quantize_params` from `Q5_1` to `Q8_0`. • Changes the default `ftype` in `llama_model_quantize_params` from `LLAMA_FTYPE_MOSTLY_Q5_1` to `LLAMA_FTYPE_MOSTLY_Q8_0`, so programs using default quantization params now produce higher-quality Q8_0 models instead of the older Q5_1 format. Breaking changes: • The default `ftype` field in `llama_model_quantize_params` is changed from `LLAMA_FTYPE_MOSTLY_Q5_1` to `LLAMA_FTYPE_MOSTLY_Q8_0`; any code that relies on the default quantization type without explicitly setting `ftype` will now produce Q8_0 output instead of Q5_1.

llama.cpp b8911

Fri, 24 Apr 2026 01:20:09 GMT

llama.cpp b8911 adds configurable Hexagon op profiling with PMU counter support via the `GGML_HEXAGON_PROFILE` environment variable. • Adds `GGML_HEXAGON_PROFILE` environment variable to enable and configure Hexagon HTP op profiling, including optional PMU counter collection (made optional because PMU counters are expensive on older devices). • Adds support for setting profiler PMU events from the environment on Hexagon HTP backends. • Adds `scripts/snapdragon/ggml-hexagon-profile.py`, a post-processing tool for Hexagon profiler output, with support for reading logs from stdin.

llama.cpp b8902

Thu, 23 Apr 2026 13:21:23 GMT

llama.cpp b8902 enables the transcriptions API for LFM2-Audio models in the server. • Enables the transcriptions API for LFM2-Audio models in the `server`.

llama.cpp b8893

Thu, 23 Apr 2026 02:36:03 GMT

llama.cpp b8893 enables HIP graphs by default on ROCm, boosting AMD GPU inference throughput. • Enables `GGML_HIP_GRAPHS` by default on ROCm backends, delivering improved inference performance on AMD GPUs without any configuration change. Breaking changes: • `GGML_HIP_GRAPHS` now defaults to on; users who previously relied on the off-by-default behavior to work around ROCm graph issues may need to explicitly set `GGML_HIP_GRAPHS=0` to restore the old behavior.

llama.cpp b8891

Wed, 22 Apr 2026 22:37:58 GMT

WebGPU backend gains fused RMS_NORM + MUL kernel with optional `GGML_WEBGPU_DISABLE_FUSION` escape hatch. • Adds fused `rms_norm_mul` kernel for the WebGPU backend, combining RMS normalisation and element-wise multiply into a single GPU dispatch to reduce overhead.

llama.cpp b8885

Wed, 22 Apr 2026 19:35:34 GMT

llama.cpp b8885 adds HunyuanVL vision-language model support with M-RoPE and GGUF conversion. • Adds `LLM_ARCH_HUNYUAN_VL` architecture with M-RoPE (XD-RoPE) multi-dimensional position encoding support for HunyuanVL image tokens. • Adds `PROJECTOR_TYPE_HUNYUANVL` with PatchMerger vision encoder for HunyuanVL multimodal inference. • Adds GGUF conversion support for HunyuanVL vision and text models (HunyuanVL and HunyuanOCR) via `convert_hf_to_gguf.py`, verified with F16 and Q8_0 quantization on Metal.

llama.cpp b8882

Wed, 22 Apr 2026 04:18:11 GMT

llama.cpp b8882 adds WebGPU conv2d shader kernels with f32 and f16 support. • Adds `conv2d` shader kernels to the WebGPU (Dawn) backend, passing f32 and f16 tests.

llama.cpp b8875

Tue, 21 Apr 2026 23:29:49 GMT

llama.cpp b8875 adds Reka Edge 2603 multimodal support with Yasa2 (ConvNeXtV2) vision encoder and reka-edge chat template. • Adds `PROJECTOR_TYPE_YASA2` (ConvNeXtV2-based) vision encoder support for Reka Edge, wired into `clip.cpp` and `mtmd.cpp` `init_vision`, using `mtmd_image_preprocessor_fixed_size` with `image_resize_algo = RESIZE_ALGO_BICUBIC` and `image_resize_pad = false`. • Adds `chat-reka.cpp`/`chat-reka.h` implementing a PEG-based parser for the reka-edge chat format, including tool-use and thinking support, with a new `Reka-Edge.jinja` chat template detected automatically via try_specialized_template().

llama.cpp b8874

Tue, 21 Apr 2026 23:23:13 GMT

llama.cpp b8874 adds the `--spec-default` CLI argument for speculative decoding configuration. • Adds `--spec-default` CLI argument to configure default speculative decoding behavior.

llama.cpp b8842

Sun, 19 Apr 2026 11:31:13 GMT

llama.cpp server gains speculative decoding with checkpoints, including multimodal (mtmd) support. • Adds speculative decoding with checkpointing to the server, enabling faster token generation by saving and restoring draft model state across speculative sessions. • Enables speculative decoding support for multimodal (mtmd) workloads in the server. Breaking changes: • The `--spec-use-checkpoints` argument has been removed.

llama.cpp b8840

Sat, 18 Apr 2026 23:20:48 GMT

llama.cpp b8840 exposes `media_tag` on the `/props` server endpoint. • Exposes `media_tag` field on the `/props` endpoint of the llama.cpp server API.

llama.cpp b8831

Fri, 17 Apr 2026 13:25:28 GMT

llama.cpp b8831 adds official Android arm64 CPU builds and fixes server EOS flag handling. • Adds official Android arm64 (CPU) binary release (`llama-b8831-bin-android-arm64.tar.gz`) via new CI pipeline. • Server now respects the ignore EOS flag, enabling proper end-of-sequence control in server mode.

llama.cpp b8828

Fri, 17 Apr 2026 11:53:44 GMT

llama.cpp b8828 adds Gemma4 model type detection support. • Adds Gemma4 model type detection, enabling llama.cpp to identify and load Gemma4 architecture models.

llama.cpp b8821

Fri, 17 Apr 2026 00:56:28 GMT

llama.cpp server gains `LLAMA_MEDIA_MARKER` env var to pin the media marker used in multimodal prompts. • Adds `LLAMA_MEDIA_MARKER` environment variable to pin the server's media marker token, overriding the randomly generated one; useful for scripted or automated multimodal prompt workflows.

llama.cpp b8815

Thu, 16 Apr 2026 15:41:56 GMT

llama.cpp b8815 adds Metal ROLL op implementation for Apple Silicon GPU acceleration. • Implements the ROLL operation for the Metal backend, enabling GPU-accelerated tensor rolling on Apple Silicon and iOS devices.

llama.cpp b8814

Thu, 16 Apr 2026 15:15:35 GMT

Adds 128-bit RISC-V Vector (RVV) implementations for i-quants, ternary quants, iq2_xs, iq3_s, iq3_xxs, and tq2_0 quantization dot products. • Adds 128-bit RVV (RISC-V Vector) kernel implementations for quantization vector dot products, covering `iq2_xs`, `iq3_s`, `iq3_xxs`, `tq2_0`, i-quants, and ternary quants — enabling hardware-accelerated inference on 128-bit RVV-capable RISC-V CPUs.

llama.cpp b8813

Thu, 16 Apr 2026 15:14:06 GMT

llama.cpp b8813 adds a SIMD GEMM kernel for the RISC-V Vector extension, accelerating inference on RVV hardware. • Adds a `simd_gemm` kernel for the RISC-V Vector (RVV) extension in ggml, enabling hardware-accelerated matrix multiplication on RVV-capable CPUs.

llama.cpp b8806

Wed, 15 Apr 2026 21:05:46 GMT

llama.cpp b8806 adds initial CUDA backend support for the Q1_0 quantization format. • Adds initial CUDA backend support for the `Q1_0` quantization format, including dp4a acceleration and AMD MMA compatibility.

llama.cpp b8802

Wed, 15 Apr 2026 16:36:54 GMT

llama.cpp b8802 adds native RDMA transport (RoCEv2) for the RPC backend. • Adds native RDMA transport (RoCEv2) to the RPC backend, enabling high-throughput, low-latency distributed inference over RDMA-capable network fabric.

llama.cpp b8794

Tue, 14 Apr 2026 22:27:09 GMT

llama.cpp b8794 adds mtmd_image_tokens_get_decoder_pos() API for multimodal token position queries. • Adds mtmd_image_tokens_get_decoder_pos() to the `mtmd` API for retrieving the decoder position of image tokens in multimodal inference pipelines.

llama.cpp b8785

Tue, 14 Apr 2026 14:33:31 GMT

Vulkan backend gains GGML_TYPE_NVFP4 support for get_rows, dequant, and mul_mat operations • Adds `GGML_TYPE_NVFP4` support to the Vulkan backend for `get_rows`, `dequant`, and `mul_mat`/`mul_mat_id` operations via fp16/fp32 paths.

llama.cpp b8784

Tue, 14 Apr 2026 13:35:01 GMT

llama.cpp server gains OpenAI-compatible `POST /v1/audio/transcriptions` API endpoint. • Adds `POST /v1/audio/transcriptions` endpoint to the llama.cpp server, implementing the OpenAI-compatible audio transcription API.

llama.cpp b8781

Mon, 13 Apr 2026 21:44:35 GMT

llama.cpp b8781 adds a dedicated DeepSeek v3.2 chat parser and its official prompt template. • Adds a dedicated DeepSeek v3.2 chat parser and 'official' prompt template for correct message formatting with that model.

llama.cpp b8779

Mon, 13 Apr 2026 15:27:12 GMT

Vulkan backend gains Flash Attention DP4A shader for quantized KV cache, accelerating inference on compatible GPUs. • Adds Vulkan Flash Attention DP4A integer dot-product shader for quantized KV cache, enabling faster attention computation on Vulkan-capable GPUs with quantized key/value types.

llama.cpp b8777

Mon, 13 Apr 2026 12:02:22 GMT

llama.cpp server now exposes build_info in router mode for deployment introspection. • Exposes `build_info` via the server in router mode, enabling operators to query build metadata from a routed llama.cpp deployment.

llama.cpp b8769

Sun, 12 Apr 2026 23:14:47 GMT

llama.cpp b8769 adds Qwen3 audio support, enabling Qwen3-Omni and Qwen3-ASR models for speech recognition. • Adds multimodal audio support for Qwen3-Omni and Qwen3-ASR models via the `mtmd` subsystem, enabling automatic speech recognition (ASR) workflows with Qwen3 architecture.

llama.cpp b8766

Sun, 12 Apr 2026 15:03:50 GMT

llama.cpp b8766 adds Gemma 4 audio conformer encoder support via a USM-style 12-layer Conformer with dedicated mel preprocessor. • Adds `mtmd_audio_preprocessor_gemma4a` — a dedicated mel preprocessor for Gemma 4 audio using HTK mel scale, 128 bins, magnitude STFT, and semicausal left-padding, enabling audio transcription with Gemma 4 E2B/E4B models on CPU and Vulkan backends.

llama.cpp b8762

Sat, 11 Apr 2026 13:17:53 GMT

llama.cpp b8762 adds MERaLiON-2 audio-language model support for speech transcription and spoken QA • Adds `PROJECTOR_TYPE_MERALION` projector type enabling MERaLiON-2 multimodal audio-language models (3B and 10B) via the `--mmproj` flag on `convert_hf_to_gguf.py` with architecture `MERaLiON2ForConditionalGeneration`. • Supports speech transcription (EN/ZH/MS/TA), translation, and spoken QA tasks using a Whisper large-v2 encoder with a gated MLP adaptor and Gemma2 decoder.

llama.cpp b8761

Sat, 11 Apr 2026 12:31:27 GMT

llama.cpp b8761 adds OpenCL support for Q5_K quantization format (matrix-vector and matrix-matrix ops). • Adds OpenCL backend support for Q5_K quantization, including general matrix-vector (mv) and flattened matrix-vector multiply, plus general matrix-matrix (mm) operations.

llama.cpp b8755

Sat, 11 Apr 2026 02:14:06 GMT

llama.cpp b8755 adds Hexagon backend support for Linux on Snapdragon (Debian on ex2). • Adds Hexagon backend support for Linux on Snapdragon, including Debian on ex2 hardware, with `-fvectorize` enabled for C/C++ CMake builds.

llama.cpp b8754

Sat, 11 Apr 2026 01:49:46 GMT

llama.cpp b8754 improves Hexagon NPU op batching, buffer management, and replaces GGML_HEXAGON_EXPERIMENTAL with GGML_HEXAGON_OPFILTER • Replaces the `GGML_HEXAGON_EXPERIMENTAL` environment variable with `GGML_HEXAGON_OPFILTER`, a regex-based op filter for selectively disabling Hexagon backend ops during debugging or validation. • Introduces op request batching on the Hexagon backend: the host now prepares batches of up to 256 requests and dispatches them via a single dspqueue message, reducing inter-op overhead. • Adds configurable opbatch sizing (`OB` and `OQ` params) and a vmem limit for op batching on the Hexagon backend. • Introduces pinned mmap-based shared buffers for packing op batches, with explicit per-tensor L2 cache management (flush/invalidate) replacing coarse global flushes. • Enables Flash Attention during token generation (TG) on the Hexagon backend to reduce graph splits. • Improves VTCM acquisition on the Hexagon backend to remove inter-op overhead, with full compatibility with QNN-HTP coexistence. Breaking changes: • The `GGML_HEXAGON_EXPERIMENTAL` environment variable is removed; use `GGML_HEXAGON_OPFILTER` instead to disable specific ops.

llama.cpp b8752

Sat, 11 Apr 2026 00:20:34 GMT

llama.cpp b8752 adds a callback interface for download progress in the common library. • Adds a callback interface for download progress in the `common` library, enabling programmatic monitoring of model download state.

llama.cpp b8742

Fri, 10 Apr 2026 09:49:20 GMT

llama.cpp b8742 adds Vulkan backend support for the Q1_0 quantization format. • Adds `Q1_0` quantization format support to the Vulkan backend, enabling GPU-accelerated inference with Q1_0 models on Vulkan-capable hardware.

llama.cpp b8739

Thu, 09 Apr 2026 23:52:47 GMT

llama.cpp b8739 adds AMD Instinct MI350X/MI355X (gfx950, CDNA4) GPU support via HIP backend • Adds `GGML_CUDA_CC_CDNA4` and `GGML_CUDA_CC_IS_CDNA4` macros in `common.cuh` to identify AMD CDNA4 (gfx950) architecture at build time. • Supports AMD Instinct MI350X/MI355X (gfx950, CDNA4) in the HIP backend, enabling GPU-accelerated inference on these accelerators when building with `-DAMDGPU_TARGETS=gfx950`. • Routes CDNA4 matrix operations to compatible MFMA instructions: `mfma_f32_16x16x4f32` for f32, `mfma_f32_16x16x16bf16_1k` for bf16, and `mfma_i32_16x16x32_i8`/`mfma_i32_32x32x16_i8` for int8. • Includes CDNA4 (gfx950) in stream-k kernel dispatch in `mmq.cuh`, enabling MMQ inference on MI355X.

llama.cpp b8738

Thu, 09 Apr 2026 23:16:42 GMT

llama.cpp b8738 adds experimental backend-agnostic tensor parallelism with NCCL/RCCL support across up to 8 GPUs. • Introduces experimental backend-agnostic tensor parallelism via `ggml-backend-meta.cpp` and the new `ggml-ext.h` staging API header, supporting arbitrary numbers of GPUs and uneven tensor splits. • Adds NCCL support for CUDA multi-GPU tensor parallelism, enabling AllReduce operations across devices using pinned (vs. pageable) memory when all backends expose the same host buffer type. • Adds RCCL support for HIP/ROCm multi-GPU tensor parallelism, extending the same backend-agnostic tensor parallel path to AMD hardware. • Extends tensor parallelism support to GPT-OSS, Qwen 3 MoE, Qwen 3.5, Qwen 3.5 MoE, and Gemma 4 MoE model architectures. • Moves most meta backend public API into `ggml-backend-impl.h`, hiding previously exposed internals and introducing `llama_device` as the new abstraction for device selection. • Uses BF16 precision for AllReduce operations in the tensor parallel path. Breaking changes: • ggml_backend_dev_is_meta() has been removed; callers must migrate to the new `llama_device` abstraction.

llama.cpp b8731

Thu, 09 Apr 2026 20:13:35 GMT

llama.cpp b8731 adds multimodal support for the Dots.OCR model via the mtmd subsystem. • Adds `dots.ocr` model support to the `mtmd` (multimodal) subsystem, including GGUF conversion and CLIP implementation.

llama.cpp b8724

Thu, 09 Apr 2026 12:38:10 GMT

SYCL Flash Attention now supports head size 512 (DKQ/DV), up from the previous 256 limit. • Extends SYCL Flash Attention (both tile and vector kernels) to support head sizes (DKQ/DV) of 512, previously capped at 256, enabling larger-model attention workloads on SYCL backends.

llama.cpp b8712

Thu, 09 Apr 2026 00:22:24 GMT

llama.cpp b8712 adds Metal GPU backend support for Q1_0 quantization on Apple Silicon. • Adds Metal GPU backend for `Q1_0` quantization, enabling hardware-accelerated inference with 1-bit weights on Apple Silicon devices.

llama.cpp b8705

Wed, 08 Apr 2026 08:38:08 GMT

llama.cpp b8705 adds support for the Step3-VL-10B vision-language model. • Adds support for the Step3-VL-10B multimodal vision-language model, including fused QKV, tensor mapping via `tensor_mapping.py`, and `understand_projector_stride` read from the global config.

llama.cpp b8703

Wed, 08 Apr 2026 05:36:39 GMT

llama.cpp b8703 adds a KleidiAI-enabled arm64 macOS release artifact for Apple Silicon. • Adds a KleidiAI-enabled arm64 macOS release artifact (`llama-b8703-bin-macos-arm64-kleidiai.tar.gz`) for Apple Silicon, providing ARM-optimized compute kernels via the KleidiAI library.

llama.cpp b8685

Tue, 07 Apr 2026 10:03:41 GMT

SYCL backend gains Q8_0 reorder optimization, delivering ~3x token-generation speedup on Intel Arc GPUs. • Extends the SYCL reorder optimization (scale-factor/weight separation for coalesced memory access) to `Q8_0`, achieving ~3.1x token-generation throughput on Intel Arc Pro B70 (Xe2) — e.g. 4.88 → 15.24 t/s on Qwen3.5-27B, with bandwidth utilization rising from 21% to 66%.

llama.cpp b8683

Mon, 06 Apr 2026 22:50:40 GMT

llama.cpp b8683 adds MUL_MAT_ID support to the WebGPU backend. • Adds `MUL_MAT_ID` operation support to the WebGPU (ggml-webgpu) backend, enabling mixture-of-experts model inference via WebGPU.

llama.cpp b8682

Mon, 06 Apr 2026 21:49:35 GMT

llama.cpp b8682 adds Q1_0 1-bit quantization support on CPU, enabling extreme model compression. • Adds `Q1_0` 1-bit quantization format (group size 128) with CPU support, including a generic fallback for x86 and other backends.

llama.cpp b8679

Mon, 06 Apr 2026 18:19:10 GMT

llama-bench gains `-fitc` and `-fitt` flags for finer benchmark filtering control. • Adds `-fitc` and `-fitt` arguments to `llama-bench` for filtering benchmark test cases and test types.

llama.cpp b8670

Mon, 06 Apr 2026 00:04:07 GMT

llama.cpp b8670 adds HunyuanOCR vision model support with a perceiver-based projector and dedicated chat template. • Adds HunyuanOCR vision projector (perceiver-based, with Conv2d merge) and tensor mappings for `mm.before_rms`, `mm.after_rms`, and related perceiver projector tensors. • Adds a dedicated `HUNYUAN_OCR` chat template (content-before-role format) for HunyuanOCR models. • Registers `HunYuanVLForConditionalGeneration` in `convert_hf_to_gguf.py` for both text and mmproj conversion. • Supports the `xdrope` RoPE scaling type introduced by HunyuanOCR.

llama.cpp b8665

Sat, 04 Apr 2026 22:01:08 GMT

llama.cpp b8665 adds a dedicated Gemma 4 chat-template parser with tool-call and interleaved-thinking support. • Adds a specialized Gemma 4 Jinja chat-template parser (`gemma4.jinja`) that emits JSON from Gemma 4 tool-call ASTs and registers `<|tool_response>` as an end-of-generation token. • Adds a custom Gemma 4 template to support interleaved thinking, preserving reasoning content in model turns.

llama.cpp b8658

Fri, 03 Apr 2026 22:24:30 GMT

llama.cpp server gains `--clear-idle` to reclaim VRAM from idle KV cache slots on new task arrival. • Adds `--clear-idle` flag to `llama-server` to automatically free VRAM used by idle KV cache slots (via `LLAMA_KV_KEEP_ONLY_ACTIVE`) when a new task arrives, reducing GPU memory pressure in multi-user deployments. Opt out with `--no-kv-clear-idle`.

llama.cpp b8648

Fri, 03 Apr 2026 13:46:44 GMT

llama.cpp b8648 adds ZenDNN MUL_MAT_ID op acceleration for Mixture-of-Experts models. • Adds `MUL_MAT_ID` op acceleration in the `ggml-zendnn` backend for Mixture-of-Experts (MoE) models, falling back to the CPU backend when total experts exceed 32. • Updates the ZenDNN library reference to ZenDNN-2026-WW13.

llama.cpp b8639

Thu, 02 Apr 2026 19:55:44 GMT

llama.cpp b8639 adds vectorized flash attention for the WebGPU backend, with q4/q8 quantization support. • Adds vectorized flash attention to the WebGPU (`ggml-webgpu`) backend, enabled when `Q->ne[1] < 20`, `Q->ne[0] % 32 == 0`, `V->ne[0] % 4 == 0`, and `K->type == f16`; also supports q4 and q8 quantized key/value types. • Introduces a fast single-workgroup (`nwg=1`) path for vectorized flash attention that skips temporary/reduce staging buffers, reducing overhead for small batch sizes. • Uses packed f16 K loads in the `flash_attn_vec_split.wgsl` shader for improved memory throughput on the WebGPU backend.

llama.cpp b8634

Thu, 02 Apr 2026 13:27:06 GMT

llama.cpp b8634 adds Granite 4.0 chat template with correct tool_call role mapping for agentic workflows. • Adds `LLM_CHAT_TEMPLATE_GRANITE_4_0` enum, map entry, and handler to support Granite 4.0 chat template, using `` XML tags and mapping the `assistant_tool_call` role to `<|start_of_role|>assistant<|end_of_role|><|tool_call|>` — fixing broken tool calling when `--jinja` is not used. • Auto-detects Granite model generation: presence of `<|start_of_role|>` plus `` or `` selects the 4.0 template; otherwise falls back to the 3.x template. Breaking changes: • The existing `LLM_CHAT_TEMPLATE_GRANITE` enum value is renamed to `LLM_CHAT_TEMPLATE_GRANITE_3_X`; any code referencing `LLM_CHAT_TEMPLATE_GRANITE` by name must be updated.

llama.cpp b8607

Wed, 01 Apr 2026 10:50:57 GMT

WebGPU backend gains quantized buffer support via u32 packing and wider browser/device compatibility. • Extends the WebGPU (ggml) backend to pack quantized buffers as u32 and use `unpackf16` for broader browser and device compatibility, removing the previous bitcast approach.

llama.cpp b8589

Mon, 30 Mar 2026 20:36:37 GMT

llama.cpp b8589 adds Q4_K GEMM and GEMV OpenCL kernels for Qualcomm Adreno GPUs • Adds `q4_K` GEMM and GEMV OpenCL kernels for Qualcomm Adreno GPUs, enabling accelerated quantized matrix operations on Adreno hardware.

llama.cpp b8581

Mon, 30 Mar 2026 08:21:43 GMT

llama.cpp server gains header wrapping for the MCP CORS proxy • Adds header wrapping in `server-cors-proxy.h` for the MCP proxy, enabling proper CORS header forwarding when the llama.cpp server is used as an MCP proxy.

llama.cpp b8570

Sat, 28 Mar 2026 16:40:46 GMT

llama.cpp b8570 adds `reasoning_format = none` support to the gpt-oss common layer. • Adds `reasoning_format = none` support to the `gpt-oss` common layer, allowing callers to suppress reasoning-format output entirely.

llama.cpp b8562

Sat, 28 Mar 2026 03:21:11 GMT

llama.cpp b8562 adds `/glob` command to the CLI for batch file selection by pattern. • Adds `/glob` command to the CLI, enabling wildcard-based file selection including paths outside the current directory, with an error output when the maximum file limit is reached.

llama.cpp b8560

Sat, 28 Mar 2026 02:30:30 GMT

llama-server gains `--reuse-port` flag to opt into SO_REUSEPORT for multi-process socket sharing • Adds `--reuse-port` flag to `llama-server`, enabling `SO_REUSEPORT` on the listening socket to allow multiple server processes to bind the same port for load-balancing setups.

llama.cpp b8558

Fri, 27 Mar 2026 20:09:32 GMT

llama.cpp b8558 adds `LLAMA_BUILD_WEBUI` CMake flag to optionally strip the embedded server web UI at build time. • Adds `LLAMA_BUILD_WEBUI` CMake build flag (default: `ON`) to allow disabling the embedded web UI in the llama-server binary, reducing attack surface in headless deployments.

llama.cpp b8557

Fri, 27 Mar 2026 19:35:23 GMT

llama.cpp b8557 adds IQ4_NL and MXFP4 quantization support to the Hexagon backend via HMX matmul. • Adds `IQ4_NL` quantization type support to the Hexagon backend, including buffer set/get tensor repack, `mul_mat`, and `mul_mat_id` dispatch. • Implements HVX `IQ4_NL` vec_dot kernels (1x1, 2x1, 2x2) with LUT-based 4-bit index to int8 dequantization on the Hexagon backend. • Adds `MXFP4` HMX dequantization path with E8M0 scale conversion, including a batch-4 fast path and single-tile fallback on the Hexagon backend. • Unifies quantized row size and scale offset logic in the DMA fetch path to handle `Q4_0`, `Q8_0`, `IQ4_NL`, and `MXFP4`.

llama.cpp b8553

Fri, 27 Mar 2026 13:02:15 GMT

llama.cpp server gains built-in tools backend with `--tools all` flag for agentic workflows. • Adds `--tools all` argument to the `llama-server` built-in tools backend, enabling server-side tool/function-call orchestration without an external middleware layer.

llama.cpp b8532

Thu, 26 Mar 2026 03:46:26 GMT

llama.cpp b8532 adds F32 kernel type support for CONV_TRANSPOSE_2D on CUDA and CPU backends. • Adds F32 kernel type support for `CONV_TRANSPOSE_2D` on CUDA and CPU backends, alongside the existing F16 type, enabling full-precision 2D transposed convolution operations.

llama.cpp b8530

Thu, 26 Mar 2026 00:20:37 GMT

llama.cpp b8530 adds DeepSeek-OCR multimodal support with dynamic resolution and flash-attn compatibility in llama-cli. • Adds DeepSeek-OCR model support via `llama-mtmd-cli`, including CLIP-ViT vision encoder, SAM-based image encoding, and DeepSeek2 language model backend with standard attention. • Supports dynamic and native resolution preprocessing for DeepSeek-OCR, with configurable min/base/max resolution tiers (640/1024/1280) for image quality vs. speed trade-offs. • Enables `--flash-attn` option compatibility with DeepSeek-OCR inference in `llama-cli`. • Adds `convert_hf_to_gguf.py` conversion support for DeepSeek-OCR models, outputting FP16 GGUF weights with corrected tensor names including `mm_fc_w` and image newline/separator tensors. • Adds a default Jinja chat template for DeepSeek-OCR for use with the llama.cpp server.

llama.cpp b8526

Wed, 25 Mar 2026 23:16:30 GMT

llama.cpp b8526 adds support for the codefuse-ai/F2LLM-v2 model. • Adds support for the codefuse-ai/F2LLM-v2 model.

llama.cpp b8498

Tue, 24 Mar 2026 10:34:19 GMT

llama.cpp b8498 adds standard Hugging Face cache support, migrating manifests and using the HF API to resolve cached model files. • Adds standard Hugging Face cache support: uses the HF API to enumerate all files, migrates existing manifests to the Hugging Face cache layout at startup, and falls back to cached files when the HF API is unavailable.

llama.cpp b8493

Mon, 23 Mar 2026 20:25:25 GMT

llama.cpp b8493 adds OpenCL q6_K GEMM and GEMV kernels for Adreno GPUs. • Adds OpenCL `q6_K` GEMM (`gemm_noshuffle_q6_k_f32`) and GEMV kernels for Adreno GPUs, enabling quantized 6-bit inference acceleration on mobile/embedded hardware via the OpenCL backend.

llama.cpp b8477

Mon, 23 Mar 2026 02:25:19 GMT

llama.cpp b8477 adds dynamic high-resolution image preprocessing support for InternVL models. • Adds dynamic high-resolution image preprocessing for InternVL models (including Qianfan-OCR), storing `min`/`max` dynamic patch counts in GGUF metadata; default values are provided for older models.

llama.cpp b8475

Sun, 22 Mar 2026 19:13:16 GMT

llama.cpp server router can now report sleep status of child instances • Enables the llama.cpp server router to expose the sleep/wake state of child server instances in its status reporting.

llama.cpp b8470

Sun, 22 Mar 2026 11:21:47 GMT

llama.cpp b8470 adds native BF16 flash attention for the CUDA vec kernel, improving GPU inference efficiency. • Adds native BF16 flash attention support for the CUDA vec kernel, avoiding prior BF16-to-FP16 conversion overhead on supported GPU backends including HIP.

llama.cpp b8416

Wed, 18 Mar 2026 23:16:16 GMT

llama.cpp b8416 adds control vector support to Qwen3.5 and Qwen-next models • Extends control vector (cvec) support to Qwen3.5 and Qwen-next models, plus fills in missing cvec compatibility across the remaining model architectures.

llama.cpp b8400

Tue, 17 Mar 2026 23:19:32 GMT

llama.cpp b8400 adds Hexagon HVX ops (neg, exp, sigmoid, softplus, CONT, REPEAT) enabling Qwen 3.5 DeltaNet linear attention on-device. • Adds `neg`, `exp`, `sigmoid`, `softplus`, `CONT`, and `REPEAT` element-wise ops to the Hexagon HVX backend, enabling Qwen 3.5 DeltaNet linear attention layers on Qualcomm hardware. • Hexagon `REPEAT` op supports `f32` and `f16` types with multi-threaded execution via the worker pool, parallelizing across output rows using `memcpy` per tile. • Hexagon `CONT` op reuses existing `CPY` infrastructure to make tensors contiguous on-device without a separate kernel.

llama.cpp b8391

Tue, 17 Mar 2026 10:31:14 GMT

Vulkan backend gains `GGML_VK_ALLOW_GRAPHICS_QUEUE` env var to opt back into graphics-queue execution on AMD GPUs. • Adds `GGML_VK_ALLOW_GRAPHICS_QUEUE` environment variable to override the new default of avoiding graphics queues on non-RADV AMD drivers and small GPUs, re-enabling the transfer queue when the graphics queue is not used. Breaking changes: • The Vulkan backend no longer uses the graphics queue by default on non-RADV AMD drivers and small GPUs; workloads that relied on graphics-queue execution will now use compute/transfer queues unless `GGML_VK_ALLOW_GRAPHICS_QUEUE` is set.

llama.cpp b8389

Tue, 17 Mar 2026 01:29:57 GMT

llama.cpp b8389 adds refusal content support to the Responses API in tools/server. • Adds refusal content support to the Responses API in `tools/server`.

llama.cpp b8388

Tue, 17 Mar 2026 00:34:09 GMT

llama.cpp b8388 adds support for the Mistral Small 4 model. • Adds Mistral Small 4 model support via `convert_hf_to_gguf.py`.

llama.cpp b8340

Sat, 14 Mar 2026 14:07:52 GMT

llama.cpp b8340 adds native AVX512-FP16 support for F16 operations, cutting instruction count by 2.7 billion on compatible CPUs. • Adds native AVX512-FP16 support for F16 operations in ggml, reducing instruction count by ~2.7 billion on AVX512-FP16-capable CPUs; enabled only for native builds or with custom compiler flags.

llama.cpp b8339

Sat, 14 Mar 2026 14:05:31 GMT

llama.cpp b8339 adds env variables to override cuBLAS compute type and uses fp32 on V100 GPUs to prevent overflows. • Adds environment variables to override the cuBLAS compute type, giving operators direct control over CUDA matrix-multiply precision without recompiling.

llama.cpp b8338

Sat, 14 Mar 2026 12:29:03 GMT

llama.cpp gains an OpenVINO backend with CPU, GPU, and NPU support including quantized model execution and stateful graph inference. • Adds `GGML_OPENVINO_DISABLE_SDPA_OPTIMIZATION` environment variable to toggle scaled dot-product attention fusion in the new OpenVINO backend. • Adds the OpenVINO backend (`ggml-openvino`) targeting Intel CPU, GPU (OpenCL), and NPU devices — enabling hardware-accelerated inference on Intel silicon without CUDA. • Supports quantized model execution on the OpenVINO backend with custom quant types `q8_1_c`, `q4_0_128`, and `Q8_0_C`; NPU always requantizes weights to `q4_0_128`. • Adds NPU prefill chunking for large-context inference on Intel NPU devices. • Supports stateful graph execution mode on the OpenVINO backend (GPU and NPU) for improved decode-phase performance. • Supports BF16 model inference on the OpenVINO backend. • Supports `-ctk f32` cache type on the OpenVINO backend. • Adds KV-cache fusion support (`kvcachefusion`) in the OpenVINO backend for improved memory and compute efficiency. • Adds OpenVINO backend documentation at `docs/backend/OPENVINO.md`. • Pre-built Ubuntu x64 binaries are now available with OpenVINO 2026.0 support.

llama.cpp b8334

Sat, 14 Mar 2026 08:26:58 GMT

llama-perplexity now supports `-kvu` flag for hellaswag and winogrande benchmarks, fixing batch decode failures on coupled sequences. • Enables the `-kvu` flag in `llama-perplexity` for `--winogrande` and hellaswag evaluation modes, allowing coupled-sequence batches that previously failed with 'failed to find a memory slot' errors.

llama.cpp b8329

Sat, 14 Mar 2026 04:58:36 GMT

Adds RISC-V Vector (RVV) accelerated kernels for quantization types iq4_nl, mxfp4, iq2_xxs, iq4_xs, iq2_xs, and iq3_xxs on RISC-V hardware. • Adds RVV (RISC-V Vector) `vec_dot` inference kernels for quantization types `iq4_nl`, `mxfp4`, `iq2_xxs`, `iq4_xs`, `iq2_xs`, and `iq3_xxs`, accelerating CPU inference on RISC-V hardware. • Adds RVV `quantize_row_q8_K` kernel for accelerated quantization on RISC-V CPUs.

llama.cpp b8320

Fri, 13 Mar 2026 15:06:10 GMT

test-backend-ops gains file-based test loading and a graph operator extraction tool via `llama_graph_reserve` • Adds `llama_graph_reserve` (moved to new `llama-ext` header) as the public API for reserving graph computations, replacing previous internal API calls. • Adds file-based test loading to `test-backend-ops`, allowing backend operation tests to be defined and loaded from a text file instead of being hardcoded. • Adds the `export-graph-ops` tool (located in `tests/`) to extract model graph operators into a file for use with the new file-based test runner. • Adds per-operator error thresholds to `test-backend-ops` for more precise correctness validation across different backend operations. • Adds `nb` parameter support in `test-backend-ops` for non-contiguous input tensors.

llama.cpp b8317

Fri, 13 Mar 2026 14:05:38 GMT

llama.cpp b8317 adds Vulkan GATED_DELTA_NET op support for AMD GPUs via fused compute shader. • Adds `GATED_DELTA_NET` op to the Vulkan backend as a fused compute shader, supporting scalar gate, KDA vector gate, GQA broadcast, multi-token sequences, and permuted (non-contiguous) q/k inputs — enabling Gated Delta Network model inference on Vulkan GPUs. • Vulkan `GATED_DELTA_NET` shader uses specialization constants to select head size (32, 64, or 128) and KDA mode at pipeline creation time, with a pipeline array layout of `[3][2]`. • Vulkan `GATED_DELTA_NET` KDA path caches exp(g) in shared memory and uses vec4 dot products on all inner loops, yielding +5.4% throughput on the KDA path on AMD Radeon 890M (RADV GFX1150).

llama.cpp b8304

Fri, 13 Mar 2026 09:28:39 GMT

llama.cpp b8304 adds chat template parser support for GigaChatV3 and GigaChatV3.1 models. • Adds GigaChatV3 and GigaChatV3.1 model support to the common chat template parser.

llama.cpp b8303

Fri, 13 Mar 2026 07:20:47 GMT

llama.cpp b8303 adds support for the Phi4ForCausalLMV multimodal model architecture. • Adds support for the `Phi4ForCausalLMV` model architecture, enabling Phi-4 vision inference with corrected SigLIP2 patch-kernel layout and NaFlex resize behavior.

llama.cpp b8300

Fri, 13 Mar 2026 03:26:14 GMT

llama.cpp b8300 adds GGML_OP_REPEAT (including i16) support to the WebGPU backend. • Adds `GGML_OP_REPEAT` operation support to the WebGPU (`ggml-webgpu`) backend, including `i16` data type handling.

llama.cpp b8299

Fri, 13 Mar 2026 02:42:59 GMT

llama.cpp b8299 adds GPU-accelerated DeltaNet inference via fused Metal and CUDA kernels for GDN, boosting Qwen3.5 throughput by up to 25%. • Adds fused Metal kernel for `GGML_OP_GATED_DELTA_NET` on Apple Silicon, enabling GPU-accelerated inference for DeltaNet-based models (Qwen3.5, etc.) with both GDA (scalar gate) and KDA (per-row gate) modes; supports head_size 64 and 128, falling back to CPU for head_size 32 or non-contiguous tensors — yields +25% token generation throughput (170 → 213 t/s) for Qwen3.5-0.8B Q4_K_M on M4 Max. • Enables chunked fused GDN path in the llama inference engine, extending DeltaNet recurrence support to prefill in addition to token generation.

llama.cpp b8297

Fri, 13 Mar 2026 00:17:10 GMT

llama.cpp b8297 adds NVFP4 quantization type with ARM NEON optimization delivering up to 3.8x throughput gains. • Adds `NVFP4` as a new quantization type option in `llama-quantize`, with CPU fallback dot product, quantization, dequantization, and conversion support across all platforms (ARM, x86, PowerPC, s390x). • Adds optimized `ggml_vec_dot_nvfp4_q8_0` ARM NEON implementation using `vqtbl1q_s8` LUT lookup, `vpaddq_s32` reduction, and `vfmaq_f32` accumulators — achieving ~31.0 t/s (tg128) versus 8.1 t/s generic, a 3.8x speedup at 77% of Q4_1 speed. • Stores NVFP4 scales as native UE4M3 format, preserving original model bits when converting from HuggingFace via `convert_hf_to_gguf.py`. • NVFP4 CPU-only backend uses a 64-block super-block layout; PPL benchmarks at 15.25 versus F16 at 14.33 at 4.70 BPW, comparable to Q4_1 (15.81) at lower bit-width.

llama.cpp b8295

Thu, 12 Mar 2026 23:56:42 GMT

llama.cpp b8295 adds support for the Nemotron 3 Super (120B.A12B) model with GGUF conversion. • Adds support for the Nemotron 3 Super model (120B.A12B), enabling conversion to GGUF format and local inference in llama.cpp.

llama.cpp b8291

Thu, 12 Mar 2026 16:17:39 GMT

llama.cpp b8291 adds an environment variable to trigger Metal graph capture on macOS. • Adds an environment variable to trigger Metal graph capture for macOS/iOS GPU backends, enabling GPU frame debugging workflows.

llama.cpp b8287

Thu, 12 Mar 2026 09:21:52 GMT

llama.cpp b8287 adds a reasoning budget sampler with CLI handling for thinking-model token control. • Adds a reasoning budget sampler to the common parser, enabling CLI-level control over reasoning token budgets for thinking models.

llama.cpp b8269

Wed, 11 Mar 2026 10:34:43 GMT

llama.cpp b8269 adds concurrent SME and NEON kernel execution via KleidiAI on ARM. • Enables concurrent SME and NEON kernel execution in the KleidiAI backend, improving throughput on ARM hardware that supports both instruction sets simultaneously.

llama.cpp b8268

Wed, 11 Mar 2026 10:21:27 GMT

llama.cpp b8268 adds RISC-V Vector (RVV) repacked GEMM/GEMV for q8_0, iq4_nl, q4_0, q4_K, and q2_K quantization types. • Adds RVV (RISC-V Vector) repacked GEMM and GEMV kernel implementations for quantization types `q8_0`, `iq4_nl`, `q4_0`, `q4_K`, and `q2_K`, enabling accelerated CPU inference on RISC-V hardware with vector extensions.

llama.cpp b8262

Tue, 10 Mar 2026 21:56:31 GMT

llama.cpp server CORS proxy now correctly parses port numbers from MCP server URLs, enabling non-standard port and SSL routing. • The server CORS proxy now parses port numbers from MCP server URLs and passes the scheme to the HTTP proxy to determine whether to use SSL, fixing routing for non-standard ports.

llama.cpp b8261

Tue, 10 Mar 2026 16:39:22 GMT

llama.cpp b8261 extends Metal mul_mv_ext small-batch kernels to BF16, Q2_K, and Q3_K quantization types. • Extends Metal `mul_mv_ext` small-batch kernels (batch sizes 2–8) to `BF16`, `Q2_K`, and `Q3_K` quantization types, which previously fell through to the slower single-row `mul_mv` path — enabling faster Apple Silicon inference for models using these formats.

llama.cpp b8248

Mon, 09 Mar 2026 11:03:07 GMT

llama.cpp b8248 displays total and free VRAM capacity during CUDA device initialization. • Displays total and free VRAM capacity during CUDA device initialization, giving practitioners immediate visibility into GPU memory availability at startup.

llama.cpp b8247

Mon, 09 Mar 2026 05:18:53 GMT

llama-bench gains `-hf` and `-hff` flags for Hugging Face model sourcing, with `--mmap 1` now default. • Adds `-hf` and `-hff` flags to `llama-bench` for specifying models directly from Hugging Face. • `llama-bench` now defaults to `--mmap 1` (memory-mapped file I/O enabled by default). Breaking changes: • `llama-bench` now uses `--mmap 1` by default; existing benchmarks that relied on the previous default (mmap off) may see different memory or performance characteristics without explicitly setting `--mmap 0`.

llama.cpp b8240

Sun, 08 Mar 2026 21:34:06 GMT

llama.cpp b8240 adds ELU op support to the Vulkan backend. • Adds ELU (Exponential Linear Unit) operator support to the ggml-Vulkan backend, expanding the set of model ops that can run GPU-accelerated on Vulkan devices.

llama.cpp b8234

Sun, 08 Mar 2026 06:44:12 GMT

SYCL backend gains Flash Attention support for fp32, fp16, Q4, Q5, and Q8 precision levels. • Adds Flash Attention support to the SYCL backend for fp32, fp16, Q4, Q5, and Q8 quantization formats, including JIT compilation support.

llama.cpp b8233

Sat, 07 Mar 2026 18:45:13 GMT

llama.cpp b8233 adds the GATED_DELTA_NET op to ggml with fused backend support, enabling Qwen3.5 dense model inference. • Adds `GATED_DELTA_NET` op to ggml, enabling support for Gated DeltaNet-based architectures including Qwen3.5 dense models. • Adds fused gated delta net backend support with a corresponding check in the llama model loader.

llama.cpp b8227

Sat, 07 Mar 2026 10:01:27 GMT

llama.cpp b8227 introduces a fully refactored Autoparser architecture with Kimi 2.5 model support. • Adds a Kimi 2.5 model parser as part of the new Autoparser architecture. • Complete refactoring of the parser architecture into a new Autoparser system.

llama.cpp b8225

Sat, 07 Mar 2026 09:52:44 GMT

llama.cpp server now preserves Anthropic thinking blocks during message conversion • Server preserves Anthropic thinking blocks when converting messages, maintaining chain-of-thought content across API interactions.

llama.cpp b8210

Thu, 05 Mar 2026 20:59:37 GMT

CUDA backend gains async CPU-to-GPU tensor copies with relaxed sync requirements, improving multi-token throughput. • Adds CPU-to-CUDA async copy capability to ggml_backend_cuda_cpy_tensor_async(), replacing synchronous copies to reduce GPU pipeline stalls. • Introduces `ggml_backend_sync_mode` opt-in mechanism to relax explicit synchronization between host-to-device copies and graph execution, extensible to backends beyond CUDA (e.g., Vulkan).

llama.cpp b8191

Tue, 03 Mar 2026 05:02:17 GMT

llama.cpp b8191 adds an optimized Q4_1 matrix-multiplication OpenCL kernel for Adreno GPUs. • Adds an optimized `q4_1` matrix-multiplication (`mm`) OpenCL kernel targeting Qualcomm Adreno GPUs, accelerating Q4_1 quantized inference on mobile/edge hardware via `ggml_cl_mul_mat_q4_1_f32_adreno`.

llama.cpp b8179

Fri, 27 Feb 2026 20:28:00 GMT

llama.cpp b8179 adds CDNA3 MFMA tensor core flash attention for AMD MI300X, yielding up to +39% prompt-processing throughput. • Adds CDNA3 MFMA tensor core support for the flash attention MMA kernel on AMD MI300X (gfx942), using `v_mfma_f32_16x16x16_f16` (FP16 input, FP32 accumulate), with head sizes 64, 80, 96, 112, and 128 supported — delivering pp512 +7%, pp1024 +13%, pp2048 +23%, pp4096 +39% on Qwen2.5-1.5B Q4_K_M. • Routes MI300X flash attention automatically: MMA kernel used for prompt processing (effective batch `eff_nq >= 128`), tile kernel for small batches — no manual dispatch required.

llama.cpp b8177

Fri, 27 Feb 2026 18:15:10 GMT

llama.cpp server now mirrors `/v1/responses` to `/responses` for API consistency. • Adds `/responses` as a mirror endpoint for `/v1/responses` in the server, matching the existing `/v1/chat/completions` pattern.

llama.cpp b8173

Fri, 27 Feb 2026 09:12:28 GMT

llama.cpp server gains comma-separated `--alias` and new `--tags` flag for multi-alias model routing via `/v1/models` • Adds comma-separated value support to `--alias` in the llama.cpp server, allowing a single model to be registered under multiple aliases simultaneously (stored as a deduplicated `std::set`). • Adds new `--tags` flag to the llama.cpp server for attaching informational metadata to a model (not used for routing). • The `/v1/models` endpoint now exposes `aliases` and `tags` fields, reflecting all registered aliases and tags for each loaded model.

llama.cpp b8164

Fri, 27 Feb 2026 00:34:49 GMT

llama.cpp b8164 adds option to merge gate and up (exp) weights for MoE models during GGUF conversion. • Adds `create_tensor_gate_up` helper and `gate_up` merged-weight support to `convert_hf_to_gguf.py` for all MoE models, enabling fused gate/exp weight tensors in converted GGUF files.

llama.cpp b8153

Wed, 25 Feb 2026 22:50:38 GMT

llama.cpp server gains multi-modal prompt caching support • Enables multi-modal prompt caching in the `server`, allowing image and other non-text prompt prefixes to be cached and reused across requests.

llama.cpp b8152

Wed, 25 Feb 2026 20:56:11 GMT

llama.cpp server gains multi-modal context checkpoint support for hybrid and recurrent memory architectures. • Adds multi-modal context checkpoint support to the `server`, enabling checkpoint/restore of KV cache state across hybrid-ISWA and recurrent memory backends.

llama.cpp b8144

Tue, 24 Feb 2026 09:57:21 GMT

llama.cpp server adds `max_completion_tokens` request property as the replacement for deprecated `max_tokens`. • Adds `max_completion_tokens` request property to the server API, setting the upper bound for reasoning+output tokens and superseding the now-deprecated `max_tokens` field.

llama.cpp b8132

Mon, 23 Feb 2026 00:01:57 GMT

llama.cpp b8132: CLI now passes the model's text filename to the model context. • The CLI provides the model with its text filename, giving the model awareness of the file it was loaded from.

llama.cpp b8128

Sun, 22 Feb 2026 15:50:12 GMT

llama.cpp b8128 adds support for the Kanana-2 model architecture. • Adds support for the Kanana-2 model architecture.

llama.cpp b8123

Sat, 21 Feb 2026 19:59:51 GMT

llama.cpp b8123 adds a ROCm 7.2 build target covering ten GPU architectures including gfx942 and gfx1201. • Adds a build target to generate ROCm 7.2 artifacts supporting GPU architectures `gfx1151`, `gfx1150`, `gfx1200`, `gfx1201`, `gfx1100`, `gfx1101`, `gfx1030`, `gfx908`, `gfx90a`, and `gfx942`.

llama.cpp b8117

Fri, 20 Feb 2026 13:16:12 GMT

llama.cpp b8117 adds RISC-V Vector (RVV) vec dot kernels for iq1_s, iq1_m, iq2_s, iq3_s, tq1_0, and tq2_0 quantization types. • Adds RVV (RISC-V Vector) vectorized dot-product kernels for quantization types `iq2_s`, `iq3_s`, `tq1_0`, `tq2_0`, `iq1_s`, and `iq1_m`, accelerating inference on RISC-V hardware with vector extensions. • Adds VLEN-aware switching logic for RVV vec dot kernels, enabling correct dispatch across RISC-V implementations with different vector register lengths.

llama.cpp b8116

Fri, 20 Feb 2026 10:16:56 GMT

llama-quantize gains --dry-run to preview quantization plans and imatrix warnings before committing. • Adds `--dry-run` flag to `llama-quantize`, letting practitioners preview tensor layout, bits-per-weight, and imatrix requirements without writing output files. • Adds a courtesy warning when a selected quantization type requires an importance matrix (imatrix) but none is provided, including an edge-case guard for `Q2_K`. • Displays model and quantization BPW (bits per weight) summary upon quantization completion.

llama.cpp b8110

Fri, 20 Feb 2026 04:04:36 GMT

llama.cpp b8110 adds support for the PaddleOCR-VL multimodal model. • Adds support for the PaddleOCR-VL vision-language model, including updated CLIP loader parameters and multimodal rope (`mtmd_decode_use_mrope`) handling via `convert_hf_to_gguf.py`.

llama.cpp b8107

Fri, 20 Feb 2026 00:44:11 GMT

llama.cpp b8107 adds flash attention toggle via `ctx_params` in the multimodal (`mtmd`) attention builder. • Adds flash attention on/off control via `ctx_params` in the `mtmd` `build_attn` path, enabling runtime toggling of flash attention for multimodal contexts.

llama.cpp b8106

Fri, 20 Feb 2026 00:06:52 GMT

llama.cpp b8106 adds JAIS-2 Arabic-English bilingual model architecture support with quantization down to Q2_K. • Adds `JAIS-2` architecture support for Inception AI's Arabic-English bilingual model family (Jais-2-8B and Jais-2-70B), including `LLAMA_VOCAB_PRE_TYPE_JAIS2` pre-tokenizer type and control vector support via `build_cvec`. • Supports JAIS-2 quantizations: BF16, Q8_0, Q6_K, Q5_K_M, Q5_0, Q4_K_M, Q4_0, Q3_K_M, and Q2_K.

llama.cpp b8102

Thu, 19 Feb 2026 18:44:16 GMT

llama.cpp b8102 adds tokenizer support for LFM2.5-Audio-1.5B, enabling lightweight audio model conversion via convert_hf_to_gguf.py. • Adds tokenizer support for the LFM2.5-Audio-1.5B model architecture, convertible via `python3 convert_hf_to_gguf.py /path/to/LFM2.5-Audio-1.5B/audio_detokenizer`. • Adds LFM2 SWA (Sliding Window Attention) model support alongside the new tokenizer.

llama.cpp b8100

Thu, 19 Feb 2026 14:04:28 GMT

llama.cpp b8100 adds full Modern BERT support including mean and rank pooling with GELU op. • Adds full Modern BERT model support, including mean pooling, rank pooling with GELU op, and a dense first layer as per the Modern BERT research paper, convertible via `convert_hf_to_gguf.py`.

llama.cpp b8099

Thu, 19 Feb 2026 11:11:32 GMT

llama.cpp b8099 adds FP16 MMA path on PowerPC for Q4/Q8 matmul, delivering 1.5–2x prompt-processing speedup. • Adds FP16×FP16→FP32 MMA acceleration path on PowerPC for Q4 and Q8 matrix multiplication, achieving 1.5–2x improvement in prompt-processing speed for Q4_0 and Q8_0 models on CPU.

llama.cpp b8094

Wed, 18 Feb 2026 21:22:52 GMT

llama.cpp server now saves generated text in `/slots` endpoint when `LLAMA_SERVER_SLOTS_DEBUG=1` • Sets `LLAMA_SERVER_SLOTS_DEBUG=1` to enable saving generated text in the `/slots` endpoint for server-side slot inspection and debugging.

llama.cpp b8091

Wed, 18 Feb 2026 21:07:56 GMT

llama.cpp b8091 adds JIT shader compilation for WebGPU, covering mul_mat, get_rows, and scale operations. • Adds JIT (just-in-time) compilation for WebGPU shaders covering `mul_mat`, `get_rows`, and `scale` operations, enabling runtime-optimized GPU kernel dispatch. • Reorganizes the WebGPU backend into an all-encompassing shader library, splitting large shaders into multiple static strings and refactoring `argmax`, `set_rows`, flash attention, and matrix multiplication into a unified format.

llama.cpp b8093

Wed, 18 Feb 2026 20:38:03 GMT

llama.cpp b8093 adds support for the GLM-OCR model. • Adds support for the GLM-OCR model, including updates to `convert_hf_to_gguf.py` for conversion.

llama.cpp b8073

Tue, 17 Feb 2026 02:28:54 GMT

llama.cpp b8073 adds support for Tiny Aya models. • Adds support for Tiny Aya models, including updated vocabulary and tokenizer regex handling.

llama.cpp b8054

Sat, 14 Feb 2026 22:24:56 GMT

llama.cpp b8054 adds multimodal support for Nemotron Nano 12B v2 VL. • Adds vision-language (VL) support for the Nemotron Nano 12B v2 model, including pre-downsampled position embeddings during GGUF conversion for fixed input sizes.

llama.cpp b8051

Sat, 14 Feb 2026 21:13:43 GMT

llama.cpp b8051 adds NetBSD build support. • Adds NetBSD build support, expanding platform coverage to include NetBSD targets.

llama.cpp b8036

Sat, 14 Feb 2026 06:16:45 GMT

llama.cpp b8036 adds support for the GLM MoE DSA model architecture. • Adds support for the GLM MoE DSA model architecture (note: the indexer component is not yet supported in this release).

llama.cpp b8033

Sat, 14 Feb 2026 05:18:32 GMT

llama.cpp b8033 adds `--verbose-prompt` flag support. • Adds `--verbose-prompt` flag to print the full tokenized prompt before inference, aiding prompt debugging and inspection.

llama.cpp b8021

Fri, 13 Feb 2026 08:35:31 GMT

llama.cpp b8021 adds OpenCL support for Q4_1 quantization in matrix-vector and matrix-matrix operations. • Adds OpenCL backend support for `q4_1` quantized matrix-vector (mv) and matrix-matrix (mm) operations, enabling GPU-accelerated inference with Q4_1 models on OpenCL devices. • Adds general `q4_0` matrix-matrix (mm) support in the OpenCL backend.

llama.cpp b8020

Fri, 13 Feb 2026 07:16:49 GMT

llama-parallel gains the `-kvu` flag in b8020 • Adds `-kvu` argument to `llama-parallel` for controlling key-value cache usage during parallel inference.

llama.cpp b8003

Thu, 12 Feb 2026 01:37:34 GMT

llama.cpp b8003 adds Kimi-K2.5 multimodal model support including image and vision capabilities. • Adds support for the Kimi-K2.5 model, including image and vision inference via updated `convert_hf_to_gguf.py` with new `kimi-k2.5` keys and `V_MMPROJ` / `V_M_IMP_NORM` tensor mappings.

llama.cpp b7990

Tue, 10 Feb 2026 21:22:13 GMT

llama.cpp b7990 adds support for the Qwen3.5 model series with FULL_ATTENTION_INTERVAL metadata. • Adds `FULL_ATTENTION_INTERVAL` metadata key to support the Qwen3.5 series hybrid attention architecture. • Supports loading and running Qwen3.5 series models.

llama.cpp b7983

Tue, 10 Feb 2026 10:35:33 GMT

llama.cpp b7983 adds quantized MoE matrix multiplication for the CANN backend, supporting Q4_0 and Q8_0 weight formats. • Adds `ggml_cann_mul_mat_id_quant` to the CANN backend, enabling quantized matrix multiplication for Mixture of Experts (MoE) models with Q4_0 and Q8_0 weight formats, automatic F16 type conversion, and both per-expert and broadcast input modes.

llama.cpp b7973

Mon, 09 Feb 2026 02:04:35 GMT

llama.cpp b7973 adds support for Qwen3.5 dense and MoE model architectures. • Adds inference support for Qwen3.5 dense and Mixture-of-Experts (MoE) model architectures (vision not included).

llama.cpp b7964

Sat, 07 Feb 2026 00:28:32 GMT

llama.cpp b7964 adds support for the Step3.5-Flash model architecture. • Adds support for the Step3.5-Flash model architecture (`LLM_ARCH_STEP35`), including GGUF conversion, `swiglu clamp` KV entries, and KV shifting disabled for this architecture.

llama.cpp b7957

Fri, 06 Feb 2026 18:55:07 GMT

llama.cpp b7957 adds Kimi-Linear model support with backend-agnostic inference and MLA KV cache. • Adds support for the Kimi-Linear (Kimi-K2) model architecture, including `convert_hf_to_gguf.py` conversion, backend-agnostic inference, and MLA KV cache via the `llm_arch_is_hybrid` path. • Introduces `llm_graph_input_mem_hybrid_k` class to support MLA-style KV cache for hybrid architectures like Kimi-Linear. • Adds Kimi-K2-specific end-of-generation (EOG) tokens to the vocabulary recognition layer. • Adds `causal_conv1d` static function abstracting convolution logic for query, key, and value projections in Kimi-Linear layers. • Supports Kimi-Linear MoE parameters including `n_experts`, `n_experts_used`, and `score_func` via the shared `TextModel` conversion path in `convert_hf_to_gguf.py`. • Skips quantization of `conv1d` weights for Kimi-Linear models during GGUF conversion.

llama.cpp b7951

Thu, 05 Feb 2026 18:57:21 GMT

llama.cpp b7951 adds adaptive CPU/GPU interleave on Metal for improved Apple Silicon inference. • Adds adaptive CPU/GPU interleave scheduling on Metal, dynamically balancing compute across CPU and GPU based on the number of graph nodes for Apple Silicon and iOS deployments.

llama.cpp b7919

Tue, 03 Feb 2026 10:58:18 GMT

llama.cpp b7919 adds Falcon-H1-Tiny-Coder FIM token support to the vocabulary. • Adds Falcon-H1-Tiny-Coder Fill-in-the-Middle (FIM) tokens to the vocabulary, enabling FIM-mode code completion with that model.

llama.cpp b7897

Sat, 31 Jan 2026 05:06:05 GMT

llama-quantize gains `--tensor-type-file` to specify per-tensor quantization types via a file. • Adds `--tensor-type-file` option to `llama-quantize`, allowing per-tensor quantization type assignments to be loaded from a file.

llama.cpp b7896

Sat, 31 Jan 2026 00:28:56 GMT

llama.cpp b7896 adds vision-only support for MiniCPM-o 4.5 via the multimodal (mtmd) subsystem. • Adds MiniCPM-o 4.5 (vision-only) support to the `mtmd` multimodal subsystem.

llama.cpp b7876

Thu, 29 Jan 2026 21:43:10 GMT

llama.cpp b7876 enables Hexagon HTP offloading on Windows on Snapdragon (WoS) devices. • Enables GPU offloading to the Qualcomm Hexagon HTP (via the `ggml-backend-dl` dynamic backend) on Windows on Snapdragon (WoS) devices, alongside existing Android support. • Adds `run-bench.ps1` and `run-tool.ps1` PowerShell scripts to simplify benchmarking and tool execution in the WoS/Hexagon build environment. • Renames the backend documentation directory from `docs/backend/hexagon` to `docs/backends/snapdragon`, consolidating Android and Windows build guides into a unified README. Breaking changes: • The backend documentation directory is renamed from `docs/backend/hexagon` to `docs/backends/snapdragon`; any scripts or CI referencing the old path will break.

llama.cpp b7870

Thu, 29 Jan 2026 07:43:23 GMT

llama-batched-bench gains the `-kvu` flag for additional benchmarking control. • Adds `-kvu` argument to `llama-batched-bench` for expanded benchmark configuration.

llama.cpp b7864

Wed, 28 Jan 2026 21:43:06 GMT

llama.cpp b7864 adds draftless self-speculative decoding via `--spec-type`, no separate draft model required. • Adds `--spec-type` CLI flag to enable self-speculative decoding without a separate draft model, using an n-gram map to generate draft tokens from the primary model itself. • Introduces `common_speculative_type` enum and a vector of speculative states to support multiple speculation modes within a single server session. • Adds common_speculative_begin() and common_speculative_state::accept() to the speculative decoding API, enabling structured draft/accept lifecycle management in code integrating llama.cpp. Breaking changes: • The `speculative.draftless_type` config field is renamed to `speculative.type`. • The `--spec-draftless` flag is replaced by `--spec-type`; any existing invocation using `--spec-draftless` will break. • The `--spec-config` flag is removed entirely. • `common_speculative_params` is removed from the API.

llama.cpp b7858

Wed, 28 Jan 2026 17:49:27 GMT

llama.cpp b7858 adds a new ggml backend for Virglrenderer API Remoting GPU acceleration. • Adds a new ggml backend for Virglrenderer API Remoting acceleration, enabling GPU-accelerated inference over virtualized graphics stacks.

llama.cpp b7851

Wed, 28 Jan 2026 06:08:04 GMT

WebGPU backend gains new unary operators (SOFTPLUS, EXPM1, FLOOR, CEIL, ROUND, TRUNC) and per-thread state isolation. • Adds SOFTPLUS, EXPM1, FLOOR, CEIL, ROUND, and TRUNC unary operators to the WebGPU backend, each with f16/f32 and inplace/non-inplace shader variants. • Splits `webgpu_context` shared state into global state (Instance, Device, Adapter, capabilities) and per-thread state (Queue, pipelines, parameter buffers), enabling safe multi-threaded WebGPU execution. • Re-enables XIELU unary operator support in the WebGPU backend after correcting IEEE 754 bit-pattern preservation in parameter passing.

llama.cpp b7842

Tue, 27 Jan 2026 05:14:45 GMT

llama.cpp b7842 enables FP16 MMA kernels on PowerPC CPUs for faster CPU inference. • Enables FP16 MMA (matrix multiply-accumulate) kernels for PowerPC CPUs via `ggml-cpu`, accelerating on-CPU inference on PPC hardware.

llama.cpp b7833

Sun, 25 Jan 2026 16:14:08 GMT

llama.cpp b7833 adds tiled flash attention for CPU prompt processing, improving long-context performance. • Adds tiled flash attention (FA) kernel for CPU prompt-processing, replacing the prior vector kernel that degraded performance on long contexts — tile sizes tuned on AMD EPYC 64-core hardware.

llama.cpp b7832

Sun, 25 Jan 2026 15:24:11 GMT

llama.cpp b7832 adds V-less KV-cache support, reducing memory overhead for compatible model architectures. • Adds V-less KV-cache mode, allowing the value cache to be omitted for model architectures that share V with K, reducing memory usage during inference.

llama.cpp b7821

Sat, 24 Jan 2026 08:25:47 GMT

llama.cpp b7821 enables CUDA graphs for `n-cpu-moe` workloads, improving GPU inference performance for MoE models. • Enables CUDA graphs support for `n-cpu-moe` (CPU-offloaded mixture-of-experts) inference, reducing GPU launch overhead and improving throughput on CUDA backends.

llama.cpp b7813

Thu, 22 Jan 2026 23:00:38 GMT

llama.cpp server now preserves reasoning_content in assistant messages, enabling round-trip chain-of-thought workflows. • Adds `reasoning_content` input support to the server, allowing assistant messages that include reasoning content to be passed back in subsequent requests.

llama.cpp b7807

Thu, 22 Jan 2026 17:43:21 GMT

llama.cpp b7807 lets manual tensor-type overrides take precedence during quantization • Manual overrides of tensor types now take precedence over automatic selection during quantization (#18952).

llama.cpp b7793

Thu, 22 Jan 2026 00:28:27 GMT

llama.cpp b7793 adds partial OpenAI Responses API support at `POST /v1/responses` including tool calls and image/file inputs. • Adds `POST /v1/responses` endpoint to the llama.cpp server with partial OpenAI Responses API compatibility, supporting text, image, and file input message types. • Adds tool call support (function calling) to the `/v1/responses` endpoint for both streaming and non-streaming modes. • Feeds reasoning/thinking model outputs through the chat template pipeline in the `/v1/responses` endpoint. • Rejects `input_file` input type on `/v1/responses` with an explicit error since it is not supported by the underlying chat completion backend.

llama.cpp b7789

Wed, 21 Jan 2026 16:27:31 GMT

llama.cpp b7789 adds hybrid ISWA memory support via `llama_memory_hybrid_iswa` • Adds `llama_memory_hybrid_iswa` hybrid sliding-window attention memory implementation in `src/llama-memory-hybrid-iswa.cpp`.

llama.cpp b7761

Sat, 17 Jan 2026 02:08:01 GMT

llama.cpp b7761 extends the WebGPU backend with sampling-critical operators including top-k, argsort, argmax, cumsum, PAD, LOG, and more. • Adds `SOFTPLUS`, `EXPM1`, `FLOOR`, `CEIL`, `ROUND`, and `TRUNC` unary operators to the WebGPU backend, each with f16/f32 and inplace/non-inplace shader variants. • Adds `argmax`, `cumsum`, `sum`, and `sum_rows` operators to the WebGPU backend. • Adds `argsort` to the WebGPU backend using a multi-pass strategy. • Adds `top-k` support to the WebGPU backend, enabling GPU-accelerated sampling. • Adds `PAD`, `LOG`, `CPY`, `GET_ROWS`, and `SET_ROWS` (with i32 indices) operators to the WebGPU backend. • Adds `FILL` operator support to the WebGPU backend via pre-WGSL porting. • Adds `get_memory` reporting for the WebGPU backend. • Ports unary operators to the pre-WGSL pipeline path in the WebGPU backend.

llama.cpp b7756

Fri, 16 Jan 2026 15:49:30 GMT

llama.cpp b7756 replaces minja with a new built-from-scratch Jinja template engine for chat template rendering. • Introduces a new built-in Jinja template engine (lexer, parser, and runtime) in `common`, replacing the prior `minja` dependency — supports filters (`selectattr`, `rejectattr`, `split`/`rsplit` with `maxsplit`, `trim` with args, `tojson` with `separators` and `indent`), keyword arguments, array slicing, macros, and `lstrip_block` behavior matched to Hugging Face JS semantics. • Adds support for rendering chat templates for additional model families including Gemma, MiniMax, GLM-4.5, Kimi-K2, Nemotron, IBM Granite, and Apertus through the new engine. • Extends `test-chat-template` to consolidate Jinja rendering tests (absorbing the former `test-chat-jinja` target) with an `--output` flag for inspecting rendered results. • Adds a fuzz test target for the Jinja engine covering both template rendering and function argument handling. Breaking changes: • `minja` is removed and its call sites are replaced by the new Jinja runtime; any build or code that depended on `minja` headers or the `test-chat-jinja` CMake target will break. • The `test-chat-jinja` CMake target is removed; tests previously run through it now run under `test-chat-template`.

llama.cpp b7752

Fri, 16 Jan 2026 12:33:50 GMT

llama.cpp b7752 adds CANN backend support for the GGML_OP_GATED_LINEAR_ATTN operator. • Adds `GGML_OP_GATED_LINEAR_ATTN` operator support to the CANN backend, enabling gated linear attention models to run on Ascend NPUs.

llama.cpp b7748

Thu, 15 Jan 2026 23:16:20 GMT

llama.cpp b7748 adds the adaptive-p sampler with EMA-based probability targeting and configurable decay. • Adds `adaptive-p` sampler to `common_params_sampling` and the `samplers` vector, with parameters including `decay` (default `0.9`), `min_target`, `max_target` (clamped to `[0.0, 1.0]`), and a `use_power_law` flag for power-law mode. • Exposes `adaptive-p` via `common_sampler_types_from_chars` so it can be selected alongside existing samplers using single-character sampler chain strings.

llama.cpp b7740

Thu, 15 Jan 2026 07:25:36 GMT

llama.cpp b7740 adds Hexagon HVX OP_CPY support and makes host buffers optional on Snapdragon backends. • Adds `GGML_HEXAGON_EXPERIMENTAL` environment variable support, now documented in the Hexagon readme. • Adds `OP_CPY` support for fp16/fp32 → fp16/fp32 copy operations on the Hexagon HVX backend, enabling broader model layer offload to Snapdragon DSPs. • Makes host buffers optional on the Hexagon backend, with repack buffers automatically disabled when host buffers are not available.

llama.cpp b7727

Wed, 14 Jan 2026 02:37:48 GMT

llama.cpp b7727 adds support for the EXAONE Mixture-of-Experts (MoE) model architecture. • Adds support for the EXAONE MoE model architecture, including Multi-Token Prediction (MTP) inference.

llama.cpp b7723

Tue, 13 Jan 2026 19:31:59 GMT

llama.cpp b7723 adds fattn-mma-f16 flash attention kernel for AMD RDNA4 GPUs via HIP. • Adds `fattn-mma-f16` flash attention kernel for RDNA4 GPUs via the HIP backend, enabling hardware-accelerated matrix multiply-accumulate attention on AMD RDNA4 hardware.

llama.cpp b7718

Tue, 13 Jan 2026 11:11:12 GMT

llama-bench gains a `direct_io` parameter for benchmarking with direct I/O enabled. • Adds `direct_io` parameter to `llama-bench` to control direct I/O during benchmarking runs.

llama.cpp b7716

Tue, 13 Jan 2026 00:24:38 GMT

llama.cpp b7716 adds a server argument to disable prompt caching for deterministic OpenAI-compat responses. • Adds a server argument to disable prompt caching, enabling deterministic responses for clients restricted to OpenAI-compatible requests.

llama.cpp b7713

Mon, 12 Jan 2026 20:56:59 GMT

llama.cpp b7713 adds `--kv-unified` flag to the batched example CLI. • Adds `--kv-unified` flag to the batched example CLI, enabling unified KV cache mode required for batches containing coupled sequences.

llama.cpp b7712

Mon, 12 Jan 2026 20:07:06 GMT

Vulkan memory logger now togglable via environment variable • Vulkan `memory_logger` is now controlled by an environment variable, allowing runtime enable/disable without recompilation.

llama.cpp b7694

Sat, 10 Jan 2026 14:40:53 GMT

llama.cpp b7694 adds support for named remote presets • Supports named remote presets, allowing preset configurations to be referenced by name from remote sources

llama.cpp b7691

Sat, 10 Jan 2026 09:15:49 GMT

llama.cpp b7691 embeds all licenses into compiled binaries, accessible via new `--license` flag. • Adds `--license` flag to display all embedded licenses directly from the binary, eliminating the need to distribute separate LICENSE files alongside executables.

llama.cpp b7689

Sat, 10 Jan 2026 00:01:22 GMT

llama.cpp b7689 adds Gemma3n multimodal support with MobileNetV5 vision encoder for image processing. • Adds `Gemma3nVisionModel` with MobileNetV5 vision encoder to `convert_hf_to_gguf.py`, enabling conversion and inference of Gemma3n multimodal models. • Adds `gemma3n` vision projector entry to `gguf-py/gguf/constants.py` and supporting tensor mappings in `tensor_mapping.py` for the new architecture. • Implements `mobilenetv5.cpp` with `clip_graph_mobilenetv5` struct to support the MobileNetV5 vision encoding pipeline within llama.cpp.

llama.cpp b7681

Fri, 09 Jan 2026 00:17:36 GMT

llama.cpp b7681 adds remote preset support, letting a single HF repo reference multiple GGUF sources. • Adds remote preset support via `--offline` option awareness, allowing argument configurations to be fetched from a remote Hugging Face repo — including one HF repo pointing to multiple HF repos for multi-GGUF use cases.

llama.cpp b7678

Thu, 08 Jan 2026 20:47:41 GMT

llama.cpp b7678 adds initial FlashAttention implementation for the WebGPU backend. • Adds initial FlashAttention implementation to the WebGPU (ggml-webgpu) backend, enabling attention computation on GPU via the WebGPU API. • Adds fast matrix and matrix-vector multiplication kernels to the WebGPU backend, improving throughput for inference on WebGPU-capable devices. • Adds Q4_0 quantized matrix multiplication support to the WebGPU backend. • Adds subgroup matrix (cooperative matrix) shader support to the WebGPU backend for improved GPU utilization where the hardware capability is present. • Adds F16 accumulation with shared-memory staging to WebGPU attention and matmul kernels. • Adds Emscripten/WASM build support for the WebGPU backend, including memory64 and pthread integration. • Adds a CI workflow (`ggml-ci`) for WebGPU backend testing.

llama.cpp b7672

Thu, 08 Jan 2026 12:00:30 GMT

llama.cpp b7672 adds per-device free memory targeting to llama-fit-params • Adds free memory target per device to `llama-fit-params`, enabling more precise multi-GPU memory allocation control.

llama.cpp b7668

Thu, 08 Jan 2026 08:30:41 GMT

llama.cpp b7668 adds `--direct-io` flag to bypass OS page cache during model loading • Adds `--direct-io` flag to enable O_DIRECT model loading, bypassing the OS page cache to reduce memory pressure on large model loads (automatically disabled when `--mmap` is explicitly enabled).

llama.cpp b7655

Wed, 07 Jan 2026 12:33:13 GMT

llama.cpp b7655 adds `llama-debug`, a new CLI utility for inspecting and validating converted GGUF models. • Adds `llama-debug` binary with `--save-logits`, `--prompt` / `-p`, `--tensor-filter`, `--verbose`, and `--embedding` flags for saving and inspecting logits, embeddings, and tensor outputs from a converted model. • Adds `compare_tokens.py` script (in `scripts/utils/`) with an optional `-v` flag to diff tokenizer output — token IDs and prompts — between an original and a converted model. • Integrates token-ID comparison into existing `compare-logits.py` and `semantic_check.py` verification scripts so tokenizer mismatches are caught before logit/embedding checks run. • Uses `llama_model_n_embd_out` in the debug example to correctly handle late-interaction retriever models (e.g. LFM2-ColBert-350M) whose output embeddings are down-projected to a lower dimension. Breaking changes: • `logits.cpp` and the `examples/model-conversion` directory have been removed; use `llama-debug --save-logits` instead.

llama.cpp b7652

Wed, 07 Jan 2026 02:15:04 GMT

llama.cpp b7652 adds f16/f32 flash attention, new matmul kernels, and GET_ROWS/SET_ROWS ops for Hexagon DSP backends. • Adds f16 and f32 flash attention (`flash_attn_ext`) support for the Hexagon backend, with double-buffered DMA prefetching for K, V, Mask, and Q tensors using `FLASH_ATTN_BLOCK_SIZE` of 128. • Adds `GET_ROWS` f32→f32 and `SET_ROWS` fp32→f16 (with i32/i64 row index) operations to the Hexagon backend. • Introduces `vec_dot_f16_f16` and `vec_dot_f16_f16_rx2` HVX kernel variants for fp16×fp16 matrix multiplication in the Hexagon backend, with automatic fallback for complex broadcasting. • Adds `quantize_fp32_f16` helper to copy and convert weights from DDR to VTCM on Hexagon, enabling the optimized matmul path when VTCM capacity allows. • Improves fp16 and fp32 matmul throughput on the Hexagon backend via aligned HVX operations, with optimized threading for SET_ROWS when processing few rows.

llama.cpp b7644

Tue, 06 Jan 2026 19:53:25 GMT

llama.cpp server adds thinking/reasoning content blocks to the Anthropic Messages API for reasoning models • Adds thinking content blocks to the Anthropic Messages API when using `--reasoning-format deepseek` with the `thinking` parameter enabled — both non-streaming (thinking block before text in content array) and streaming (`thinking_delta` events with correct block indices) are supported. • Tracks reasoning state across streaming chunks via the `anthropic_has_reasoning` member variable to correctly handle partial streaming of thinking content.

llama.cpp b7639

Mon, 05 Jan 2026 23:26:10 GMT

llama.cpp b7639 adds support for the LFM2-ColBERT-350M embedding model. • Adds llama_model_n_embd_out() API function that returns `hparams.n_embd_out` when set, falling back to `hparams.n_embd`. • Adds support for the LFM2-ColBERT-350M model architecture.

llama.cpp b7630

Mon, 05 Jan 2026 09:42:05 GMT

CANN backend gains ADD + RMS_NORM operator fusion, toggled via `GGML_CANN_OPERATOR_FUSION` env var. • Adds `GGML_CANN_OPERATOR_FUSION` environment variable (default: `false`) to enable fused ADD + RMS_NORM kernel execution on the CANN backend, reducing memory access overhead via a single ACLNN `AddRmsNorm` call.

llama.cpp b7614

Fri, 02 Jan 2026 19:56:56 GMT

llama.cpp b7614 adds support for the Maincoder-1B model architecture. • Adds support for the Maincoder-1B model architecture via a new `maincoder.cpp` backend.

llama.cpp b7607

Thu, 01 Jan 2026 20:07:18 GMT

llama.cpp b7607 adds support for the Youtu-VL vision-language model. • Adds support for the Youtu-VL vision-language model.

llama.cpp b7605

Thu, 01 Jan 2026 19:40:22 GMT

llama.cpp b7605 adds support for JinaBertModel with non-gated feed-forward networks, including German tokenizer variant. • Adds support for `JinaBertModel` with non-gated (standard GELU) feed-forward network variant, detected automatically from tensor dimensions. • Adds `jina-v2-de` tokenizer variant in the converter to support the German Semantic V3 model.

llama.cpp b7603

Thu, 01 Jan 2026 19:39:33 GMT

llama.cpp b7603 adds support for the Solar-Open model with chat template and reasoning tags. • Adds support for the Solar-Open model, including its LLM type, vocabulary end-of-generation handling, and a `solar-open` chat template in `llama-chat`.

llama.cpp b7574

Mon, 29 Dec 2025 17:40:39 GMT

llama.cpp b7574 adds `-to` flag for configurable HTTP read timeouts on the inference server. • Adds `-to` command-line argument to `server` to set a configurable HTTP read timeout (previously fixed at 600 seconds), with internal TTFT crash protection raised to 90 days. • Enables per-child-model HTTP timeout configuration when running server with child models.

llama.cpp b7567

Mon, 29 Dec 2025 10:16:17 GMT

llama.cpp b7567 adds CUDA Blackwell feature support for non-native builds. • Enables CUDA Blackwell GPU features for non-native (cross-compiled) builds via PR #18436.

llama.cpp b7563

Sun, 28 Dec 2025 17:02:17 GMT

llama.cpp b7563 adds support for the Plamo3 model architecture. • Adds support for the Plamo3 model architecture, including chat template integration.

llama.cpp b7557

Sun, 28 Dec 2025 08:57:32 GMT

llama.cpp b7557 adds six new x86-64 CPU architecture targets when building with `GGML_CPU_ALL_VARIANTS=On` • Adds `ivybridge`, `piledriver`, `cannonlake`, `cascadelake`, `cooperlake`, and `zen4` as supported x86-64 CPU backends when building with `GGML_CPU_ALL_VARIANTS=On`

llama.cpp b7531

Wed, 24 Dec 2025 16:44:44 GMT

llama.cpp b7531 adds support for the LlamaBidirectionalModel architecture, enabling llama-embed-nemotron embedding models. • Adds support for the `LlamaBidirectionalModel` architecture (`llama-embed` arch), enabling use of bidirectional embedding models such as llama-embed-nemotron.

llama.cpp b7525

Wed, 24 Dec 2025 06:50:29 GMT

llama.cpp b7525 adds `LLAMA_ARG_OVERRIDE_TENSOR` env var to expose the `-ot` tensor-override flag to environment-based config. • Adds `LLAMA_ARG_OVERRIDE_TENSOR` environment variable as an alternative way to supply the `-ot` (override tensor) argument, enabling tensor-routing control without modifying CLI invocations.

llama.cpp b7524

Tue, 23 Dec 2025 21:52:05 GMT

llama.cpp server now reports 0% processing state via `return_progress` • Extends `return_progress` in the llama.cpp server to also report the 0% processing state, giving clients visibility into the earliest stage of request handling.

llama.cpp b7515

Tue, 23 Dec 2025 01:43:47 GMT

llama.cpp b7515 adds Granite Embedding and ModernBERT model support with HuggingFace-to-GGUF conversion. • Adds support for Granite Embedding models, including HuggingFace-to-GGUF conversion via `convert_hf_to_gguf.py` and inference via `llama-embedding`. • Adds ModernBERT architecture support (encoder-only, without `head.norm`) with alternating local/global sliding window attention, GEGLU activation, and alternating RoPE, implemented in `src/models/modern-bert.cpp`. • Adds `swa_period` (sliding window attention period) GGUF hyperparameter, written and read via `gguf-py/gguf/gguf_writer.py` and `src/llama-hparams.h`, enabling dense-every-N-layers SWA patterns for ModernBERT-family models.

llama.cpp b7502

Sun, 21 Dec 2025 22:34:30 GMT

Vulkan backend gains async tensor loading with `GGML_VK_DISABLE_HOST_VISIBLE_VIDMEM` for faster model load times • Implements `set_tensor_async` and event interfaces for the Vulkan backend, enabling async load code paths in `llama_model_loader::load_all_data`.

llama.cpp b7492

Sun, 21 Dec 2025 02:38:25 GMT

llama.cpp server gains auto-sleep after configurable idle timeout to free resources automatically • Adds auto-sleep to the `llama-server` that suspends the server context and frees model resources after a configurable number of idle seconds, reducing memory and GPU pressure when the server is not actively serving requests.

llama.cpp b7487

Sat, 20 Dec 2025 08:52:58 GMT

llama.cpp server gains autoload-on-startup and preset-only option support • Adds load-on-startup support to `llama-server`, allowing a model to be loaded automatically when the server starts rather than requiring a separate API call. • Adds preset-only options to `llama-server`, enabling configuration presets to be applied without requiring a full model specification.

llama.cpp b7484

Fri, 19 Dec 2025 19:19:29 GMT

llama.cpp b7484 adds true Q8_0 quantization on the Hexagon NPU for more accurate mixed-precision matmul. • Adds a CMake option for configuring FP32 quantize group size on the Hexagon NPU backend (`ggml-hexagon`). • Implements true Q8_0 quantization on the Hexagon NPU (`ggml-hexagon`) for more accurate mixed-precision matrix multiply operations, replacing the previous approximate path. Breaking changes: • Linux releases will migrate from .zip archives to .tar.gz archives — deployment scripts that fetch or unpack Linux release artifacts will need to be updated.

llama.cpp b7480

Fri, 19 Dec 2025 12:28:44 GMT

llama.cpp b7480 adds cascading presets with a global section and multi-source preset merging. • Adds a `global` section to preset configuration, enabling cascade presets sourced from different origins to merge in a defined order. • Filters server-controlled args before to_ini() serialization, preventing server-managed values from being written back into preset files. • Linux releases will switch from `.zip` to `.tar.gz` archives in an upcoming release. Breaking changes: • Linux release archives will soon change from `.zip` to `.tar.gz` format, requiring updates to any deployment scripts that reference or extract the current archive format.

llama.cpp b7476

Fri, 19 Dec 2025 06:44:46 GMT

llama.cpp b7476 adds Vulkan concurrent perf logger mode and promotes sync logging to a runtime env var. • Adds `GGML_VK_PERF_LOGGER_CONCURRENT=1` env var to enable concurrent Vulkan perf logger mode, which times groups of overlapping GPU work rather than individual operations with barriers — useful for understanding whether operation-level or group-level optimization is needed (requires `GGML_VK_PERF_LOGGER` also set). • Adds `GGML_VK_SYNC_LOGGER=1` env var, replacing the previous `ENABLE_SYNC_LOGGING` compile-time switch and making sync logging configurable at runtime without recompilation. Breaking changes: • Linux releases will switch from `.zip` archives to `.tar.gz` archives — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7475

Fri, 19 Dec 2025 00:47:32 GMT

llama.cpp b7475 adds ASR support for the LFM2-Audio-1.5B conformer model. • Adds automatic speech recognition (ASR) support for the LFM2-Audio-1.5B (conformer) audio model. Breaking changes: • Linux releases will soon switch from .zip to .tar.gz archives — update any deployment scripts that reference .zip download URLs.

llama.cpp b7445

Wed, 17 Dec 2025 12:25:53 GMT

llama.cpp b7445 enables `-kvu` flag in `llama-perplexity` for hellaswag and winogrande benchmarks with coupled sequences. • Adds `--kv-unified` (`-kvu`) flag support to `llama-perplexity`, enabling hellaswag and winogrande benchmarks that use coupled sequences. Breaking changes: • Linux releases will switch from `.zip` archives to `.tar.gz` archives — update any deployment scripts that unzip Linux builds.

llama.cpp b7429

Tue, 16 Dec 2025 18:48:52 GMT

llama.cpp b7429 adds GLM4V vision encoder support including Metal acceleration and MoE variant. • Adds support for the GLM4V vision encoder model, including Metal (Apple GPU) acceleration and MoE variant support. • Introduces `GGML_ROPE_TYPE_MRNORM` rope type to support the GLM4V text model's multi-resolution positional encoding. Breaking changes: • Linux releases will change from .zip archives to .tar.gz archives — update any deployment scripts that unzip Linux release artifacts.

llama.cpp b7426

Tue, 16 Dec 2025 16:16:35 GMT

llama.cpp b7426 adds Nemotron Nano 3 model parsing support • Adds Nemotron Nano 3 chat template parsing and support in the common layer. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives — deployment scripts that fetch or unpack Linux binaries must be updated.

llama.cpp b7418

Tue, 16 Dec 2025 08:25:39 GMT

llama.cpp b7418 adds support for the NVIDIA Nemotron Nano 3 model. • Adds support for running and converting the NVIDIA Nemotron Nano 3 model. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives — update deployment scripts accordingly.

llama.cpp b7414

Tue, 16 Dec 2025 01:09:16 GMT

llama.cpp b7414 adds support for the KORMo causal language model architecture. • Adds support for the `KORMoForCausalLM` model architecture, enabling inference on KORMo causal language models. • Adds a KORMo-specific tokenizer (`KORMo Tokenizer`) using the Qwen2 pretokenizer under the hood. Breaking changes: • Linux release archives will switch from .zip to .tar.gz format — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7406

Mon, 15 Dec 2025 04:15:33 GMT

SYCL backend gains GPU support for gpt-oss models via mxfp4 matrix multiply, add-id, and swiglu_oai ops • Adds SYCL GPU support for gpt-oss models via the `add-id`, `mul_mat` (mxfp4), and `swiglu_oai` operations, enabling hardware-accelerated inference for this model family on Intel SYCL devices. Breaking changes: • Linux release archives will change from .zip to .tar.gz format — update any deployment scripts that download or unpack Linux builds.

llama.cpp b7405

Mon, 15 Dec 2025 04:09:00 GMT

llama.cpp b7405 adds GLM-ASR model support and announces Linux archive format change. • Adds support for the GLM-ASR model architecture, including a new conversion script for HuggingFace checkpoints. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives — update any deployment scripts that unzip Linux builds.

llama.cpp b7378

Sat, 13 Dec 2025 13:35:11 GMT

llama.cpp b7378 adds `llama-completion` to bash completion executables and warns of upcoming Linux archive format change. • Adds `llama-completion` to the completion-bash executables, enabling tab-completion support for the `llama-completion` command in Bash. • Linux releases will switch from `.zip` to `.tar.gz` archives in an upcoming release — deployment scripts should be updated accordingly.

llama.cpp b7376

Sat, 13 Dec 2025 03:08:37 GMT

llama.cpp b7376 adds support for negated CLI arguments across common args. • Adds negated argument support to common args, allowing boolean flags to be explicitly disabled at the command line (e.g. `--no-*` style negation), including `LLAMA_ARG_NO_HOST` and related options. Breaking changes: • Linux releases will change from .zip archives to .tar.gz archives — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7366

Sat, 13 Dec 2025 03:05:19 GMT

llama.cpp b7366 adds `-mm` and `-mmu` as short forms for `--mmproj` and `--mmproj-url`. • Adds `-mm` and `-mmu` as short-form aliases for `--mmproj` and `--mmproj-url` respectively. Breaking changes: • Linux releases will change from .zip archives to .tar.gz archives — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7364

Fri, 12 Dec 2025 18:36:16 GMT

llama.cpp b7364 adds a minimalist multi-thread progress bar and upcoming Linux archive format change. • Adds a minimalist multi-thread progress bar to common tooling for tracking parallel operation progress. Breaking changes: • Linux releases will soon switch from .zip archives to .tar.gz archives — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7353

Thu, 11 Dec 2025 04:15:08 GMT

llama.cpp b7353 enables Jinja templating by default in the CLI. • Jinja templating is now enabled by default in the CLI (previously required explicit opt-in via `common/arg.cpp`). Breaking changes: • Linux releases will move from .zip to .tar.gz archives — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7352

Thu, 11 Dec 2025 04:09:38 GMT

llama-server gains INI-based per-model presets and recursive GGUF directory loading for multi-model deployments. • Adds INI-based per-model configuration presets to `llama-server` via `config.ini`, with auto-creation and template generation; keys must follow the `LLAMA_ARG_*` naming convention and section names starting with `LLAMA_ARG_*` are rejected. • Adds `LLAMA_ARG_*` environment variable passthrough so global settings reach child model processes without CLI duplication — children receive only minimal args (executable, model, port, alias). • Adds `LLAMA_ARG_HOST` and `CHILD_ADDR` support for controlling child process network binding in multi-model router mode. • Replaces flat `--models-dir` directory scan with recursive traversal using nested vendor/model layouts (e.g. `vendor/model/*.gguf`); model names now reflect the relative path within `--models-dir` rather than just the filename. Breaking changes: • Linux releases will move from `.zip` archives to `.tar.gz` archives — deployment scripts that fetch or unpack Linux release artifacts will need to be updated.

llama.cpp b7348

Wed, 10 Dec 2025 19:59:38 GMT

llama.cpp b7348 ships a new interactive CLI with `--show-timings`, `--image`, `--audio`, `/read`, and speculative-decoding args. • Adds `--show-timings` flag to the new CLI (enabled by default) to display inference timing information after each completion. • Adds `--image` and `--audio` flags to the new CLI for passing media inputs directly on the command line. • Adds `/read` command in the interactive CLI session to read input from a file. • Adds arguments for speculative decoding in the new CLI experience. • Introduces a new interactive CLI experience (the old CLI is moved to `llama-completion`), with a loading animation, colored output (default `auto`), and `reasoning_content` support. • Linux releases now ship as `.tar.gz` archives instead of `.zip`. Breaking changes: • The old `llama.cpp` CLI binary is replaced by the new CLI; the previous CLI is now available as `llama-completion` — any scripts invoking the old CLI entry point will need to be updated. • Linux release archives are changing from `.zip` to `.tar.gz` format — deployment scripts that unzip Linux builds will break and must be updated.

llama.cpp b7347

Wed, 10 Dec 2025 18:48:05 GMT

llama.cpp b7347 adds support for the Qwen3-Next-80B-A3B model architecture with correct 48-layer configuration. • Adds support for the Qwen3-Next-80B-A3B model architecture, correctly configured with 48 layers and a new `80B-A3B` type name. Breaking changes: • Linux releases will switch from `.zip` archives to `.tar.gz` archives — deployment scripts that fetch or unpack Linux builds will need to be updated.

llama.cpp b7342

Wed, 10 Dec 2025 01:31:14 GMT

llama.cpp b7342 adds tokenizer/parser support for Ministral, Mistral Large 3, and Devstral 2 models. • Adds parser support for Ministral, Mistral Large 3, and Devstral 2 model families, enabling correct tokenization and prompt formatting for these models. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives — deployment scripts that fetch or unpack Linux builds will need to be updated.

llama.cpp b7337

Tue, 09 Dec 2025 21:39:04 GMT

llama.cpp b7337 adds `GGML_BACKTRACE_LLDB` env var for lldb-based backtraces on macOS. • Adds `GGML_BACKTRACE_LLDB` environment variable to enable lldb-based backtrace printing on macOS, replacing the default that could cause terminal crashes. Breaking changes: • Linux releases will switch from .zip to .tar.gz archives — deployment scripts that reference .zip paths will need to be updated.

llama.cpp b7332

Tue, 09 Dec 2025 14:41:36 GMT

llama.cpp b7332 adds full readline-style console editing with history navigation, word movement, and Delete key support. • Adds `KEY_CTRL_ARROW_LEFT` and `KEY_CTRL_ARROW_RIGHT` word-jump navigation to the interactive console, parsing ANSI sequences `1;5D`/`1;5C` on Linux and `dwControlKeyState` CTRL modifier on Windows. • Adds Delete key support to the console — `VK_DELETE` on Windows and `ESC[3~` sequence on Linux — enabling forward character deletion with full UTF-8 support. • Adds arrow-left/right in-line editing to the console with UTF-8 support, plus Home/End key support on both Windows and Linux. • Adds bash-style history navigation: Up/Down arrows browse history, edits to any history line persist per-session, and pressing Enter appends the edited version as a new history entry. Breaking changes: • Linux release archives will change from .zip to .tar.gz format — deployment scripts that unzip Linux releases will need to be updated.

llama.cpp b7331

Tue, 09 Dec 2025 14:16:53 GMT

llama.cpp b7331 adds CANN backend support for partial RoPE and Vision mode rotation in vision-language models. • Adds CANN backend support for partial RoPE (`rope_dims < ne0`), splitting tensors into rotated head and unrotated tail portions to handle modern model architectures. • Adds CANN backend support for `GGML_ROPE_TYPE_VISION` mode, enabling full-tensor rotation with Vision-style dimension pairing (dimension i paired with i+n_dims) for vision-language models. • CANN RoPE implementation now supports both F32 and F16 data types with intermediate F32 conversion, and handles both contiguous and non-contiguous tensor layouts. • Improves CANN cache invalidation logic to include `rope_dims` and `indep_sects` parameters. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives — deployment scripts that unzip Linux release artifacts will need to be updated.

llama.cpp b7329

Tue, 09 Dec 2025 07:02:07 GMT

llama.cpp b7329 adds token matching support to llama-grammar, enabling token-level trigger patterns. • Adds token matching support to `llama-grammar`, allowing grammar rules to match on individual tokens in addition to string patterns. • Refactors `trigger_patterns` in `llama-grammar` to replay tokens instead of replaying the entire string, improving pattern-matching fidelity. Breaking changes: • Linux releases will switch from `.zip` archives to `.tar.gz` archives — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7328

Tue, 09 Dec 2025 05:13:43 GMT

llama.cpp b7328 adds support for the Rnj-1 model architecture. • Adds support for the Rnj-1 model architecture (built on a refactored Gemma3 backend). Breaking changes: • Linux releases will switch from .zip to .tar.gz archives — update any deployment scripts that reference .zip paths.

llama.cpp b7318

Mon, 08 Dec 2025 12:39:48 GMT

llama.cpp b7318 lets you configure `cache_reuse` per request on the server. • Adds `cache_reuse` as a per-request configurable field on the llama.cpp server, allowing fine-grained control over KV-cache reuse on a request-by-request basis. Breaking changes: • Linux releases will move from .zip archives to .tar.gz archives; deployment scripts that fetch or unpack Linux builds must be updated.

llama.cpp b7315

Sun, 07 Dec 2025 22:07:23 GMT

llama.cpp b7315 adds Llama 4 scaling support for Mistral-Large (DeepSeek architecture). • Adds Llama 4 scaling support for Mistral-Large models using the DeepSeek architecture. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives — update any deployment scripts that reference the .zip format.

llama.cpp b7312

Sun, 07 Dec 2025 03:18:23 GMT

llama.cpp b7312: `--color` gains `on/off/auto` modes, defaulting to `auto`; Linux releases switching to `.tar.gz`. • Changes `--color` flag to accept explicit values `on`, `off`, or `auto` (default: `auto`), replacing the previous boolean toggle. Breaking changes: • Linux releases will use `.tar.gz` archives instead of `.zip` — deployment scripts that unzip Linux builds must be updated.

llama.cpp b7310

Sat, 06 Dec 2025 21:38:33 GMT

llama.cpp b7310 adds a Vulkan perf-logger env var to control stats-dump frequency and improves fusion-op tracking. • Adds an environment variable to control how frequently the Vulkan performance logger dumps stats (set a very large value to defer output until context destruction). • Adds fusion info strings to Vulkan perf-logger tracking, logging only one item per fused op for cleaner output. Breaking changes: • Linux releases will soon switch from .zip to .tar.gz archives — update any deployment scripts that unzip Linux builds.

llama.cpp b7276

Fri, 05 Dec 2025 01:21:16 GMT

llama.cpp b7276 adds CUDA/HIP GPU acceleration for CUMSUM and TRI tensor operations. • Adds GPU-accelerated `CUMSUM` and `TRI` tensor operations for CUDA and HIP (ROCm) backends, enabling models that use these ops to run on-device without CPU fallback. Breaking changes: • Linux releases will switch from `.zip` archives to `.tar.gz` archives — deployment scripts that unzip Linux builds will break when the format change takes effect.

llama.cpp b7275

Thu, 04 Dec 2025 23:04:06 GMT

llama.cpp b7275 adds Metal GPU kernels for TRI, FILL, EXPM1, and SOFTPLUS ops on Apple Silicon. • Adds Metal GPU kernel implementations for the TRI, FILL, EXPM1, and SOFTPLUS operations, enabling these ops to run natively on Apple Silicon GPUs. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives in an upcoming release — deployment scripts that unzip Linux builds will need to be updated.

llama.cpp b7256

Wed, 03 Dec 2025 23:52:10 GMT

llama.cpp b7256 extends CUDA Flash Attention to Volta GPUs via generalized MMA kernel support. • Adds Volta GPU architecture support to the CUDA Flash Attention (FA) implementation via a generalized matrix multiply-accumulate (MMA) kernel, expanding hardware compatibility for CUDA-accelerated inference. Breaking changes: • Linux releases will soon switch from .zip archives to .tar.gz archives — deployment scripts that fetch or unpack Linux release artifacts will need to be updated.

llama.cpp b7247

Wed, 03 Dec 2025 12:12:46 GMT

llama.cpp b7247 adds WebGPU backend support for Emscripten/Wasm builds with fast matrix multiply and Q4_0 quantization. • Adds WebGPU backend support for Emscripten (Wasm) builds, enabling browser-side inference with fast f16/f32 matrix and matrix-vector multiplication via subgroup matrix shaders. • Supports Q4_0 quantized matrix multiply in the WebGPU backend, bringing quantized inference to Wasm targets. • Adds `memory64` support via Emscripten's `memory64` option to enable `get_memory` in Wasm builds. • Enables subgroup matrix (cooperative matrix) acceleration in the WebGPU backend, with automatic detection of available subgroup matrix configurations at runtime. • Adds toggles for subgroup matrix and f16 support specifically on Nvidia+Vulkan targets. Breaking changes: • Linux releases will switch from `.zip` archives to `.tar.gz` archives — deployment scripts that unzip Linux release artifacts will need to be updated.

llama.cpp b7243

Wed, 03 Dec 2025 04:22:17 GMT

llama.cpp b7243 adds `--media-path` for serving local media files from the built-in server. • Adds `--media-path` flag to the `server` subcommand to specify a local directory from which the server will serve media files. Breaking changes: • Linux releases will switch from .zip archives to .tar.gz archives — deployment scripts that unzip Linux builds will need to be updated.