◆ Local LLM Runtimes

oobabooga textgen

v4.9 open-source

# Default — localhost only, no extra flag needed
python server.py

# Opt into network exposure
python server.py --listen
# or for public API access
python server.py --public-api
textgen --no-electron
textgen --split-mode tensor
textgen --listen --nowebui
python server.py --preserve-thinking
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/projects"]
    }
  }
}
curl -X POST http://localhost:5000/v1/internal/model/load \
  -H 'Content-Type: application/json' \
  -d '{"model_name": "Mistral-7B-Instruct", "instruction_template": "Mistral", "args": {}}'
python server.py --ik --model my-model.gguf
curl http://localhost:5000/v1/completions -H 'Content-Type: application/json' -d '{"model": "my-model", "prompt": "Explain KV cache", "max_tokens": 64, "echo": true, "logprobs": 5}'
ANTHROPIC_BASE_URL=http://127.0.0.1:5000 claude
python server.py --loader llama.cpp --spec-type ngram-mod --spec-ngram-size-n 8 --spec-ngram-size-m 2 --spec-ngram-min-hits 2
python server.py --loader llama.cpp --parallel 4 --n-ctx 32768
python server.py --temperature 0.7 --min-p 0.05 --top-k 40 --repetition-penalty 1.1 --chat-template-file my_template.jinja
python server.py --cpu-moe
python server.py --loader exllamav3 --enable-tp --tp-backend cuda
python server.py --model mistral-7b-instruct-v0.2.Q5_K_M.gguf --ctx-size 32768 --cache-type q4_0
python server.py --torch-compile
python download-model.py TheBloke/Mixtral-8x7B-v0.1-GGUF --exclude-pattern '*.Q8_0.gguf'
python server.py --old-colors
python server.py --loader exllamav2 --num_experts_per_token 2 --model <your-moe-model>
python server.py --nowebui --admin-key mysecretkey
curl -X POST http://localhost:5000/v1/internal/logits -H 'Authorization: Bearer mysecretkey' -H 'Content-Type: application/json' -d '{"prompt": "The capital of France is"}'
curl http://localhost:5000/v1/internal/lora -H 'Authorization: Bearer mysecretkey'
curl -X POST http://localhost:5000/v1/internal/stop-generation

Summary

oobabooga textgen is an open-source desktop application that functions as a local LLM runtime for developers and chat users. As it has no licensing cost, it can be run via a self-contained, portable build for Linux, Windows, and macOS. It is designed for users who interact with local-llm-runtime tools, offering features such as chat interfaces, multimodal vision capabilities via attached images, and file ingestion for PDF or DOCX content. The application supports multiple backends like llama.cpp and Transformers, and exposes an OpenAI/Anthropic-compatible API with native tool-calling support. The project is actively maintained, with portable builds available across common operating systems.

What oobabooga textgen answers

What model formats does it support running?

GGUF (llama.cpp) models and others through attached backends like Transformers, ExLlamaV3, and TensorRT-LLM

How can I integrate it with existing workflows that expect specific API endpoints?

It exposes an OpenAI/Anthropic-compatible API with Chat, Completions, and Messages endpoints that support tool-calling

What types of files can I include as context for the conversation?

It processes text files, PDF documents, and .docx documents, in addition to attached images

Does changing the underlying model structure require restarting the application?

No, you can switch between backends and models without restarting

What kinds of external functions can the models interact with?

Models can call custom functions defined in single .py files, including those for web search, page fetching, and math

Command line

composed command

option surface across analysed releases

options at that release added, per release removed, per release

option	found in	since	description

Values are placeholders taken from each option’s declared default. Nothing is executed here — the output shown is a recording of a run that already happened.

Release history

v4.9 May 20, 2026 · issue -091

v4.9 adds MTP speculative decoding, web search snippets, live token-speed display, drag-and-drop upload, and localhost-only CORS hardening.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.9 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.9
```
└──▷ TRY IT
Lock the API surface to localhost only (default after v4.9) or expose it to the network when needed.
```
$ # Default — localhost only, no extra flag needed
python server.py

# Opt into network exposure
python server.py --listen
# or for public API access
python server.py --public-api
```
- ›Adds draft-mtp as a new --spec-type option for MTP speculative decoding; auto-enabled when loading MTP GGUFs (e.g. Qwen 3.6 MoE MTP builds).
- ›Treats negative --ctx-size values as auto (0), allowing automatic context-size selection.
- ›Adds snippet support to the web_search tool so results include a short text excerpt, often eliminating the need for a follow-up fetch_webpage call.
- ›Shows live generation speed (tokens/s) and context size in the UI while generating.
- ›Adds drag-and-drop file upload support to the chat input.
+8 moreshow less
- ›Adds Linux aarch64 portable builds for DGX Spark support.
- ›Stores Electron app data in user_data/cache/electron instead of the OS default location.
- ›Auto-detects and auto-selects sibling mmproj-*.gguf files when loading a model; these appear in the mmproj dropdown and are hidden from the regular model dropdown.
- ›Drops link URLs from fetch_webpage output (links render as plain text instead of [text](url) markdown), significantly reducing tokens used per page.
- ›Adds 'Check for updates' button, a folder picker for the models directory, right-click copy context menu, and a spellcheck toggle in the Electron Session tab.
- ›Improves web search security by rejecting non-HTTP links.
- ›Hides reasoning and tools controls in chat mode (shown only in instruct / chat-instruct modes).
- ›One-click installer now tracks the latest release tag instead of bleeding-edge main.
└──▷ BREAKING ON UPGRADE
- !Electron app data location changed from the OS default to user_data/cache/electron; existing cached data in the old location will not be picked up automatically.
v4.8 May 7, 2026 · issue -104

oobabooga textgen v4.8 adds a --no-electron flag, list-format API content support, and a redesigned chat composer.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.8 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.8
```
└──▷ TRY IT
Launch textgen headlessly in a server environment where you want browser access only, bypassing the Electron desktop window.
```
$ textgen --no-electron
```
- ›Adds --no-electron flag to skip the Electron desktop window and use the web UI in the browser instead.
- ›API now supports list-format content in tool and assistant messages.
- ›Redesigned chat composer with a taller input area and pinned paperclip and message-action buttons.
- ›Electron window now persists bounds and maximize state across launches.
- ›Smooth scroll animation when sending a new message in the chat UI.
v4.7.3 May 3, 2026 · issue -108

v4.7.3 adds a native Electron desktop app, tensor-parallel multi-GPU flag, and .jinja template support alongside a full UI overhaul.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.7.3 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.7.3
```
└──▷ TRY IT
Enable tensor parallelism across multiple GPUs to get 60%+ faster inference on a multi-GPU workstation.
```
$ textgen --split-mode tensor
```
Run textgen as a headless API server (no Electron window) for automated or remote access workflows.
```
$ textgen --listen --nowebui
```
- ›Adds --split-mode flag for llama.cpp with a tensor option enabling tensor parallelism that can make multi-GPU inference 60%+ faster; on ik_llama.cpp, tensor and row fall back to graph.
- ›Adds --listen flag to skip the native Electron window and expose the server directly; --nowebui also skips the window for headless server use.
- ›Adds support for standalone .jinja and .jinja2 instruction template files in the UI, alongside the existing .yaml format.
- ›Portable builds now bundle Electron and open as a native desktop window; launch with textgen (Linux/macOS) or textgen.bat (Windows) instead of the previous start scripts.
- ›Replaces DuckDuckGo HTML scraping in the web search tool with the ddgs Python library for more robust bot-blocking resistance.
+1 moreshow less
- ›Redesigns the chat UI with Inter font, Lucide SVG icons, a 3-button segmented mode selector (chat / chat-instruct / instruct), a rounded card chat input with accent-colored send button, and hairline sidebar handles.
└──▷ BREAKING ON UPGRADE
- !The --row-split flag is replaced by --split-mode; existing launch scripts using --row-split will need to be updated.
v4.6.2 Apr 23, 2026 · issue -118

oobabooga textgen v4.6.2 adds tool-call confirmation buttons, stdio MCP server support, and a --preserve-thinking flag.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.6.2 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.6.2
```
└──▷ TRY IT
Launch the server with thinking-block preservation enabled so prior-turn reasoning stays in context for multi-turn sessions.
```
$ python server.py --preserve-thinking
```
Connect a local stdio-based MCP server (e.g. a filesystem tool) so textgen can call it as an agent tool — same config format as Claude Desktop.
json
```
{
  "mcpServers": {
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/home/user/projects"]
    }
  }
}
```
- ›Adds --preserve-thinking CLI flag and matching chat template parameter preserve_thinking to control whether thinking blocks from prior turns are kept in context.
- ›Adds stdio MCP server support via user_data/mcp.json, using the same format as Claude Desktop and Cursor, enabling local subprocess-based MCP servers alongside existing HTTP MCP servers.
- ›Adds inline approve/reject/always-approve tool-call confirmation buttons, enabled via the new 'Confirm tool calls' checkbox in the Chat tab.
- ›Adds UI checkbox for preserve_thinking to control thinking-block retention without the CLI.
- ›Passes --draft-min 48 by default for llama.cpp draftless speculative decoding.
+1 moreshow less
- ›Limits 'Reasoning effort' and 'Enable thinking' controls to models whose chat template actually uses them.
v4.4 Apr 7, 2026 · issue -134

v4.4 adds MCP server integration, image-gen metadata in API responses, and new model-load API parameters.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.4 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.4
```
└──▷ TRY IT
Specify a prompt template when loading a model via the API, so the server applies the correct instruction format from the start.
```
$ curl -X POST http://localhost:5000/v1/internal/model/load \
  -H 'Content-Type: application/json' \
  -d '{"model_name": "Mistral-7B-Instruct", "instruction_template": "Mistral", "args": {}}'
```
Connect a remote MCP server so its tools are automatically discovered and available alongside local tools during chat.
📍In the Chat tab, locate the 'MCP servers' field and enter your server URL (one per line), e.g.: https://mcp.example.com/my-server Then send a message — tools are discovered and invoked automatically.
- ›Adds instruction_template and instruction_template_str parameters to the model load API endpoint, allowing callers to specify prompt format at load time.
- ›Adds MCP server support: enter one remote MCP server URL per line in the new 'MCP servers' field in the Chat tab — tools are discovered automatically and used alongside local tools.
- ›PNG images returned by the image generation API now include generation metadata (model, seed, dimensions, steps, CFG scale, sampler) embedded in the file.
- ›Adds Windows + ROCm portable builds (ROCm 7.2) alongside the existing Windows CUDA and Vulkan builds.
- ›Updates ik_llama.cpp backend, adding Gemma 4 model support.
└──▷ BREAKING ON UPGRADE
- !The deprecated settings parameter is removed from the model load endpoint — any client passing settings will need to be updated.
v4.3.3 Apr 4, 2026 · issue -137

oobabooga textgen v4.3.3 adds ik_llama.cpp backend, Gemma 4 tool-calling, and token-level logprobs in the completions API.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.3.3 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.3.3
```
└──▷ TRY IT
Run a full install using the ik_llama.cpp backend to get improved KV cache quantization and MoE/CPU optimizations.
```
$ python server.py --ik --model my-model.gguf
```
Retrieve token-level log probabilities for both the prompt and generated output via the completions API.
```
$ curl http://localhost:5000/v1/completions -H 'Content-Type: application/json' -d '{"model": "my-model", "prompt": "Explain KV cache", "max_tokens": 64, "echo": true, "logprobs": 5}'
```
- ›Adds --ik flag for full installs to switch to the ik_llama.cpp backend, which brings new quant types, more accurate KV cache quantization via Hadamard rotation (enabled by default), and MoE/CPU inference optimizations.
- ›Adds textgen-portable-ik portable builds as a no-install distribution of the ik_llama.cpp backend alongside the existing llama.cpp portables.
- ›Adds echo and logprobs parameters to the /v1/completions endpoint, returning token-level log probabilities for both prompt and generated tokens; token IDs are also returned in a new top_logprobs_ids field.
- ›Adds Gemma 4 support with tool-calling in both the API and UI.
- ›Transformers backend now autodetects torch_dtype from model config instead of always forcing bfloat16/float16; --bf16 still works as a manual override.
+1 moreshow less
- ›Removes the obsolete models/config.yaml file; instruction templates are now detected from model metadata.
└──▷ BREAKING ON UPGRADE
- !The models/config.yaml file is removed; instruction templates are now auto-detected from model metadata, so any local customizations stored in that file will no longer be applied.
v4.2 Mar 24, 2026 · issue -148

oobabooga textgen v4.2 adds an Anthropic-compatible /v1/messages endpoint and expands --extra-flags to accept literal flags.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.2 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.2
```
└──▷ TRY IT
Point Claude Code at your local textgen instance to use local models via the new Anthropic-compatible endpoint.
```
$ ANTHROPIC_BASE_URL=http://127.0.0.1:5000 claude
```
- ›Adds /v1/messages endpoint for Anthropic API compatibility, enabling Claude Code, Cursor, and other Anthropic clients to connect; supports system messages, content blocks, tool use, tool results, image inputs, and thinking blocks.
- ›Expands --extra-flags to accept literal flags directly (e.g. --extra-flags "--rpc 192.168.1.100:50052 --jinja"), in addition to the existing key=value format.
- ›Moves the OpenAI-compatible API from extensions/openai to modules/api; the old --extensions openai flag remains accepted as an alias for --api.
- ›Sets top_p=0.95 as the new default sampling parameter for API requests.
- ›Enables gradient_checkpointing by default during training to reduce VRAM usage.
+4 moreshow less
- ›Removes the higher_rank_limit training parameter.
- ›Updated UI theme with new colors, borders, and button styles across light and dark modes.
- ›Removes 52 obsolete instruction templates from 2023 (Airoboros, Baichuan, Guanaco, Koala, Vicuna v0, MOSS, and others).
- ›Reduces portable build sizes by switching to a stripped Python distribution.
└──▷ BREAKING ON UPGRADE
- !The default top_p for API requests is now 0.95; any setup relying on the previous default will receive different sampling behavior without explicit configuration.
- !The higher_rank_limit training parameter has been removed; configs or scripts referencing it will need to be updated.
- !52 instruction templates (Airoboros, Baichuan, Guanaco, Koala, Vicuna v0, MOSS, and others) have been removed; any saved configurations referencing these templates will lose them.
v4.1.1 Mar 18, 2026 · issue -154

Tool-calling in the UI, incognito chat, OpenAI API spec overhaul, and llama.cpp/ExLlamaV3 capability expansions in v4.1.1
└──▷ GET THIS VERSION
```
$ git clone --branch v4.1.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.1.1
```
└──▷ HOW TO FIND IT
Start the UI and keep a sensitive conversation entirely in RAM — no chat history ever written to disk.
📍In the chat interface, click the 'Incognito chat' button next to the 'New chat' button to open a temporary session that is never saved to disk.
- ›Adds tool-calling support in the chat UI: each tool is a single .py file dropped into user_data/tools; five built-in examples ship — web_search, fetch_webpage, calculate, get_datetime, and roll_dice — and each tool call renders as a collapsible accordion during streaming showing the function name, LLM-chosen arguments, and output.
- ›Adds reasoning_content field to OpenAI API streaming and non-streaming chat completions, isolating thinking-block content from content (which now carries only the post-thinking reply) even when tool calls are present.
- ›Adds tool_choice support and corrects the tool_calls response format in the OpenAI API for strict spec compliance, including returning finish_reason: tool_calls when tool calls are detected.
- ›Adds stream_options support with include_usage for streaming responses in the OpenAI API.
- ›Adds max_completion_tokens as an alias for max_tokens in the OpenAI API.
+20 moreshow less
- ›Adds support for the developer role in the OpenAI API, mapped to system.
- ›Makes the /v1/models endpoint show only the currently loaded model.
- ›Includes /v1 in the API URL printed to the terminal.
- ›Rewrites logprobs support in the OpenAI API for full spec compliance across llama.cpp, ExLlamaV3, and Transformers backends — both streaming and non-streaming responses now return token-by-token logprobs.
- ›Sets llama.cpp ctx-size to 0 (auto) by default when --gpu-layers is -1 (the default); on other loaders, 0 maps to 8192.
- ›Uses --fit-ctx 8192 to establish 8192 as the minimum acceptable context size for llama.cpp's --fit on mode (previously 4096).
- ›Reduces the llama.cpp --fit-target default from 1024 MiB to 512 MiB.
- ›Makes logit_bias and logprobs functional in llama.cpp API calls.
- ›Adds missing custom_token_bans parameter to the llama.cpp section of the UI.
- ›Adds native logit_bias and logprobs support for the ExLlamaV3 backend.
- ›Loads the ExLlamaV3 vision model and draft model before the main model so memory auto-splitting accounts for them.
- ›Adds 'Incognito chat' button next to the existing 'New chat' button — incognito chats are temporary, live in RAM, and are never written to disk.
- ›Introduces a new default sampling preset 'Top-P' (top_p: 0.95); removes the 'Qwen3 - Thinking', 'Qwen3 - No Thinking', 'min_p', and 'Instruct' presets.
- ›Removes rope scaling parameters alpha_value, rope_freq_base, and compress_pos_emb from the UI; they can still be passed to llama.cpp via --extra-flags.
- ›Increases the ctx-size slider maximum to 1M tokens in the UI (1024-token step).
- ›Adds a drag-and-drop UI component for reordering 'Sampler priority' items.
- ›Adds security hardening: server-side file save roots, image URL SSRF protection, and an extension allowlist.
- ›Replaces html2text with trafilatura for webpage text extraction, reducing navigation-bar boilerplate and lowering token usage in agentic tool-calling loops.
- ›Refactors thinking/reasoning extraction into a standalone module supporting Qwen, GPT-OSS, Solar, seed:think, and other formats, and auto-detects when a chat template appends <think> to prepend it to the reply during streaming.
- ›Removes the Gradio import in --nowebui mode, saving approximately 0.5–0.8 seconds on startup.
└──▷ BREAKING ON UPGRADE
- !The alpha_value, rope_freq_base, and compress_pos_emb rope scaling parameters are removed from the UI; pass them to llama.cpp via --extra-flags if still needed.
- !The 'Qwen3 - Thinking', 'Qwen3 - No Thinking', 'min_p', and 'Instruct' presets are removed; the new default preset is 'Top-P' (top_p: 0.95).
- !Thinking-block content now goes exclusively into the reasoning_content field; content in chat completions no longer includes thinking-block text, even when tool calls are present.
- !The /v1/models endpoint now returns only the currently loaded model instead of all available models.
- !llama.cpp ctx-size now defaults to 0 (auto) when --gpu-layers is -1; setups that relied on the previous default context size will see a different context window.
v4.0 Mar 7, 2026 · issue -164

oobabooga textgen v4.0 ships parallel API requests, a tool-calling overhaul, N-gram speculative decoding, new CLI generation-parameter flags, and major security hardening.
└──▷ GET THIS VERSION
```
$ git clone --branch v4.0 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v4.0
```
└──▷ TRY IT
Enable N-gram speculative decoding in llama.cpp for faster generation without a draft model.
```
$ python server.py --loader llama.cpp --spec-type ngram-mod --spec-ngram-size-n 8 --spec-ngram-size-m 2 --spec-ngram-min-hits 2
```
Run concurrent API requests through llama.cpp with 4-way parallelism, multiplying context by 4 to provide enough KV-cache headroom.
```
$ python server.py --loader llama.cpp --parallel 4 --n-ctx 32768
```
Set default API generation parameters and a custom jinja2 chat template at server startup so every API caller inherits them.
```
$ python server.py --temperature 0.7 --min-p 0.05 --top-k 40 --repetition-penalty 1.1 --chat-template-file my_template.jinja
```
- ›Enables parallel API requests via --parallel N for llama.cpp (multiply context length by N), ExLlamaV3, and TensorRT-LLM loaders for maximum throughput.
- ›Adds --spec-type, --spec-ngram-size-n, --spec-ngram-size-m, and --spec-ngram-min-hits parameters to llama.cpp for N-gram speculative decoding — set --spec-type ngram-mod to activate faster generation without a draft model.
- ›Adds --fit-target parameter to control per-GPU VRAM headroom (default: 1024 MiB) alongside automatic GPU-layer selection (-1 auto) for llama.cpp.
- ›Adds new CLI flags --temperature, --min-p, --top-k, --repetition-penalty, --enable-thinking, --reasoning-effort, and --chat-template-file to set default API generation parameters; --chat-template-file accepts .jinja or .yaml files.
- ›Adds --user-data-dir flag to customize the user data directory location; also auto-detects a ../user_data folder in portable mode for easier updates.
+17 moreshow less
- ›Restricts model loading over API to block extra_flags and trust_remote_code parameters, reducing attack surface.
- ›Migrates TensorRT-LLM from the old ModelRunner API to the new LLM API, which accepts any Transformers model as input and exposes more sampling parameters.
- ›Tool-calling now works correctly for Qwen 3.5, Devstral 2, GPT-OSS, DeepSeek V3.2, GLM 5, MiniMax M2.5, Kimi K2/K2.5, and Llama 4 models with strict OpenAI format compliance.
- ›Adds 'Target all linear layers' checkbox in the Training tab to apply LoRA to every nn.Linear layer except lm_head, working across any model architecture.
- ›Adds checkpoint resumption to the Training tab: HF Trainer checkpoint directories are detected automatically and training resumes with full optimizer/scheduler state.
- ›Adds dynamic padding for chat training datasets — batches pad to the longest sequence in the batch rather than always padding to cutoff_len.
- ›Conversations exceeding the cutoff length are now dropped instead of silently truncated (configurable).
- ›Introduces an adaptive-p sampler for llama.cpp, Transformers, ExLlamaV3, and ExLlamaV3_HF loaders that reshapes the logit distribution to favor tokens near a target probability.
- ›Adds a user persona dropdown in the Character settings tab to save and load user profiles (name, bio, profile picture) for switching personas.
- ›Adds Qwen 3.5 and Solar Open thinking block support to the UI.
- ›Replaces PyPDF2 with pymupdf for more accurate conversion of PDF inputs to text.
- ›Adds ROCm portable builds for Windows and CUDA 13.1 portable builds; restores macOS x86_64 (Intel) portable builds.
- ›llama.cpp binaries now autodetect CPU instruction set (AVX, AVX2, AVX-512) at runtime, replacing separate AVX/AVX2 builds.
- ›llama-server now spawns on port 5005 by default instead of a random port.
- ›SSE separator for streaming over the API changed from \r\n to \n to match OpenAI.
- ›Replaces Gradio with a custom fork (oobabooga/gradio) delivering faster startup, instant SSE message delivery (replacing 50 ms polling), and a new zero-rendering gr.Headless component that reduces chat-streaming overhead.
- ›Removes the ExLlamaV2 loader (archived upstream); EXL2 users must migrate to EXL3.
└──▷ BREAKING ON UPGRADE
- !The ExLlamaV2 loader has been removed; EXL2 model users must migrate to the EXL3 loader.
- !The Training_PRO extension has been removed after being made obsolete by the Training tab rewrite.
- !The SSE streaming separator over the API changed from \r\n to \n; clients that relied on \r\n framing will break.
- !Conversations exceeding cutoff_len during training are now dropped rather than silently truncated, changing training dataset behavior for affected entries.
- !Model loading over API no longer accepts extra_flags or trust_remote_code parameters; callers that pass these will be blocked.
- !TensorRT-LLM has been migrated from the old ModelRunner API to the new LLM API; existing TensorRT-LLM integrations built against ModelRunner will break.
v3.23 Jan 8, 2026 · issue -221

oobabooga textgen v3.23 improves table and separator styling in chat messages.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.23 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.23
```
- ›Improves the visual style of tables and horizontal separators rendered inside chat messages.
v3.20 Dec 7, 2025 · issue -253

oobabooga textgen v3.20 adds a full image generation tab with diffusers models, quantization, and an OpenAI-compatible image API.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.20 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.20
```
└──▷ HOW TO FIND IT
Access the new image generation UI to generate images with a diffusers model, enable quantization, and browse past generations in the gallery.
📍1. Launch textgen as usual and open the web UI. 2. Click the 'Image AI' tab. 3. Select a diffusers model (e.g. Z-Image-Turbo) and choose 4-bit or 8-bit quantization. 4. Enter a prompt and click Generate; past generations appear in the gallery below.
- ›New 'Image AI' tab supports image generation via diffusers models (e.g. Z-Image-Turbo) with 4-bit/8-bit quantization, torch.compile support, PNG metadata embedding, a gallery of past generations, and a progress bar.
- ›Adds an OpenAI-compatible API endpoint for image generation.
- ›Passes bos_token and eos_token to jinja2 templates, enabling template support for Seed-OSS-36B-Instruct and similar models.
- ›Enables flash_attention_2 by default for Transformers models.
- ›Adds LLM-generated prompt variations as an image generation feature within the new 'Image AI' tab.
v3.19 Nov 29, 2025 · issue -261

oobabooga textgen v3.19 adds a --ubatch-size slider for llama.cpp and updates the backend with Qwen3-Next support.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.19 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.19
```
- ›Adds --ubatch-size slider in the llama.cpp loader UI with new defaults tuned for MoE models, significantly improving prompt processing speeds in full-GPU and GPU+CPU configurations.
- ›Updates llama.cpp backend to commit ff55414c adding Qwen3-Next model support.
- ›Updates ExLlamaV3 to version 0.0.16.
v3.18 Nov 19, 2025 · issue -271

oobabooga text-gen v3.18 adds --cpu-moe flag for VRAM reduction and ROCm portable builds for AMD GPUs on Linux.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.18 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.18
```
└──▷ TRY IT
Run a large MoE model when VRAM is insufficient to hold all experts on the GPU.
```
$ python server.py --cpu-moe
```
- ›Adds --cpu-moe flag for the llama.cpp backend to offload MoE model experts to CPU, reducing VRAM usage on large mixture-of-experts models.
- ›Adds ROCm portable builds for AMD GPUs on Linux, enabling no-install GGUF inference on AMD hardware.
└──▷ BREAKING ON UPGRADE
- !macOS 13 wheels are removed and no longer supported.
v3.17 Nov 6, 2025 · issue -284

v3.17 adds Qwen3-VL support via llama.cpp update and bumps exllamav3 to 0.0.12.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.17 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.17
```
- ›Supports Qwen3-VL multimodal models through the updated llama.cpp backend.
- ›Updates exllamav3 backend to version 0.0.12.
- ›Adds weights_only=True to torch.load in Training_PRO to harden model checkpoint loading.
v3.16 Oct 23, 2025 · issue -298

oobabooga text-generation-webui v3.16 adds symlink support for portable builds and updates llama.cpp with Ling-mini-2.0 and Ring-mini-2.0 model support.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.16 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.16
```
- ›Enables running a portable Web UI build via a symlink, allowing flexible install layouts without copying the full package.
- ›Updates llama.cpp backend to add support for Ling-mini-2.0 and Ring-mini-2.0 model architectures.
- ›Updates exllamav3 to version 0.0.11.
- ›Updates triton-windows to version 3.5.0.post21.
v3.15 Oct 15, 2025 · issue -306

oobabooga textgen v3.15 locks down --trust-remote-code from UI/API and adds llama-server context-size error logging.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.15 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.15
```
- ›Logs an error when a llama-server request exceeds the context size, making oversized-context failures visible instead of silent.
- ›Makes --trust-remote-code immutable from the UI and API, preventing runtime elevation of code-execution trust.
v3.14 Oct 10, 2025 · issue -311

v3.14 adds /v1/internal/logits endpoint for exllamav3 loaders and qwen3-next model support via fla.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.14 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.14
```
- ›Implements the /v1/internal/logits API endpoint for the exllamav3 and exllamav3_hf loaders.
- ›Adds fla to requirements for Exllamav3 to enable support for qwen3-next models.
- ›Improves handling of multi-GPU setups when using Transformers with bitsandbytes (load-in-8bit and load-in-4bit).
v3.12 Sep 2, 2025 · issue -348

oobabooga textgen v3.12 adds thinking support for characters in chat-instruct mode and absolute-path GGUF model loading.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.12 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.12
```
- ›Extends the --model flag to accept absolute paths for GGUF models, e.g. --model /tmp/gemma-3-270m-it-IQ4_NL.gguf.
- ›Adds thinking support for characters in chat-instruct mode, enabled by Jinja2 template handling improvements.
- ›Adds a chat prompt template for the Seed-OSS-36B-Instruct model.
- ›Optimizes LaTeX rendering during streaming for long replies.
- ›Statically links the MSVC runtime in llama.cpp Windows binaries, removing the Visual C++ Redistributable dependency.
+2 moreshow less
- ›Enables --verbose llama.cpp terminal output with reduced verbosity.
- ›Makes venv portable installs compatible with Python 3.13.
└──▷ BREAKING ON UPGRADE
- !The --flash-attn flag has been removed; flash attention is now always enabled in llama.cpp and passing this flag will no longer work.
v3.11 Aug 19, 2025 · issue -362

oobabooga text-generation-webui v3.11 adds Tensor Parallelism support to ExLlamav3 loaders via new CLI options.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.11 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.11
```
└──▷ TRY IT
Launch the web UI with Tensor Parallelism enabled on ExLlamav3 to spread a large model across multiple GPUs.
```
$ python server.py --loader exllamav3 --enable-tp --tp-backend cuda
```
- ›Adds --enable-tp and --tp-backend CLI options to enable Tensor Parallelism for the ExLlamav3 and ExLlamav3_HF loaders.
- ›Sets multimodal status at model load time rather than checking on every generation, improving multimodal pipeline efficiency.
v3.10 Aug 12, 2025 · issue -363

oobabooga text-generation-webui v3.10 adds multimodal support to UI and API across llama.cpp and ExLlamaV3 loaders, plus speculative decoding.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.10 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.10
```
- ›Supports loading chat templates from chat_template.json files for EXL3, EXL2, and Transformers models.
- ›Passes --swa-full to llama-server when the streaming-llm option is checked, enabling SWA model compatibility in llama.cpp.
- ›Adds multimodal (image + text) support to the UI and API via the llama.cpp loader.
- ›Adds multimodal support to the UI and API via a new ExLlamaV3 loader.
- ›Adds speculative decoding to the new ExLlamaV3 loader.
+3 moreshow less
- ›Defaults max_tokens to 512 in the API (previously 16).
- ›Uses ExLlamaV3 instead of ExLlamaV3_HF as the default loader for EXL3 models, since it now supports multimodal and speculative decoding.
- ›Reorganizes the right sidebar in the UI for better layout.
└──▷ BREAKING ON UPGRADE
- !The default API value of max_tokens changes from 16 to 512; clients relying on the old default will now receive longer responses.
- !EXL3 models now default to the ExLlamaV3 loader instead of ExLlamaV3_HF; setups that depended on ExLlamaV3_HF behavior by default will need to explicitly select that loader.
v3.9.1 Aug 7, 2025 · issue -363

GPT-OSS template 'Continue' and 'Impersonate' actions now work correctly, plus a llama.cpp update.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.9.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.9.1
```
- ›GPT-OSS template special actions 'Continue' and 'Impersonate' now function correctly.
v3.9 Aug 6, 2025 · issue -363

oobabooga textgen v3.9 adds experimental GPT-OSS support with reasoning effort controls and standalone Jinja chat templates.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.9 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.9
```
- ›Adds a Reasoning effort UI element in the chat tab with low, medium, and high options for GPT-OSS models.
- ›Supports standalone .jinja chat templates, enabling GPT-OSS to be loaded through the Transformers backend.
- ›Makes web search functional with thinking models.
- ›Experimental support for GPT-OSS GGUF models (20B and 120B variants) via updated llama.cpp and Transformers 4.55.
v3.8 Jul 19, 2025 · issue -364

oobabooga textgen v3.8 unifies attention config, adds speculative decoding 'None' option, and bumps core backends.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.8 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.8
```
- ›Replaces use_flash_attention_2 and use_eager_attention with a unified attn_implementation setting in the Transformers loader, consolidating attention backend selection into a single config key.
- ›Adds a 'None' option for the speculative decoding model, allowing users to explicitly disable speculative decoding from the UI.
- ›Ignores add_bos_token in instruct prompts, deferring BOS token handling to the jinja2 template instead.
- ›Updates ExLlamaV3 to 0.0.5 and ExLlamaV2 to 0.3.2.
- ›Updates Transformers to 4.53 alongside the latest bitsandbytes, Accelerate, and PEFT versions.
+1 moreshow less
- ›Updates llama.cpp backend to commit 90083283ec254fa8d33897746dea229aee401b37.
└──▷ BREAKING ON UPGRADE
- !The use_flash_attention_2 and use_eager_attention Transformers loader settings are replaced by attn_implementation; existing configs using either removed key will need to be updated.
v3.7.1 Jul 9, 2025 · issue -364

textgen v3.7.1 adds user extension installs, moves 'Enable thinking' to sidebar, and switches to miniforge for org-friendly installs.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.7.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.7.1
```
- ›Supports installing user extensions in user_data/extensions/ without modifying the core installation.
- ›Moves the 'Enable thinking' checkbox from the Parameters tab to the right sidebar for faster access during chat.
- ›Switches the one-click installer from miniconda to miniforge, removing Anaconda licensing restrictions for organizations with 200+ users.
- ›Replaces the 'Generate' button label with 'Send' in the main chat interface.
- ›Standardizes margins and paddings across all chat styles.
v3.6.1 Jun 19, 2025 · issue -365

oobabooga text-generation-webui v3.6.1 merges tabs, adds autosave, a new Character tab, and exposes real models via /v1/models
└──▷ GET THIS VERSION
```
$ git clone --branch v3.6.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.6.1
```
- ›Exposes real model list via the GET /v1/models endpoint instead of a static placeholder.
- ›Merges the Default and Notebook tabs into a single Notebook tab, with a Session tab option to switch between one and two columns.
- ›Autosaves text in the Notebook tab (both generated and manually typed); adds 'New' and 'Rename' buttons for prompt management.
- ›Saved prompts relocated from user_data/prompts to user_data/logs/notebook.
- ›Adds a new Character tab for character settings.
+9 moreshow less
- ›Adds an option in the Session tab to exclude attachments from previous messages in the chat prompt.
- ›Moves 'Custom system message' to the Parameters > Generation tab.
- ›Truncates web search results to at most 8192 tokens to handle edge cases such as infinite-scrolling pages.
- ›Shows file sizes in the Model tab on 'Get file list'.
- ›Removes images and links from web search results to reduce noise and focus on relevant text content.
- ›Remembers the last selected chat for each chat mode and character.
- ›Forces dark theme on the Gradio login page.
- ›Hides the navigation bar on Ctrl+S / Show controls click.
- ›Updates llama.cpp backend and exllamav3 to 0.0.4.
└──▷ BREAKING ON UPGRADE
- !Saved prompts have been moved from user_data/prompts to user_data/logs/notebook; existing prompts must be manually relocated to the new path.
v3.5 Jun 11, 2025 · issue -365

oobabooga textgen v3.5 adds persistent UI settings, .docx attachments, Qwen3 presets, and RTX 50XX CUDA 12.8 support.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.5 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.5
```
- ›Adds CUDA 12.8 installation option supporting RTX 50XX NVIDIA Blackwell GPUs via ExLlamaV2/V3 and Transformers backends.
- ›Uses user_data/cache/gradio for Gradio temporary files instead of the system temporary folder.
- ›Adds Qwen3 presets ('Thinking' and 'No Thinking'); sets Qwen3 - Thinking as the new default preset.
- ›Adds support for .docx file attachments.
- ›Adds an option (configurable in the Session tab) to automatically convert long pasted text into an attachment; disabled by default.
+8 moreshow less
- ›Adds model name to each message's metadata, displayed in the UI on hover over the message date/time.
- ›Adds 'Restore preset', 'Neutralize samplers', and 'Restore character' buttons to the Parameters tab.
- ›Adds trash () buttons for deleting individual past chats.
- ›Reorganizes the Parameters tab with preset-saved parameters on the left and all other settings on the right.
- ›Optimizes chat streaming to update only the last message during generation, enabling smooth streaming even at 100k token context lengths.
- ›Extracts web search result text with formatting instead of collapsing all content to a single line.
- ›Shows llama.cpp prompt processing progress on a single line.
- ›Adds informative tooltips on hover for the file upload icon and web search checkbox.
v3.4.1 May 31, 2025 · issue -366

oobabooga textgen v3.4.1 adds file attachments, DuckDuckGo web search, message versioning, and a chat token counter.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.4.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.4.1
```
- ›Adds max_updates_second back to cap UI refresh rate during high-speed streaming (~200 tokens/second), resolving a performance bottleneck.
- ›Adds file attachment support to chat — text files and PDF documents are fully injected into the prompt (not RAG).
- ›Adds a web search feature powered by DuckDuckGo; the LLM auto-generates the search query from your input.
- ›Adds message version navigation ('swipes') — press left/right to browse previous reply versions, or press right at the latest to generate a new one.
- ›Adds a token counter to the chat tab covering input, history, and attachments.
+3 moreshow less
- ›Adds date/time display on chat messages.
- ›Adds footer buttons for editing individual chat messages.
- ›Adds a 'Branch here' footer button on chat messages to fork the conversation.
v3.3 May 16, 2025 · issue -366

oobabooga textgen v3.3 adds VRAM estimation with auto gpu-layers for GGUF and Tools support for the OpenAI-compatible API.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.3 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.3
```
└──▷ TRY IT
Load a large GGUF model for inference without guessing GPU layers — let the auto-calculator fit as many layers as free VRAM allows.
```
$ python server.py --model mistral-7b-instruct-v0.2.Q5_K_M.gguf --ctx-size 32768 --cache-type q4_0
```
- ›Automatically estimates VRAM usage for GGUF models and sets gpu-layers based on free VRAM on NVIDIA GPUs; recalculates in real time when ctx-size or cache-type changes in the UI.
- ›When loading a GGUF model via CLI (e.g. --model model.gguf --ctx-size 32768 --cache-type q4_0), --gpu-layers is now calculated automatically — no manual flag required.
- ›Adds Tools support for the OpenAI-compatible API.
- ›Adds the top_n_sigma sampler to the llama.cpp loader.
- ›Renders max_updates_second obsolete with a new dynamic Chat Message UI update speed that substantially reduces CPU usage in Chat mode.
+4 moreshow less
- ›Simplifies the Model tab by splitting settings into 'Main options' and 'Other options', with 'Other options' hidden in a closed accordion by default.
- ›Streamlines the UI in portable builds: hides non-functional items such as training, shows only the llama.cpp loader, and excludes non-working extensions to reduce build size.
- ›Shows the list of available files when a user attempts to download an entire GGUF repository instead of a specific file.
- ›Handles short arguments (e.g. ot) in the --extra-flags option for the llama.cpp loader.
v3.2 May 1, 2025 · issue -366

oobabooga textgen v3.2 adds Qwen3 thinking toggle, auto API port selection, and verbose llama-server logging
└──▷ GET THIS VERSION
```
$ git clone --branch v3.2 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.2
```
- ›Adds enable_thinking checkbox under Parameters to enable or disable thinking for Qwen3 models (and future models with the feature); thinking is enabled by default and works via the Jinja2 template.
- ›Automatically finds a new API port if the default one is already taken.
- ›Makes --verbose print the llama-server launch command to the console.
- ›Makes <think> UI blocks closed by default, reducing visual noise from reasoning model output.
v3.1 Apr 27, 2025 · issue -367

textgen v3.1 adds speculative decoding (up to +88.7% tokens/sec), Vulkan builds, and a universal --ctx-size flag across all loaders.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.1
```
└──▷ HOW TO FIND IT
Boost generation speed on a large GGUF model by pairing it with a small draft model for speculative decoding in the llama.cpp loader.
📍# In the UI, load google_gemma-3-27b-it-Q8_0.gguf and set the draft model to google_gemma-3-1b-it-Q4_K_M.gguf under the llama.cpp speculative decoding settings.
- ›Adds --extra-flags parameter to the llama.cpp loader for passing additional flags directly to llama-server (e.g. override-tensor=exps=CPU for MoE models).
- ›Adds --streaming-llm flag to llama.cpp (mapped to --cache-reuse in llama.cpp internals) to skip full prompt reprocessing when context length is filled, useful for long role-playing sessions.
- ›Adds universal --ctx-size flag to specify context size across all loaders.
- ›Adds speculative decoding to the llama.cpp loader; benchmarks show +88.7% tokens/second with a 27B model using a 1B draft model, with gains of +34–88% observed across different model combinations.
- ›Adds speculative decoding to the non-HF ExLlamaV2 loader.
+8 moreshow less
- ›Adds KV cache quantization to the ExLlamaV3 loader.
- ›Restructures all user data (models, characters, presets, saved settings) under text-generation-webui/user_data/ to enable portable install updates by moving a single folder.
- ›Adds Vulkan portable builds supporting AMD and Intel Arc GPUs on both Windows and Linux.
- ›Adds prompt processing progress messages to the llama.cpp loader.
- ›UI: Adds a collapsible thinking block for messages containing <think> steps.
- ›UI: Sets 'instruct' as the default chat mode.
- ›UI: Adds a greeting when the web UI launches in instruct mode with an empty chat history.
- ›UI: Model menu now displays only part 00001 of multipart GGUF files.
└──▷ BREAKING ON UPGRADE
- !All user data has moved: models must be manually relocated from models/ to user_data/models/, presets from presets/ to user_data/presets/, and other user data (characters, saved settings) to their corresponding paths under user_data/ after upgrading.
v3.0 Apr 22, 2025 · issue -367

oobabooga text-generation-webui v3.0 ships portable zip builds with llama.cpp built in and makes llama.cpp the default loader.
└──▷ GET THIS VERSION
```
$ git clone --branch v3.0 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v3.0
```
- ›Adds portable zip builds (~700 MB) for Windows, Linux, and macOS (variants: cuda12.4, cuda11.7, cpu, macOS arm64, macOS x86_64) bundling text-generation-webui + llama.cpp with no installation required.
- ›API now starts by default on localhost in portable builds without requiring the --api flag.
- ›The --gpu-split flag (previously EXL2-only) is now also reused for Transformers GPU memory control.
- ›Makes llama.cpp the default loader across the project.
- ›Adds support for llama.cpp builds from the ggml-org/llama.cpp fork.
└──▷ BREAKING ON UPGRADE
- !The --gpu-memory flag has been removed; use --gpu-split instead for Transformers GPU memory configuration.
v2.8 Apr 18, 2025 · issue -367

oobabooga text-generation-webui v2.8 replaces llama-cpp-python with a new llama-server loader and adds smoother chat streaming.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.8 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.8
```
- ›New llama-server-based llama.cpp loader replaces llama-cpp-python, adding sampling parameters xtc, dry, and dynatemp and reducing installation size.
- ›Supports GGUF model subfolder organization inside text-generation-webui/models, enabling direct import from LM Studio directory layouts.
- ›Adds support for the Llama-4-Scout-17B-16E-Instruct model via the updated llama.cpp backend.
- ›Smoother per-word chat streaming in the Chat tab replaces chunked token delivery.
- ›The llamacpp_HF loader has been removed; only one llama.cpp loader now exists.
└──▷ BREAKING ON UPGRADE
- !The llamacpp_HF loader has been removed — any workflow that selected it will need to switch to the single remaining llama.cpp loader.
- !Some command-line flags have been removed — existing launch scripts using those flags will break on upgrade.
v2.7 Apr 9, 2025 · issue -367

oobabooga text-gen v2.7 adds ExLlamaV3 support via new ExLlamav3_HF loader, a Dark chat style, and default 8192 context-length cap.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.7 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.7
```
- ›Adds ExLlamav3_HF loader, providing ExLlamaV3 inference with the same sampler stack as Transformers and ExLlamav2_HF; pre-built wheels for Linux and Windows are included, removing manual installation (requires compute capability 8+).
- ›Adds a new 'Dark' chat style option.
v2.6 Mar 15, 2025 · issue -368

oobabooga text-gen v2.6 adds a top N-sigma sampler, SuperboogaV2 GPU/date-time upgrades, and a refreshed Perplexity Colors extension.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.6 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.6
```
- ›Adds the top N-sigma sampler for token sampling.
- ›Improves SuperboogaV2 with Date/Time Embeddings, GPU support, and multiple file format handling.
- ›Updates the Perplexity Colors extension to v2 with additional improvements.
v2.5 Feb 15, 2025 · issue -369

Adds a 'Show after' parameter to the UI for controlling DeepSeek </think> token display.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.5 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.5
```
- ›Adds a Show after parameter to the UI for use with DeepSeek </think> token handling.
- ›Minor UI improvements to list styles and the light theme.
v2.4 Jan 29, 2025 · issue -370

Adds 'Continue' and 'Remove' chat buttons, strftime_now in JINJA, and smarter installer caching in v2.4
└──▷ GET THIS VERSION
```
$ git clone --branch v2.4 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.4
```
- ›Adds 'Continue' and 'Remove' buttons below the last chat message in the UI for quick in-place edits.
- ›Adds strftime_now to JINJA template support, enabling LLAMA 3.1, 3.2, and Granite model chat templates to render correctly.
- ›Installer now skips re-downloading .whl requirements during updates unless the files or local repo commit have changed, speeding up updates after manual branch switches.
- ›Adds a descriptive error message when llama.cpp fails to load a model, prompting users to lower the context length.
- ›Extends OpenAI-compatible leniency to SillyTavern API clients.
v2.3 Jan 15, 2025 · issue -370

oobabooga text-gen v2.3 cuts Chat tab CPU usage via morphdom incremental rendering and adds per-message copy/regenerate buttons.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.3 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.3
```
- ›Adopts the morphdom library for incremental DOM updates in the Chat tab during streaming, drastically reducing CPU usage on long contexts or high token throughput and keeping the UI responsive.
- ›Enables text and code selection/copying from previous chat messages while a reply is still streaming, since only changed elements are updated.
- ›Adds a 'copy raw message content' button below each chat message.
- ›Adds a 'regenerate reply' button below the last chat message.
- ›Activates auto_max_new_tokens by default, removing the need to manually continue replies every 512 tokens.
v2.2 Jan 9, 2025 · issue -370

oobabooga textgen v2.2 adds branch/search chat UI, --torch-compile, --exclude-pattern, IPv6 API support, and ExLlamaV2 sampler expansions.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.2 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.2
```
└──▷ TRY IT
Speed up transformers inference on repeated prompt patterns by compiling the model graph at first run.
```
$ python server.py --torch-compile
```
Download a model while skipping large or unwanted shard files to save disk space.
```
$ python download-model.py TheBloke/Mixtral-8x7B-v0.1-GGUF --exclude-pattern '*.Q8_0.gguf'
```
- ›Adds --torch-compile flag for the transformers loader to improve inference performance.
- ›Adds --exclude-pattern flag to the download-model.py script to filter files during model downloads.
- ›Connects XTC, DRY, smoothing_factor, and dynatemp sampling parameters to the ExLlamaV2 loader (non-HF).
- ›Adds IPv6 support to the API.
- ›Adds a 'Static KV cache' option for transformers to improve performance.
+5 moreshow less
- ›Adds a 'Branch chat' option to the chat tab, enabling conversation branching from any point.
- ›Adds a 'Search chats' menu to the chat tab for finding past conversations.
- ›Removes a 0.2-second startup delay for llama.cpp and ExLlamaV2, increasing reported tokens/second.
- ›Adds a horizontal scrollbar to code blocks wider than the chat area.
- ›Removes the AutoGPTQ loader; GPTQ models can still be loaded through ExLlamaV2.
└──▷ BREAKING ON UPGRADE
- !The AutoGPTQ loader has been removed; any setup relying on it must switch to the ExLlamaV2 loader to load GPTQ models.
v2.1 Dec 31, 2024 · issue -371

Parameters tab reorganized with grouped controls and new Instruct and Creative presets in oobabooga textgen v2.1
└──▷ GET THIS VERSION
```
$ git clone --branch v2.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.1
```
- ›Adds two new built-in presets, Instruct and Creative, replacing 5 removed presets to reduce the default preset list from 11 to 6.
- ›Reorganizes the Parameters tab with grouped input fields (sliders, checkboxes) and headings for parameter types (curve shape, curve cutoff) to reduce visual clutter.
- ›Reorganizes the Model tab with similar grouping and heading structure.
- ›Improves heading, list, and link styling in chat messages.
- ›Improves the streaming typing cursor | appearance during chat.
+1 moreshow less
- ›Improves chat colors in light mode.
v2.0 Dec 19, 2024 · issue -371

text-generation-webui v2.0 overhauls the UI with collapsible sidebars, vertical nav, mobile support, and a new --old-colors flag.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.0 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v2.0
```
└──▷ TRY IT
Preserve your team's familiar workflow after upgrading by launching with the legacy color palette.
```
$ python server.py --old-colors
```
- ›Adds --old-colors launch flag to restore the previous UI color scheme for users who prefer the legacy look.
- ›Allows more granular KV cache settings.
- ›Redesigns the UI with a vertical left sidebar replacing the top navigation bar (Chat / Default / Notebook tabs).
- ›Adds collapsible sidebars for improved mobile navigation.
- ›Moves the 'past chats' menu into its own sidebar, visually separated from the chat area.
+1 moreshow less
- ›Improves HTML rendering for nested lists with sub-items.
v1.16 Oct 25, 2024 · issue -373

oobabooga text-gen v1.16 adds Whisper Turbo transcription and RWKV-World instruction template support.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.16 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.16
```
- ›Adds Whisper Turbo as a selectable transcription model option.
- ›Adds RWKV-World instruction template for use with RWKV-World models.
v1.15 Oct 1, 2024 · issue -373

oobabooga textgen v1.15 adds XTC sampler, sortable repetition penalties, and ExLlamaV2 tensor parallelism for multi-GPU inference.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.15 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.15
```
- ›Adds xtc to the Sampler priority ordering keywords, alongside new keywords repetition_penalty, presence_penalty, frequency_penalty, dry, encoder_repetition_penalty, and no_repeat_ngram, enabling fine-grained control over the order repetition penalties are applied.
- ›Adds Exclude Top Choices (XTC) sampler to boost creativity, break writing clichés, and inhibit non-verbatim repetition.
- ›Enables ExLlamaV2 tensor parallelism to increase multi-GPU inference speeds.
- ›Removes AutoAWQ, AutoGPTQ, HQQ, and AQLM from requirements.txt; these can still be installed manually.
- ›Adds a beforeunload confirmation dialog when leaving the page to prevent accidental session loss.
+2 moreshow less
- ›Updates API documentation with examples to list and load models.
- ›Defers PEFT import until needed, reducing web UI launch time.
└──▷ BREAKING ON UPGRADE
- !AutoAWQ and AutoGPTQ are removed from requirements.txt due to lack of support for PyTorch 2.4.1 and CUDA 12.1; setups relying on these quantization backends must install them manually.
- !HQQ and AQLM are removed from requirements.txt; setups using these experimental backends must install them manually.
v1.14 Aug 20, 2024 · issue -375

textgen v1.14 adds --tokenizer-dir flag for llamacpp_HF and faster multi-file model downloads.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.14 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.14
```
- ›Adds --tokenizer-dir flag for use with llamacpp_HF to specify a custom tokenizer directory.
- ›Model downloader now uses a single session for all files in a download, reducing per-file startup latency.
v1.13 Aug 1, 2024 · issue -375

textgen v1.13 adds Llama 3.1 support, manual context-length entry, and improved model download progress.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.13 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.13
```
- ›Makes compress_pos_emb, n_ctx, max_seq_len, and truncation_length editable number fields instead of sliders, allowing precise manual entry of context lengths.
- ›Adds Llama 3.1 support via llama-cpp-python 0.2.85 bump, including better handling of the Llama 3.1 Jinja2 template (optional 'tools' headers now suppressed).
- ›Model downloader progress bar now shows filename, file size, and download speed per file.
v1.12 Jul 25, 2024 · issue -376

oobabooga textgen v1.12 adds Llama 3.1 support across Transformers, ExLlamaV2, and AutoAWQ backends with UI markdown and coloring improvements.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.12 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.12
```
- ›Removes AutoAWQ as a standalone loader; AWQ models (including Llama 3.1 AWQ-INT4) are now loaded directly through Transformers for better compatibility.
- ›Adds Llama 3.1 support via Transformers 4.43, ExLlamaV2 0.1.8, and AutoAWQ 0.2.6 backend bumps.
- ›Colors text between quote characters in chat and chat-instruct modes for improved readability.
- ›Makes the markdown rendering cache infinite (cleared on chat switch) to keep the UI responsive during long 128k-context conversations.
└──▷ BREAKING ON UPGRADE
- !AutoAWQ is no longer available as a standalone loader; AWQ models must now be loaded through the Transformers loader.
v1.11 Jul 23, 2024 · issue -376

textgen v1.11 defaults to chat-instruct mode, adds Mistral-Nemo support, and enables gradio subpath customization for reverse proxies.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.11 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.11
```
- ›Adds support for customizing the gradio subpath for use with a reverse proxy.
- ›Adds ChatML-format.json format example for training.
- ›Supports Mistral-Nemo via llama-cpp-python bump to 0.2.83.
- ›Defaults new sessions to chat-instruct mode, which automatically applies the model's Jinja2 template for higher-quality outputs.
- ›Significantly faster and more responsive UI through refactored event handling that reduces connections between the UI and the server.
+1 moreshow less
- ›Excludes the instruction template from exports when using 'Save UI defaults to settings.yaml', preventing it from being overwritten by the model template.
v1.10 Jul 11, 2024 · issue -376

oobabooga textgen v1.10 adds Gemma-2 support via ExLlamaV2 and new attention-disable flags.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.10 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.10
```
- ›Adds --no_xformers and --no_sdpa flags for ExLlamaV2 (complement the existing --no_flash_attn flag); all three must be used together — or their UI checkboxes enabled — to run Gemma-2 without severe performance degradation.
- ›ExLlamaV2 updated to 0.1.7, enabling Gemma-2 model support.
v1.9 Jul 5, 2024 · issue -376

oobabooga text-gen v1.9 adds 4-bit/8-bit KV cache for llama.cpp, Gemma-2 support, and a rewritten Whisper STT extension.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.9 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.9
```
- ›Adds 4-bit and 8-bit KV cache options to llama.cpp and llamacpp_HF backends via the existing --cache_4bit and --cache_8bit flags.
- ›Adds native Gemma-2 support in Transformers: automatically detects Gemma2ForCausalLM architecture and applies optimal settings (bfloat16/float16 precision and eager attention) without requiring --bf16 --use_eager_attention manually.
- ›Automatically obtains the EOT token from Jinja2 templates and adds it to stopping strings, so Llama-3-Instruct stops correctly without manually adding <eot> to custom stopping strings.
- ›Rewrites the Whisper STT extension, replacing the Gradio microphone component with a custom microphone element for improved reliability.
- ›Increases the chat area size on mobile devices.
v1.8 Jun 27, 2024 · issue -377

oobabooga textgen v1.8 adds TensorRT-LLM as its fastest backend and overhauls the chat UI with persistent browser history.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.8 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.8
```
- ›Adds TensorRT-LLM backend support, now the fastest inference backend in the project, installed in a separate Python 3.10 environment with a provided Dockerfile.
- ›Stores chat history in the browser so conversations survive server restarts without requiring a page refresh.
- ›Redesigned 'past chats' menu from a dropdown to a vertical text list, reducing conversation switching to a single click.
- ›Moves the 'Character' dropdown menu to the main Chat tab for faster character switching.
- ›Enables installing past releases by downloading the .zip source and running the start_ script, without auto-updating to latest.
└──▷ BREAKING ON UPGRADE
- !The installation script no longer updates to the latest version automatically on run.
snapshot-2024-04-21 Apr 21, 2024 · issue -379

Adds /v1/internal/chat-prompt API endpoint for retrieving the formatted chat prompt.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-04-21 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-04-21
```
- ›Adds POST /v1/internal/chat-prompt API endpoint to retrieve the fully formatted chat prompt string before generation.
snapshot-2024-04-14 Apr 14, 2024 · issue -379

Adds min_p sampling preset as default, Ascend NPU support, and HF_ENDPOINT awareness for model downloads.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-04-14 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-04-14
```
- ›Respects model and LoRA directory settings when downloading files, so downloads land in the configured paths rather than defaults.
- ›Reads the HF_ENDPOINT environment variable when downloading models, enabling use of Hugging Face mirror endpoints.
- ›Adds Ascend NPU as a supported hardware backend.
- ›Adds a min_p sampling preset and makes it the default generation preset.
snapshot-2024-03-31 Mar 31, 2024 · issue -380

Parameters tab reorganized and ChatML support added for Hyperion and Hercules models.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-03-31 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-03-31
```
- ›Adds ChatML format configuration for Hyperion and Hercules models.
- ›Reorganizes the Parameters tab layout in the UI.
snapshot-2024-03-17 Mar 17, 2024 · issue -380

Adds AQLM quantization support, StreamingLLM for llama.cpp backends, and a new User Description field in the UI.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-03-17 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-03-17
```
- ›Adds a new 'User description' field in the UI for setting user personality/biography context.
- ›Adds experimental AQLM quantization support for loading AQLM-quantized models.
- ›Adds StreamingLLM support for the llamacpp and llamacpp_HF backends, enabling efficient long-context streaming inference.
- ›Bumps AutoAWQ to 0.2.3 on Linux, enabling use of newer AWQ model variants on that platform.
snapshot-2024-03-10 Mar 10, 2024 · issue -380

Adds cache_q4 KV-cache quantization for ExLlamaV2 and introduces an update wizard in textgen WebUI.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-03-10 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-03-10
```
- ›Adds cache_q4 option for ExLlamaV2 to enable 4-bit KV-cache quantization, reducing VRAM usage during inference.
- ›Introduces an update wizard to guide users through upgrading the WebUI.
- ›Extensions requirements are no longer installed by default, reducing setup overhead for users who don't need them.
snapshot-2024-03-03 Mar 3, 2024 · issue -380

Adds cubic sampling with a curve parameter and Gemma model compatibility via transformers 4.38.1.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-03-03 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-03-03
```
- ›Adds cubic sampling with a configurable curve parameter as a new sampling method for text generation.
- ›Supports Google Gemma models via a bump to transformers 4.38.1.
snapshot-2024-02-18 Feb 18, 2024 · issue -381

Adds --autosplit for ExLlamaV2, a llamacpp_HF creator menu, and per-model instruction template customization.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-02-18 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-02-18
```
- ›Adds --autosplit flag for ExLlamaV2 to automatically split model layers across GPUs.
- ›Adds a 'llamacpp_HF creator' menu in the UI for building llama.cpp HuggingFace-compatible model configs.
- ›Adds a menu for customizing the instruction template on a per-model basis.
└──▷ BREAKING ON UPGRADE
- !The 'Maximum UI updates/second' parameter has been removed.
snapshot-2024-02-11 Feb 11, 2024 · issue -381

Adds quadratic sampling and custom sampler order support; llama.cpp multi-GPU now splits by rows instead of layers.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-02-11 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-02-11
```
- ›Adds custom sampler order support, letting users control the sequence in which samplers are applied during text generation.
- ›Adds quadratic sampling as a new sampling method for text generation.
- ›Splits by rows instead of layers for llama.cpp multi-GPU configurations, improving multi-GPU utilization.
- ›Removes the non-HuggingFace ExLlamaV2 loader, consolidating to the HF-based loader only.
└──▷ BREAKING ON UPGRADE
- !The non-HuggingFace ExLlamaV2 loader has been removed; setups using that loader must switch to the HuggingFace ExLlamaV2 loader.
snapshot-2024-02-04 Feb 4, 2024 · issue -381

Adds CPU support for Coqui TTS, new Docker GPU options, and a roleplay GBNF grammar file.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-02-04 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-02-04
```
- ›Adds roleplay.gbnf grammar file for constrained roleplay-format generation.
- ›Adds Docker options for CPU-only, Intel GPU, and AMD GPU deployments.
- ›Enables the Coqui TTS extension to run on CPU, removing the GPU requirement.
- ›Supports running the tool as a git submodule.
snapshot-2024-01-28 Jan 28, 2024 · issue -382

Adds prompt_lookup_num_tokens, multimodal content arrays, RGBA image support, and trust_remote_code for DeepSpeed loading.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-01-28 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-01-28
```
- ›Adds prompt_lookup_num_tokens parameter for prompt lookup decoding.
- ›Adds trust_remote_code support when loading models with DeepSpeed.
- ›Updates n_gpu_layers default to 256 to support larger models.
- ›Supports content arrays in multimodal OpenAI API requests.
- ›Supports RGBA color format for image inputs.
snapshot-2024-01-21 Jan 21, 2024 · issue -382

Adds dynatemp parameters, past-chat sidebar, and Tab-key navigation between tabs.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-01-21 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-01-21
```
- ›Adds dynatemp_low, dynatemp_high, and dynatemp_exponent parameters for dynamic temperature control during generation.
- ›Adds a past chat histories sidebar on desktop for quick access to previous conversations.
- ›Adds Tab key shortcut to switch between the current tab and the Parameters tab.
snapshot-2024-01-14 Jan 14, 2024 · issue -382

Adds dynamic temperature parameters and a desktop chat history sidebar to oobabooga text-generation-webui.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-01-14 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-01-14
```
- ›Adds dynatemp_low, dynatemp_high, and dynatemp_exponent parameters for dynamic temperature control during text generation.
- ›New sidebar on desktop displays past chat histories for quick access.
- ›Press Tab to switch between the current tab and the Parameters tab in the UI.
snapshot-2024-01-07 Jan 7, 2024 · issue -382

Adds dynamic_temperature_low parameter and Dynamic Temperature support for the HF loader.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2024-01-07 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2024-01-07
```
- ›Adds dynamic_temperature_low parameter for finer control over dynamic temperature sampling ranges.
- ›Adds Dynamic Temperature sampling support for the HuggingFace (HF) loader.
- ›Adds Docker build args to support non-AVX2 CPUs.
snapshot-2023-12-31 Dec 31, 2023 · issue -383

Adds image input to OpenAI chat API and Synthia instruction templates; removes --sdp-attention, --xformers, ExLlamaV1, and RWKV loaders.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-12-31 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-12-31
```
- ›Adds image sending support to the OpenAI chat API endpoint, enabling multimodal chat requests.
- ›Adds Synthia instruction templates for Synthia-series models.
└──▷ BREAKING ON UPGRADE
- !The --sdp-attention and --xformers flags are removed; any launch scripts using them will fail on upgrade.
- !ExLlamaV1 loaders are removed; models configured to use ExLlamaV1 must be switched to an alternative loader.
- !The RWKV loader is removed; RWKV models can no longer be loaded.
- !Additional deprecated command-line flags are removed; any scripts relying on them will break.
snapshot-2023-12-24 Dec 24, 2023 · issue -383

Adds HQQ quant loader, GGUF instruction-template auto-detection, tensor-core llama.cpp wheels, and safetensor LoRA support for ExLlama v1.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-12-24 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-12-24
```
- ›Adds HQQ quantization loader for loading HQQ-quantized models.
- ›Adds auto-detection of Eric Hartford's Dolphin models via models/config.yaml.
- ›Adds Orca-Vicuna instruction template.
- ›Reads instruction templates automatically from GGUF metadata for llama.cpp models, removing the need to set them manually.
- ›Adds llama-cpp-python wheels built with tensor cores support, unlocking faster inference on compatible GPUs.
+4 moreshow less
- ›Enables ExLlama v1 models to load safetensor-format LoRAs.
- ›Organizes CMD arguments by group for clearer CLI navigation.
- ›Updates Training PRO with new capabilities.
- ›Bumps ExLlamaV2 to 0.0.11.
snapshot-2023-12-17 Dec 17, 2023 · issue -383

Adds Jinja templates for chat/instruct, OpenAI embeddings endpoint, max_logits param, and --num_experts_per_token for ExLlamav2.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-12-17 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-12-17
```
└──▷ TRY IT
Run the server with expert routing tuned for a MoE model loaded via ExLlamav2.
```
$ python server.py --loader exllamav2 --num_experts_per_token 2 --model <your-moe-model>
```
- ›Adds --num_experts_per_token CLI parameter for ExLlamav2 to control mixture-of-experts routing per token.
- ›Adds OpenAI-compatible embeddings endpoint via the OpenAI extension.
- ›Adds max_logits parameter to the OpenAI extension logits endpoint.
- ›Introduces Jinja2 template support for Instruct and Chat prompt formatting.
- ›Supports symlinked folders within the root directory for model/asset organization.
+2 moreshow less
- ›Bypasses Coqui TTS EULA check to streamline TTS extension setup.
- ›Removes the ElevenLabs extension.
└──▷ BREAKING ON UPGRADE
- !The ElevenLabs extension has been removed; any setup relying on it will stop working after upgrade.
snapshot-2023-12-10 Dec 10, 2023 · issue -383

Adds QuIP# quantization support and gallery improvements; changes defaults for max_new_tokens and repetition_penalty_range.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-12-10 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-12-10
```
- ›Changes default values for max_new_tokens and repetition_penalty_range parameters.
- ›Adds QuIP# quantization format support for model loading.
- ›Improves the model Gallery UI.
└──▷ BREAKING ON UPGRADE
- !Default values for max_new_tokens and repetition_penalty_range have changed; existing workflows relying on the previous defaults may produce different outputs after upgrade.
snapshot-2023-12-03 Dec 3, 2023 · issue -383

Adds OpenHermes Mistral model support and Docker container CLI improvements.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-12-03 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-12-03
```
- ›Adds support for the OpenHermes Mistral model.
- ›Improves Docker container CLI arguments for more practical containerized deployments.
snapshot-2023-11-19 Nov 19, 2023 · issue -384

Adds --admin-key, --nowebui, /v1/internal/logits, and /v1/internal/lora endpoints plus a random preset button.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-11-19 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-11-19
```
└──▷ TRY IT
Run the server headlessly for API-only deployments, secured with an admin key.
```
$ python server.py --nowebui --admin-key mysecretkey
```
Retrieve per-token logit scores for a prompt to inspect model confidence.
```
$ curl -X POST http://localhost:5000/v1/internal/logits -H 'Authorization: Bearer mysecretkey' -H 'Content-Type: application/json' -d '{"prompt": "The capital of France is"}'
```
List currently loaded LoRA adapters via the API.
```
$ curl http://localhost:5000/v1/internal/lora -H 'Authorization: Bearer mysecretkey'
```
- ›Adds --admin-key flag to require an API key for all API access.
- ›Adds --nowebui flag to run the server in pure API mode with no browser UI.
- ›Adds GET /v1/internal/logits endpoint for retrieving token logits via the API.
- ›Adds /v1/internal/lora endpoints for managing LoRA adapters via the API.
- ›New 'random preset' button in the UI to randomly select a generation preset.
+2 moreshow less
- ›Character pictures in chat mode can now be enlarged on click.
- ›System message support added to chat instruct mode, shared between UI and API.
snapshot-2023-11-12 Nov 12, 2023 · issue -384

OpenAI API becomes the default, gains /v1/internal/stop-generation endpoint, and now supports trust_remote_code for embeddings.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-11-12 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-11-12
```
└──▷ TRY IT
Abort a running generation mid-stream from a script or integration that uses the OpenAI-compatible API.
```
$ curl -X POST http://localhost:5000/v1/internal/stop-generation
```
- ›Adds POST /v1/internal/stop-generation endpoint to the OpenAI-compatible API, allowing programmatic cancellation of in-progress generation.
- ›Makes the OpenAI-compatible API the default API (previously non-default).
- ›Enables trust_remote_code support in the OpenAI API embedder, allowing embedding models that require remote code execution.
- ›Separates context and system message fields in instruction formats, enabling independent control of each in prompt templates.
snapshot-2023-11-05 Nov 5, 2023 · issue -384

Adds Min P sampler, temperature_last parameter, and use_flash_attention_2 flag to oobabooga text-generation-webui.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-11-05 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-11-05
```
- ›Adds temperature_last parameter to control the order in which temperature sampling is applied relative to other samplers.
- ›Adds use_flash_attention_2 parameter to the Transformers model loader to enable Flash Attention 2 support.
- ›Implements Min P as a new sampler option in HF loaders for controlling minimum probability thresholds during generation.
- ›Adds a flag to force loading models from safetensors format in the Transformers loader.
snapshot-2023-10-29 Oct 29, 2023 · issue -385

Adds presence_penalty and frequency_penalty sampler settings, plus initial Intel GPU support.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-10-29 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-10-29
```
- ›Adds presence_penalty sampler setting (renamed from additive_repetition_penalty) to control token repetition based on presence in the output so far.
- ›Adds frequency_penalty sampler setting to penalize tokens proportional to how frequently they have already appeared in the output.
- ›Adds initial Intel GPU support.
snapshot-2023-10-22 Oct 22, 2023 · issue -385

Adds LLaVA v1.5 support, multi-LoRA blending, flash-attention 2 on Windows, and expanded Silero language coverage.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-10-22 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-10-22
```
- ›Supports LLaVA v1.5 and LLaVA v1.5 7B multimodal models for vision-language inference.
- ›Allows multiple loaded LoRAs to simultaneously influence generation output, enabling LoRA blending workflows.
- ›Adds option to select additional linear modules and layers as targets during LoRA training.
- ›Adds flash-attention 2 support on Windows.
- ›Adds Python 3.11 support with structured requirements.
+2 moreshow less
- ›Expands Silero text-to-speech language coverage with more supported languages.
- ›Enables special token support for the ExLlamav2 backend.
snapshot-2023-10-15 Oct 15, 2023 · issue -385

textgen snapshot adds LLaVA-LLaMA-2 support, HTTPS for OpenAI extension, ChatML, and ExLlamaV2 LoRA.
└──▷ GET THIS VERSION
```
$ git clone --branch snapshot-2023-10-15 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout snapshot-2023-10-15
```
- ›Adds HTTPS support to the OpenAI extension, enabling secure API connections.
- ›Adds ChatML prompt format support, including Mistral-OpenOrca compatibility.
- ›Adds LoRA support for the ExLlamaV2 backend.
- ›Supports LLaVA-LLaMA-2 multimodal model loading.
v1.7 Oct 8, 2023 · issue -385

v1.7 adds AutoAWQ loader support, grammar for Transformers/HF loaders, NUMA for llama.cpp, and EOS-abort for text generation.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.7 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.7
```
- ›Adds grammar support to the transformers and _HF loaders, enabling constrained decoding for locally loaded models.
- ›Adds initial AutoAWQ loader support, allowing AWQ-quantized models to be loaded directly.
- ›Enables NUMA feature for llama_cpp_python, improving CPU memory locality on multi-socket systems.
- ›Model downloader now downloads *.tiktoken tokenizer files alongside model weights.
- ›Adds EOS-token abort behavior in text generation — output stops immediately when the end-of-sequence token is reached.
+3 moreshow less
- ›Updates on the one-click installer now preserve custom changes to CMD_FLAGS.txt rather than overwriting them.
- ›Expands macOS llama.cpp support in requirements.txt.
- ›Bumps ExLlamaV2 to 0.0.5 and AutoAWQ to v0.1.4.
1.6.1 Sep 26, 2023 · issue -386

Adds a grammar editor to the UI and alternative requirements.txt with AMD and Metal wheels.
└──▷ GET THIS VERSION
```
$ git clone --branch 1.6.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout 1.6.1
```
- ›Adds a grammar editor to the UI for direct grammar authoring during inference.
- ›Adds an alternative requirements.txt with AMD and Metal wheels for broader hardware support.
v1.6 Sep 22, 2023 · issue -386

oobabooga textgen v1.6 adds GGUF support, ExLlamaV2 loaders, CFG sampling, SSL, named Cloudflare tunnels, and a logit viewer.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.6 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.6
```
- ›Adds --cpu option for llama.cpp to force CPU-only inference and prevent CUDA from being used.
- ›Adds --disable_exllama option for AutoGPTQ to bypass ExLlama kernels.
- ›Adds auto_max_new_tokens parameter to automatically size the maximum new tokens based on context.
- ›Adds max_tokens_second parameter to rate-limit token generation speed.
- ›Adds SSL certificate support for securing the web UI endpoint.
+25 moreshow less
- ›Adds option for named Cloudflare tunnels, enabling persistent public URLs.
- ›Adds ExLlamaV2 and ExLlamaV2_HF loaders for next-generation ExLlama inference.
- ›Adds GGUF format support, including reading GGUF metadata fields such as scale_linear and freq_base.
- ›Adds Classifier Free Guidance (CFG) sampling for Transformers, ExLlama, ExLlama_HF, and llamacpp_HF loaders.
- ›Adds new RoPE settings for ExLlama, with NTK (alpha) and RoPE scaling also supported for ExLlamaV2 and ExLlamaV2_HF.
- ›Adds mlock and no-mmap options to ctransformers loader.
- ›Adds GGUF support to ctransformers loader.
- ›Adds array (batched) input support to the openai extension API.
- ›Handles Chunked Transfer Encoding in the openai extension for streaming requests.
- ›Adds chat_instruct_command support in the API.
- ›Adds save defaults to settings.yaml button so current UI parameters persist across sessions.
- ›Adds Show controls button to the chat UI to toggle the parameter panel.
- ›Adds a simple logit viewer to inspect per-token probability distributions.
- ›Adds probability dropdown to the perplexity_colors extension.
- ›Adds language option to the silero_tts extension.
- ›Adds silero TTS audio preview back to the UI.
- ›Unifies the three interface modes (chat, notebook, default) into a single cohesive UI.
- ›Adds support for downloading a single file from a model repository via the UI and download-model.py.
- ›Merges one-click-installers into the main repository, reducing installation size by several GB with a more reliable update procedure.
- ›Adds a standalone Dockerfile for NVIDIA Jetson devices.
- ›Adds ffmpeg to the Docker image.
- ›Adds chat instruction configuration for BaiChuan-chat, starchat, Vicuna-v1.5, pygmalion-2, mythalion, and StableBeluga2 model series.
- ›Adds scrollbar to notebook/default textboxes and enables scrolling during token streaming.
- ›Refactors the training tab UI for improved usability.
- ›Uses separate llama-cpp-python packages to maintain GGML support alongside GGUF.
v1.5 Jul 26, 2023 · issue -388

oobabooga text-generation-webui v1.5 adds a new extension hook, automatic instruction templates, and drops FlexGen support.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.5 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.5
```
- ›Introduces chat_input_modifier extension function in extensions/example/script.py as the replacement for the deprecated input_hijack hook.
- ›Switches to the dark theme as the new default UI appearance.
- ›Removes FlexGen backend support, which has been made obsolete by llama.cpp and 4-bit quantization.
- ›Adds a detailed extension authoring example at extensions/example/script.py with updated extension documentation.
└──▷ BREAKING ON UPGRADE
- !input_hijack is deprecated in favour of chat_input_modifier; existing extensions using input_hijack will need to migrate to the new function.
- !FlexGen support is removed; any setup relying on the FlexGen backend will no longer work after upgrading.
v1.4 Jul 24, 2023 · issue -388

oobabooga textgen v1.4 adds llama-2-70b GGML support and expands OpenAI extension with images and logit_bias/logprobs.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.4 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.4
```
- ›OpenAI extension gains support for logit_bias and logprobs parameters, image handling, embeddings, and token endpoints — with improved error reporting and updated docs.
- ›Adds llama-2-70b GGML model support.
- ›Bumps exllama module to 0.0.8 with expanded LoRA support.
- ›Bumps bitsandbytes to 0.41.0 for faster inference speeds.
v1.3 Jul 18, 2023 · issue -388

oobabooga textgen v1.3 adds Llama-v2 support, customizable RoPE for GGML, and LoRA loading via AutoGPTQ 0.3.0.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.3 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.3
```
- ›Adds Llama-v2 instruction template with auto-detection of truncation length.
- ›Adds customizable RoPE (Rotary Position Embedding) support for GGML models.
- ›Adds Airoboros-v1.2 instruction template.
- ›AutoGPTQ updated to 0.3.0, enabling LoRA loading out of the box for GPTQ models.
- ›LoRA menu selection is now preserved when loading a model (no longer reset on model load).
+1 moreshow less
- ›Disables 'Autoload the model' by default, and disables auto-loading at startup even when only one model is available.
└──▷ BREAKING ON UPGRADE
- !The 'Autoload the model' setting is now disabled by default; setups that relied on automatic model loading at startup will need to re-enable it manually.
v1.2 Jul 16, 2023 · issue -388

v1.2 adds a llamacpp_HF loader, exllama perplexity evaluation, logits processor support in extensions, and token probability/perplexity colorization.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.2 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.2
```
- ›New llamacpp_HF loader adds a llama.cpp backend variant compatible with the Hugging Face tokenizer interface.
- ›Extensions can now register custom logits processors, enabling token-level generation control from extension code.
- ›Enables perplexity evaluation for models loaded with the exllama backend.
- ›Increases the alpha value limit for NTK RoPE scaling on exllama and exllama_HF loaders, supporting longer context extrapolation.
- ›New 'Color tokens by probability and/or perplexity' extension visualizes per-token generation confidence in the UI.
v1.1 Jul 12, 2023 · issue -388

oobabooga text-generation-webui v1.1 adds low-VRAM llama.cpp mode, token auth for model downloads, and TensorBoard/W&B training integration.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.1 https://github.com/oobabooga/textgen.git
# already have the repo? check out this version:
$ git checkout v1.1
```
- ›Adds low VRAM mode support to the llama.cpp module for running larger models on constrained hardware.
- ›Adds token authorization for downloading models, enabling authenticated access to gated or private model repositories.
- ›Adds context_instruct field to the API and loads the default model instruction template automatically.
- ›Expands models/config.yaml with entries for platypus/gplatty, longchat, vicuna-33b, Redmond-Hermes-Coder, wizardcoder, and additional models.
- ›Adds TensorBoard and Weights & Biases integration for monitoring training runs.
+6 moreshow less
- ›Adds default environment variable values to the Docker Compose file for easier out-of-box deployment.
- ›Adds ability to load all text files from a subdirectory for training datasets.
- ›Adds a feature to log a sample of the training dataset for inspection during fine-tuning.
- ›Chat history downloads now generate more detailed file names.
- ›Disables wandb remote HTTP requests by default.
- ›--load-in-4bit quantization is significantly faster following a bitsandbytes Windows wheel update.

oobabooga textgen

Summary

What oobabooga textgen answers

Examples

Command line

Release history