oobabooga textgen — The AI Toolchain

oobabooga textgen v4.9

Wed, 20 May 2026 19:02:34 GMT

v4.9 adds MTP speculative decoding, web search snippets, live token-speed display, drag-and-drop upload, and localhost-only CORS hardening. • Adds `draft-mtp` as a new `--spec-type` option for MTP speculative decoding; auto-enabled when loading MTP GGUFs (e.g. Qwen 3.6 MoE MTP builds). • Treats negative `--ctx-size` values as auto (0), allowing automatic context-size selection. • Adds snippet support to the `web_search` tool so results include a short text excerpt, often eliminating the need for a follow-up `fetch_webpage` call. • Shows live generation speed (tokens/s) and context size in the UI while generating. • Adds drag-and-drop file upload support to the chat input. • Adds Linux aarch64 portable builds for DGX Spark support. • Stores Electron app data in `user_data/cache/electron` instead of the OS default location. • Auto-detects and auto-selects sibling `mmproj-*.gguf` files when loading a model; these appear in the mmproj dropdown and are hidden from the regular model dropdown. • Drops link URLs from `fetch_webpage` output (links render as plain text instead of [text](url) markdown), significantly reducing tokens used per page. • Adds 'Check for updates' button, a folder picker for the models directory, right-click copy context menu, and a spellcheck toggle in the Electron Session tab. • Improves web search security by rejecting non-HTTP links. • Hides reasoning and tools controls in chat mode (shown only in instruct / chat-instruct modes). • One-click installer now tracks the latest release tag instead of bleeding-edge `main`. Breaking changes: • Electron app data location changed from the OS default to `user_data/cache/electron`; existing cached data in the old location will not be picked up automatically.

oobabooga textgen v4.8

Thu, 07 May 2026 17:43:03 GMT

oobabooga textgen v4.8 adds a `--no-electron` flag, list-format API content support, and a redesigned chat composer. • Adds `--no-electron` flag to skip the Electron desktop window and use the web UI in the browser instead. • API now supports list-format content in tool and assistant messages. • Redesigned chat composer with a taller input area and pinned paperclip and message-action buttons. • Electron window now persists bounds and maximize state across launches. • Smooth scroll animation when sending a new message in the chat UI.

oobabooga textgen v4.7.3

Sun, 03 May 2026 06:18:44 GMT

v4.7.3 adds a native Electron desktop app, tensor-parallel multi-GPU flag, and .jinja template support alongside a full UI overhaul. • Adds `--split-mode` flag for llama.cpp with a `tensor` option enabling tensor parallelism that can make multi-GPU inference 60%+ faster; on ik_llama.cpp, `tensor` and `row` fall back to `graph`. • Adds `--listen` flag to skip the native Electron window and expose the server directly; `--nowebui` also skips the window for headless server use. • Adds support for standalone `.jinja` and `.jinja2` instruction template files in the UI, alongside the existing `.yaml` format. • Portable builds now bundle Electron and open as a native desktop window; launch with `textgen` (Linux/macOS) or `textgen.bat` (Windows) instead of the previous start scripts. • Replaces DuckDuckGo HTML scraping in the web search tool with the `ddgs` Python library for more robust bot-blocking resistance. • Redesigns the chat UI with Inter font, Lucide SVG icons, a 3-button segmented mode selector (chat / chat-instruct / instruct), a rounded card chat input with accent-colored send button, and hairline sidebar handles. Breaking changes: • The `--row-split` flag is replaced by `--split-mode`; existing launch scripts using `--row-split` will need to be updated.

oobabooga textgen v4.6.2

Thu, 23 Apr 2026 14:55:28 GMT

oobabooga textgen v4.6.2 adds tool-call confirmation buttons, stdio MCP server support, and a `--preserve-thinking` flag. • Adds `--preserve-thinking` CLI flag and matching chat template parameter `preserve_thinking` to control whether thinking blocks from prior turns are kept in context. • Adds stdio MCP server support via `user_data/mcp.json`, using the same format as Claude Desktop and Cursor, enabling local subprocess-based MCP servers alongside existing HTTP MCP servers. • Adds inline approve/reject/always-approve tool-call confirmation buttons, enabled via the new 'Confirm tool calls' checkbox in the Chat tab. • Adds UI checkbox for `preserve_thinking` to control thinking-block retention without the CLI. • Passes `--draft-min 48` by default for llama.cpp draftless speculative decoding. • Limits 'Reasoning effort' and 'Enable thinking' controls to models whose chat template actually uses them.

oobabooga textgen v4.4

Tue, 07 Apr 2026 00:56:56 GMT

v4.4 adds MCP server integration, image-gen metadata in API responses, and new model-load API parameters. • Adds `instruction_template` and `instruction_template_str` parameters to the model load API endpoint, allowing callers to specify prompt format at load time. • Adds MCP server support: enter one remote MCP server URL per line in the new 'MCP servers' field in the Chat tab — tools are discovered automatically and used alongside local tools. • PNG images returned by the image generation API now include generation metadata (model, seed, dimensions, steps, CFG scale, sampler) embedded in the file. • Adds Windows + ROCm portable builds (ROCm 7.2) alongside the existing Windows CUDA and Vulkan builds. • Updates ik_llama.cpp backend, adding Gemma 4 model support. Breaking changes: • The deprecated `settings` parameter is removed from the model load endpoint — any client passing `settings` will need to be updated.

oobabooga textgen v4.3.3

Sat, 04 Apr 2026 00:05:43 GMT

oobabooga textgen v4.3.3 adds ik_llama.cpp backend, Gemma 4 tool-calling, and token-level logprobs in the completions API. • Adds `--ik` flag for full installs to switch to the ik_llama.cpp backend, which brings new quant types, more accurate KV cache quantization via Hadamard rotation (enabled by default), and MoE/CPU inference optimizations. • Adds `textgen-portable-ik` portable builds as a no-install distribution of the ik_llama.cpp backend alongside the existing llama.cpp portables. • Adds `echo` and `logprobs` parameters to the `/v1/completions` endpoint, returning token-level log probabilities for both prompt and generated tokens; token IDs are also returned in a new `top_logprobs_ids` field. • Adds Gemma 4 support with tool-calling in both the API and UI. • Transformers backend now autodetects `torch_dtype` from model config instead of always forcing bfloat16/float16; `--bf16` still works as a manual override. • Removes the obsolete `models/config.yaml` file; instruction templates are now detected from model metadata. Breaking changes: • The `models/config.yaml` file is removed; instruction templates are now auto-detected from model metadata, so any local customizations stored in that file will no longer be applied.

oobabooga textgen v4.2

Tue, 24 Mar 2026 19:39:58 GMT

oobabooga textgen v4.2 adds an Anthropic-compatible `/v1/messages` endpoint and expands `--extra-flags` to accept literal flags. • Adds `/v1/messages` endpoint for Anthropic API compatibility, enabling Claude Code, Cursor, and other Anthropic clients to connect; supports system messages, content blocks, tool use, tool results, image inputs, and thinking blocks. • Expands `--extra-flags` to accept literal flags directly (e.g. `--extra-flags "--rpc 192.168.1.100:50052 --jinja"`), in addition to the existing `key=value` format. • Moves the OpenAI-compatible API from `extensions/openai` to `modules/api`; the old `--extensions openai` flag remains accepted as an alias for `--api`. • Sets `top_p=0.95` as the new default sampling parameter for API requests. • Enables `gradient_checkpointing` by default during training to reduce VRAM usage. • Removes the `higher_rank_limit` training parameter. • Updated UI theme with new colors, borders, and button styles across light and dark modes. • Removes 52 obsolete instruction templates from 2023 (Airoboros, Baichuan, Guanaco, Koala, Vicuna v0, MOSS, and others). • Reduces portable build sizes by switching to a stripped Python distribution. Breaking changes: • The default `top_p` for API requests is now `0.95`; any setup relying on the previous default will receive different sampling behavior without explicit configuration. • The `higher_rank_limit` training parameter has been removed; configs or scripts referencing it will need to be updated. • 52 instruction templates (Airoboros, Baichuan, Guanaco, Koala, Vicuna v0, MOSS, and others) have been removed; any saved configurations referencing these templates will lose them.

oobabooga textgen v4.1.1

Wed, 18 Mar 2026 05:33:05 GMT

Tool-calling in the UI, incognito chat, OpenAI API spec overhaul, and llama.cpp/ExLlamaV3 capability expansions in v4.1.1 • Adds tool-calling support in the chat UI: each tool is a single `.py` file dropped into `user_data/tools`; five built-in examples ship — `web_search`, `fetch_webpage`, `calculate`, `get_datetime`, and `roll_dice` — and each tool call renders as a collapsible accordion during streaming showing the function name, LLM-chosen arguments, and output. • Adds `reasoning_content` field to OpenAI API streaming and non-streaming chat completions, isolating thinking-block content from `content` (which now carries only the post-thinking reply) even when tool calls are present. • Adds `tool_choice` support and corrects the `tool_calls` response format in the OpenAI API for strict spec compliance, including returning `finish_reason: tool_calls` when tool calls are detected. • Adds `stream_options` support with `include_usage` for streaming responses in the OpenAI API. • Adds `max_completion_tokens` as an alias for `max_tokens` in the OpenAI API. • Adds support for the `developer` role in the OpenAI API, mapped to `system`. • Makes the `/v1/models` endpoint show only the currently loaded model. • Includes `/v1` in the API URL printed to the terminal. • Rewrites `logprobs` support in the OpenAI API for full spec compliance across llama.cpp, ExLlamaV3, and Transformers backends — both streaming and non-streaming responses now return token-by-token logprobs. • Sets llama.cpp `ctx-size` to `0` (auto) by default when `--gpu-layers` is `-1` (the default); on other loaders, `0` maps to 8192. • Uses `--fit-ctx 8192` to establish 8192 as the minimum acceptable context size for llama.cpp's `--fit on` mode (previously 4096). • Reduces the llama.cpp `--fit-target` default from 1024 MiB to 512 MiB. • Makes `logit_bias` and `logprobs` functional in llama.cpp API calls. • Adds missing `custom_token_bans` parameter to the llama.cpp section of the UI. • Adds native `logit_bias` and `logprobs` support for the ExLlamaV3 backend. • Loads the ExLlamaV3 vision model and draft model before the main model so memory auto-splitting accounts for them. • Adds 'Incognito chat' button next to the existing 'New chat' button — incognito chats are temporary, live in RAM, and are never written to disk. • Introduces a new default sampling preset 'Top-P' (`top_p: 0.95`); removes the 'Qwen3 - Thinking', 'Qwen3 - No Thinking', 'min_p', and 'Instruct' presets. • Removes rope scaling parameters `alpha_value`, `rope_freq_base`, and `compress_pos_emb` from the UI; they can still be passed to llama.cpp via `--extra-flags`. • Increases the `ctx-size` slider maximum to 1M tokens in the UI (1024-token step). • Adds a drag-and-drop UI component for reordering 'Sampler priority' items. • Adds security hardening: server-side file save roots, image URL SSRF protection, and an extension allowlist. • Replaces `html2text` with `trafilatura` for webpage text extraction, reducing navigation-bar boilerplate and lowering token usage in agentic tool-calling loops. • Refactors thinking/reasoning extraction into a standalone module supporting Qwen, GPT-OSS, Solar, seed:think, and other formats, and auto-detects when a chat template appends `` to prepend it to the reply during streaming. • Removes the Gradio import in `--nowebui` mode, saving approximately 0.5–0.8 seconds on startup. Breaking changes: • The `alpha_value`, `rope_freq_base`, and `compress_pos_emb` rope scaling parameters are removed from the UI; pass them to llama.cpp via `--extra-flags` if still needed. • The 'Qwen3 - Thinking', 'Qwen3 - No Thinking', 'min_p', and 'Instruct' presets are removed; the new default preset is 'Top-P' (`top_p: 0.95`). • Thinking-block content now goes exclusively into the `reasoning_content` field; `content` in chat completions no longer includes thinking-block text, even when tool calls are present. • The `/v1/models` endpoint now returns only the currently loaded model instead of all available models. • llama.cpp `ctx-size` now defaults to `0` (auto) when `--gpu-layers` is `-1`; setups that relied on the previous default context size will see a different context window.

oobabooga textgen v4.0

Sat, 07 Mar 2026 14:34:59 GMT

oobabooga textgen v4.0 ships parallel API requests, a tool-calling overhaul, N-gram speculative decoding, new CLI generation-parameter flags, and major security hardening. • Enables parallel API requests via `--parallel N` for llama.cpp (multiply context length by N), ExLlamaV3, and TensorRT-LLM loaders for maximum throughput. • Adds `--spec-type`, `--spec-ngram-size-n`, `--spec-ngram-size-m`, and `--spec-ngram-min-hits` parameters to llama.cpp for N-gram speculative decoding — set `--spec-type ngram-mod` to activate faster generation without a draft model. • Adds `--fit-target` parameter to control per-GPU VRAM headroom (default: 1024 MiB) alongside automatic GPU-layer selection (`-1` auto) for llama.cpp. • Adds new CLI flags `--temperature`, `--min-p`, `--top-k`, `--repetition-penalty`, `--enable-thinking`, `--reasoning-effort`, and `--chat-template-file` to set default API generation parameters; `--chat-template-file` accepts `.jinja` or `.yaml` files. • Adds `--user-data-dir` flag to customize the user data directory location; also auto-detects a `../user_data` folder in portable mode for easier updates. • Restricts model loading over API to block `extra_flags` and `trust_remote_code` parameters, reducing attack surface. • Migrates TensorRT-LLM from the old ModelRunner API to the new LLM API, which accepts any Transformers model as input and exposes more sampling parameters. • Tool-calling now works correctly for Qwen 3.5, Devstral 2, GPT-OSS, DeepSeek V3.2, GLM 5, MiniMax M2.5, Kimi K2/K2.5, and Llama 4 models with strict OpenAI format compliance. • Adds 'Target all linear layers' checkbox in the Training tab to apply LoRA to every `nn.Linear` layer except `lm_head`, working across any model architecture. • Adds checkpoint resumption to the Training tab: HF Trainer checkpoint directories are detected automatically and training resumes with full optimizer/scheduler state. • Adds dynamic padding for chat training datasets — batches pad to the longest sequence in the batch rather than always padding to `cutoff_len`. • Conversations exceeding the cutoff length are now dropped instead of silently truncated (configurable). • Introduces an adaptive-p sampler for llama.cpp, Transformers, ExLlamaV3, and ExLlamaV3_HF loaders that reshapes the logit distribution to favor tokens near a target probability. • Adds a user persona dropdown in the Character settings tab to save and load user profiles (name, bio, profile picture) for switching personas. • Adds Qwen 3.5 and Solar Open thinking block support to the UI. • Replaces PyPDF2 with pymupdf for more accurate conversion of PDF inputs to text. • Adds ROCm portable builds for Windows and CUDA 13.1 portable builds; restores macOS x86_64 (Intel) portable builds. • llama.cpp binaries now autodetect CPU instruction set (AVX, AVX2, AVX-512) at runtime, replacing separate AVX/AVX2 builds. • llama-server now spawns on port 5005 by default instead of a random port. • SSE separator for streaming over the API changed from `\r\n` to `\n` to match OpenAI. • Replaces Gradio with a custom fork (`oobabooga/gradio`) delivering faster startup, instant SSE message delivery (replacing 50 ms polling), and a new zero-rendering `gr.Headless` component that reduces chat-streaming overhead. • Removes the ExLlamaV2 loader (archived upstream); EXL2 users must migrate to EXL3. Breaking changes: • The ExLlamaV2 loader has been removed; EXL2 model users must migrate to the EXL3 loader. • The Training_PRO extension has been removed after being made obsolete by the Training tab rewrite. • The SSE streaming separator over the API changed from `\r\n` to `\n`; clients that relied on `\r\n` framing will break. • Conversations exceeding `cutoff_len` during training are now dropped rather than silently truncated, changing training dataset behavior for affected entries. • Model loading over API no longer accepts `extra_flags` or `trust_remote_code` parameters; callers that pass these will be blocked. • TensorRT-LLM has been migrated from the old ModelRunner API to the new LLM API; existing TensorRT-LLM integrations built against ModelRunner will break.

oobabooga textgen v3.23

Thu, 08 Jan 2026 20:54:35 GMT

oobabooga textgen v3.23 improves table and separator styling in chat messages. • Improves the visual style of tables and horizontal separators rendered inside chat messages.

oobabooga textgen v3.20

Sun, 07 Dec 2025 20:58:45 GMT

oobabooga textgen v3.20 adds a full image generation tab with diffusers models, quantization, and an OpenAI-compatible image API. • New 'Image AI' tab supports image generation via `diffusers` models (e.g. Z-Image-Turbo) with 4-bit/8-bit quantization, `torch.compile` support, PNG metadata embedding, a gallery of past generations, and a progress bar. • Adds an OpenAI-compatible API endpoint for image generation. • Passes `bos_token` and `eos_token` to jinja2 templates, enabling template support for `Seed-OSS-36B-Instruct` and similar models. • Enables `flash_attention_2` by default for Transformers models. • Adds LLM-generated prompt variations as an image generation feature within the new 'Image AI' tab.

oobabooga textgen v3.19

Sat, 29 Nov 2025 02:00:44 GMT

oobabooga textgen v3.19 adds a `--ubatch-size` slider for llama.cpp and updates the backend with Qwen3-Next support. • Adds `--ubatch-size` slider in the llama.cpp loader UI with new defaults tuned for MoE models, significantly improving prompt processing speeds in full-GPU and GPU+CPU configurations. • Updates llama.cpp backend to commit `ff55414c` adding Qwen3-Next model support. • Updates ExLlamaV3 to version 0.0.16.

oobabooga textgen v3.18

Wed, 19 Nov 2025 14:04:23 GMT

oobabooga text-gen v3.18 adds `--cpu-moe` flag for VRAM reduction and ROCm portable builds for AMD GPUs on Linux. • Adds `--cpu-moe` flag for the llama.cpp backend to offload MoE model experts to CPU, reducing VRAM usage on large mixture-of-experts models. • Adds ROCm portable builds for AMD GPUs on Linux, enabling no-install GGUF inference on AMD hardware. Breaking changes: • macOS 13 wheels are removed and no longer supported.

oobabooga textgen v3.17

Thu, 06 Nov 2025 03:39:00 GMT

v3.17 adds Qwen3-VL support via llama.cpp update and bumps exllamav3 to 0.0.12. • Supports Qwen3-VL multimodal models through the updated llama.cpp backend. • Updates exllamav3 backend to version 0.0.12. • Adds `weights_only=True` to `torch.load` in Training_PRO to harden model checkpoint loading.

oobabooga textgen v3.16

Thu, 23 Oct 2025 15:50:37 GMT

oobabooga text-generation-webui v3.16 adds symlink support for portable builds and updates llama.cpp with Ling-mini-2.0 and Ring-mini-2.0 model support. • Enables running a portable Web UI build via a symlink, allowing flexible install layouts without copying the full package. • Updates llama.cpp backend to add support for Ling-mini-2.0 and Ring-mini-2.0 model architectures. • Updates exllamav3 to version 0.0.11. • Updates triton-windows to version 3.5.0.post21.

oobabooga textgen v3.15

Wed, 15 Oct 2025 20:15:51 GMT

oobabooga textgen v3.15 locks down `--trust-remote-code` from UI/API and adds llama-server context-size error logging. • Logs an error when a `llama-server` request exceeds the context size, making oversized-context failures visible instead of silent. • Makes `--trust-remote-code` immutable from the UI and API, preventing runtime elevation of code-execution trust.

oobabooga textgen v3.14

Fri, 10 Oct 2025 13:47:17 GMT

v3.14 adds `/v1/internal/logits` endpoint for exllamav3 loaders and `qwen3-next` model support via `fla`. • Implements the `/v1/internal/logits` API endpoint for the `exllamav3` and `exllamav3_hf` loaders. • Adds `fla` to requirements for Exllamav3 to enable support for `qwen3-next` models. • Improves handling of multi-GPU setups when using Transformers with bitsandbytes (`load-in-8bit` and `load-in-4bit`).

oobabooga textgen v3.12

Tue, 02 Sep 2025 19:55:29 GMT

oobabooga textgen v3.12 adds thinking support for characters in chat-instruct mode and absolute-path GGUF model loading. • Extends the `--model` flag to accept absolute paths for GGUF models, e.g. `--model /tmp/gemma-3-270m-it-IQ4_NL.gguf`. • Adds thinking support for characters in `chat-instruct` mode, enabled by Jinja2 template handling improvements. • Adds a chat prompt template for the **Seed-OSS-36B-Instruct** model. • Optimizes LaTeX rendering during streaming for long replies. • Statically links the MSVC runtime in llama.cpp Windows binaries, removing the Visual C++ Redistributable dependency. • Enables `--verbose` llama.cpp terminal output with reduced verbosity. • Makes venv portable installs compatible with Python 3.13. Breaking changes: • The `--flash-attn` flag has been removed; flash attention is now always enabled in llama.cpp and passing this flag will no longer work.

oobabooga textgen v3.11

Tue, 19 Aug 2025 14:52:49 GMT

oobabooga text-generation-webui v3.11 adds Tensor Parallelism support to ExLlamav3 loaders via new CLI options. • Adds `--enable-tp` and `--tp-backend` CLI options to enable Tensor Parallelism for the ExLlamav3 and ExLlamav3_HF loaders. • Sets multimodal status at model load time rather than checking on every generation, improving multimodal pipeline efficiency.

oobabooga textgen v3.10

Tue, 12 Aug 2025 21:18:29 GMT

oobabooga text-generation-webui v3.10 adds multimodal support to UI and API across llama.cpp and ExLlamaV3 loaders, plus speculative decoding. • Supports loading chat templates from `chat_template.json` files for EXL3, EXL2, and Transformers models. • Passes `--swa-full` to `llama-server` when the `streaming-llm` option is checked, enabling SWA model compatibility in llama.cpp. • Adds multimodal (image + text) support to the UI and API via the llama.cpp loader. • Adds multimodal support to the UI and API via a new ExLlamaV3 loader. • Adds speculative decoding to the new ExLlamaV3 loader. • Defaults `max_tokens` to 512 in the API (previously 16). • Uses ExLlamaV3 instead of ExLlamaV3_HF as the default loader for EXL3 models, since it now supports multimodal and speculative decoding. • Reorganizes the right sidebar in the UI for better layout. Breaking changes: • The default API value of `max_tokens` changes from 16 to 512; clients relying on the old default will now receive longer responses. • EXL3 models now default to the ExLlamaV3 loader instead of ExLlamaV3_HF; setups that depended on ExLlamaV3_HF behavior by default will need to explicitly select that loader.

oobabooga textgen v3.9.1

Thu, 07 Aug 2025 03:33:13 GMT

GPT-OSS template 'Continue' and 'Impersonate' actions now work correctly, plus a llama.cpp update. • GPT-OSS template special actions 'Continue' and 'Impersonate' now function correctly.

oobabooga textgen v3.9

Wed, 06 Aug 2025 02:55:23 GMT

oobabooga textgen v3.9 adds experimental GPT-OSS support with reasoning effort controls and standalone Jinja chat templates. • Adds a `Reasoning effort` UI element in the chat tab with `low`, `medium`, and `high` options for GPT-OSS models. • Supports standalone `.jinja` chat templates, enabling GPT-OSS to be loaded through the Transformers backend. • Makes web search functional with thinking models. • Experimental support for GPT-OSS GGUF models (20B and 120B variants) via updated llama.cpp and Transformers 4.55.

oobabooga textgen v3.8

Sat, 19 Jul 2025 20:54:48 GMT

oobabooga textgen v3.8 unifies attention config, adds speculative decoding 'None' option, and bumps core backends. • Replaces `use_flash_attention_2` and `use_eager_attention` with a unified `attn_implementation` setting in the Transformers loader, consolidating attention backend selection into a single config key. • Adds a 'None' option for the speculative decoding model, allowing users to explicitly disable speculative decoding from the UI. • Ignores `add_bos_token` in instruct prompts, deferring BOS token handling to the jinja2 template instead. • Updates ExLlamaV3 to 0.0.5 and ExLlamaV2 to 0.3.2. • Updates Transformers to 4.53 alongside the latest bitsandbytes, Accelerate, and PEFT versions. • Updates llama.cpp backend to commit `90083283ec254fa8d33897746dea229aee401b37`. Breaking changes: • The `use_flash_attention_2` and `use_eager_attention` Transformers loader settings are replaced by `attn_implementation`; existing configs using either removed key will need to be updated.

oobabooga textgen v3.7.1

Wed, 09 Jul 2025 03:17:22 GMT

textgen v3.7.1 adds user extension installs, moves 'Enable thinking' to sidebar, and switches to miniforge for org-friendly installs. • Supports installing user extensions in `user_data/extensions/` without modifying the core installation. • Moves the 'Enable thinking' checkbox from the Parameters tab to the right sidebar for faster access during chat. • Switches the one-click installer from miniconda to miniforge, removing Anaconda licensing restrictions for organizations with 200+ users. • Replaces the 'Generate' button label with 'Send' in the main chat interface. • Standardizes margins and paddings across all chat styles.

oobabooga textgen v3.6.1

Thu, 19 Jun 2025 22:49:50 GMT

oobabooga text-generation-webui v3.6.1 merges tabs, adds autosave, a new Character tab, and exposes real models via /v1/models • Exposes real model list via the `GET /v1/models` endpoint instead of a static placeholder. • Merges the Default and Notebook tabs into a single Notebook tab, with a Session tab option to switch between one and two columns. • Autosaves text in the Notebook tab (both generated and manually typed); adds 'New' and 'Rename' buttons for prompt management. • Saved prompts relocated from `user_data/prompts` to `user_data/logs/notebook`. • Adds a new Character tab for character settings. • Adds an option in the Session tab to exclude attachments from previous messages in the chat prompt. • Moves 'Custom system message' to the Parameters > Generation tab. • Truncates web search results to at most 8192 tokens to handle edge cases such as infinite-scrolling pages. • Shows file sizes in the Model tab on 'Get file list'. • Removes images and links from web search results to reduce noise and focus on relevant text content. • Remembers the last selected chat for each chat mode and character. • Forces dark theme on the Gradio login page. • Hides the navigation bar on Ctrl+S / Show controls click. • Updates llama.cpp backend and exllamav3 to 0.0.4. Breaking changes: • Saved prompts have been moved from `user_data/prompts` to `user_data/logs/notebook`; existing prompts must be manually relocated to the new path.

oobabooga textgen v3.5

Wed, 11 Jun 2025 02:15:14 GMT

oobabooga textgen v3.5 adds persistent UI settings, .docx attachments, Qwen3 presets, and RTX 50XX CUDA 12.8 support. • Adds CUDA 12.8 installation option supporting RTX 50XX NVIDIA Blackwell GPUs via ExLlamaV2/V3 and Transformers backends. • Uses `user_data/cache/gradio` for Gradio temporary files instead of the system temporary folder. • Adds Qwen3 presets ('Thinking' and 'No Thinking'); sets `Qwen3 - Thinking` as the new default preset. • Adds support for `.docx` file attachments. • Adds an option (configurable in the Session tab) to automatically convert long pasted text into an attachment; disabled by default. • Adds model name to each message's metadata, displayed in the UI on hover over the message date/time. • Adds 'Restore preset', 'Neutralize samplers', and 'Restore character' buttons to the Parameters tab. • Adds trash () buttons for deleting individual past chats. • Reorganizes the Parameters tab with preset-saved parameters on the left and all other settings on the right. • Optimizes chat streaming to update only the last message during generation, enabling smooth streaming even at 100k token context lengths. • Extracts web search result text with formatting instead of collapsing all content to a single line. • Shows llama.cpp prompt processing progress on a single line. • Adds informative tooltips on hover for the file upload icon and web search checkbox.

oobabooga textgen v3.4.1

Sat, 31 May 2025 02:12:49 GMT

oobabooga textgen v3.4.1 adds file attachments, DuckDuckGo web search, message versioning, and a chat token counter. • Adds `max_updates_second` back to cap UI refresh rate during high-speed streaming (~200 tokens/second), resolving a performance bottleneck. • Adds file attachment support to chat — text files and PDF documents are fully injected into the prompt (not RAG). • Adds a web search feature powered by DuckDuckGo; the LLM auto-generates the search query from your input. • Adds message version navigation ('swipes') — press left/right to browse previous reply versions, or press right at the latest to generate a new one. • Adds a token counter to the chat tab covering input, history, and attachments. • Adds date/time display on chat messages. • Adds footer buttons for editing individual chat messages. • Adds a 'Branch here' footer button on chat messages to fork the conversation.

oobabooga textgen v3.3

Fri, 16 May 2025 20:14:45 GMT

oobabooga textgen v3.3 adds VRAM estimation with auto `gpu-layers` for GGUF and Tools support for the OpenAI-compatible API. • Automatically estimates VRAM usage for GGUF models and sets `gpu-layers` based on free VRAM on NVIDIA GPUs; recalculates in real time when `ctx-size` or `cache-type` changes in the UI. • When loading a GGUF model via CLI (e.g. `--model model.gguf --ctx-size 32768 --cache-type q4_0`), `--gpu-layers` is now calculated automatically — no manual flag required. • Adds Tools support for the OpenAI-compatible API. • Adds the `top_n_sigma` sampler to the llama.cpp loader. • Renders `max_updates_second` obsolete with a new dynamic Chat Message UI update speed that substantially reduces CPU usage in Chat mode. • Simplifies the Model tab by splitting settings into 'Main options' and 'Other options', with 'Other options' hidden in a closed accordion by default. • Streamlines the UI in portable builds: hides non-functional items such as training, shows only the llama.cpp loader, and excludes non-working extensions to reduce build size. • Shows the list of available files when a user attempts to download an entire GGUF repository instead of a specific file. • Handles short arguments (e.g. `ot`) in the `--extra-flags` option for the llama.cpp loader.

oobabooga textgen v3.2

Thu, 01 May 2025 03:18:51 GMT

oobabooga textgen v3.2 adds Qwen3 thinking toggle, auto API port selection, and verbose llama-server logging • Adds `enable_thinking` checkbox under Parameters to enable or disable thinking for Qwen3 models (and future models with the feature); thinking is enabled by default and works via the Jinja2 template. • Automatically finds a new API port if the default one is already taken. • Makes `--verbose` print the `llama-server` launch command to the console. • Makes `` UI blocks closed by default, reducing visual noise from reasoning model output.

oobabooga textgen v3.1

Sun, 27 Apr 2025 03:03:48 GMT

textgen v3.1 adds speculative decoding (up to +88.7% tokens/sec), Vulkan builds, and a universal `--ctx-size` flag across all loaders. • Adds `--extra-flags` parameter to the llama.cpp loader for passing additional flags directly to `llama-server` (e.g. `override-tensor=exps=CPU` for MoE models). • Adds `--streaming-llm` flag to llama.cpp (mapped to `--cache-reuse` in llama.cpp internals) to skip full prompt reprocessing when context length is filled, useful for long role-playing sessions. • Adds universal `--ctx-size` flag to specify context size across all loaders. • Adds speculative decoding to the llama.cpp loader; benchmarks show +88.7% tokens/second with a 27B model using a 1B draft model, with gains of +34–88% observed across different model combinations. • Adds speculative decoding to the non-HF ExLlamaV2 loader. • Adds KV cache quantization to the ExLlamaV3 loader. • Restructures all user data (models, characters, presets, saved settings) under `text-generation-webui/user_data/` to enable portable install updates by moving a single folder. • Adds Vulkan portable builds supporting AMD and Intel Arc GPUs on both Windows and Linux. • Adds prompt processing progress messages to the llama.cpp loader. • UI: Adds a collapsible thinking block for messages containing `` steps. • UI: Sets 'instruct' as the default chat mode. • UI: Adds a greeting when the web UI launches in instruct mode with an empty chat history. • UI: Model menu now displays only part 00001 of multipart GGUF files. Breaking changes: • All user data has moved: models must be manually relocated from `models/` to `user_data/models/`, presets from `presets/` to `user_data/presets/`, and other user data (characters, saved settings) to their corresponding paths under `user_data/` after upgrading.

oobabooga textgen v3.0

Tue, 22 Apr 2025 15:11:56 GMT

oobabooga text-generation-webui v3.0 ships portable zip builds with llama.cpp built in and makes llama.cpp the default loader. • Adds portable zip builds (~700 MB) for Windows, Linux, and macOS (variants: `cuda12.4`, `cuda11.7`, `cpu`, macOS `arm64`, macOS `x86_64`) bundling `text-generation-webui` + `llama.cpp` with no installation required. • API now starts by default on `localhost` in portable builds without requiring the `--api` flag. • The `--gpu-split` flag (previously EXL2-only) is now also reused for Transformers GPU memory control. • Makes llama.cpp the default loader across the project. • Adds support for llama.cpp builds from the `ggml-org/llama.cpp` fork. Breaking changes: • The `--gpu-memory` flag has been removed; use `--gpu-split` instead for Transformers GPU memory configuration.

oobabooga textgen v2.8

Fri, 18 Apr 2025 16:37:36 GMT

oobabooga text-generation-webui v2.8 replaces llama-cpp-python with a new llama-server loader and adds smoother chat streaming. • New `llama-server`-based llama.cpp loader replaces `llama-cpp-python`, adding sampling parameters `xtc`, `dry`, and `dynatemp` and reducing installation size. • Supports GGUF model subfolder organization inside `text-generation-webui/models`, enabling direct import from LM Studio directory layouts. • Adds support for the **Llama-4-Scout-17B-16E-Instruct** model via the updated llama.cpp backend. • Smoother per-word chat streaming in the Chat tab replaces chunked token delivery. • The `llamacpp_HF` loader has been removed; only one llama.cpp loader now exists. Breaking changes: • The `llamacpp_HF` loader has been removed — any workflow that selected it will need to switch to the single remaining llama.cpp loader. • Some command-line flags have been removed — existing launch scripts using those flags will break on upgrade.

oobabooga textgen v2.7

Wed, 09 Apr 2025 17:49:49 GMT

oobabooga text-gen v2.7 adds ExLlamaV3 support via new `ExLlamav3_HF` loader, a Dark chat style, and default 8192 context-length cap. • Adds `ExLlamav3_HF` loader, providing ExLlamaV3 inference with the same sampler stack as `Transformers` and `ExLlamav2_HF`; pre-built wheels for Linux and Windows are included, removing manual installation (requires compute capability 8+). • Adds a new 'Dark' chat style option.

oobabooga textgen v2.6

Sat, 15 Mar 2025 03:11:44 GMT

oobabooga text-gen v2.6 adds a top N-sigma sampler, SuperboogaV2 GPU/date-time upgrades, and a refreshed Perplexity Colors extension. • Adds the top N-sigma sampler for token sampling. • Improves SuperboogaV2 with Date/Time Embeddings, GPU support, and multiple file format handling. • Updates the Perplexity Colors extension to v2 with additional improvements.

oobabooga textgen v2.5

Sat, 15 Feb 2025 02:26:14 GMT

Adds a 'Show after' parameter to the UI for controlling DeepSeek token display. • Adds a `Show after` parameter to the UI for use with DeepSeek `` token handling. • Minor UI improvements to list styles and the light theme.

oobabooga textgen v2.4

Wed, 29 Jan 2025 22:14:49 GMT

Adds 'Continue' and 'Remove' chat buttons, `strftime_now` in JINJA, and smarter installer caching in v2.4 • Adds 'Continue' and 'Remove' buttons below the last chat message in the UI for quick in-place edits. • Adds `strftime_now` to JINJA template support, enabling LLAMA 3.1, 3.2, and Granite model chat templates to render correctly. • Installer now skips re-downloading `.whl` requirements during updates unless the files or local repo commit have changed, speeding up updates after manual branch switches. • Adds a descriptive error message when llama.cpp fails to load a model, prompting users to lower the context length. • Extends OpenAI-compatible leniency to SillyTavern API clients.

oobabooga textgen v2.3

Wed, 15 Jan 2025 01:05:29 GMT

oobabooga text-gen v2.3 cuts Chat tab CPU usage via morphdom incremental rendering and adds per-message copy/regenerate buttons. • Adopts the `morphdom` library for incremental DOM updates in the Chat tab during streaming, drastically reducing CPU usage on long contexts or high token throughput and keeping the UI responsive. • Enables text and code selection/copying from previous chat messages while a reply is still streaming, since only changed elements are updated. • Adds a 'copy raw message content' button below each chat message. • Adds a 'regenerate reply' button below the last chat message. • Activates `auto_max_new_tokens` by default, removing the need to manually continue replies every 512 tokens.

oobabooga textgen v2.2

Thu, 09 Jan 2025 21:48:25 GMT

oobabooga textgen v2.2 adds branch/search chat UI, `--torch-compile`, `--exclude-pattern`, IPv6 API support, and ExLlamaV2 sampler expansions. • Adds `--torch-compile` flag for the transformers loader to improve inference performance. • Adds `--exclude-pattern` flag to the `download-model.py` script to filter files during model downloads. • Connects XTC, DRY, `smoothing_factor`, and `dynatemp` sampling parameters to the ExLlamaV2 loader (non-HF). • Adds IPv6 support to the API. • Adds a 'Static KV cache' option for transformers to improve performance. • Adds a 'Branch chat' option to the chat tab, enabling conversation branching from any point. • Adds a 'Search chats' menu to the chat tab for finding past conversations. • Removes a 0.2-second startup delay for llama.cpp and ExLlamaV2, increasing reported tokens/second. • Adds a horizontal scrollbar to code blocks wider than the chat area. • Removes the AutoGPTQ loader; GPTQ models can still be loaded through ExLlamaV2. Breaking changes: • The AutoGPTQ loader has been removed; any setup relying on it must switch to the ExLlamaV2 loader to load GPTQ models.

oobabooga textgen v2.1

Tue, 31 Dec 2024 23:48:57 GMT

Parameters tab reorganized with grouped controls and new `Instruct` and `Creative` presets in oobabooga textgen v2.1 • Adds two new built-in presets, `Instruct` and `Creative`, replacing 5 removed presets to reduce the default preset list from 11 to 6. • Reorganizes the Parameters tab with grouped input fields (sliders, checkboxes) and headings for parameter types (curve shape, curve cutoff) to reduce visual clutter. • Reorganizes the Model tab with similar grouping and heading structure. • Improves heading, list, and link styling in chat messages. • Improves the streaming typing cursor `|` appearance during chat. • Improves chat colors in light mode.

oobabooga textgen v2.0

Thu, 19 Dec 2024 02:37:32 GMT

text-generation-webui v2.0 overhauls the UI with collapsible sidebars, vertical nav, mobile support, and a new `--old-colors` flag. • Adds `--old-colors` launch flag to restore the previous UI color scheme for users who prefer the legacy look. • Allows more granular KV cache settings. • Redesigns the UI with a vertical left sidebar replacing the top navigation bar (Chat / Default / Notebook tabs). • Adds collapsible sidebars for improved mobile navigation. • Moves the 'past chats' menu into its own sidebar, visually separated from the chat area. • Improves HTML rendering for nested lists with sub-items.

oobabooga textgen v1.16

Fri, 25 Oct 2024 04:10:35 GMT

oobabooga text-gen v1.16 adds Whisper Turbo transcription and RWKV-World instruction template support. • Adds Whisper Turbo as a selectable transcription model option. • Adds RWKV-World instruction template for use with RWKV-World models.

oobabooga textgen v1.15

Tue, 01 Oct 2024 17:48:52 GMT

oobabooga textgen v1.15 adds XTC sampler, sortable repetition penalties, and ExLlamaV2 tensor parallelism for multi-GPU inference. • Adds `xtc` to the `Sampler priority` ordering keywords, alongside new keywords `repetition_penalty`, `presence_penalty`, `frequency_penalty`, `dry`, `encoder_repetition_penalty`, and `no_repeat_ngram`, enabling fine-grained control over the order repetition penalties are applied. • Adds Exclude Top Choices (XTC) sampler to boost creativity, break writing clichés, and inhibit non-verbatim repetition. • Enables ExLlamaV2 tensor parallelism to increase multi-GPU inference speeds. • Removes `AutoAWQ`, `AutoGPTQ`, `HQQ`, and `AQLM` from `requirements.txt`; these can still be installed manually. • Adds a `beforeunload` confirmation dialog when leaving the page to prevent accidental session loss. • Updates API documentation with examples to list and load models. • Defers PEFT import until needed, reducing web UI launch time. Breaking changes: • `AutoAWQ` and `AutoGPTQ` are removed from `requirements.txt` due to lack of support for PyTorch 2.4.1 and CUDA 12.1; setups relying on these quantization backends must install them manually. • `HQQ` and `AQLM` are removed from `requirements.txt`; setups using these experimental backends must install them manually.

oobabooga textgen v1.14

Tue, 20 Aug 2024 04:29:43 GMT

textgen v1.14 adds `--tokenizer-dir` flag for llamacpp_HF and faster multi-file model downloads. • Adds `--tokenizer-dir` flag for use with `llamacpp_HF` to specify a custom tokenizer directory. • Model downloader now uses a single session for all files in a download, reducing per-file startup latency.

oobabooga textgen v1.13

Thu, 01 Aug 2024 05:28:53 GMT

textgen v1.13 adds Llama 3.1 support, manual context-length entry, and improved model download progress. • Makes `compress_pos_emb`, `n_ctx`, `max_seq_len`, and `truncation_length` editable number fields instead of sliders, allowing precise manual entry of context lengths. • Adds Llama 3.1 support via llama-cpp-python 0.2.85 bump, including better handling of the Llama 3.1 Jinja2 template (optional 'tools' headers now suppressed). • Model downloader progress bar now shows filename, file size, and download speed per file.

oobabooga textgen v1.12

Thu, 25 Jul 2024 15:19:22 GMT

oobabooga textgen v1.12 adds Llama 3.1 support across Transformers, ExLlamaV2, and AutoAWQ backends with UI markdown and coloring improvements. • Removes AutoAWQ as a standalone loader; AWQ models (including Llama 3.1 AWQ-INT4) are now loaded directly through Transformers for better compatibility. • Adds Llama 3.1 support via Transformers 4.43, ExLlamaV2 0.1.8, and AutoAWQ 0.2.6 backend bumps. • Colors text between quote characters in chat and chat-instruct modes for improved readability. • Makes the markdown rendering cache infinite (cleared on chat switch) to keep the UI responsive during long 128k-context conversations. Breaking changes: • AutoAWQ is no longer available as a standalone loader; AWQ models must now be loaded through the Transformers loader.

oobabooga textgen v1.11

Tue, 23 Jul 2024 05:34:27 GMT

textgen v1.11 defaults to chat-instruct mode, adds Mistral-Nemo support, and enables gradio subpath customization for reverse proxies. • Adds support for customizing the gradio subpath for use with a reverse proxy. • Adds `ChatML-format.json` format example for training. • Supports Mistral-Nemo via `llama-cpp-python` bump to 0.2.83. • Defaults new sessions to chat-instruct mode, which automatically applies the model's Jinja2 template for higher-quality outputs. • Significantly faster and more responsive UI through refactored event handling that reduces connections between the UI and the server. • Excludes the instruction template from exports when using 'Save UI defaults to settings.yaml', preventing it from being overwritten by the model template.

oobabooga textgen v1.10

Thu, 11 Jul 2024 23:43:59 GMT

oobabooga textgen v1.10 adds Gemma-2 support via ExLlamaV2 and new attention-disable flags. • Adds `--no_xformers` and `--no_sdpa` flags for ExLlamaV2 (complement the existing `--no_flash_attn` flag); all three must be used together — or their UI checkboxes enabled — to run Gemma-2 without severe performance degradation. • ExLlamaV2 updated to 0.1.7, enabling Gemma-2 model support.

oobabooga textgen v1.9

Fri, 05 Jul 2024 03:24:05 GMT

oobabooga text-gen v1.9 adds 4-bit/8-bit KV cache for llama.cpp, Gemma-2 support, and a rewritten Whisper STT extension. • Adds 4-bit and 8-bit KV cache options to llama.cpp and llamacpp_HF backends via the existing `--cache_4bit` and `--cache_8bit` flags. • Adds native Gemma-2 support in Transformers: automatically detects `Gemma2ForCausalLM` architecture and applies optimal settings (bfloat16/float16 precision and eager attention) without requiring `--bf16 --use_eager_attention` manually. • Automatically obtains the EOT token from Jinja2 templates and adds it to stopping strings, so Llama-3-Instruct stops correctly without manually adding `` to custom stopping strings. • Rewrites the Whisper STT extension, replacing the Gradio microphone component with a custom microphone element for improved reliability. • Increases the chat area size on mobile devices.

oobabooga textgen v1.8

Thu, 27 Jun 2024 02:38:24 GMT

oobabooga textgen v1.8 adds TensorRT-LLM as its fastest backend and overhauls the chat UI with persistent browser history. • Adds TensorRT-LLM backend support, now the fastest inference backend in the project, installed in a separate Python 3.10 environment with a provided Dockerfile. • Stores chat history in the browser so conversations survive server restarts without requiring a page refresh. • Redesigned 'past chats' menu from a dropdown to a vertical text list, reducing conversation switching to a single click. • Moves the 'Character' dropdown menu to the main Chat tab for faster character switching. • Enables installing past releases by downloading the `.zip` source and running the `start_` script, without auto-updating to latest. Breaking changes: • The installation script no longer updates to the latest version automatically on run.

oobabooga textgen snapshot-2024-04-21

Sun, 21 Apr 2024 20:19:56 GMT

Adds `/v1/internal/chat-prompt` API endpoint for retrieving the formatted chat prompt. • Adds `POST /v1/internal/chat-prompt` API endpoint to retrieve the fully formatted chat prompt string before generation.

oobabooga textgen snapshot-2024-04-14

Sun, 14 Apr 2024 22:22:11 GMT

Adds min_p sampling preset as default, Ascend NPU support, and HF_ENDPOINT awareness for model downloads. • Respects model and LoRA directory settings when downloading files, so downloads land in the configured paths rather than defaults. • Reads the `HF_ENDPOINT` environment variable when downloading models, enabling use of Hugging Face mirror endpoints. • Adds Ascend NPU as a supported hardware backend. • Adds a `min_p` sampling preset and makes it the default generation preset.

oobabooga textgen snapshot-2024-03-31

Sun, 31 Mar 2024 20:20:47 GMT

Parameters tab reorganized and ChatML support added for Hyperion and Hercules models. • Adds ChatML format configuration for Hyperion and Hercules models. • Reorganizes the Parameters tab layout in the UI.

oobabooga textgen snapshot-2024-03-17

Sun, 17 Mar 2024 20:19:14 GMT

Adds AQLM quantization support, StreamingLLM for llama.cpp backends, and a new User Description field in the UI. • Adds a new 'User description' field in the UI for setting user personality/biography context. • Adds experimental AQLM quantization support for loading AQLM-quantized models. • Adds StreamingLLM support for the `llamacpp` and `llamacpp_HF` backends, enabling efficient long-context streaming inference. • Bumps AutoAWQ to 0.2.3 on Linux, enabling use of newer AWQ model variants on that platform.

oobabooga textgen snapshot-2024-03-10

Sun, 10 Mar 2024 20:20:24 GMT

Adds `cache_q4` KV-cache quantization for ExLlamaV2 and introduces an update wizard in textgen WebUI. • Adds `cache_q4` option for ExLlamaV2 to enable 4-bit KV-cache quantization, reducing VRAM usage during inference. • Introduces an update wizard to guide users through upgrading the WebUI. • Extensions requirements are no longer installed by default, reducing setup overhead for users who don't need them.

oobabooga textgen snapshot-2024-03-03

Sun, 03 Mar 2024 20:24:50 GMT

Adds cubic sampling with a curve parameter and Gemma model compatibility via transformers 4.38.1. • Adds cubic sampling with a configurable curve parameter as a new sampling method for text generation. • Supports Google Gemma models via a bump to transformers 4.38.1.

oobabooga textgen snapshot-2024-02-18

Sun, 18 Feb 2024 20:19:24 GMT

Adds `--autosplit` for ExLlamaV2, a llamacpp_HF creator menu, and per-model instruction template customization. • Adds `--autosplit` flag for ExLlamaV2 to automatically split model layers across GPUs. • Adds a 'llamacpp_HF creator' menu in the UI for building llama.cpp HuggingFace-compatible model configs. • Adds a menu for customizing the instruction template on a per-model basis. Breaking changes: • The 'Maximum UI updates/second' parameter has been removed.

oobabooga textgen snapshot-2024-02-11

Sun, 11 Feb 2024 20:19:30 GMT

Adds quadratic sampling and custom sampler order support; llama.cpp multi-GPU now splits by rows instead of layers. • Adds custom sampler order support, letting users control the sequence in which samplers are applied during text generation. • Adds quadratic sampling as a new sampling method for text generation. • Splits by rows instead of layers for llama.cpp multi-GPU configurations, improving multi-GPU utilization. • Removes the non-HuggingFace ExLlamaV2 loader, consolidating to the HF-based loader only. Breaking changes: • The non-HuggingFace ExLlamaV2 loader has been removed; setups using that loader must switch to the HuggingFace ExLlamaV2 loader.

oobabooga textgen snapshot-2024-02-04

Sun, 04 Feb 2024 20:19:08 GMT

Adds CPU support for Coqui TTS, new Docker GPU options, and a roleplay GBNF grammar file. • Adds `roleplay.gbnf` grammar file for constrained roleplay-format generation. • Adds Docker options for CPU-only, Intel GPU, and AMD GPU deployments. • Enables the Coqui TTS extension to run on CPU, removing the GPU requirement. • Supports running the tool as a git submodule.

oobabooga textgen snapshot-2024-01-28

Sun, 28 Jan 2024 20:18:40 GMT

Adds `prompt_lookup_num_tokens`, multimodal content arrays, RGBA image support, and `trust_remote_code` for DeepSpeed loading. • Adds `prompt_lookup_num_tokens` parameter for prompt lookup decoding. • Adds `trust_remote_code` support when loading models with DeepSpeed. • Updates `n_gpu_layers` default to 256 to support larger models. • Supports content arrays in multimodal OpenAI API requests. • Supports RGBA color format for image inputs.

oobabooga textgen snapshot-2024-01-21

Sun, 21 Jan 2024 20:20:18 GMT

Adds dynatemp parameters, past-chat sidebar, and Tab-key navigation between tabs. • Adds `dynatemp_low`, `dynatemp_high`, and `dynatemp_exponent` parameters for dynamic temperature control during generation. • Adds a past chat histories sidebar on desktop for quick access to previous conversations. • Adds Tab key shortcut to switch between the current tab and the Parameters tab.

oobabooga textgen snapshot-2024-01-14

Sun, 14 Jan 2024 20:20:10 GMT

Adds dynamic temperature parameters and a desktop chat history sidebar to oobabooga text-generation-webui. • Adds `dynatemp_low`, `dynatemp_high`, and `dynatemp_exponent` parameters for dynamic temperature control during text generation. • New sidebar on desktop displays past chat histories for quick access. • Press Tab to switch between the current tab and the Parameters tab in the UI.

oobabooga textgen snapshot-2024-01-07

Sun, 07 Jan 2024 20:19:53 GMT

Adds `dynamic_temperature_low` parameter and Dynamic Temperature support for the HF loader. • Adds `dynamic_temperature_low` parameter for finer control over dynamic temperature sampling ranges. • Adds Dynamic Temperature sampling support for the HuggingFace (HF) loader. • Adds Docker build args to support non-AVX2 CPUs.

oobabooga textgen snapshot-2023-12-31

Sun, 31 Dec 2023 20:19:47 GMT

Adds image input to OpenAI chat API and Synthia instruction templates; removes `--sdp-attention`, `--xformers`, ExLlamaV1, and RWKV loaders. • Adds image sending support to the OpenAI chat API endpoint, enabling multimodal chat requests. • Adds Synthia instruction templates for Synthia-series models. Breaking changes: • The `--sdp-attention` and `--xformers` flags are removed; any launch scripts using them will fail on upgrade. • ExLlamaV1 loaders are removed; models configured to use ExLlamaV1 must be switched to an alternative loader. • The RWKV loader is removed; RWKV models can no longer be loaded. • Additional deprecated command-line flags are removed; any scripts relying on them will break.

oobabooga textgen snapshot-2023-12-24

Sun, 24 Dec 2023 20:19:40 GMT

Adds HQQ quant loader, GGUF instruction-template auto-detection, tensor-core llama.cpp wheels, and safetensor LoRA support for ExLlama v1. • Adds HQQ quantization loader for loading HQQ-quantized models. • Adds auto-detection of Eric Hartford's Dolphin models via `models/config.yaml`. • Adds Orca-Vicuna instruction template. • Reads instruction templates automatically from GGUF metadata for llama.cpp models, removing the need to set them manually. • Adds llama-cpp-python wheels built with tensor cores support, unlocking faster inference on compatible GPUs. • Enables ExLlama v1 models to load safetensor-format LoRAs. • Organizes CMD arguments by group for clearer CLI navigation. • Updates Training PRO with new capabilities. • Bumps ExLlamaV2 to 0.0.11.

oobabooga textgen snapshot-2023-12-17

Sun, 17 Dec 2023 20:19:46 GMT

Adds Jinja templates for chat/instruct, OpenAI embeddings endpoint, `max_logits` param, and `--num_experts_per_token` for ExLlamav2. • Adds `--num_experts_per_token` CLI parameter for ExLlamav2 to control mixture-of-experts routing per token. • Adds OpenAI-compatible embeddings endpoint via the OpenAI extension. • Adds `max_logits` parameter to the OpenAI extension logits endpoint. • Introduces Jinja2 template support for Instruct and Chat prompt formatting. • Supports symlinked folders within the root directory for model/asset organization. • Bypasses Coqui TTS EULA check to streamline TTS extension setup. • Removes the ElevenLabs extension. Breaking changes: • The ElevenLabs extension has been removed; any setup relying on it will stop working after upgrade.

oobabooga textgen snapshot-2023-12-10

Sun, 10 Dec 2023 20:19:46 GMT

Adds QuIP# quantization support and gallery improvements; changes defaults for max_new_tokens and repetition_penalty_range. • Changes default values for `max_new_tokens` and `repetition_penalty_range` parameters. • Adds QuIP# quantization format support for model loading. • Improves the model Gallery UI. Breaking changes: • Default values for `max_new_tokens` and `repetition_penalty_range` have changed; existing workflows relying on the previous defaults may produce different outputs after upgrade.

oobabooga textgen snapshot-2023-12-03

Sun, 03 Dec 2023 20:19:48 GMT

Adds OpenHermes Mistral model support and Docker container CLI improvements. • Adds support for the OpenHermes Mistral model. • Improves Docker container CLI arguments for more practical containerized deployments.

oobabooga textgen snapshot-2023-11-19

Sun, 19 Nov 2023 20:19:46 GMT

Adds `--admin-key`, `--nowebui`, `/v1/internal/logits`, and `/v1/internal/lora` endpoints plus a random preset button. • Adds `--admin-key` flag to require an API key for all API access. • Adds `--nowebui` flag to run the server in pure API mode with no browser UI. • Adds `GET /v1/internal/logits` endpoint for retrieving token logits via the API. • Adds `/v1/internal/lora` endpoints for managing LoRA adapters via the API. • New 'random preset' button in the UI to randomly select a generation preset. • Character pictures in chat mode can now be enlarged on click. • System message support added to chat instruct mode, shared between UI and API.

oobabooga textgen snapshot-2023-11-12

Sun, 12 Nov 2023 20:19:20 GMT

OpenAI API becomes the default, gains `/v1/internal/stop-generation` endpoint, and now supports `trust_remote_code` for embeddings. • Adds `POST /v1/internal/stop-generation` endpoint to the OpenAI-compatible API, allowing programmatic cancellation of in-progress generation. • Makes the OpenAI-compatible API the default API (previously non-default). • Enables `trust_remote_code` support in the OpenAI API embedder, allowing embedding models that require remote code execution. • Separates context and system message fields in instruction formats, enabling independent control of each in prompt templates.

oobabooga textgen snapshot-2023-11-05

Sun, 05 Nov 2023 20:19:32 GMT

Adds `Min P` sampler, `temperature_last` parameter, and `use_flash_attention_2` flag to oobabooga text-generation-webui. • Adds `temperature_last` parameter to control the order in which temperature sampling is applied relative to other samplers. • Adds `use_flash_attention_2` parameter to the Transformers model loader to enable Flash Attention 2 support. • Implements `Min P` as a new sampler option in HF loaders for controlling minimum probability thresholds during generation. • Adds a flag to force loading models from safetensors format in the Transformers loader.

oobabooga textgen snapshot-2023-10-29

Sun, 29 Oct 2023 20:19:20 GMT

Adds `presence_penalty` and `frequency_penalty` sampler settings, plus initial Intel GPU support. • Adds `presence_penalty` sampler setting (renamed from `additive_repetition_penalty`) to control token repetition based on presence in the output so far. • Adds `frequency_penalty` sampler setting to penalize tokens proportional to how frequently they have already appeared in the output. • Adds initial Intel GPU support.

oobabooga textgen snapshot-2023-10-22

Sun, 22 Oct 2023 20:19:18 GMT

Adds LLaVA v1.5 support, multi-LoRA blending, flash-attention 2 on Windows, and expanded Silero language coverage. • Supports LLaVA v1.5 and LLaVA v1.5 7B multimodal models for vision-language inference. • Allows multiple loaded LoRAs to simultaneously influence generation output, enabling LoRA blending workflows. • Adds option to select additional linear modules and layers as targets during LoRA training. • Adds flash-attention 2 support on Windows. • Adds Python 3.11 support with structured requirements. • Expands Silero text-to-speech language coverage with more supported languages. • Enables special token support for the ExLlamav2 backend.

oobabooga textgen snapshot-2023-10-15

Sun, 15 Oct 2023 20:27:48 GMT

textgen snapshot adds LLaVA-LLaMA-2 support, HTTPS for OpenAI extension, ChatML, and ExLlamaV2 LoRA. • Adds HTTPS support to the OpenAI extension, enabling secure API connections. • Adds ChatML prompt format support, including Mistral-OpenOrca compatibility. • Adds LoRA support for the ExLlamaV2 backend. • Supports LLaVA-LLaMA-2 multimodal model loading.

oobabooga textgen v1.7

Sun, 08 Oct 2023 20:26:50 GMT

v1.7 adds AutoAWQ loader support, grammar for Transformers/HF loaders, NUMA for llama.cpp, and EOS-abort for text generation. • Adds `grammar` support to the `transformers` and `_HF` loaders, enabling constrained decoding for locally loaded models. • Adds initial AutoAWQ loader support, allowing AWQ-quantized models to be loaded directly. • Enables NUMA feature for `llama_cpp_python`, improving CPU memory locality on multi-socket systems. • Model downloader now downloads `*.tiktoken` tokenizer files alongside model weights. • Adds EOS-token abort behavior in text generation — output stops immediately when the end-of-sequence token is reached. • Updates on the one-click installer now preserve custom changes to `CMD_FLAGS.txt` rather than overwriting them. • Expands macOS `llama.cpp` support in `requirements.txt`. • Bumps ExLlamaV2 to 0.0.5 and AutoAWQ to v0.1.4.

oobabooga textgen 1.6.1

Tue, 26 Sep 2023 03:37:57 GMT

Adds a grammar editor to the UI and alternative requirements.txt with AMD and Metal wheels. • Adds a grammar editor to the UI for direct grammar authoring during inference. • Adds an alternative `requirements.txt` with AMD and Metal wheels for broader hardware support.

oobabooga textgen v1.6

Fri, 22 Sep 2023 22:17:32 GMT

oobabooga textgen v1.6 adds GGUF support, ExLlamaV2 loaders, CFG sampling, SSL, named Cloudflare tunnels, and a logit viewer. • Adds `--cpu` option for llama.cpp to force CPU-only inference and prevent CUDA from being used. • Adds `--disable_exllama` option for AutoGPTQ to bypass ExLlama kernels. • Adds `auto_max_new_tokens` parameter to automatically size the maximum new tokens based on context. • Adds `max_tokens_second` parameter to rate-limit token generation speed. • Adds SSL certificate support for securing the web UI endpoint. • Adds option for named Cloudflare tunnels, enabling persistent public URLs. • Adds `ExLlamaV2` and `ExLlamaV2_HF` loaders for next-generation ExLlama inference. • Adds GGUF format support, including reading GGUF metadata fields such as `scale_linear` and `freq_base`. • Adds Classifier Free Guidance (CFG) sampling for Transformers, ExLlama, ExLlama_HF, and llamacpp_HF loaders. • Adds new RoPE settings for ExLlama, with NTK (alpha) and RoPE scaling also supported for ExLlamaV2 and ExLlamaV2_HF. • Adds `mlock` and `no-mmap` options to ctransformers loader. • Adds GGUF support to ctransformers loader. • Adds array (batched) input support to the `openai` extension API. • Handles Chunked Transfer Encoding in the `openai` extension for streaming requests. • Adds `chat_instruct_command` support in the API. • Adds `save defaults to settings.yaml` button so current UI parameters persist across sessions. • Adds `Show controls` button to the chat UI to toggle the parameter panel. • Adds a simple logit viewer to inspect per-token probability distributions. • Adds probability dropdown to the `perplexity_colors` extension. • Adds language option to the `silero_tts` extension. • Adds silero TTS audio preview back to the UI. • Unifies the three interface modes (chat, notebook, default) into a single cohesive UI. • Adds support for downloading a single file from a model repository via the UI and `download-model.py`. • Merges one-click-installers into the main repository, reducing installation size by several GB with a more reliable update procedure. • Adds a standalone Dockerfile for NVIDIA Jetson devices. • Adds ffmpeg to the Docker image. • Adds chat instruction configuration for BaiChuan-chat, starchat, Vicuna-v1.5, pygmalion-2, mythalion, and StableBeluga2 model series. • Adds scrollbar to notebook/default textboxes and enables scrolling during token streaming. • Refactors the training tab UI for improved usability. • Uses separate llama-cpp-python packages to maintain GGML support alongside GGUF.

oobabooga textgen v1.5

Wed, 26 Jul 2023 14:14:49 GMT

oobabooga text-generation-webui v1.5 adds a new extension hook, automatic instruction templates, and drops FlexGen support. • Introduces `chat_input_modifier` extension function in `extensions/example/script.py` as the replacement for the deprecated `input_hijack` hook. • Switches to the dark theme as the new default UI appearance. • Removes FlexGen backend support, which has been made obsolete by llama.cpp and 4-bit quantization. • Adds a detailed extension authoring example at `extensions/example/script.py` with updated extension documentation. Breaking changes: • `input_hijack` is deprecated in favour of `chat_input_modifier`; existing extensions using `input_hijack` will need to migrate to the new function. • FlexGen support is removed; any setup relying on the FlexGen backend will no longer work after upgrading.

oobabooga textgen v1.4

Mon, 24 Jul 2023 19:42:39 GMT

oobabooga textgen v1.4 adds llama-2-70b GGML support and expands OpenAI extension with images and logit_bias/logprobs. • OpenAI extension gains support for `logit_bias` and `logprobs` parameters, image handling, embeddings, and token endpoints — with improved error reporting and updated docs. • Adds llama-2-70b GGML model support. • Bumps exllama module to 0.0.8 with expanded LoRA support. • Bumps bitsandbytes to 0.41.0 for faster inference speeds.

oobabooga textgen v1.3

Tue, 18 Jul 2023 20:33:25 GMT

oobabooga textgen v1.3 adds Llama-v2 support, customizable RoPE for GGML, and LoRA loading via AutoGPTQ 0.3.0. • Adds Llama-v2 instruction template with auto-detection of truncation length. • Adds customizable RoPE (Rotary Position Embedding) support for GGML models. • Adds Airoboros-v1.2 instruction template. • AutoGPTQ updated to 0.3.0, enabling LoRA loading out of the box for GPTQ models. • LoRA menu selection is now preserved when loading a model (no longer reset on model load). • Disables 'Autoload the model' by default, and disables auto-loading at startup even when only one model is available. Breaking changes: • The 'Autoload the model' setting is now disabled by default; setups that relied on automatic model loading at startup will need to re-enable it manually.

oobabooga textgen v1.2

Sun, 16 Jul 2023 05:44:27 GMT

v1.2 adds a llamacpp_HF loader, exllama perplexity evaluation, logits processor support in extensions, and token probability/perplexity colorization. • New `llamacpp_HF` loader adds a llama.cpp backend variant compatible with the Hugging Face tokenizer interface. • Extensions can now register custom logits processors, enabling token-level generation control from extension code. • Enables perplexity evaluation for models loaded with the exllama backend. • Increases the alpha value limit for NTK RoPE scaling on exllama and exllama_HF loaders, supporting longer context extrapolation. • New 'Color tokens by probability and/or perplexity' extension visualizes per-token generation confidence in the UI.

oobabooga textgen v1.1

Wed, 12 Jul 2023 18:56:03 GMT

oobabooga text-generation-webui v1.1 adds low-VRAM llama.cpp mode, token auth for model downloads, and TensorBoard/W&B training integration. • Adds low VRAM mode support to the llama.cpp module for running larger models on constrained hardware. • Adds token authorization for downloading models, enabling authenticated access to gated or private model repositories. • Adds `context_instruct` field to the API and loads the default model instruction template automatically. • Expands `models/config.yaml` with entries for platypus/gplatty, longchat, vicuna-33b, Redmond-Hermes-Coder, wizardcoder, and additional models. • Adds TensorBoard and Weights & Biases integration for monitoring training runs. • Adds default environment variable values to the Docker Compose file for easier out-of-box deployment. • Adds ability to load all text files from a subdirectory for training datasets. • Adds a feature to log a sample of the training dataset for inspection during fine-tuning. • Chat history downloads now generate more detailed file names. • Disables wandb remote HTTP requests by default. • `--load-in-4bit` quantization is significantly faster following a bitsandbytes Windows wheel update.