Triton Inference Server — The AI Toolchain

Triton Inference Server v2.71.0

Wed, 29 Jul 2026 16:09:26 GMT

Triton v2.71.0 adds multi-GPU TensorRT inference, PyTorch 2 batching, NV embedding layers, and tighter OpenAI frontend security. • Redacts restriction header values from the OpenAI frontend startup log and enforces inference restrictions in the SageMaker MME invoke handler. • Adds a streaming tool-call parse buffer limit to the OpenAI-compatible frontend to cap memory usage during streaming tool calls. • TensorRT backend gains multi-device (multi-GPU) inference support. • PyTorch backend enables PyTorch 2 batching and adds support for loading NV embedding layers. • Adds support for serving HSTU generative-recommender models via the PyTorch AOTI serving path. • Python client now preserves the output datatype for empty (zero-element) tensors.

Triton Inference Server v2.71.0

Wed, 29 Jul 2026 16:09:26 GMT

Triton v2.71.0 adds TensorRT multi-GPU inference, PyTorch 2 batching, NV embedding layers, and OpenAI frontend memory safeguards. • Adds a streaming tool-call parse buffer limit to the OpenAI-compatible frontend to cap memory usage during streaming tool calls. • Redacts restriction header values from the OpenAI frontend startup log and enforces inference restrictions in the SageMaker MME invoke handler. • TensorRT backend gains multi-device (multi-GPU) inference support. • PyTorch backend enables PyTorch 2 batching and adds support for loading NV embedding layers. • Supports HSTU generative-recommender models via the PyTorch AOTI serving path. • Python client now preserves output datatype for empty (zero-element) tensors.

Triton Inference Server v2.70.0

Fri, 26 Jun 2026 18:41:12 GMT

Triton v2.70.0 adds multi-GPU TensorRT inference, PyTorch 2 dynamic batching, and OpenAI frontend request-size limits. • Adds multi-device (multi-GPU) inference support to the TensorRT backend. • Enables PyTorch 2 dynamic batching in the PyTorch backend. • Adds an HTTP request body size limit to the OpenAI-compatible frontend. • Hardens model environment unpacking and file-path handling against path traversal attacks. • Changes the container install tree under `/opt/tritonserver/` to root-owned, with the `triton-server` user accessing it via group/other permissions. Breaking changes: • Windows server build is removed; Windows is no longer supported in the core build or documentation. • The Python client now requires `ml_dtypes.bfloat16` arrays for BF16 input/output tensors and no longer casts to/from `np.float32`; scripts passing BF16 data as `np.float32` arrays must switch to `ml_dtypes.bfloat16`.

Triton Inference Server v2.69.0

Tue, 02 Jun 2026 17:33:03 GMT

Triton v2.69.0 adds a Rust gRPC client, Azure Managed Identity auth, GPU_DEVICE_IDS for vLLM, and gRPC request cancellation. • Adds `GPU_DEVICE_IDS` to the vLLM backend config to pin multi-GPU models to specific GPU devices. • Adds Azure Managed Identity authentication support for Azure Storage model repositories, enabling credential-free model loading. • Adds a Rust gRPC client library as a new supported client language. • Supports request cancellation in the gRPC C++ client. • Adds `MemberIsNullIs[type]` helpers to `TritonJson` for safer JSON field-presence and null checks. • Adds complex I/O type support for PT2 models in the PyTorch backend. • Adds `auditwheel` to the `Dockerfile.sdk` `sdk_build` stage and the Model Analyzer devel container for arch-specific `manylinux` wheel tagging. • Triton now rejects inference requests whose parameters or forward headers use Triton-reserved keys; the Python client enforces the same rejection on the send side. • Triton validates and rejects inference requests containing duplicate output names in both HTTP and gRPC. • Replaces `pickle` with `json` for inter-process payloads in the vLLM backend.

Triton Inference Server v2.68.0

Tue, 28 Apr 2026 17:10:50 GMT

Triton v2.68.0 disables client shared memory by default, adds ensemble queue limits, and brings OpenAI-compatible model control mode. • Adds `--allow-client-shm=true` flag required to re-enable client shared memory, which is now disabled by default. • Enforces `max_inflight_requests` as a shared limit across ensemble requests, preventing unbounded queue growth in ensemble pipelines. • Adds support for explicit model control mode and model management in the OpenAI-compatible frontend. • Adds model name validation at load time following the POSIX standard. • Introduces safe `GetElementCount` and `GetByteSize` APIs with overflow protection across core and common libraries. • PyTorch backend adds Meta OpTree support. Breaking changes: • Client shared memory is now disabled by default; servers that relied on the previous default must add `--allow-client-shm=true` to their startup command or shared-memory functionality will be unavailable.

Triton Inference Server v2.67.0

Fri, 27 Mar 2026 16:47:26 GMT

Triton v2.67.0 adds PyTorch AOT Inductor (torch_aoti) platform, Python backend is_ready() health hooks, and bfloat16 ONNX I/O support. • Adds new platform `torch_aoti` for PyTorch PT2 format models using AOT Inductor, with default model file `model.pt2`; introduces provider classes `InductorModel` and `InductorModelInstance`; separates `pytorch_libtorch` and `torch_aoti` into distinct namespaces; includes helper utilities, macros, `TritonException`, and optional debug trace logging via `ENABLE_DEBUG_TRACE_*` flags. • Adds support for a user-defined is_ready() method in Python backend models to enable custom readiness/health logic. • Enables bfloat16 I/O tensor dtype support in the ONNX Runtime backend. • Adds model name validation for model management requests. • Introduces safe `GetElementCount` and `GetByteSize` APIs with overflow protection and proper validation. • Adds validation for OpenAI frontend LoRA paths. • Relaxes the upper bound on the gRPC dependency in the client library for greater version flexibility.

Triton Inference Server v2.65.0

Tue, 03 Feb 2026 20:20:28 GMT

Triton v2.65.0 adds first-response latency histogram metrics for coupled models and makes Perf Analyzer dependencies optional. • Adds support for the `nv_inference_first_response_histogram_ms` metric when running models in coupled (non-decoupled) mode, enabling latency histogram tracking for standard inference workloads. • Makes Perf Analyzer dependencies optional when installing `tritonclient` via pip, reducing install footprint for deployments that don't need performance analysis. • Exposes HTTP errors for non-JSON format responses, improving error visibility for clients sending malformed requests.

Triton Inference Server v2.64.0

Wed, 24 Dec 2025 02:01:09 GMT

Triton v2.64.0 promotes OpenAI-compatible frontend to stable, adds multi-LoRA and `echo` parameter support. • Adds `echo` request parameter for TensorRT-LLM and Python backends to the OpenAI-compatible API frontend `v1/completions` endpoint. • Enables multi-LoRA support for the TensorRT-LLM backend via the OpenAI-compatible API frontend. • Adds `TRITONBACKEND_ModelInstanceReady` function for backends to implement, enabling accurate model readiness reporting. • Promotes the OpenAI-compatible frontend from beta to stable release. • Updates the Python backend to accurately report model readiness status.

Triton Inference Server v2.63.0

Wed, 26 Nov 2025 01:26:44 GMT

Triton v2.63.0 adds `v1/embeddings` endpoint for vLLM and `echo` parameter support in the OpenAI-compatible API server. • Enables the `v1/embeddings` endpoint for the vLLM backend in the OpenAI-compatible API server. • Enables the `echo` parameter for TensorRT-LLM and Python backends in the OpenAI-compatible API server. • Provides more specific, OpenAI-compliant error codes from the OpenAI-compatible API server.

Triton Inference Server v2.61.0

Tue, 07 Oct 2025 22:10:06 GMT

Triton v2.61.0 adds static key authentication and repository scoping for OpenAI Frontend APIs. • Adds static key authentication support for OpenAI Frontend APIs.

Triton Inference Server v2.60.0

Tue, 26 Aug 2025 22:15:33 GMT

Triton Inference Server v2.60.0 adds CUDA 13 support. • Adds CUDA 13 support to Triton Inference Server.

Triton Inference Server v2.59.0

Thu, 26 Jun 2025 23:35:58 GMT

Triton Inference Server v2.59.0 improves ensemble model throughput and latency for out-of-order response scenarios. • Improves ensemble model performance in out-of-order response scenarios, increasing maximum throughput and reducing latency.

Triton Inference Server v2.58.0

Sat, 31 May 2025 09:16:11 GMT

Triton v2.58.0 adds tool calling for Llama 3/Mistral, TensorRT memory strategy config, and expanded GenAI-Perf GPU metrics. • Adds `execution_context_allocation_strategy` optional parameter to TensorRT backend configuration to control memory allocation behavior. • GenAI-Perf now supports a configuration file as an alternative to command-line arguments. • GenAI-Perf collects GPU metrics from the `/metrics` endpoint exposed by DCGM Exporter. • GenAI-Perf adds support for Power, Utilization, ECC, Errors, and PCIe metrics. • Adds tool calling support for Llama 3 and Mistral models via the OpenAI frontend.

Triton Inference Server v2.57.0

Mon, 12 May 2025 18:13:57 GMT

Triton v2.57.0 adds gRPC infer thread count exposure, BLS decoupled cancellation, and major GenAI-Perf enhancements including config file support and TPS/user metric. • Exposes gRPC infer thread count as a configurable server option. • Adds BLS decoupled request cancellation support in the Python Backend. • GenAI-Perf now supports a configuration file as an alternative to command-line arguments. • GenAI-Perf adds support for the Hugging Face TGI (Text Generation Inference) generated endpoint. • GenAI-Perf adds a Token per Second per User (TPS/user) metric. • GenAI-Perf metric parsing speed increased by 60%, unlocking faster benchmarking at scale.

Triton Inference Server v2.56.0

Mon, 07 Apr 2025 19:30:21 GMT

Triton v2.56.0 adds SageMaker generate/stream inference types and live KV-cache metrics in HTTP response headers for TRT-LLM. • Adds `SAGEMAKER_TRITON_INFERENCE_TYPE` environment variable to select inference type (`infer`, `generate`, or `generate_stream`) on SageMaker server launch, enabling generate and generate_stream endpoints for SageMaker deployments. • When used with TRT-LLM, Triton now includes live KV-cache utilization and capacity metrics in the HTTP response header during inference requests, enabling on-demand metric retrieval for external load balancers such as the Kubernetes Inference Gateway API. Breaking changes: • The TensorFlow Backend is deprecated as of 25.03; the '25.03-tf2-python-py3' container is no longer available. Users must build the TensorFlow Backend from source and install it into `/opt/tritonserver/backends/` to continue using it.

Triton Inference Server v2.55.0

Wed, 26 Feb 2025 19:55:19 GMT

Triton v2.55.0 adds inference response parameters in Python backend, guided generation support, and major GenAI-Perf upgrades including Jinja2 output templating. • Adds support for the `guided_generation` request parameter in vLLM interactions for constrained decoding workflows. • Python backend now supports setting and retrieving Inference Response Parameters on `InferenceResponse` objects in `model.py`. • GenAI-Perf adds Jinja2 template support for formatting output reports. • GenAI-Perf telemetry now supports multiple metric endpoints. • GenAI-Perf supports corpus sizes up to 90x larger than previously supported. • GenAI-Perf now accepts keys without values as input. • GenAI-Perf adds a chat template option for the TRT-LLM engine. • Adds dynamic sampling parameter handling for vLLM interactions, improving flexibility across requests. • Optimized core Python binding architecture for improved OpenAI frontend performance. • Improved Multi-LoRA handling in TRTLLM gRPC client `end_to_end_grpc_client.py`.

Triton Inference Server v2.54.0

Wed, 29 Jan 2025 23:09:44 GMT

Triton v2.54.0 adds Blackwell GPU support, vLLM ZMQ engine comms, and major GenAI-Perf expansions including a new 'analyze' subcommand. • GenAI-Perf gains `--num-system-prompts` and `--system-prompt-length` flags to create a prefix pool that emulates system prompts in benchmarking runs. • GenAI-Perf gains `--request-count` to send a specific number of requests and `--header` to attach custom headers to every request. • GenAI-Perf gains a new `analyze` subcommand (migrated from Model Analyzer) to sweep configurations and find the optimal model setup. • GenAI-Perf gains a `compare` subcommand that accepts a custom tokenizer for comparing multiple benchmark profiles. • GenAI-Perf now reports time-to-second-token and request count as first-class metrics. • GenAI-Perf now surfaces more detailed errors when OpenAI frontends return an error or metric generation fails. • GenAI-Perf now provides the exact input sequence length requested for synthetic text generation. • vLLM backend can now communicate with the vLLM engine via ZMQ, enabling vLLM v0.6 performance improvements. • Supports Blackwell GPU architectures starting with this release. • The vLLM container shipped with Triton is now NVIDIA-optimized; users who prefer the public vLLM build can continue to construct their own Triton-vLLM container. • Triton Windows beta release now includes CUDA context sharing support in the TensorRT Backend.

Triton Inference Server v2.53.0

Mon, 23 Dec 2024 21:20:20 GMT

v2.53.0 adds vLLM health checks, additional vLLM outputs, multi-node Perf Analyzer support, and signed Windows binaries with long-path and UTF-16 model repo support. • Adds optional vLLM backend health check that automatically unloads the model when the vLLM engine health check fails. • Adds support for requesting additional outputs from the vLLM backend beyond default inference results. • Adds multi-node process support to Perf Analyzer for TensorRT-LLM workloads. • Windows executables and DLLs are now signed by NVIDIA, removing the untrusted-software popup when starting Triton outside administrator mode. • Triton on Windows now supports long path notation for model repositories. • Triton on Windows now supports wide character encoding (UTF-16) for model repositories. • Improved server stability during gRPC client cancellation.

Triton Inference Server v2.52.0

Tue, 26 Nov 2024 23:13:19 GMT

Triton v2.52.0 adds a 'Request to First Response Time' histogram metric for decoupled models and per-model histogram bucket overrides. • Adds `--metrics-config histogram_latencies=true` flag to enable a new 'Request to First Response Time' histogram metric for decoupled models. • Adds new `model_metrics` model configuration field to override default buckets for histogram metric families on a per-model basis. • Adds a comprehensive tutorial on Semantic Caching optimization for LLM workloads to the Conceptual Guides.

Triton Inference Server v2.51.0

Tue, 29 Oct 2024 15:38:56 GMT

Triton Inference Server v2.51.0 optimizes vLLM performance with custom metrics support. • Optimizes vLLM performance with custom metrics.

Triton Inference Server v2.50.0

Fri, 27 Sep 2024 16:50:59 GMT

vLLM backend gains five new metrics including e2e latency and token counts; new LLM agent and Kubernetes scaling tutorials added. • vLLM backend now exposes five additional Triton metrics: `vllm:e2e_request_latency_seconds`, `vllm:request_prompt_tokens`, `vllm:request_generation_tokens`, `vllm:request_params_best_of`, and `vllm:request_params_n`. • New tutorials added covering constrained decoding in the TensorRT-LLM Python backend and function/tool calling for AI agents. • New tutorials added for Kubernetes multi-node and multi-instance scaling with Triton and TRT-LLM.

Triton Inference Server v2.49.0

Tue, 27 Aug 2024 18:03:16 GMT

Triton v2.49.0 adds vLLM metrics reporting, BF16 via DLPack, GPU device ID pinning per instance, and expanded GenAI-Perf profiling. • Enables vLLM metrics reporting (`vllm:prompt_tokens_total`, `vllm:generation_tokens_total`, `vllm:time_to_first_token_seconds`) by setting `REPORT_CUSTOM_METRICS` to `yes` in `config.pbtxt`. • Adds `gpu_device_ids` field to TensorRT-LLM backend instance configuration, allowing GPU device IDs to be specified per model instance. • GenAI-Perf now accepts multiple user-specified prompts via `--input-file`, enabling custom prompt-set benchmarking. • Python backend now supports BF16 tensors via DLPack. • GenAI-Perf can now profile OpenAI-compatible embeddings and Hugging Face TEI re-ranker API-compatible rankings. • Loaded model versions whose model files are unmodified are no longer reloaded when the model config is updated to load new versions, reducing unnecessary reload overhead.

Triton Inference Server v2.48.0

Wed, 24 Jul 2024 22:52:33 GMT

Triton v2.48.0 adds custom backend tracing, enhanced failure metrics, and GenAI-Perf support for embeddings, re-rankers, and multi-prompt input. • Adds `--input-file` flag to GenAI-Perf to accept multiple user-specified prompts in a single run. • Supports tracing custom backend activities via the backend tracing API (see trace.md#custom-backend-tracing). • Enhances Failure Count Metrics to include the failure reason of each inference request. • Enables profiling of OpenAI-compatible embeddings and Hugging Face TEI re-ranker API-compatible rankings via GenAI-Perf. • Updates request-rate handling in OpenAI and HTTP clients so async requests are dispatched at exactly the specified rate. • Adds a warning in Perf Analyzer when latency does not stabilize for async models, while still allowing the run to complete. • Adds input/output validation in Perf Analyzer, returning an error if user-supplied inputs or outputs are not present in the model.

Triton Inference Server v2.47.0

Fri, 28 Jun 2024 00:34:40 GMT

Triton v2.47.0 adds BF16 support in TensorRT backend, GenAI-Perf compare/multi-LoRA/custom-viz, and `--enable-peer-access` GPU flag. • Adds `--enable-peer-access` flag to control whether Triton attempts to enable GPU peer access at startup (default: TRUE). • Adds `compare` subcommand to GenAI-Perf for side-by-side comparison across multiple profiling runs. • Adds multi-LoRA and multi-model support in GenAI-Perf. • Adds custom visualizations in GenAI-Perf. • Adds fixed request count support in Perf Analyzer. • Adds ensemble top-level response caching support in Perf Analyzer. • TensorRT Backend now supports the BF16 datatype. • Python models in default mode can now send responses using `InferenceResponseSender`, matching the capability previously available only in decoupled mode.

Triton Inference Server v2.46.0

Sat, 25 May 2024 05:34:12 GMT

Triton v2.46.0 adds namespace metrics, multi-config model loading, LoRA adapter support, and new GenAI-Perf compare subcommand. • Adds `namespace` label to metrics output when the server is launched with `--model-namespacing=true`, enabling per-namespace metric disambiguation for models sharing the same name. • Adds `--model-config-name` server launch option to select among multiple model configuration files (`configs/.pbtxt`) stored in the model repository for a given model. • Adds `INTER_OP_THREAD_COUNT` and `INTRA_OP_THREAD_COUNT` parameters to `config.pbtxt` for the PyTorch Backend to control thread counts during model execution. • GenAI-Perf gains a new `compare` subcommand for generating visual comparisons across different profile runs. • GenAI-Perf now accepts an input file containing a single prompt string to drive input generation. • Extends response caching support to top-level requests targeting ensemble models. • Triton's vLLM Backend now supports deployment of models with multiple LoRA adapters. • FIL backend is now included in Triton's ARM-SBSA container image. • Triton logging format has been updated; see the logging format extension documentation for details. Breaking changes: • Triton logging format has been modified; existing log parsers or monitoring pipelines that depend on the previous format may need to be updated.

Triton Inference Server v2.45.0

Tue, 30 Apr 2024 17:47:21 GMT

Triton v2.45.0 adds AsyncIO decoupled mode, OpenTelemetry trace retrieval, and GenAI-Perf LLM profiling with output token distribution control. • Trace settings API now returns `trace_mode` and `trace_config` fields when querying trace configuration via gRPC/HTTP endpoints. • Supports retrieving OpenTelemetry trace settings from the gRPC/HTTP endpoints. • Beta support for AsyncIO in decoupled mode in the Python backend. • GenAI-Perf gains the ability to select an output token distribution for load generation. • Model Analyzer adds support for profiling LLMs with GenAI-Perf. • GenAI-Perf adds metric visualizations. • Python backend shared memory region naming now uses UUIDs, allowing multiple Triton servers to run on the same machine without requiring different shared memory region prefixes. • Enhances server shutdown to account for both HTTP live connections and in-flight inferences. • The TensorRT-LLM container now includes the `tensorrt_llm` Python package for creating engines. • Adds an iterative scheduling tutorial demonstrating GPT2-based iterative scheduling workflows. • Adds Python Client API reference docs to the Triton documentation website. Breaking changes: • Log file and trace file locations can no longer be updated via the gRPC/HTTP endpoints. • Some GenAI-Perf CLI arguments have been renamed in this release. • Perf Analyzer no longer supports the `--trace-file` option. • There is no Windows release for 24.04; the latest Windows release remains v2.44.0.

Triton Inference Server v2.44.0

Wed, 27 Mar 2024 01:30:21 GMT

Triton v2.44.0 adds per-response metrics, request cancellation stats, OpenTelemetry Python Backend access, and a new GenAI-Perf LLM benchmarking tool. • New `GenAI-Perf` tool (alpha) for LLM benchmarking, available via the triton-inference-server/client repository. • Triton's backend API now supports collecting per-response metrics. • Triton now publishes request cancellation events in response statistics. • OpenTelemetry context for traces started on the Triton server side is now accessible from the Python Backend. • Python Backend now supports correlation strings in BLS (Business Logic Scripting) models. • HTTP header matching for the header forwarding feature is now case-insensitive.

Triton Inference Server v2.43.0

Fri, 01 Mar 2024 01:13:32 GMT

Triton v2.43.0 adds Windows Python backend support and switches OpenTelemetry tracing to Batch Span Processor with configurable batching parameters. • OpenTelemetry trace mode switches to Batch Span Processor, enabling bulk export of completed spans with size-based batching controlled by `bsp_max_export_batch_size` and `bsp_max_queue_size`, and time-based batching controlled by `bsp_schedule_delay`. • Adds base Python backend functionality for Windows, enabling Python model serving on Windows deployments. • Removes Wait/Read(avg) and `Overhead` metrics for gRPC from the Trace Summary Tool to eliminate inaccurate readings.

Triton Inference Server v2.42.0

Tue, 30 Jan 2024 01:16:56 GMT

Triton v2.42.0 adds a Python in-process API, model-load retry, OpenTelemetry context propagation, pinned-memory metrics, and experimental PyTorch 2.0 serving. • Adds a command-line option to retry loading failed models a configurable number of attempts. • Adds Triton Python API for in-process integration within a Python environment. • Adds support for OpenTelemetry context propagation in trace mode. • Adds pinned memory pool usage reporting to Triton metrics. • Adds experimental support for serving PyTorch 2.0 models via the PyTorch backend. • Improves HTTP endpoint error responses so that status codes other than 400 may be returned to align with the actual error type. • Model Analyzer now loads and optimizes ensemble models. • Model Analyzer now supports optimizing a model on a remote Triton server without requiring a local GPU. Breaking changes: • The FasterTransformer backend is deprecated as of 24.01 and is no longer supported or released with this and future versions of Triton.

Triton Inference Server v2.41.0

Wed, 20 Dec 2023 01:02:35 GMT

Triton v2.41.0 adds TRTLLM backend metrics, OpenTelemetry request ID tracing, and new Jetpack 6.0 iGPU containers. • Adds metrics support to the TensorRT-LLM backend (`tensorrtllm_backend`) when running within Triton. • Includes request ID in OpenTelemetry tracing spans for improved distributed trace correlation. • Publishes new Jetson iGPU containers on NGC with `-igpu` suffix (`XX.YY-py3-igpu` and `XX.YY-py3-sdk-igpu`) targeting Jetpack 6.0 and above on supported Jetson devices.

Triton Inference Server v2.40.0

Fri, 01 Dec 2023 23:32:35 GMT

Triton v2.40.0 adds iterative sequence scheduling, HTTP endpoint access restrictions, vLLM auto-complete, and TRT-LLM request cancellation. • Adds `backend: "vllm"` auto-complete support so Triton can automatically fill in vLLM backend model configuration without a full manual config. • Enhances the HTTP endpoint to support access restriction (beta), limiting which endpoints are reachable — see the inference protocols customization guide. • Enhances implicit state management to support growing buffers and a single shared buffer for both input and output states, configurable via `model_config.proto`. • Adds iterative scheduling support to the sequence batcher, enabling sequences that require multiple scheduling passes. • Extends the backend API to support request rescheduling, available in the Python backend and custom C++ backends. • Adds request cancellation support to the TRT-LLM backend. • Adds parameters support in BLS (Business Logic Scripting) requests from the Python backend. • Adds Kubernetes `StartupProbe` support to the on-premises Kubernetes deploy example, giving Triton pods time to finish startup before health probes run. • Improves Python backend GPU tensor performance. • Adds a Secure Deployment Guide covering how to deploy Triton in a hardened configuration. • Adds a new tutorial for deploying LLaMA2 using TRT-LLM. • Publishes Triton iGPU containers for Jetson/IGX devices, supporting TensorFlow 2.14.0, TensorRT 8.6.2.3, ONNX Runtime 1.16.3, PyTorch 2.2.0a0+6a974be, and Python 3.10. Breaking changes: • `reuse-grpc-port` and `reuse-http-port` are now properly parsed as booleans; any integer value other than 0 or 1 will throw an error (previously non-boolean integers may have been silently accepted). • The client model loading API no longer allows uploading files outside the model repository; any workflow that relied on out-of-repository file uploads will break.

Triton Inference Server v2.39.0

Fri, 27 Oct 2023 15:48:15 GMT

Triton v2.39.0 adds TensorRT-LLM and vLLM backends, a Generate REST extension for LLMs, client-side request cancellation, and Python-based custom backends. • Adds the `Generate` extension (beta) providing REST APIs optimized for Large Language Model inference. • Adds support for handling client-side request cancellation in the server and backends. • Adds the TensorRT-LLM backend (release/0.5.0), replacing the FasterTransformer backend, with a dedicated NGC container for 23.10. • Adds the vLLM backend (r23.10) for deploying supported models on the vLLM engine, with a dedicated NGC container for 23.10. • Adds support for writing custom backends in Python via the new Python-based backends framework. • Adds Scalar I/O support in the ONNXRuntime backend. Breaking changes: • The FasterTransformer backend is replaced by the TensorRT-LLM backend; FasterTransformer is only officially supported through Triton 22.12 (buildable up to 23.07).

Triton Inference Server v2.38.0

Wed, 27 Sep 2023 21:40:36 GMT

Triton v2.38.0 adds Python C API bindings, ensemble request parameter forwarding, queue-depth metrics, and TensorRT version compatibility. • Enables TensorRT version compatibility across models built with the same major TensorRT version via the `--backend-config=tensorrt,version-compatible=true` flag. • Adds pending-request queue size per model to the metrics API. • New Python bindings for the Triton C API. • Forwards request parameters to each composing model in an ensemble pipeline. • Filesystem API now supports named temporary cache directories when downloading models via the repository agent. • Backend API adds access to inference response outputs by name or by index. • Python backend models can now return structured error codes in addition to error messages. • Python backend gains experimental (Beta) support for loading PyTorch models directly.

Triton Inference Server v2.37.0

Wed, 30 Aug 2023 18:06:53 GMT

Triton v2.37.0 adds parallel model instance loading, PyTorch serving via torch.compile(), OpenTelemetry resource settings, and BLS trace capture. • Exposes `preserve_ordering` field to the oldest-strategy sequence batcher, letting operators control whether response order is preserved across independent requests belonging to different sequences. • Triton OpenTelemetry Tracing exposes resource settings to configure service name and version via OpenTelemetry trace API settings. • Supports loading model instances in parallel for backends that opt in via `TRITONBACKEND_BackendAttributeSetParallelModelInstanceLoading`; Python and ONNXRuntime backends enabled as of this release. • Python backend models can now capture traces for composing child models when executing BLS (Business Logic Scripting) requests. • Python backend supports directly loading and serving PyTorch models using torch.compile() (experimental PyTorch platform). Breaking changes: • The default value of `preserve_ordering` in the oldest-strategy sequence batcher changes from `true` to `false`, meaning response order across independent requests from different sequences is no longer preserved by default on upgrade.

Triton Inference Server v2.36.0

Fri, 28 Jul 2023 00:48:38 GMT

Triton v2.36.0 adds PyTorch implicit state, TF SavedModel direct serving, OpenTelemetry ensemble tracing, DLPack CUDA shared memory, Java bindings, and S3 repo scaling past 1000 files. • Adds model loading APIs to `python_backend` for BLS (Business Logic Scripting) usage. • Supports direct serving of TensorFlow SavedModel via `python_backend` without a dedicated TF backend. • Supports unpacked Conda execution environments in `python_backend` for custom dependency management. • Adds DLPack tensor support to Triton Python client CUDA shared memory utilities. • Adds implicit state management to `pytorch_backend`. • Extends OpenTelemetry trace mode to cover ensemble model tracing. • Adds Java binding of the Triton in-process C++ API via javacpp-presets. • Supports S3 model repositories containing more than 1000 files.

Triton Inference Server v2.35.0

Fri, 30 Jun 2023 01:39:56 GMT

Triton v2.35.0 adds Redis cache support, gRPC response flags, KIND_MODEL for PyTorch, and serial sequences in Perf Analyzer. • Adds `--serial-sequences` mode to Perf Analyzer for sequential load testing of inference requests. • Adds beta support for using Redis as a cache for inference requests via the `redis_cache` backend. • Extends `KIND_MODEL` instance group type support to the PyTorch backend. • The statistics extension now includes memory usage of loaded models (currently implemented for TensorRT and ONNXRuntime backends). • gRPC clients can now request response flags per response to programmatically determine when all responses for a decoupled model request have been received. • Adds support for batch inputs in ragged batching for the PyTorch backend.

Triton Inference Server v2.34.0

Wed, 31 May 2023 00:18:03 GMT

Triton 2.34.0 adds custom metrics in Python backend, a client plugin API, dynamic instance scaling, and a new `--metrics-address` flag. • Adds `--metrics-address=

` CLI option to bind the metrics server to a different address than the default `0.0.0.0`. • Reduces the default number of model load threads from 2*(number of CPU cores) to 4; the `--model-load-thread-count` CLI option overrides this default. • Python backend now supports Custom Metrics, letting users define and report counters and gauges via the same interface as the C API. • Python Triton Client introduces the Triton Client Plugin API (beta) for registering custom plugins that add or modify request headers. • Adds DLPack Python specification support in the Python backend via pb_utils.Tensor.from_dlpack(). • Improves model instance scaling: when only the instance group changes in a model config, Triton updates instance counts in-place without a full model reload (non-sequence models only). Breaking changes: • The default model load thread count is reduced from 2*(number of CPU cores) to 4; deployments on large-core systems that relied on the previous default parallelism will need to set `--model-load-thread-count` explicitly to restore prior behavior.

Triton Inference Server v2.33.0

Wed, 26 Apr 2023 01:02:28 GMT

Triton v2.33.0 adds concurrent model loading, OpenTelemetry tracing, HTTP/gRPC header forwarding, configurable latency quantiles, and protocol access restrictions. • Adds experimental latency metrics as configurable quantiles over a sliding time window via metrics summary support (see `metrics.md#summaries`). • Adds beta support for restricting access to specific protocols on a given Triton endpoint (see `inference_protocols.md#limit-endpoint-access-beta`). • Adds experimental support for schedule policy in the sequence batcher with direct scheduling strategy. • Adds limited support for tracing inference requests using OpenTelemetry Trace APIs. • Enables forwarding of HTTP/gRPC headers as inference request parameters to the backend. • Extends ragged batching support to the PyTorch backend. • Enables concurrent model loading to reduce server start-up times. • Python backend business logic scripting (BLS) now allows selecting a specific device to receive output tensors from a BLS call. • Model Analyzer adds support for BLS model config search.

Triton Inference Server v2.32.0

Tue, 28 Mar 2023 22:16:32 GMT

Triton v2.32.0 adds Parameters Extension, decoupled BLS, model namespacing, pluggable cache API, and request_id tracing • Adds `--model-namespacing` flag to allow the same model name to be used across different model repositories. • Adds `--cache-config` flag as the preferred method for configuring the response cache, backed by the new `TRITONCACHE` shared-library API; `--response-cache-byte-size` continues to work. • Introduces the Parameters Extension, enabling inference requests to supply custom parameters that cannot be passed as inputs, accessible in the Python backend via inference request parameters. • Adds support for models using the decoupled API for Business Scripting Logic (BLS) in the Python backend. • Extends the trace tool to support tracing by `request_id`. • Converts the Response Cache to a pluggable shared-library architecture (`TRITONCACHE` APIs), with `local_cache` as the default implementation.

Triton Inference Server v2.31.0

Wed, 01 Mar 2023 05:21:01 GMT

Triton v2.31.0 adds ensemble model support in Model Analyzer and GRPC Standard Health Check Protocol. • Adds support for the GRPC Standard Health Check Protocol on the inference server endpoint. • Adds ensemble model support in Model Analyzer, enabling config search across ensemble pipelines.

Triton Inference Server v2.30.0

Wed, 01 Feb 2023 04:47:24 GMT

Triton v2.30.0 adds user-defined custom batching strategies to the dynamic batcher. • Enables user-defined batching constraints in the dynamic batcher via custom batching strategies, letting operators control how requests are grouped into batches. • Relaxes the Python client gRPC version requirement, broadening compatible client environments.

Triton Inference Server v2.29.0

Tue, 20 Dec 2022 19:59:11 GMT

Triton Inference Server v2.29.0 adds improved Windows builds and a beta Windows package with HTTP/REST, GRPC, ONNX, and OpenVINO support. • Releases a beta Windows package (`tritonserver2.29.0-win.zip`) supporting HTTP/REST and GRPC endpoints, ONNX models via ONNX Runtime 1.13.1 (CPU, CUDA, and TensorRT execution providers), and OpenVINO models via OpenVINO 2021.4. • Bundles CUDA 11.8.0, cuDNN 8.7.0.84, and TensorRT 8.5.1.7 dependencies for Windows deployments. • Improves container and non-container builds on Windows. • Includes Ubuntu 20.04 client libraries and examples in `v2.29.0_ubuntu2004.clients.tar.gz` and Windows SDK in `tritonserver2.29.0-sdk-win.zip`.

Triton Inference Server v2.28.0

Tue, 22 Nov 2022 21:26:00 GMT

Triton v2.28.0 adds TensorRT 8.5 UINT8 I/O, data-dependent dynamic shapes, and external execution environment paths. • Adds `EXECUTION_ENV_PATH` parameter in `config.pbtxt` to specify execution environment paths outside the model directory for the Python backend. • Supports TensorRT 8.5 UINT8 input/output tensors. • Supports TensorRT 8.5 'data dependent dynamic shapes' operators, including ONNX NMS and NonZero operations. • Windows beta release adds ONNX Runtime 1.13.1 support with CPU, CUDA, and TensorRT execution providers. • Windows beta release adds OpenVINO 2021.4 model support. • Windows beta release supports HTTP/REST and GRPC endpoints.

Triton Inference Server v2.27.0

Wed, 02 Nov 2022 22:20:48 GMT

Triton 2.27.0 adds JAX model support, enhanced Server Wrapper API with decoupled model and tracing, and multi-model concurrent profiling in Model Analyzer. • Adds multi-model concurrent profiling to Model Analyzer via Multi-Model Search Mode, enabling simultaneous profiling and analysis of multiple models. • Enhances the Server Wrapper API to support decoupled model workflows and tracing, covering previously missing features. • Adds an example demonstrating JAX model usage inside Python backend models.

Triton Inference Server v2.26.0

Tue, 04 Oct 2022 00:55:25 GMT

Triton v2.26.0 adds CPU metrics, dynamic logging config, GPU metrics in Perf Analyzer, and a quick-search mode for Model Analyzer. • Adds CPU metrics reporting to Triton's Prometheus metrics endpoint. • Adds a logging protocol extension enabling users to change logging configuration dynamically at runtime. • Adds `--disable-auto-complete-config` launch flag awareness note: users can specify custom TensorRT plugins via command-line option in addition to `LD_PRELOAD`. • Adds GPU metrics gathering to Perf Analyzer, improving accuracy of metrics also consumed by Model Analyzer. • Adds a 'quick search' algorithm to Model Analyzer to drastically reduce model configuration search time. • Adds a developer tools GitHub repository providing a simplified interface to the Triton Core shared library (beta). • Enables auto-completion of model configuration for the OpenVINO backend. • Enables the Python backend to emit log messages through Triton's native logger.

Triton Inference Server v2.25.0

Fri, 26 Aug 2022 17:32:34 GMT

Triton v2.25.0 adds multi-cloud credential support, GPU memory load limits, and configless custom backend loading • Adds `--model-load-gpu-limit` server option and `TRITONSERVER_ServerOptionsSetModelLoadDeviceLimit` C API function to cap GPU memory usage when loading models. • Adds beta support for multiple cloud storage credentials via a credential file, enabling per-repository cloud auth configuration. • Enables loading custom backend models without an explicit `config.pbtxt` when the model is named in the form `.` and the backend implements auto-complete configuration. • Defaults TensorFlow backend to version 2.X; TensorFlow 1.X can still be selected via `--backend-config=tensorflow,version=`. • Model Analyzer's `profile` subcommand now automatically runs analysis after profiling completes, removing the need to invoke the separate `analyze` subcommand. • PyTorch backend now uses a separate CUDA stream per GPU model instance, improving GPU inference throughput. • Adds new 'Performance Tuning' user guide with a step-by-step production optimization walkthrough. Breaking changes: • TensorFlow backend now defaults to version 2.X; workloads relying on the previous default of TensorFlow 1.X must explicitly set `--backend-config=tensorflow,version=1`. • Model Analyzer's `analyze` subcommand is deprecated; analysis is now integrated into `profile` and the standalone subcommand may be removed in a future release.

Triton Inference Server v2.24.0

Sat, 30 Jul 2022 00:26:23 GMT

Triton v2.24.0 enables auto-complete by default, adds asyncio Python APIs, a C++ BLS example, and a concurrent model-load thread pool. • Replaces the soft-deprecated `--strict-model-config` option with the new `--disable-auto-complete-config` CLI option, and enables Auto-Complete model configuration by default. • Adds `--model-load-thread-count` tritonserver CLI option to configure the size of a new thread pool for concurrently loading models, reducing service downtime. • Adds a new asyncio-compatible API (beta) to the Python gRPC/HTTP client libraries. • Supports `init_ops` parameters for TensorFlow TF1.x GraphDef models, configurable via JSON file. • New example backend demonstrating Business Logic Scripting (BLS) in C++. • Model Analyzer no longer requires a `config.pbtxt` file for models that support Triton auto-completion. Breaking changes: • `--strict-model-config` is soft-deprecated; replace it with `--disable-auto-complete-config`. Auto-Complete is now enabled by default, which may increase server start time for existing deployments that relied on full model configurations.

Triton Inference Server v2.23.0

Thu, 30 Jun 2022 01:24:36 GMT

Triton v2.23.0 enables dynamic batching by default, brings Python backend auto-config and Decoupled API to GA. • Auto-generated model configuration now enables dynamic batching in supported models by default. • Python backend models now support auto-generated model configuration via `auto_complete_config`. • Decoupled API support in the Python backend is promoted out of beta to general availability. • Updated I/O tensor naming convention for serving TorchScript models via the PyTorch backend.

Triton Inference Server v2.22.0

Thu, 26 May 2022 19:35:23 GMT

Triton v2.22.0 adds Java in-process API, BF16 support, decoupled Python backend, and dynamic model repository registration. • Adds Java bindings for the Triton In-Process API, enabling the full in-process server to be embedded in Java applications. • Supports loading models with file content supplied inline at API invocation time via the model-repository load extension. • Adds `BF16` (bfloat16) as a supported data type in model configuration. • Adds support for customized gRPC channel settings in the Python gRPC client library (`tritonclient.grpc`). • Python backend adds decoupled API support (BETA), enabling models to send multiple, asynchronous responses per request. • Explicit model control mode now supports loading all models at startup. • PyTorch backend adds support for 1-dimensional String tensor I/O. • ONNX Runtime backend updated to version 1.11.1 on both Ubuntu and Windows.

Triton Inference Server v2.21.0

Fri, 29 Apr 2022 00:31:14 GMT

Triton v2.21.0 adds raw binary requests, custom metrics via C API, multi-cloud model repos, and ensemble optional inputs. • Adds `--tmp-dir` argument to `build.py` to specify a custom temporary directory during container builds. • Supports raw binary HTTP requests to the inference endpoint, eliminating the need to specify an inference header. • Adds custom metrics support via the Triton C API, allowing custom backends and applications to publish metrics to the existing Triton metrics endpoint. • Supports loading models from multiple cloud storage repositories simultaneously — including multiple S3 buckets, GCS buckets, and Azure Storage containers — in a single Triton instance. • Ensembles now recognize optional inputs, enabling more flexible ensemble pipeline configurations. • ONNX Runtime backend now uses available execution providers (e.g. GPU/TensorRT) when autocomplete is enabled, rather than always defaulting to the CPU execution provider. • Adds PyTorch and TensorFlow 1 backend support to `build.py` and `compose.py` for CPU-only builds.

Triton Inference Server v2.20.0

Tue, 29 Mar 2022 21:27:30 GMT

Triton v2.20.0 adds ragged batching for ONNX/TensorRT/TF, cache-miss Prometheus metrics, and trace config via API. • Adds cache miss statistics to the Prometheus metrics endpoint, as documented in the metrics reference. • Enables trace settings to be configured at runtime via the Triton Server Trace Protocol extension. • Supports loading models from a serialized `model_config` message via the Triton Server model-repository API. • ONNX Runtime, TensorRT, and TensorFlow backends now support server-side, multi-dimensional ragged batching.

Triton Inference Server v2.19.0

Fri, 25 Feb 2022 17:19:03 GMT

Triton v2.19.0 adds SSL in the HTTP C++ client, new Prometheus cache metrics, and PyTorch dictionary-tensor inputs. • Adds full SSL/TLS support to the HTTP C++ client library. • PyTorch Backend now supports passing model inputs as a dictionary of tensors.

Triton Inference Server v2.18.0

Mon, 31 Jan 2022 21:03:52 GMT

Triton 2.18.0 adds implicit state management for ONNX/TensorRT, Inferentia batching, Jetson backend support, and ARM analyzer support. • Implicit state management is now supported for the ONNX Runtime and TensorRT backends. • State initialization from a constant is now supported within implicit state management. • PyTorch and TensorFlow models now support batching on Inferentia. • PyTorch and Python backends are now supported on Jetson. • ARM support added for the Performance Analyzer and Model Analyzer. • CPU-only build now supports the TensorFlow2 backend for Linux x86.

Triton Inference Server v2.17.0

Tue, 21 Dec 2021 06:34:47 GMT

Triton v2.17.0 adds MLflow deployment, TorchTRT model support, Neuron Runtime 2.x for Inferentia, and ONNX Runtime 1.10.0. • Adds MLflow plugin at `deploy/mlflow-triton-plugin` to deploy MLflow models directly to Triton. • Supports TorchTRT (preview): PyTorch models optimized with TensorRT can now be loaded the same way as regular TorchScript models. • Upgrades ONNX Runtime backend to version 1.10.0 on both Ubuntu and Windows builds of Triton. • Improves Inferentia support to use Neuron Runtime 2.x and enables multiple model instances. • Adds end-of-phase example command-line output in Model Analyzer to guide users through the next analysis phase. • Expands Windows alpha support: HTTP/REST and GRPC endpoints are now available alongside ONNX Runtime 1.10.0 (CPU, CUDA, TensorRT execution providers) and OpenVINO 2021.2. • Jetson JetPack 4.6 release adds support for TensorFlow 2.6.0, TensorFlow 1.15.5, TensorRT 8.0.1.6, and ONNX Runtime 1.10.0, including ensemble models. Breaking changes: • The `byte_contents` field in the GRPC protobuf implementation is renamed to `bytes_contents`; client code using `byte_contents` must be updated.

Triton Inference Server v2.16.0

Fri, 19 Nov 2021 23:09:24 GMT

Triton v2.16.0 adds LightGBM categorical feature support, Inferentia PoC, ARM Model Analyzer, and Jetson examples. • Adds support for LightGBM models with categorical features in the FIL backend. • Adds ARM support for Model Analyzer. • Completes proof-of-concept Inferentia support via the Python backend. • Adds Jetson examples under `docs/examples/jetson` in documentation.

Triton Inference Server v2.15.0

Thu, 28 Oct 2021 00:17:52 GMT

Triton v2.15.0 adds rate limiting, response caching, string sequence IDs, ARM SBSA beta, and Windows HTTP support. • Adds rate limiter that manages the rate at which requests are scheduled on model instances, configurable via `docs/rate_limiter.md`. • Adds support for caching responses to inference requests. • Sequence IDs (via the sequence extension protocol) can now accept string values in addition to integers. • Container composer tool gains the ability to generate CPU-only Triton containers (see `docs/compose.md#cpu-only-container-composition`). • Windows Triton build now supports the HTTP/REST protocol (previously GRPC only). • Beta release of Triton is available for ARM SBSA platforms. • Windows alpha release now supports OpenVINO models at version 2021.2 and ONNX models via ONNX Runtime 1.9 with CPU, CUDA, and TensorRT execution providers. • Jetson JetPack 4.6 release adds support for TensorFlow 2.6.0, TensorFlow 1.15.5, TensorRT 8.0.1.6, OnnxRuntime 1.8.1, and ensembles.

Triton Inference Server v2.14.0

Wed, 29 Sep 2021 21:27:12 GMT

Triton v2.14.0 ships BLS beta, Java client beta, PyPI SDK wheel, and TensorRT as an optional backend. • Adds `--gpus` flag support for CUDA Device Index in addition to GPU UUID in Model Analyzer, giving teams more flexibility when targeting specific devices. • Full-featured beta of Business Logic Scripting (BLS) released for the Python backend, enabling custom inference logic within the server. • Beta Java client released with initial support for Triton's inference API. • Triton Client SDK wheel now available directly from PyPI for both Ubuntu and Windows, installable via pip. • TensorRT backend is now optional; the `compose` utility can build a Triton container without it, just like all other backends. • Model Analyzer can now profile using perf_analyzer's C-API.

Triton Inference Server v2.13.0

Sat, 28 Aug 2021 03:42:11 GMT

Triton v2.13.0 adds Business Logic Scripting beta, a Container Composition Utility, and Model Analyzer support for custom ops. • Adds `--backend-directory` flag and `--backend-config=tensorflow,version=2` argument to `tritonserver` for explicit backend path and TensorFlow version selection on Jetson. • Initial beta release of Business Logic Scripting — a new set of utility functions allowing a Python model to execute inference requests on other models being served by Triton. • New Container Composition Utility (`docs/compose.md`) for building custom Triton containers with specific backends and repository agents. • Publishes two new focused NGC containers: `nvcr.io/nvidia/tritonserver:21.08-tf-python-py3` (TensorFlow 2.x + Python backends) and `nvcr.io/nvidia/tritonserver:21.08-pyt-python-py3` (PyTorch + Python backends). • Adds Model Analyzer support for models with custom operations. Breaking changes: • In the Python client shared-memory support library, utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return `np.object_` numpy arrays instead of `np.bytes_` arrays; code depending on `np.bytes_` must be updated.

Triton Inference Server v2.12.0

Tue, 27 Jul 2021 19:00:34 GMT

Triton v2.12.0 adds CPU support for RAPIDS FIL, multi-copy C API inputs, TensorRT precision/workspace config for ONNX, and Model Analyzer offline mode. • The `tensorrt_accelerator` option in the ONNX model configuration can now specify `precision` and `workspace size` for TensorRT-accelerated ONNX models. • Adds CPU support to the RAPIDS FIL Backend, extending inference beyond GPU-only deployments. • C API inference requests can now supply multiple copies of an input tensor in different memory locations; Triton selects the most performant copy based on execution context. • Model Analyzer gains an offline mode that prioritizes throughput over latency, with a new set of reports and graphs tailored to offline inferencing use cases.

Triton Inference Server v2.11.0

Fri, 25 Jun 2021 20:38:30 GMT

Triton v2.11.0 adds FIL backend for forest models, OpenVINO on Windows, C API benchmarking, conda Python environments, and MIG GPU metrics. • Adds the Forest Inference Library (FIL) backend, enabling deployment of forest models trained with XGBoost, LightGBM, Scikit-Learn, and cuML. • Adds `perf_analyzer` support for benchmarking Triton directly via the C API, in addition to HTTP/gRPC endpoints. • Python backend now supports conda to create isolated execution environments per Python model. • Adds passive model instance support in model configuration (via `ModelInstanceGroup` in `model_config.proto`), allowing instances to be loaded and initialized without receiving inference requests. • Model repositories hosted in S3 storage can now be accessed using the HTTPS protocol. • Triton now collects GPU metrics for MIG (Multi-Instance GPU) partitions. • NVDLA support added to the TensorRT backend. • Windows release of Triton now includes the OpenVINO backend (OpenVINO version 2021.2). • DALI backend now accepts GPU inputs and adds support for dynamic batching and ragged inputs. • Model Analyzer gains an offline mode and improved detailed and summary reports. • ONNX Runtime updated to version 1.8.0. Breaking changes: • In the Python client shared-memory support library, utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return `np.object_` numpy arrays instead of `np.bytes_` arrays; code depending on `np.bytes_` must be updated.

Triton Inference Server v2.10.0

Fri, 21 May 2021 14:38:44 GMT

Triton v2.10.0 adds ONNX on Jetson, HTTP compression, ragged batching, and Model Analyzer CLI subcommands. • Adds `profile`, `analyze`, and `report` subcommands to the Model Analyzer CLI for structured profiling and reporting workflows. • Adds `--backend-directory` flag to `tritonserver` to explicitly set the backend directory, required on Jetson deployments. • Adds `--backend-config=tensorflow,version=2` flag to select TensorFlow 2.x on Jetson (where TF 1.x is the default). • Triton server and HTTP clients (Python and C++) now support request/response compression. • Ragged batching is now supported for ONNX models. • Triton on Jetson (JetPack 4.5) now supports ONNX via the ONNX Runtime backend (version 1.7.1), including the TensorRT execution provider. • Model Analyzer gains a `report` subcommand to generate a detailed report for any specific model configuration. • Model Analyzer now supports CPU-only mode. • Windows alpha release adds GRPC endpoint support for TensorRT and ONNX Runtime (CPU, CUDA, TensorRT execution providers) models. Breaking changes: • Legacy custom backend support is removed; all custom backends must be reimplemented using the TRITONBACKEND API. • Model Analyzer CLI subcommands (`profile`, `analyze`, `report`) require updating existing Model Analyzer config files and CLI flags. • The Triton client libraries have moved to a separate repository at https://github.com/triton-inference-server/client. • utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() in the Python client shared-memory support library now return `np.object_` arrays instead of `np.bytes_` arrays; code depending on `np.bytes_` must be updated.

Triton Inference Server v2.9.0

Tue, 27 Apr 2021 17:19:30 GMT

Triton v2.9.0 adds GKE Marketplace deployment, GRPC compression, ragged batching for TensorFlow, and ensemble unload control. • The Model Control API gains an option to unload all models contained within an ensemble when unloading the ensemble itself — see the model-repository extension protocol docs. • GRPC client libraries now support enabling compression. • Ragged batching is now supported for TensorFlow models. • For TensorFlow SavedModel-format models, adds the ability to select which graph and `signature_def` to load via backend parameters. • Triton Server is now available as a GKE Marketplace Application. • Adds a Helm Chart example for AWS deployments. • Python backend performance is significantly increased. • ONNX Runtime updated to version 1.7.1. • Model Analyzer summary reports now include GPU power usage. • Model Analyzer can now find the Top N model configurations across multiple models. Breaking changes: • In the Python client shared-memory support library, utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return `np.object_` numpy arrays instead of `np.bytes_` numpy arrays for tensors of type BYTES — code depending on `np.bytes_` must be updated.

Triton Inference Server v2.8.0

Sat, 27 Mar 2021 00:16:37 GMT

Triton 2.8.0 adds Repository Agent C API, OpenVINO backend, SSL/TLS mutual auth for gRPC, and ONNX support on Windows. • Adds `gather_kernel_buffer_threshold` Model Configuration option to instruct Triton to use a CUDA kernel to gather input buffers onto the GPU, improving inference performance for some models. • Introduces Repository Agent, a new extensibility C API for implementing custom authentication, decryption, conversion, or similar operations at model-load time. • Adds an OpenVINO backend enabling execution of OpenVINO models on CPUs. • Adds SSL/TLS Mutual Authentication support to the GRPC client library. • Adds system shared-memory protocol support between clients and the Triton server on Jetson deployments. • Windows alpha release now supports ONNX models via the ONNX Runtime 1.6.0 backend, with CPU, CUDA, and TensorRT execution providers (OpenVINO execution provider not supported). Breaking changes: • The Python client shared-memory support library functions utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return `np.object_` numpy arrays instead of `np.bytes_` arrays; code depending on `np.bytes_` must be updated.

Triton Inference Server v2.7.0

Sat, 27 Feb 2021 01:20:07 GMT

Triton v2.7.0 adds Java/Scala gRPC clients, TFServing/TorchServe perf_analyzer support, and YAML-based Model Analyzer configuration. • Extends `perf_analyzer` to work with TFServing and TorchServe, enabling benchmarking across non-Triton serving backends. • Model Analyzer parameters and test model configurations can now be specified via a YAML configuration file. • Model Analyzer now reports performance metrics for end-to-end latency and CPU memory usage. • Adds example Java and Scala clients based on the gRPC-generated API. • Alpha Windows release (`tritonserver2.7.0-win.zip`) ships with TensorRT 7.2.2 support and a GRPC endpoint. • JetPack 4.5 release (`tritonserver2.7.0-jetpack4.5.tgz`) adds support for TensorFlow 2.4.0, TensorFlow 1.15.5, TensorRT 7.1, Custom backends, ensembles, and system shared memory on Jetson.

Triton Inference Server v2.6.0

Fri, 18 Dec 2020 23:54:58 GMT

Triton 2.6.0 adds alpha Windows support, Model Analyzer, and a reorganized SDK container with renamed image tags. • Renames the `*-py3-clientsdk` container to `*-py3-sdk`; the renamed image now bundles the Model Analyzer alongside client libraries and examples. • Initial release of the Model Analyzer tool, available in the Triton SDK container and as the `nvidia-triton-model-analyzer` PIP package on the NVIDIA Py Index. • Moves the PyTorch backend to a separate repository (https://github.com/triton-inference-server/pytorch_backend), enabling it to be added or removed without rebuilding Triton via the compose workflow. • Alpha release of Triton for Windows (`tritonserver2.6.0-win.zip`), supporting TensorRT 7.2.2 models over the GRPC endpoint (HTTP/REST, Prometheus metrics, and shared memory are not yet supported). • Adds `--backend-directory` flag to explicitly set the backend path, enabling Jetson/JetPack deployments where the default path is not used. • Adds `--backend-config=tensorflow,version=2` flag to select TensorFlow 2.x on Jetson, where TensorFlow 1.x is the default. • Releases Triton for JetPack 4.4 (`tritonserver2.6.0-jetpack4.4.tgz`) with support for TensorFlow 2.3.1, TensorFlow 1.15.4, TensorRT 7.1, custom backends, and ensembles. Breaking changes: • The `*-py3-clientsdk` container image is renamed to `*-py3-sdk`; any pipelines or scripts referencing the old image name will break. • The ONNX Runtime OpenVINO execution provider is disabled in this release due to Ubuntu 20.04 interactions; workloads relying on it will not function until it is re-enabled in a future release.

Triton Inference Server v2.5.0

Fri, 20 Nov 2020 23:24:04 GMT

Triton v2.5.0 releases ModelAnalyzer, drops Caffe2, and moves PyTorch to a dedicated backend repo. • Adds `--backend-directory` flag to explicitly set the backend directory path, required on Jetson deployments. • Adds `--backend-config=tensorflow,version=2` to select TensorFlow 2.x when Triton defaults to TensorFlow 1.x. • Releases Triton ModelAnalyzer in the Triton SDK container and as a PIP package available in NVIDIA PyIndex. • Moves the PyTorch backend to a dedicated repository at `triton-inference-server/pytorch_backend`. • Updates ONNX Runtime backend to use ONNX Runtime 1.5.3. • Adds Jetson JetPack 4.4 support with TensorFlow 2.3.1, TensorFlow 1.15.4, TensorRT 7.1, custom backends, and ensembles (GPU metrics, GCS, and S3 storage not supported on Jetson). Breaking changes: • The Caffe2 backend is removed; Caffe2 models are no longer supported.

Triton Inference Server v2.4.0

Tue, 27 Oct 2020 01:21:44 GMT

Triton 2.4.0 adds Python and DALI backends, a new Model Analyzer tool, and moves backends to separate repositories. • New `--backend-directory` flag lets you explicitly set the backend directory path when running Triton, required on Jetson deployments. • New `--backend-config=tensorflow,version=2` argument allows selecting TensorFlow 2.x when Triton defaults to TensorFlow 1.x. • New Python backend enables running arbitrary Python code as a model within Triton (see github.com/triton-inference-server/python_backend). • New DALI backend enables running pre-processing and data augmentation pipelines directly within Triton (see github.com/triton-inference-server/dali_backend). • New Model Analyzer project provides analysis and guidance for optimizing single or multiple models within Triton; initial release analyzes GPU memory usage (see github.com/triton-inference-server/model_analyzer). • TensorFlow, ONNX Runtime, Python, and DALI backends are now hosted in separate repositories, enabling independent versioning and distribution. Breaking changes: • The `perf_client` application is renamed to `perf_analyzer`; any scripts or automation invoking `perf_client` will break.

Triton Inference Server v2.3.0

Fri, 02 Oct 2020 02:10:39 GMT

Triton v2.3.0 ships the Python client as a pip package and drops the legacy custom backend SDK in favor of the new Triton backend API. • Python client library is now installable as a pip package from the NVIDIA PyPI index, replacing the previous tarball-only distribution. • The legacy custom backend API (`custom.h`) and its SDK are no longer shipped with Triton releases; users are directed to migrate to the new Triton backend API. Breaking changes: • The custom backend SDK (`custom.h` and associated tooling) is no longer included in the Triton release package; new custom backends must use the new Triton backend API, though existing compiled legacy backends continue to work.

Triton Inference Server v2.2.0

Thu, 27 Aug 2020 19:57:28 GMT

Triton v2.2.0 adds TensorFlow 2.x support, per-request inference timeouts, and API versioning • Adds TensorFlow 2.x support alongside TensorFlow 1.x, selectable at launch via `--backend-config=tensorflow,version=` (set `` to `1` or `2`; default is `1`). • Adds `TRITON_CLIENT_SKIP_EXAMPLES` CMake option to disable building client examples. • Adds inference request timeout option to Python and C++ client libraries. • Adds explicit major/minor versioning to `TRITONSERVER` and `TRITONBACKEND` APIs.

Triton Inference Server v2.1.0

Thu, 30 Jul 2020 22:28:11 GMT

Triton v2.1.0 adds automatic FP16 TensorFlow optimization and TorchVision support in the PyTorch backend. • Adds a TensorFlow optimization option that enables automatic FP16 optimization of models served by the server. • Adds TorchVision operations support to the PyTorch backend.

Triton Inference Server v1.15.0

Thu, 30 Jul 2020 22:27:14 GMT

Triton v1.15.0 publishes legacy V1 HTTP/REST and GRPC support on a dedicated branch and NGC container. • Releases legacy V1 HTTP/REST and GRPC endpoints, plus corresponding client libraries, on GitHub branch `r20.07-v1` and NGC container `20.07-v1-py3` for teams that cannot yet migrate to the V2 API.

Triton Inference Server v2.0.0

Tue, 30 Jun 2020 01:06:34 GMT

Triton Inference Server v2.0.0 brings updated KFServing HTTP/REST and gRPC protocols with new Python and C++ client libraries. • Updates KFServing HTTP/REST and gRPC protocols with corresponding V2 Python and C++ client libraries. • Adds Jetson JetPack 4.4 support via `v2.0.0-jetpack4.4ga.tgz`, with TensorFlow 1.15.2, TensorRT 7.1, Custom backends, and ensemble support on edge devices. • Client libraries and examples for Ubuntu 18.04 are packaged in `v2.0.0_ubuntu1804.clients.tar.gz` and installable via the `triton*.whl` Python wheel. • Custom backend SDK for Ubuntu 18.04 is packaged in `v2.0.0_ubuntu1804.custombackend.tar.gz`. • Client SDK is also available as an NGC container image. Breaking changes: • Migration from Triton V1 to Triton V2 requires significant changes — see the 'Backwards Compatibility' and 'Roadmap' sections of the GitHub README.

Triton Inference Server v1.14.0

Tue, 30 Jun 2020 01:06:25 GMT

Triton v1.14.0 publishes legacy V1 HTTP/REST and GRPC support on a dedicated branch and NGC container. • Releases legacy V1 HTTP/REST and GRPC endpoints, along with corresponding client libraries, on GitHub branch `r20.06-v1` and as NGC container `20.06-v1-py3`.

Triton Inference Server v1.13.0

Mon, 01 Jun 2020 21:39:59 GMT

Triton v1.13.0 updates KFServing HTTP/REST and gRPC protocols and bumps gRPC to 1.24.0 • Updates KFServing HTTP/REST and gRPC protocols with corresponding Python and C++ client libraries. • Upgrades gRPC dependency to version 1.24.0. Breaking changes: • The Caffe2 backend is deprecated; support for Caffe2 models will be removed in a future release.

Triton Inference Server v1.12.0

Thu, 26 Mar 2020 21:17:23 GMT

Triton v1.12.0 adds dynamic batching queue policies, ONNX large-model support, and an experimental GRPC inferencing API. • Adds `--min-supported-compute-capability` flag to allow Triton Server to run on older, otherwise unsupported GPUs. • Adds queuing policies for the dynamic batching scheduler, configurable per model via the model configuration: maximum queue size, timeouts, and priority levels for inference requests. • Supports configuring ONNX Runtime optimization level via the model configuration `optimization` setting. • Supports large ONNX models where weights are stored in separate files. • Experimental Python client and server support for the community standard GRPC inferencing API. • Experimental Triton release for JetPack 4.4 (Developer Preview), supporting TensorFlow 1.15.2, TensorRT 7.1, Custom backends, and ensembles on Jetson hardware.

Triton Inference Server v1.11.0

Wed, 26 Feb 2020 21:47:42 GMT

Triton v1.11.0 adds ragged batching for custom backends, local S3 endpoint support, and TensorRT boolean/shape tensor types. • Adds support for local S3 storage endpoints in model repositories, specified as `s3://host:port/path/to/repository`. • New model configuration option enables the dynamic batcher to create 'ragged' batches for custom backend models, where input/output tensors may have different shapes across batch entries. • Removes `LD_LIBRARY_PATH` from the inference server container in favor of `RUNPATH` for shared library resolution. • Adds support for boolean data-type and shape tensors in TensorRT models. • Significantly improves TensorRT backend performance by reducing thread contention, using pinned memory for faster CPU<->GPU transfers, and increasing compute/memory copy overlap. • Reduces memory usage of TensorRT models by sharing weights across multiple model instances. • Updates the Helm chart example Kubernetes deployment to include Prometheus and Grafana support for metrics collection and visualization. Breaking changes: • Python 2 support has been fully removed; only Python 3 is supported.

Triton Inference Server v1.10.0

Tue, 28 Jan 2020 17:56:22 GMT

Triton v1.10.0 adds JSON-format server status via REST and ordered batching across multiple model instances. • Adds JSON-format server status via the HTTP/REST endpoint `GET /api/status?format=json`. • Adds a preserve-ordering option to the dynamic batcher so batched requests maintain sequence across multiple model instances (configured via `model_config.proto`).

Triton Inference Server v1.9.0

Sat, 21 Dec 2019 01:24:49 GMT

Triton v1.9.0 adds model warmup, a new repository index API, Oldest sequence-batcher strategy, and enhanced tracing for ensembles. • Adds a `model warmup` option to model configuration, allowing models to be pre-tuned with synthetic or realistic workloads before serving inference requests — particularly useful for frameworks like TensorFlow that optimize on initial inference. • New repository index API (available via shared library API, HTTP, and GRPC) returns an index of all models visible in the model repositories, enabling discovery of models available for loading. • Server status API response now includes the timestamp of the last inference request received for each model. • Adds an `Oldest` scheduling strategy to the sequence batcher, integrating with the dynamic batcher to improve throughput for models that do not require all requests in a sequence to share the same batch slot. • Adds a `perf_client` option to generate requests using a Poisson distribution or a user-provided distribution for more realistic load testing. • Extends tracing support to cover ensembles and their contained models. • Adds a community-contributed Dockerfile for building inference server clients on CentOS. Breaking changes: • The beta custom backend API v2 `CustomGetNextInputV2Fn_t` function signature adds the `memory_type_id` argument — existing custom backends using this function will break. • The beta custom backend API v2 `CustomGetOutputV2Fn_t` function signature adds the `memory_type_id` argument — existing custom backends using this function will break. • The beta library API `TRTSERVER_ResponseAllocatorAllocFn_t` function signature and behavior has changed — see `src/core/trtserver.h` for the new behavior; existing callers will break. • The beta library API `TRTSERVER_InferenceRequestProviderSetInputData` function signature adds the `memory_type_id` argument — existing callers will break. • The beta library API `TRTSERVER_InferenceResponseOutputData` function signature adds the `memory_type_id` argument — existing callers will break.

Triton Inference Server v1.8.0

Wed, 27 Nov 2019 18:13:39 GMT

Triton v1.8.0 adds CUDA shared memory support and string tensor handling in perf_client • Adds CUDA shared memory support, expanding shared-memory capabilities beyond CPU/pinned memory. • Adds string tensor support to `perf_client`, enabling performance benchmarking of string-typed inputs and outputs. • Improves `perf_client` command-line arguments for easier use while maintaining compatibility with existing arguments. • Improves pinned-memory efficiency for ensemble models. Breaking changes: • In the C++ client library, the non-callback version of `AsyncRun` is removed. • In the C++ client library, `GetReadyAsyncRequest` is removed. • In the C++ client library, the signature of `GetAsyncRunResults` is changed — the `is_ready` and `wait` arguments are removed. • In the Python client library, the non-callback version of `async_run` is removed. • In the Python client library, `get_ready_async_request` is removed. • In the Python client library, the signature of `get_async_run_results` is changed — the `wait` argument is removed.

Triton Inference Server v1.7.0

Wed, 30 Oct 2019 00:03:52 GMT

Triton v1.7.0 adds TensorRT optimization for TensorFlow, ONNXRuntime TensorRT/OpenVINO providers, multi-repo support, and a Go GRPC client example. • Adds `--strict-model-config=false` support for TensorRT models with variable-sized inputs and/or outputs, enabling automatic configuration generation for those models. • Multiple model repositories can now be specified on the command line, with optional flags to explicitly load specific models from each repository. • TensorRT optimization can now be enabled for any TensorFlow model via the `optimization` section of the model configuration. • The ONNXRuntime backend now includes TensorRT and OpenVINO execution providers, configurable via the `optimization` section of the model configuration. • Ensemble models are now pruned dynamically so only models required to compute the requested outputs are executed. • A Client SDK container is now available on NGC, packaging client libraries and examples separately from the inference server container. • Example clients now include a Go example using the GRPC API.

Triton Inference Server v1.6.0

Fri, 27 Sep 2019 21:50:58 GMT

Triton v1.6.0 adds TensorRT 6 dynamic shapes, shared memory tensors, S3 model repos, and a new library API. • Supports Amazon S3 as a remote model repository via the `s3://` prefix on model repository paths. • Adds shared memory support (alpha) so input and output tensors can be communicated via system (CPU) shared memory instead of over the network, reducing data-copy overhead. • Introduces a `libtrtserver.so` library API (beta) that lets applications link the full inference server functionality directly instead of communicating over HTTP/gRPC. • Adds TensorRT 6 support, including dynamic shapes. • Extends `perf_client` to support models with variable-sized input tensors. • The gRPC endpoint now uses significantly less memory while delivering higher throughput. • The ensemble scheduler now allows batching and non-batching models to be composed together. • The ensemble scheduler retains tensors in GPU memory between models when possible, avoiding round-trips through system memory.

Triton Inference Server v1.5.0

Tue, 03 Sep 2019 23:53:58 GMT

Triton v1.5.0 adds a GRPC/HTTP model control API, multi-GPU TF distribution, and a C++ custom backend SDK wrapper. • Adds a new GRPC/HTTP model control API for dynamically loading and unloading models at runtime, enabling an execution mode where the server starts with no models loaded from the model repository. • Adds a new instance-group mode for TensorFlow models to explicitly distribute inferencing across multiple GPUs within the inference server. • Extends `perf_client` to report per-model statistics and aggregate ensemble statistics for ensemble models. • Adds a C++ wrapper around the custom backend C API in the custom backend SDK to simplify authoring custom backends. • Improves input/output tensor reshape to support variable-sized dimensions.

Triton Inference Server v1.4.0

Tue, 30 Jul 2019 23:06:40 GMT

Triton v1.4.0 adds PyTorch/TorchScript backend, Custom Backend SDK, and richer perf_client latency reporting. • Adds `--strict-model-config=false` improvements to automatic model configuration creation, reducing manual config overhead. • New `libtorch` backend enables running PyTorch models decorated or traced to TorchScript directly in the inference server. • Custom Backend SDK now available with improved documentation to simplify building custom backends, distributed as `v1.4.0_ubuntu1604.custombackend.tar.gz` and `v1.4.0_ubuntu1804.custombackend.tar.gz`. • `perf_client` now accepts user-supplied input data instead of being limited to random or zero values. • `perf_client` now reports latency at multiple confidence percentiles (p50, p90, p95, p99) plus a user-supplied percentile that also drives latency stabilization. • C++ and Python client libraries now support specifying additional HTTP headers when using the HTTP protocol. • Improved `AsyncRun` API in C++ and Python client libraries. • Build system migrated from Bazel to CMake for improved portability and modularity.

Triton Inference Server v1.3.0

Fri, 28 Jun 2019 16:36:10 GMT

Triton v1.3.0 adds ONNX Runtime integration and an independent `--http-health-port` flag. • Adds `--http-health-port` flag to specify the HTTP health endpoint port independently from the inference and status HTTP port. • Integrates the ONNX Runtime, enabling ONNX models to be used directly in a model repository without conversion.

Triton Inference Server v1.2.0

Fri, 24 May 2019 16:20:19 GMT

Triton v1.2.0 adds model ensembling pipelines, Kubernetes Helm chart, and new perf_client latency flags • Adds `--percentile` flag to `perf_client` to report Nth-percentile latency (e.g. `--percentile=99` for p99) instead of average latency. • Adds `-z` flag to `perf_client` to use zero-valued input tensors instead of random values during benchmarking. • Adds `--allow-gpu-metrics` option to enable or disable reporting of GPU metrics. • Introduces ensemble model support: a single inference request to an ensemble triggers execution of an entire pipeline of chained models with connected input/output tensors. • Adds a Helm chart for deploying the inference server into a Kubernetes cluster. • Client library builds now support both Ubuntu 16.04 and Ubuntu 18.04, with Python wheels compatible with both Python 2 and Python 3.

Triton Inference Server v1.1.0

Wed, 24 Apr 2019 00:07:16 GMT

Triton v1.1.0 adds zero-sized tensor support, TensorFlow custom C++ ops, and a standalone client build system. • Supports zero-sized input and output tensors for batching models, enabling models that require inputs and outputs with shape `[ batch-size ]`. • Supports variable-size tensor dimensions (indicated by `-1` in model configuration) where the variable dimension has value 0. • TensorFlow custom operations (C++) can now be built directly into the inference server, with an included example and documentation. • Client libraries and examples now build via a separate `Makefile` (with an accompanying `Dockerfile`), decoupled from the main server build.

Triton Inference Server v1.0.0

Mon, 18 Mar 2019 20:11:14 GMT

TensorRT Inference Server 1.0.0 GA adds stateful model scheduling, GRPC streaming, and async HTTP for lower-latency inference. • Adds the *sequence batcher* to schedule and batch stateful models that require multiple inference requests routed to the same model instance/batch slot. • Adds GRPC streaming protocol support for inference requests. • HTTP front-end is now asynchronous, enabling lower-latency and higher-throughput inference handling. • Enhances `perf_client` to support stateful models and backends.

Triton Inference Server v0.11.0

Thu, 28 Feb 2019 02:32:29 GMT

Triton v0.11.0 adds variable-size tensor support, string datatype tensors, and CPU-only (non-GPU) deployment. • Supports variable-size input and output tensors in model configuration by setting a dimension size of `-1` for any dimension that can take on any size. • Adds string datatype support for input and output tensors in TensorFlow models and custom backends. • Enables inference server operation on systems without GPUs, nvidia-docker, or CUDA installed.

Triton Inference Server v0.10.0

Mon, 28 Jan 2019 21:03:27 GMT

Triton Inference Server v0.10.0 adds custom backend support for models implemented outside deep-learning frameworks. • Adds custom backend support, allowing individual models to implement arbitrary logic while retaining GPU support, concurrent execution, and dynamic batching from the server.

Triton Inference Server v0.9.0

Thu, 20 Dec 2018 01:10:45 GMT

TRTIS 0.9.0 adds live model repository reloading, priority controls, and a `--strict-model-config=false` auto-config flag. • Adds `--strict-model-config=false` flag to allow automatic derivation of model configuration properties, removing the need for a `config.pbtxt` file for some model types. • Adds a model priority parameter to model configuration, controlling CPU thread priority for all models and CUDA stream priority for TensorRT models. • Live model repository monitoring now dynamically reloads models without a server restart — supports adding/removing model versions, adding/removing entire models, modifying `config.pbtxt`, and updating model labels at runtime. • Asynchronous GRPC frontend delivers improved inference throughput. Breaking changes: • The GRPC API model version parameter type has changed from string to int — existing clients passing a string model version will break.