◆ AI Model & Data Infrastructure

Triton Inference Server

v2.71.0 open-source

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

source ↗ website ↗ rss ↗ last repo push Aug 20, 2026

tritonserver --model-repository=/models --allow-client-shm=true
# config.pbtxt for an AOT Inductor model
platform: "torch_aoti"
default_model_filename: "model.pt2"
import triton_python_backend_utils as pb_utils

class TritonPythonModel:
    def initialize(self, args):
        self.ready = False
        # ... setup ...
        self.ready = True

    def is_ready(self):
        return self.ready

    def execute(self, requests):
        ...
genai-perf profile --model my-model --request-count 1000 --header 'Authorization: Bearer <token>'
tritonserver --model-repository=/models --metrics-config histogram_latencies=true
parameters: {
  key: "REPORT_CUSTOM_METRICS"
  value: {
    string_value:"yes"
  }
}
genai-perf --input-file prompts.jsonl
tritonserver --model-repository=/models --enable-peer-access=false
tritonserver --model-repository=/models --model-config-name=low_latency
parameters {
  key: "INTER_OP_THREAD_COUNT"
  value: { string_value: "2" }
}
parameters {
  key: "INTRA_OP_THREAD_COUNT"
  value: { string_value: "4" }
}
backend: "vllm"
tritonserver --model-repository=/models --backend-config=tensorrt,version-compatible=true
sequence_batching {
  oldest {
    preserve_ordering: true
  }
}
perf_analyzer -m my_model --serial-sequences
tritonserver --model-repository=/models --metrics-address=127.0.0.1
tritonserver --model-repository=/models --model-load-thread-count=8
tritonserver --model-repository=/models/teamA --model-repository=/models/teamB --model-namespacing
tritonserver --model-repository=/models --cache-config local_cache,size=1073741824
parameters {
  key: "EXECUTION_ENV_PATH"
  value: { string_value: "/shared/envs/my_env.tar.gz" }
}
tritonserver --model-repository=/models --backend-config=tensorflow,version=1
tritonserver --model-repository=/models --disable-auto-complete-config
tritonserver --model-repository=/models --model-load-thread-count=8
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
parameters {
  key: "TF_GRAPH_TAG"
  value: { string_value: "serve" }
}
parameters {
  key: "TF_SIGNATURE_DEF"
  value: { string_value: "serving_default" }
}
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
python3 -m pip install --upgrade clients/python/tritonclient-2.6.0-py3-none-linux_aarch64.whl[all]
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
tritonserver --backend-config=tensorflow,version=2 --model-repository=/models
curl http://<server>:8000/api/status?format=json
trtserver --http-health-port=8080 --http-port=8000 --grpc-port=8001 --model-store=/models
perf_client --percentile=99 -m <model_name> -u <server_url>
perf_client -z -m <model_name> -u <server_url>
trtserver --strict-model-config=false --model-store=/models

Summary

Triton Inference Server is an open source inference serving software that streamlines AI inferencing. It is licensed under BSD-3 and is run as a service across cloud, data center, edge, and embedded devices supporting NVIDIA GPUs, x86, and ARM CPUs. It is for teams needing to deploy AI models from frameworks like PyTorch, ONNX, and TensorRT. Its documentation positions it alongside tools for model-serving. The repository shows recent activity, indicating ongoing development.

The Triton Inference Server provides an optimized cloud and edge inferencing solution.

What Triton Inference Server answers

Which hardware accelerators does it support?

NVIDIA GPUs, x86, ARM CPUs, or AWS Inferentia

What models can it serve?

Models from frameworks including PyTorch, ONNX, TensorRT, OpenVINO, Python, and RAPIDS FIL

Where can it be deployed?

Cloud, data center, edge, and embedded devices

What is its licensing status?

It uses the BSD-3 license

Command line

composed command

option surface across analysed releases

options at that release added, per release removed, per release

option	found in	since	description

Values are placeholders taken from each option’s declared default. Nothing is executed here — the output shown is a recording of a run that already happened.

Release history

v2.71.0 Jul 29, 2026 · issue -021

Triton v2.71.0 adds multi-GPU TensorRT inference, PyTorch 2 batching, NV embedding layers, and tighter OpenAI frontend security.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.71.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.71.0
```
- ›Redacts restriction header values from the OpenAI frontend startup log and enforces inference restrictions in the SageMaker MME invoke handler.
- ›Adds a streaming tool-call parse buffer limit to the OpenAI-compatible frontend to cap memory usage during streaming tool calls.
- ›TensorRT backend gains multi-device (multi-GPU) inference support.
- ›PyTorch backend enables PyTorch 2 batching and adds support for loading NV embedding layers.
- ›Adds support for serving HSTU generative-recommender models via the PyTorch AOTI serving path.
+1 moreshow less
- ›Python client now preserves the output datatype for empty (zero-element) tensors.
v2.71.0 Jul 29, 2026 · issue 002

Triton v2.71.0 adds TensorRT multi-GPU inference, PyTorch 2 batching, NV embedding layers, and OpenAI frontend memory safeguards.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.71.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.71.0
```
- ›Adds a streaming tool-call parse buffer limit to the OpenAI-compatible frontend to cap memory usage during streaming tool calls.
- ›Redacts restriction header values from the OpenAI frontend startup log and enforces inference restrictions in the SageMaker MME invoke handler.
- ›TensorRT backend gains multi-device (multi-GPU) inference support.
- ›PyTorch backend enables PyTorch 2 batching and adds support for loading NV embedding layers.
- ›Supports HSTU generative-recommender models via the PyTorch AOTI serving path.
+1 moreshow less
- ›Python client now preserves output datatype for empty (zero-element) tensors.
v2.70.0 Jun 26, 2026 · issue -054

Triton v2.70.0 adds multi-GPU TensorRT inference, PyTorch 2 dynamic batching, and OpenAI frontend request-size limits.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.70.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.70.0
```
- ›Adds multi-device (multi-GPU) inference support to the TensorRT backend.
- ›Enables PyTorch 2 dynamic batching in the PyTorch backend.
- ›Adds an HTTP request body size limit to the OpenAI-compatible frontend.
- ›Hardens model environment unpacking and file-path handling against path traversal attacks.
- ›Changes the container install tree under /opt/tritonserver/ to root-owned, with the triton-server user accessing it via group/other permissions.
└──▷ BREAKING ON UPGRADE
- !Windows server build is removed; Windows is no longer supported in the core build or documentation.
- !The Python client now requires ml_dtypes.bfloat16 arrays for BF16 input/output tensors and no longer casts to/from np.float32; scripts passing BF16 data as np.float32 arrays must switch to ml_dtypes.bfloat16.
v2.69.0 Jun 2, 2026 · issue -078

Triton v2.69.0 adds a Rust gRPC client, Azure Managed Identity auth, GPU_DEVICE_IDS for vLLM, and gRPC request cancellation.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.69.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.69.0
```
- ›Adds GPU_DEVICE_IDS to the vLLM backend config to pin multi-GPU models to specific GPU devices.
- ›Adds Azure Managed Identity authentication support for Azure Storage model repositories, enabling credential-free model loading.
- ›Adds a Rust gRPC client library as a new supported client language.
- ›Supports request cancellation in the gRPC C++ client.
- ›Adds MemberIsNullIs[type] helpers to TritonJson for safer JSON field-presence and null checks.
+5 moreshow less
- ›Adds complex I/O type support for PT2 models in the PyTorch backend.
- ›Adds auditwheel to the Dockerfile.sdk sdk_build stage and the Model Analyzer devel container for arch-specific manylinux wheel tagging.
- ›Triton now rejects inference requests whose parameters or forward headers use Triton-reserved keys; the Python client enforces the same rejection on the send side.
- ›Triton validates and rejects inference requests containing duplicate output names in both HTTP and gRPC.
- ›Replaces pickle with json for inter-process payloads in the vLLM backend.
v2.68.0 Apr 28, 2026 · issue -113

Triton v2.68.0 disables client shared memory by default, adds ensemble queue limits, and brings OpenAI-compatible model control mode.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.68.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.68.0
```
└──▷ TRY IT
Re-enable client shared memory on upgrade, since it is now off by default and will silently break workloads that depend on it.
```
$ tritonserver --model-repository=/models --allow-client-shm=true
```
- ›Adds --allow-client-shm=true flag required to re-enable client shared memory, which is now disabled by default.
- ›Enforces max_inflight_requests as a shared limit across ensemble requests, preventing unbounded queue growth in ensemble pipelines.
- ›Adds support for explicit model control mode and model management in the OpenAI-compatible frontend.
- ›Adds model name validation at load time following the POSIX standard.
- ›Introduces safe GetElementCount and GetByteSize APIs with overflow protection across core and common libraries.
+1 moreshow less
- ›PyTorch backend adds Meta OpTree support.
└──▷ BREAKING ON UPGRADE
- !Client shared memory is now disabled by default; servers that relied on the previous default must add --allow-client-shm=true to their startup command or shared-memory functionality will be unavailable.
v2.67.0 Mar 27, 2026 · issue -145

Triton v2.67.0 adds PyTorch AOT Inductor (torch_aoti) platform, Python backend is_ready() health hooks, and bfloat16 ONNX I/O support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.67.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.67.0
```
└──▷ USE IT
Serve a PyTorch AOT Inductor (PT2) compiled model using the new torch_aoti platform — replace model.pt workflows with model.pt2 for ahead-of-time compiled inference.
yaml
```
# config.pbtxt for an AOT Inductor model
platform: "torch_aoti"
default_model_filename: "model.pt2"
```
Add a custom readiness check to a Python backend model so Triton only marks the model ready once your internal state is fully initialized.
python
```
import triton_python_backend_utils as pb_utils

class TritonPythonModel:
    def initialize(self, args):
        self.ready = False
        # ... setup ...
        self.ready = True

    def is_ready(self):
        return self.ready

    def execute(self, requests):
        ...
```
- ›Adds new platform torch_aoti for PyTorch PT2 format models using AOT Inductor, with default model file model.pt2; introduces provider classes InductorModel and InductorModelInstance; separates pytorch_libtorch and torch_aoti into distinct namespaces; includes helper utilities, macros, TritonException, and optional debug trace logging via ENABLE_DEBUG_TRACE_* flags.
- ›Adds support for a user-defined is_ready() method in Python backend models to enable custom readiness/health logic.
- ›Enables bfloat16 I/O tensor dtype support in the ONNX Runtime backend.
- ›Adds model name validation for model management requests.
- ›Introduces safe GetElementCount and GetByteSize APIs with overflow protection and proper validation.
+2 moreshow less
- ›Adds validation for OpenAI frontend LoRA paths.
- ›Relaxes the upper bound on the gRPC dependency in the client library for greater version flexibility.
v2.65.0 Feb 3, 2026 · issue -195

Triton v2.65.0 adds first-response latency histogram metrics for coupled models and makes Perf Analyzer dependencies optional.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.65.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.65.0
```
- ›Adds support for the nv_inference_first_response_histogram_ms metric when running models in coupled (non-decoupled) mode, enabling latency histogram tracking for standard inference workloads.
- ›Makes Perf Analyzer dependencies optional when installing tritonclient via pip, reducing install footprint for deployments that don't need performance analysis.
- ›Exposes HTTP errors for non-JSON format responses, improving error visibility for clients sending malformed requests.
v2.64.0 Dec 24, 2025 · issue -236

Triton v2.64.0 promotes OpenAI-compatible frontend to stable, adds multi-LoRA and echo parameter support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.64.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.64.0
```
- ›Adds echo request parameter for TensorRT-LLM and Python backends to the OpenAI-compatible API frontend v1/completions endpoint.
- ›Enables multi-LoRA support for the TensorRT-LLM backend via the OpenAI-compatible API frontend.
- ›Adds TRITONBACKEND_ModelInstanceReady function for backends to implement, enabling accurate model readiness reporting.
- ›Promotes the OpenAI-compatible frontend from beta to stable release.
- ›Updates the Python backend to accurately report model readiness status.
v2.63.0 Nov 26, 2025 · issue -264

Triton v2.63.0 adds v1/embeddings endpoint for vLLM and echo parameter support in the OpenAI-compatible API server.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.63.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.63.0
```
- ›Enables the v1/embeddings endpoint for the vLLM backend in the OpenAI-compatible API server.
- ›Enables the echo parameter for TensorRT-LLM and Python backends in the OpenAI-compatible API server.
- ›Provides more specific, OpenAI-compliant error codes from the OpenAI-compatible API server.
v2.61.0 Oct 7, 2025 · issue -314

Triton v2.61.0 adds static key authentication and repository scoping for OpenAI Frontend APIs.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.61.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.61.0
```
- ›Adds static key authentication support for OpenAI Frontend APIs.
v2.60.0 Aug 26, 2025 · issue -355

Triton Inference Server v2.60.0 adds CUDA 13 support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.60.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.60.0
```
- ›Adds CUDA 13 support to Triton Inference Server.
v2.59.0 Jun 26, 2025 · issue -365

Triton Inference Server v2.59.0 improves ensemble model throughput and latency for out-of-order response scenarios.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.59.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.59.0
```
- ›Improves ensemble model performance in out-of-order response scenarios, increasing maximum throughput and reducing latency.
v2.58.0 May 31, 2025 · issue -366

Triton v2.58.0 adds tool calling for Llama 3/Mistral, TensorRT memory strategy config, and expanded GenAI-Perf GPU metrics.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.58.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.58.0
```
- ›Adds execution_context_allocation_strategy optional parameter to TensorRT backend configuration to control memory allocation behavior.
- ›GenAI-Perf now supports a configuration file as an alternative to command-line arguments.
- ›GenAI-Perf collects GPU metrics from the /metrics endpoint exposed by DCGM Exporter.
- ›GenAI-Perf adds support for Power, Utilization, ECC, Errors, and PCIe metrics.
- ›Adds tool calling support for Llama 3 and Mistral models via the OpenAI frontend.
v2.57.0 May 12, 2025 · issue -366

Triton v2.57.0 adds gRPC infer thread count exposure, BLS decoupled cancellation, and major GenAI-Perf enhancements including config file support and TPS/user metric.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.57.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.57.0
```
- ›Exposes gRPC infer thread count as a configurable server option.
- ›Adds BLS decoupled request cancellation support in the Python Backend.
- ›GenAI-Perf now supports a configuration file as an alternative to command-line arguments.
- ›GenAI-Perf adds support for the Hugging Face TGI (Text Generation Inference) generated endpoint.
- ›GenAI-Perf adds a Token per Second per User (TPS/user) metric.
+1 moreshow less
- ›GenAI-Perf metric parsing speed increased by 60%, unlocking faster benchmarking at scale.
v2.56.0 Apr 7, 2025 · issue -367

Triton v2.56.0 adds SageMaker generate/stream inference types and live KV-cache metrics in HTTP response headers for TRT-LLM.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.56.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.56.0
```
- ›Adds SAGEMAKER_TRITON_INFERENCE_TYPE environment variable to select inference type (infer, generate, or generate_stream) on SageMaker server launch, enabling generate and generate_stream endpoints for SageMaker deployments.
- ›When used with TRT-LLM, Triton now includes live KV-cache utilization and capacity metrics in the HTTP response header during inference requests, enabling on-demand metric retrieval for external load balancers such as the Kubernetes Inference Gateway API.
└──▷ BREAKING ON UPGRADE
- !The TensorFlow Backend is deprecated as of 25.03; the '25.03-tf2-python-py3' container is no longer available. Users must build the TensorFlow Backend from source and install it into /opt/tritonserver/backends/ to continue using it.
v2.55.0 Feb 26, 2025 · issue -369

Triton v2.55.0 adds inference response parameters in Python backend, guided generation support, and major GenAI-Perf upgrades including Jinja2 output templating.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.55.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.55.0
```
- ›Adds support for the guided_generation request parameter in vLLM interactions for constrained decoding workflows.
- ›Python backend now supports setting and retrieving Inference Response Parameters on InferenceResponse objects in model.py.
- ›GenAI-Perf adds Jinja2 template support for formatting output reports.
- ›GenAI-Perf telemetry now supports multiple metric endpoints.
- ›GenAI-Perf supports corpus sizes up to 90x larger than previously supported.
+5 moreshow less
- ›GenAI-Perf now accepts keys without values as input.
- ›GenAI-Perf adds a chat template option for the TRT-LLM engine.
- ›Adds dynamic sampling parameter handling for vLLM interactions, improving flexibility across requests.
- ›Optimized core Python binding architecture for improved OpenAI frontend performance.
- ›Improved Multi-LoRA handling in TRTLLM gRPC client end_to_end_grpc_client.py.
v2.54.0 Jan 29, 2025 · issue -370

Triton v2.54.0 adds Blackwell GPU support, vLLM ZMQ engine comms, and major GenAI-Perf expansions including a new 'analyze' subcommand.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.54.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.54.0
```
└──▷ TRY IT
Send a fixed number of requests with a custom auth header to a secured inference endpoint during CI load testing.
```
$ genai-perf profile --model my-model --request-count 1000 --header 'Authorization: Bearer <token>'
```
- ›GenAI-Perf gains --num-system-prompts and --system-prompt-length flags to create a prefix pool that emulates system prompts in benchmarking runs.
- ›GenAI-Perf gains --request-count to send a specific number of requests and --header to attach custom headers to every request.
- ›GenAI-Perf gains a new analyze subcommand (migrated from Model Analyzer) to sweep configurations and find the optimal model setup.
- ›GenAI-Perf gains a compare subcommand that accepts a custom tokenizer for comparing multiple benchmark profiles.
- ›GenAI-Perf now reports time-to-second-token and request count as first-class metrics.
+6 moreshow less
- ›GenAI-Perf now surfaces more detailed errors when OpenAI frontends return an error or metric generation fails.
- ›GenAI-Perf now provides the exact input sequence length requested for synthetic text generation.
- ›vLLM backend can now communicate with the vLLM engine via ZMQ, enabling vLLM v0.6 performance improvements.
- ›Supports Blackwell GPU architectures starting with this release.
- ›The vLLM container shipped with Triton is now NVIDIA-optimized; users who prefer the public vLLM build can continue to construct their own Triton-vLLM container.
- ›Triton Windows beta release now includes CUDA context sharing support in the TensorRT Backend.
v2.53.0 Dec 23, 2024 · issue -371

v2.53.0 adds vLLM health checks, additional vLLM outputs, multi-node Perf Analyzer support, and signed Windows binaries with long-path and UTF-16 model repo support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.53.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.53.0
```
- ›Adds optional vLLM backend health check that automatically unloads the model when the vLLM engine health check fails.
- ›Adds support for requesting additional outputs from the vLLM backend beyond default inference results.
- ›Adds multi-node process support to Perf Analyzer for TensorRT-LLM workloads.
- ›Windows executables and DLLs are now signed by NVIDIA, removing the untrusted-software popup when starting Triton outside administrator mode.
- ›Triton on Windows now supports long path notation for model repositories.
+2 moreshow less
- ›Triton on Windows now supports wide character encoding (UTF-16) for model repositories.
- ›Improved server stability during gRPC client cancellation.
v2.52.0 Nov 26, 2024 · issue -372

Triton v2.52.0 adds a 'Request to First Response Time' histogram metric for decoupled models and per-model histogram bucket overrides.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.52.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.52.0
```
└──▷ TRY IT
Enable per-request latency histograms for decoupled models to observe time-to-first-response distribution in Triton metrics.
```
$ tritonserver --model-repository=/models --metrics-config histogram_latencies=true
```
- ›Adds --metrics-config histogram_latencies=true flag to enable a new 'Request to First Response Time' histogram metric for decoupled models.
- ›Adds new model_metrics model configuration field to override default buckets for histogram metric families on a per-model basis.
- ›Adds a comprehensive tutorial on Semantic Caching optimization for LLM workloads to the Conceptual Guides.
v2.51.0 Oct 29, 2024 · issue -373

Triton Inference Server v2.51.0 optimizes vLLM performance with custom metrics support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.51.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.51.0
```
- ›Optimizes vLLM performance with custom metrics.
v2.50.0 Sep 27, 2024 · issue -374

vLLM backend gains five new metrics including e2e latency and token counts; new LLM agent and Kubernetes scaling tutorials added.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.50.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.50.0
```
- ›vLLM backend now exposes five additional Triton metrics: vllm:e2e_request_latency_seconds, vllm:request_prompt_tokens, vllm:request_generation_tokens, vllm:request_params_best_of, and vllm:request_params_n.
- ›New tutorials added covering constrained decoding in the TensorRT-LLM Python backend and function/tool calling for AI agents.
- ›New tutorials added for Kubernetes multi-node and multi-instance scaling with Triton and TRT-LLM.
v2.49.0 Aug 27, 2024 · issue -375

Triton v2.49.0 adds vLLM metrics reporting, BF16 via DLPack, GPU device ID pinning per instance, and expanded GenAI-Perf profiling.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.49.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.49.0
```
└──▷ USE IT
Enable vLLM token and latency metrics reporting so Triton exposes them via its metrics endpoint.
protobuf
```
parameters: {
  key: "REPORT_CUSTOM_METRICS"
  value: {
    string_value:"yes"
  }
}
```
Benchmark a custom prompt set against an OpenAI-compatible embeddings endpoint instead of synthetic data.
```
$ genai-perf --input-file prompts.jsonl
```
- ›Enables vLLM metrics reporting (vllm:prompt_tokens_total, vllm:generation_tokens_total, vllm:time_to_first_token_seconds) by setting REPORT_CUSTOM_METRICS to yes in config.pbtxt.
- ›Adds gpu_device_ids field to TensorRT-LLM backend instance configuration, allowing GPU device IDs to be specified per model instance.
- ›GenAI-Perf now accepts multiple user-specified prompts via --input-file, enabling custom prompt-set benchmarking.
- ›Python backend now supports BF16 tensors via DLPack.
- ›GenAI-Perf can now profile OpenAI-compatible embeddings and Hugging Face TEI re-ranker API-compatible rankings.
+1 moreshow less
- ›Loaded model versions whose model files are unmodified are no longer reloaded when the model config is updated to load new versions, reducing unnecessary reload overhead.
v2.48.0 Jul 24, 2024 · issue -376

Triton v2.48.0 adds custom backend tracing, enhanced failure metrics, and GenAI-Perf support for embeddings, re-rankers, and multi-prompt input.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.48.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.48.0
```
- ›Adds --input-file flag to GenAI-Perf to accept multiple user-specified prompts in a single run.
- ›Supports tracing custom backend activities via the backend tracing API (see trace.md#custom-backend-tracing).
- ›Enhances Failure Count Metrics to include the failure reason of each inference request.
- ›Enables profiling of OpenAI-compatible embeddings and Hugging Face TEI re-ranker API-compatible rankings via GenAI-Perf.
- ›Updates request-rate handling in OpenAI and HTTP clients so async requests are dispatched at exactly the specified rate.
+2 moreshow less
- ›Adds a warning in Perf Analyzer when latency does not stabilize for async models, while still allowing the run to complete.
- ›Adds input/output validation in Perf Analyzer, returning an error if user-supplied inputs or outputs are not present in the model.
v2.47.0 Jun 28, 2024 · issue -377

Triton v2.47.0 adds BF16 support in TensorRT backend, GenAI-Perf compare/multi-LoRA/custom-viz, and --enable-peer-access GPU flag.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.47.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.47.0
```
└──▷ TRY IT
Disable automatic GPU peer access attempts at startup when your topology or permissions make it undesirable.
```
$ tritonserver --model-repository=/models --enable-peer-access=false
```
- ›Adds --enable-peer-access flag to control whether Triton attempts to enable GPU peer access at startup (default: TRUE).
- ›Adds compare subcommand to GenAI-Perf for side-by-side comparison across multiple profiling runs.
- ›Adds multi-LoRA and multi-model support in GenAI-Perf.
- ›Adds custom visualizations in GenAI-Perf.
- ›Adds fixed request count support in Perf Analyzer.
+3 moreshow less
- ›Adds ensemble top-level response caching support in Perf Analyzer.
- ›TensorRT Backend now supports the BF16 datatype.
- ›Python models in default mode can now send responses using InferenceResponseSender, matching the capability previously available only in decoupled mode.
v2.46.0 May 25, 2024 · issue -378

Triton v2.46.0 adds namespace metrics, multi-config model loading, LoRA adapter support, and new GenAI-Perf compare subcommand.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.46.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.46.0
```
└──▷ TRY IT
Select a non-default model configuration file at server startup, e.g. to load a low-latency tuned config alongside the default one.
```
$ tritonserver --model-repository=/models --model-config-name=low_latency
```
Control PyTorch inter- and intra-op thread counts per model to tune CPU parallelism without restarting the server.
yaml
```
parameters {
  key: "INTER_OP_THREAD_COUNT"
  value: { string_value: "2" }
}
parameters {
  key: "INTRA_OP_THREAD_COUNT"
  value: { string_value: "4" }
}
```
- ›Adds namespace label to metrics output when the server is launched with --model-namespacing=true, enabling per-namespace metric disambiguation for models sharing the same name.
- ›Adds --model-config-name server launch option to select among multiple model configuration files (configs/<model-config-name>.pbtxt) stored in the model repository for a given model.
- ›Adds INTER_OP_THREAD_COUNT and INTRA_OP_THREAD_COUNT parameters to config.pbtxt for the PyTorch Backend to control thread counts during model execution.
- ›GenAI-Perf gains a new compare subcommand for generating visual comparisons across different profile runs.
- ›GenAI-Perf now accepts an input file containing a single prompt string to drive input generation.
+4 moreshow less
- ›Extends response caching support to top-level requests targeting ensemble models.
- ›Triton's vLLM Backend now supports deployment of models with multiple LoRA adapters.
- ›FIL backend is now included in Triton's ARM-SBSA container image.
- ›Triton logging format has been updated; see the logging format extension documentation for details.
└──▷ BREAKING ON UPGRADE
- !Triton logging format has been modified; existing log parsers or monitoring pipelines that depend on the previous format may need to be updated.
v2.45.0 Apr 30, 2024 · issue -379

Triton v2.45.0 adds AsyncIO decoupled mode, OpenTelemetry trace retrieval, and GenAI-Perf LLM profiling with output token distribution control.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.45.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.45.0
```
- ›Trace settings API now returns trace_mode and trace_config fields when querying trace configuration via gRPC/HTTP endpoints.
- ›Supports retrieving OpenTelemetry trace settings from the gRPC/HTTP endpoints.
- ›Beta support for AsyncIO in decoupled mode in the Python backend.
- ›GenAI-Perf gains the ability to select an output token distribution for load generation.
- ›Model Analyzer adds support for profiling LLMs with GenAI-Perf.
+6 moreshow less
- ›GenAI-Perf adds metric visualizations.
- ›Python backend shared memory region naming now uses UUIDs, allowing multiple Triton servers to run on the same machine without requiring different shared memory region prefixes.
- ›Enhances server shutdown to account for both HTTP live connections and in-flight inferences.
- ›The TensorRT-LLM container now includes the tensorrt_llm Python package for creating engines.
- ›Adds an iterative scheduling tutorial demonstrating GPT2-based iterative scheduling workflows.
- ›Adds Python Client API reference docs to the Triton documentation website.
└──▷ BREAKING ON UPGRADE
- !Log file and trace file locations can no longer be updated via the gRPC/HTTP endpoints.
- !Some GenAI-Perf CLI arguments have been renamed in this release.
- !Perf Analyzer no longer supports the --trace-file option.
- !There is no Windows release for 24.04; the latest Windows release remains v2.44.0.
v2.44.0 Mar 27, 2024 · issue -380

Triton v2.44.0 adds per-response metrics, request cancellation stats, OpenTelemetry Python Backend access, and a new GenAI-Perf LLM benchmarking tool.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.44.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.44.0
```
- ›New GenAI-Perf tool (alpha) for LLM benchmarking, available via the triton-inference-server/client repository.
- ›Triton's backend API now supports collecting per-response metrics.
- ›Triton now publishes request cancellation events in response statistics.
- ›OpenTelemetry context for traces started on the Triton server side is now accessible from the Python Backend.
- ›Python Backend now supports correlation strings in BLS (Business Logic Scripting) models.
+1 moreshow less
- ›HTTP header matching for the header forwarding feature is now case-insensitive.
v2.43.0 Mar 1, 2024 · issue -380

Triton v2.43.0 adds Windows Python backend support and switches OpenTelemetry tracing to Batch Span Processor with configurable batching parameters.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.43.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.43.0
```
- ›OpenTelemetry trace mode switches to Batch Span Processor, enabling bulk export of completed spans with size-based batching controlled by bsp_max_export_batch_size and bsp_max_queue_size, and time-based batching controlled by bsp_schedule_delay.
- ›Adds base Python backend functionality for Windows, enabling Python model serving on Windows deployments.
- ›Removes Wait/Read(avg) and Overhead metrics for gRPC from the Trace Summary Tool to eliminate inaccurate readings.
v2.42.0 Jan 30, 2024 · issue -382

Triton v2.42.0 adds a Python in-process API, model-load retry, OpenTelemetry context propagation, pinned-memory metrics, and experimental PyTorch 2.0 serving.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.42.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.42.0
```
- ›Adds a command-line option to retry loading failed models a configurable number of attempts.
- ›Adds Triton Python API for in-process integration within a Python environment.
- ›Adds support for OpenTelemetry context propagation in trace mode.
- ›Adds pinned memory pool usage reporting to Triton metrics.
- ›Adds experimental support for serving PyTorch 2.0 models via the PyTorch backend.
+3 moreshow less
- ›Improves HTTP endpoint error responses so that status codes other than 400 may be returned to align with the actual error type.
- ›Model Analyzer now loads and optimizes ensemble models.
- ›Model Analyzer now supports optimizing a model on a remote Triton server without requiring a local GPU.
└──▷ BREAKING ON UPGRADE
- !The FasterTransformer backend is deprecated as of 24.01 and is no longer supported or released with this and future versions of Triton.
v2.41.0 Dec 20, 2023 · issue -383

Triton v2.41.0 adds TRTLLM backend metrics, OpenTelemetry request ID tracing, and new Jetpack 6.0 iGPU containers.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.41.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.41.0
```
- ›Adds metrics support to the TensorRT-LLM backend (tensorrtllm_backend) when running within Triton.
- ›Includes request ID in OpenTelemetry tracing spans for improved distributed trace correlation.
- ›Publishes new Jetson iGPU containers on NGC with -igpu suffix (XX.YY-py3-igpu and XX.YY-py3-sdk-igpu) targeting Jetpack 6.0 and above on supported Jetson devices.
v2.40.0 Dec 1, 2023 · issue -383

Triton v2.40.0 adds iterative sequence scheduling, HTTP endpoint access restrictions, vLLM auto-complete, and TRT-LLM request cancellation.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.40.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.40.0
```
└──▷ USE IT
Enable vLLM backend auto-complete so Triton fills in the model config automatically, avoiding the need to write a full config.pbtxt.
yaml
```
backend: "vllm"
```
- ›Adds backend: "vllm" auto-complete support so Triton can automatically fill in vLLM backend model configuration without a full manual config.
- ›Enhances the HTTP endpoint to support access restriction (beta), limiting which endpoints are reachable — see the inference protocols customization guide.
- ›Enhances implicit state management to support growing buffers and a single shared buffer for both input and output states, configurable via model_config.proto.
- ›Adds iterative scheduling support to the sequence batcher, enabling sequences that require multiple scheduling passes.
- ›Extends the backend API to support request rescheduling, available in the Python backend and custom C++ backends.
+7 moreshow less
- ›Adds request cancellation support to the TRT-LLM backend.
- ›Adds parameters support in BLS (Business Logic Scripting) requests from the Python backend.
- ›Adds Kubernetes StartupProbe support to the on-premises Kubernetes deploy example, giving Triton pods time to finish startup before health probes run.
- ›Improves Python backend GPU tensor performance.
- ›Adds a Secure Deployment Guide covering how to deploy Triton in a hardened configuration.
- ›Adds a new tutorial for deploying LLaMA2 using TRT-LLM.
- ›Publishes Triton iGPU containers for Jetson/IGX devices, supporting TensorFlow 2.14.0, TensorRT 8.6.2.3, ONNX Runtime 1.16.3, PyTorch 2.2.0a0+6a974be, and Python 3.10.
└──▷ BREAKING ON UPGRADE
- !reuse-grpc-port and reuse-http-port are now properly parsed as booleans; any integer value other than 0 or 1 will throw an error (previously non-boolean integers may have been silently accepted).
- !The client model loading API no longer allows uploading files outside the model repository; any workflow that relied on out-of-repository file uploads will break.
v2.39.0 Oct 27, 2023 · issue -385

Triton v2.39.0 adds TensorRT-LLM and vLLM backends, a Generate REST extension for LLMs, client-side request cancellation, and Python-based custom backends.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.39.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.39.0
```
- ›Adds the Generate extension (beta) providing REST APIs optimized for Large Language Model inference.
- ›Adds support for handling client-side request cancellation in the server and backends.
- ›Adds the TensorRT-LLM backend (release/0.5.0), replacing the FasterTransformer backend, with a dedicated NGC container for 23.10.
- ›Adds the vLLM backend (r23.10) for deploying supported models on the vLLM engine, with a dedicated NGC container for 23.10.
- ›Adds support for writing custom backends in Python via the new Python-based backends framework.
+1 moreshow less
- ›Adds Scalar I/O support in the ONNXRuntime backend.
└──▷ BREAKING ON UPGRADE
- !The FasterTransformer backend is replaced by the TensorRT-LLM backend; FasterTransformer is only officially supported through Triton 22.12 (buildable up to 23.07).
v2.38.0 Sep 27, 2023 · issue -386

Triton v2.38.0 adds Python C API bindings, ensemble request parameter forwarding, queue-depth metrics, and TensorRT version compatibility.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.38.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.38.0
```
└──▷ TRY IT
Enable TensorRT version compatibility so models built with any TensorRT 8.x release can run on a TensorRT 8.x server without rebuilding.
```
$ tritonserver --model-repository=/models --backend-config=tensorrt,version-compatible=true
```
- ›Enables TensorRT version compatibility across models built with the same major TensorRT version via the --backend-config=tensorrt,version-compatible=true flag.
- ›Adds pending-request queue size per model to the metrics API.
- ›New Python bindings for the Triton C API.
- ›Forwards request parameters to each composing model in an ensemble pipeline.
- ›Filesystem API now supports named temporary cache directories when downloading models via the repository agent.
+3 moreshow less
- ›Backend API adds access to inference response outputs by name or by index.
- ›Python backend models can now return structured error codes in addition to error messages.
- ›Python backend gains experimental (Beta) support for loading PyTorch models directly.
v2.37.0 Aug 30, 2023 · issue -387

Triton v2.37.0 adds parallel model instance loading, PyTorch serving via torch.compile(), OpenTelemetry resource settings, and BLS trace capture.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.37.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.37.0
```
└──▷ USE IT
Explicitly restore pre-v2.37.0 response ordering behavior for an oldest-strategy sequence batcher after the default changes to false.
protobuf
```
sequence_batching {
  oldest {
    preserve_ordering: true
  }
}
```
- ›Exposes preserve_ordering field to the oldest-strategy sequence batcher, letting operators control whether response order is preserved across independent requests belonging to different sequences.
- ›Triton OpenTelemetry Tracing exposes resource settings to configure service name and version via OpenTelemetry trace API settings.
- ›Supports loading model instances in parallel for backends that opt in via TRITONBACKEND_BackendAttributeSetParallelModelInstanceLoading; Python and ONNXRuntime backends enabled as of this release.
- ›Python backend models can now capture traces for composing child models when executing BLS (Business Logic Scripting) requests.
- ›Python backend supports directly loading and serving PyTorch models using torch.compile() (experimental PyTorch platform).
└──▷ BREAKING ON UPGRADE
- !The default value of preserve_ordering in the oldest-strategy sequence batcher changes from true to false, meaning response order across independent requests from different sequences is no longer preserved by default on upgrade.
v2.36.0 Jul 28, 2023 · issue -388

Triton v2.36.0 adds PyTorch implicit state, TF SavedModel direct serving, OpenTelemetry ensemble tracing, DLPack CUDA shared memory, Java bindings, and S3 repo scaling past 1000 files.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.36.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.36.0
```
- ›Adds model loading APIs to python_backend for BLS (Business Logic Scripting) usage.
- ›Supports direct serving of TensorFlow SavedModel via python_backend without a dedicated TF backend.
- ›Supports unpacked Conda execution environments in python_backend for custom dependency management.
- ›Adds DLPack tensor support to Triton Python client CUDA shared memory utilities.
- ›Adds implicit state management to pytorch_backend.
+3 moreshow less
- ›Extends OpenTelemetry trace mode to cover ensemble model tracing.
- ›Adds Java binding of the Triton in-process C++ API via javacpp-presets.
- ›Supports S3 model repositories containing more than 1000 files.
v2.35.0 Jun 30, 2023 · issue -389

Triton v2.35.0 adds Redis cache support, gRPC response flags, KIND_MODEL for PyTorch, and serial sequences in Perf Analyzer.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.35.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.35.0
```
└──▷ TRY IT
Run Perf Analyzer in serial sequences mode to avoid interleaving requests across sequences during load testing.
```
$ perf_analyzer -m my_model --serial-sequences
```
- ›Adds --serial-sequences mode to Perf Analyzer for sequential load testing of inference requests.
- ›Adds beta support for using Redis as a cache for inference requests via the redis_cache backend.
- ›Extends KIND_MODEL instance group type support to the PyTorch backend.
- ›The statistics extension now includes memory usage of loaded models (currently implemented for TensorRT and ONNXRuntime backends).
- ›gRPC clients can now request response flags per response to programmatically determine when all responses for a decoupled model request have been received.
+1 moreshow less
- ›Adds support for batch inputs in ragged batching for the PyTorch backend.
v2.34.0 May 31, 2023 · issue -390

Triton 2.34.0 adds custom metrics in Python backend, a client plugin API, dynamic instance scaling, and a new --metrics-address flag.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.34.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.34.0
```
└──▷ TRY IT
Bind the Prometheus metrics endpoint to a specific internal interface instead of the default 0.0.0.0, useful when you need to restrict metrics exposure on a multi-homed host.
```
$ tritonserver --model-repository=/models --metrics-address=127.0.0.1
```
On a high-core-count host where the old default caused resource exhaustion, pin model load parallelism to a safe value while keeping the new lower default as a baseline.
```
$ tritonserver --model-repository=/models --model-load-thread-count=8
```
- ›Adds --metrics-address=<address> CLI option to bind the metrics server to a different address than the default 0.0.0.0.
- ›Reduces the default number of model load threads from 2*(number of CPU cores) to 4; the --model-load-thread-count CLI option overrides this default.
- ›Python backend now supports Custom Metrics, letting users define and report counters and gauges via the same interface as the C API.
- ›Python Triton Client introduces the Triton Client Plugin API (beta) for registering custom plugins that add or modify request headers.
- ›Adds DLPack Python specification support in the Python backend via pb_utils.Tensor.from_dlpack().
+1 moreshow less
- ›Improves model instance scaling: when only the instance group changes in a model config, Triton updates instance counts in-place without a full model reload (non-sequence models only).
└──▷ BREAKING ON UPGRADE
- !The default model load thread count is reduced from 2*(number of CPU cores) to 4; deployments on large-core systems that relied on the previous default parallelism will need to set --model-load-thread-count explicitly to restore prior behavior.
v2.33.0 Apr 26, 2023 · issue -391

Triton v2.33.0 adds concurrent model loading, OpenTelemetry tracing, HTTP/gRPC header forwarding, configurable latency quantiles, and protocol access restrictions.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.33.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.33.0
```
- ›Adds experimental latency metrics as configurable quantiles over a sliding time window via metrics summary support (see metrics.md#summaries).
- ›Adds beta support for restricting access to specific protocols on a given Triton endpoint (see inference_protocols.md#limit-endpoint-access-beta).
- ›Adds experimental support for schedule policy in the sequence batcher with direct scheduling strategy.
- ›Adds limited support for tracing inference requests using OpenTelemetry Trace APIs.
- ›Enables forwarding of HTTP/gRPC headers as inference request parameters to the backend.
+4 moreshow less
- ›Extends ragged batching support to the PyTorch backend.
- ›Enables concurrent model loading to reduce server start-up times.
- ›Python backend business logic scripting (BLS) now allows selecting a specific device to receive output tensors from a BLS call.
- ›Model Analyzer adds support for BLS model config search.
v2.32.0 Mar 28, 2023 · issue -392

Triton v2.32.0 adds Parameters Extension, decoupled BLS, model namespacing, pluggable cache API, and request_id tracing
└──▷ GET THIS VERSION
```
$ git clone --branch v2.32.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.32.0
```
└──▷ TRY IT
Run Triton with model namespacing enabled so identical model names in different repositories do not collide.
```
$ tritonserver --model-repository=/models/teamA --model-repository=/models/teamB --model-namespacing
```
Configure the response cache using the new preferred --cache-config flag instead of the legacy byte-size flag.
```
$ tritonserver --model-repository=/models --cache-config local_cache,size=1073741824
```
- ›Adds --model-namespacing flag to allow the same model name to be used across different model repositories.
- ›Adds --cache-config flag as the preferred method for configuring the response cache, backed by the new TRITONCACHE shared-library API; --response-cache-byte-size continues to work.
- ›Introduces the Parameters Extension, enabling inference requests to supply custom parameters that cannot be passed as inputs, accessible in the Python backend via inference request parameters.
- ›Adds support for models using the decoupled API for Business Scripting Logic (BLS) in the Python backend.
- ›Extends the trace tool to support tracing by request_id.
+1 moreshow less
- ›Converts the Response Cache to a pluggable shared-library architecture (TRITONCACHE APIs), with local_cache as the default implementation.
v2.31.0 Mar 1, 2023 · issue -392

Triton v2.31.0 adds ensemble model support in Model Analyzer and GRPC Standard Health Check Protocol.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.31.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.31.0
```
- ›Adds support for the GRPC Standard Health Check Protocol on the inference server endpoint.
- ›Adds ensemble model support in Model Analyzer, enabling config search across ensemble pipelines.
v2.30.0 Feb 1, 2023 · issue -393

Triton v2.30.0 adds user-defined custom batching strategies to the dynamic batcher.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.30.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.30.0
```
- ›Enables user-defined batching constraints in the dynamic batcher via custom batching strategies, letting operators control how requests are grouped into batches.
- ›Relaxes the Python client gRPC version requirement, broadening compatible client environments.
v2.29.0 Dec 20, 2022 · issue -395

Triton Inference Server v2.29.0 adds improved Windows builds and a beta Windows package with HTTP/REST, GRPC, ONNX, and OpenVINO support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.29.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.29.0
```
- ›Releases a beta Windows package (tritonserver2.29.0-win.zip) supporting HTTP/REST and GRPC endpoints, ONNX models via ONNX Runtime 1.13.1 (CPU, CUDA, and TensorRT execution providers), and OpenVINO models via OpenVINO 2021.4.
- ›Bundles CUDA 11.8.0, cuDNN 8.7.0.84, and TensorRT 8.5.1.7 dependencies for Windows deployments.
- ›Improves container and non-container builds on Windows.
- ›Includes Ubuntu 20.04 client libraries and examples in v2.29.0_ubuntu2004.clients.tar.gz and Windows SDK in tritonserver2.29.0-sdk-win.zip.
v2.28.0 Nov 22, 2022 · issue -396

Triton v2.28.0 adds TensorRT 8.5 UINT8 I/O, data-dependent dynamic shapes, and external execution environment paths.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.28.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.28.0
```
└──▷ USE IT
Point the Python backend to an execution environment (e.g. a conda pack) stored outside the model directory.
yaml
```
parameters {
  key: "EXECUTION_ENV_PATH"
  value: { string_value: "/shared/envs/my_env.tar.gz" }
}
```
- ›Adds EXECUTION_ENV_PATH parameter in config.pbtxt to specify execution environment paths outside the model directory for the Python backend.
- ›Supports TensorRT 8.5 UINT8 input/output tensors.
- ›Supports TensorRT 8.5 'data dependent dynamic shapes' operators, including ONNX NMS and NonZero operations.
- ›Windows beta release adds ONNX Runtime 1.13.1 support with CPU, CUDA, and TensorRT execution providers.
- ›Windows beta release adds OpenVINO 2021.4 model support.
+1 moreshow less
- ›Windows beta release supports HTTP/REST and GRPC endpoints.
v2.27.0 Nov 2, 2022 · issue -396

Triton 2.27.0 adds JAX model support, enhanced Server Wrapper API with decoupled model and tracing, and multi-model concurrent profiling in Model Analyzer.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.27.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.27.0
```
- ›Adds multi-model concurrent profiling to Model Analyzer via Multi-Model Search Mode, enabling simultaneous profiling and analysis of multiple models.
- ›Enhances the Server Wrapper API to support decoupled model workflows and tracing, covering previously missing features.
- ›Adds an example demonstrating JAX model usage inside Python backend models.
v2.26.0 Oct 4, 2022 · issue -397

Triton v2.26.0 adds CPU metrics, dynamic logging config, GPU metrics in Perf Analyzer, and a quick-search mode for Model Analyzer.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.26.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.26.0
```
- ›Adds CPU metrics reporting to Triton's Prometheus metrics endpoint.
- ›Adds a logging protocol extension enabling users to change logging configuration dynamically at runtime.
- ›Adds --disable-auto-complete-config launch flag awareness note: users can specify custom TensorRT plugins via command-line option in addition to LD_PRELOAD.
- ›Adds GPU metrics gathering to Perf Analyzer, improving accuracy of metrics also consumed by Model Analyzer.
- ›Adds a 'quick search' algorithm to Model Analyzer to drastically reduce model configuration search time.
+3 moreshow less
- ›Adds a developer tools GitHub repository providing a simplified interface to the Triton Core shared library (beta).
- ›Enables auto-completion of model configuration for the OpenVINO backend.
- ›Enables the Python backend to emit log messages through Triton's native logger.
v2.25.0 Aug 26, 2022 · issue -399

Triton v2.25.0 adds multi-cloud credential support, GPU memory load limits, and configless custom backend loading
└──▷ GET THIS VERSION
```
$ git clone --branch v2.25.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.25.0
```
└──▷ TRY IT
Keep using TensorFlow 1.X after upgrading to v2.25.0, which now defaults to TF 2.X.
```
$ tritonserver --model-repository=/models --backend-config=tensorflow,version=1
```
- ›Adds --model-load-gpu-limit server option and TRITONSERVER_ServerOptionsSetModelLoadDeviceLimit C API function to cap GPU memory usage when loading models.
- ›Adds beta support for multiple cloud storage credentials via a credential file, enabling per-repository cloud auth configuration.
- ›Enables loading custom backend models without an explicit config.pbtxt when the model is named in the form <model_name>.<backend_name> and the backend implements auto-complete configuration.
- ›Defaults TensorFlow backend to version 2.X; TensorFlow 1.X can still be selected via --backend-config=tensorflow,version=<int>.
- ›Model Analyzer's profile subcommand now automatically runs analysis after profiling completes, removing the need to invoke the separate analyze subcommand.
+2 moreshow less
- ›PyTorch backend now uses a separate CUDA stream per GPU model instance, improving GPU inference throughput.
- ›Adds new 'Performance Tuning' user guide with a step-by-step production optimization walkthrough.
└──▷ BREAKING ON UPGRADE
- !TensorFlow backend now defaults to version 2.X; workloads relying on the previous default of TensorFlow 1.X must explicitly set --backend-config=tensorflow,version=1.
- !Model Analyzer's analyze subcommand is deprecated; analysis is now integrated into profile and the standalone subcommand may be removed in a future release.
v2.24.0 Jul 30, 2022 · issue -400

Triton v2.24.0 enables auto-complete by default, adds asyncio Python APIs, a C++ BLS example, and a concurrent model-load thread pool.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.24.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.24.0
```
└──▷ TRY IT
Disable auto-complete config to surface models that fail to load cleanly, replacing the old --strict-model-config flag.
```
$ tritonserver --model-repository=/models --disable-auto-complete-config
```
Speed up server startup and reduce downtime when loading many models simultaneously by increasing the model-load thread pool.
```
$ tritonserver --model-repository=/models --model-load-thread-count=8
```
- ›Replaces the soft-deprecated --strict-model-config option with the new --disable-auto-complete-config CLI option, and enables Auto-Complete model configuration by default.
- ›Adds --model-load-thread-count tritonserver CLI option to configure the size of a new thread pool for concurrently loading models, reducing service downtime.
- ›Adds a new asyncio-compatible API (beta) to the Python gRPC/HTTP client libraries.
- ›Supports init_ops parameters for TensorFlow TF1.x GraphDef models, configurable via JSON file.
- ›New example backend demonstrating Business Logic Scripting (BLS) in C++.
+1 moreshow less
- ›Model Analyzer no longer requires a config.pbtxt file for models that support Triton auto-completion.
└──▷ BREAKING ON UPGRADE
- !--strict-model-config is soft-deprecated; replace it with --disable-auto-complete-config. Auto-Complete is now enabled by default, which may increase server start time for existing deployments that relied on full model configurations.
v2.23.0 Jun 30, 2022 · issue -401

Triton v2.23.0 enables dynamic batching by default, brings Python backend auto-config and Decoupled API to GA.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.23.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.23.0
```
- ›Auto-generated model configuration now enables dynamic batching in supported models by default.
- ›Python backend models now support auto-generated model configuration via auto_complete_config.
- ›Decoupled API support in the Python backend is promoted out of beta to general availability.
- ›Updated I/O tensor naming convention for serving TorchScript models via the PyTorch backend.
v2.22.0 May 26, 2022 · issue -402

Triton v2.22.0 adds Java in-process API, BF16 support, decoupled Python backend, and dynamic model repository registration.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.22.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.22.0
```
- ›Adds Java bindings for the Triton In-Process API, enabling the full in-process server to be embedded in Java applications.
- ›Supports loading models with file content supplied inline at API invocation time via the model-repository load extension.
- ›Adds BF16 (bfloat16) as a supported data type in model configuration.
- ›Adds support for customized gRPC channel settings in the Python gRPC client library (tritonclient.grpc).
- ›Python backend adds decoupled API support (BETA), enabling models to send multiple, asynchronous responses per request.
+3 moreshow less
- ›Explicit model control mode now supports loading all models at startup.
- ›PyTorch backend adds support for 1-dimensional String tensor I/O.
- ›ONNX Runtime backend updated to version 1.11.1 on both Ubuntu and Windows.
v2.21.0 Apr 29, 2022 · issue -403

Triton v2.21.0 adds raw binary requests, custom metrics via C API, multi-cloud model repos, and ensemble optional inputs.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.21.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.21.0
```
- ›Adds --tmp-dir argument to build.py to specify a custom temporary directory during container builds.
- ›Supports raw binary HTTP requests to the inference endpoint, eliminating the need to specify an inference header.
- ›Adds custom metrics support via the Triton C API, allowing custom backends and applications to publish metrics to the existing Triton metrics endpoint.
- ›Supports loading models from multiple cloud storage repositories simultaneously — including multiple S3 buckets, GCS buckets, and Azure Storage containers — in a single Triton instance.
- ›Ensembles now recognize optional inputs, enabling more flexible ensemble pipeline configurations.
+2 moreshow less
- ›ONNX Runtime backend now uses available execution providers (e.g. GPU/TensorRT) when autocomplete is enabled, rather than always defaulting to the CPU execution provider.
- ›Adds PyTorch and TensorFlow 1 backend support to build.py and compose.py for CPU-only builds.
v2.20.0 Mar 29, 2022 · issue -404

Triton v2.20.0 adds ragged batching for ONNX/TensorRT/TF, cache-miss Prometheus metrics, and trace config via API.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.20.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.20.0
```
- ›Adds cache miss statistics to the Prometheus metrics endpoint, as documented in the metrics reference.
- ›Enables trace settings to be configured at runtime via the Triton Server Trace Protocol extension.
- ›Supports loading models from a serialized model_config message via the Triton Server model-repository API.
- ›ONNX Runtime, TensorRT, and TensorFlow backends now support server-side, multi-dimensional ragged batching.
v2.19.0 Feb 25, 2022 · issue -405

Triton v2.19.0 adds SSL in the HTTP C++ client, new Prometheus cache metrics, and PyTorch dictionary-tensor inputs.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.19.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.19.0
```
- ›Adds full SSL/TLS support to the HTTP C++ client library.
- ›PyTorch Backend now supports passing model inputs as a dictionary of tensors.
v2.18.0 Jan 31, 2022 · issue -406

Triton 2.18.0 adds implicit state management for ONNX/TensorRT, Inferentia batching, Jetson backend support, and ARM analyzer support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.18.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.18.0
```
- ›Implicit state management is now supported for the ONNX Runtime and TensorRT backends.
- ›State initialization from a constant is now supported within implicit state management.
- ›PyTorch and TensorFlow models now support batching on Inferentia.
- ›PyTorch and Python backends are now supported on Jetson.
- ›ARM support added for the Performance Analyzer and Model Analyzer.
+1 moreshow less
- ›CPU-only build now supports the TensorFlow2 backend for Linux x86.
v2.17.0 Dec 21, 2021 · issue -407

Triton v2.17.0 adds MLflow deployment, TorchTRT model support, Neuron Runtime 2.x for Inferentia, and ONNX Runtime 1.10.0.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.17.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.17.0
```
└──▷ TRY IT
Run Triton on Jetson with an explicit backend directory and TensorFlow 2.x, using the new JetPack 4.6 release.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
```
- ›Adds MLflow plugin at deploy/mlflow-triton-plugin to deploy MLflow models directly to Triton.
- ›Supports TorchTRT (preview): PyTorch models optimized with TensorRT can now be loaded the same way as regular TorchScript models.
- ›Upgrades ONNX Runtime backend to version 1.10.0 on both Ubuntu and Windows builds of Triton.
- ›Improves Inferentia support to use Neuron Runtime 2.x and enables multiple model instances.
- ›Adds end-of-phase example command-line output in Model Analyzer to guide users through the next analysis phase.
+2 moreshow less
- ›Expands Windows alpha support: HTTP/REST and GRPC endpoints are now available alongside ONNX Runtime 1.10.0 (CPU, CUDA, TensorRT execution providers) and OpenVINO 2021.2.
- ›Jetson JetPack 4.6 release adds support for TensorFlow 2.6.0, TensorFlow 1.15.5, TensorRT 8.0.1.6, and ONNX Runtime 1.10.0, including ensemble models.
└──▷ BREAKING ON UPGRADE
- !The byte_contents field in the GRPC protobuf implementation is renamed to bytes_contents; client code using byte_contents must be updated.
v2.16.0 Nov 19, 2021 · issue -408

Triton v2.16.0 adds LightGBM categorical feature support, Inferentia PoC, ARM Model Analyzer, and Jetson examples.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.16.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.16.0
```
- ›Adds support for LightGBM models with categorical features in the FIL backend.
- ›Adds ARM support for Model Analyzer.
- ›Completes proof-of-concept Inferentia support via the Python backend.
- ›Adds Jetson examples under docs/examples/jetson in documentation.
v2.15.0 Oct 28, 2021 · issue -409

Triton v2.15.0 adds rate limiting, response caching, string sequence IDs, ARM SBSA beta, and Windows HTTP support.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.15.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.15.0
```
└──▷ TRY IT
Run Triton on Jetson with an explicit backend directory and TensorFlow 2.x to leverage the new JetPack 4.6 TF2 support.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
```
- ›Adds rate limiter that manages the rate at which requests are scheduled on model instances, configurable via docs/rate_limiter.md.
- ›Adds support for caching responses to inference requests.
- ›Sequence IDs (via the sequence extension protocol) can now accept string values in addition to integers.
- ›Container composer tool gains the ability to generate CPU-only Triton containers (see docs/compose.md#cpu-only-container-composition).
- ›Windows Triton build now supports the HTTP/REST protocol (previously GRPC only).
+3 moreshow less
- ›Beta release of Triton is available for ARM SBSA platforms.
- ›Windows alpha release now supports OpenVINO models at version 2021.2 and ONNX models via ONNX Runtime 1.9 with CPU, CUDA, and TensorRT execution providers.
- ›Jetson JetPack 4.6 release adds support for TensorFlow 2.6.0, TensorFlow 1.15.5, TensorRT 8.0.1.6, OnnxRuntime 1.8.1, and ensembles.
v2.14.0 Sep 29, 2021 · issue -410

Triton v2.14.0 ships BLS beta, Java client beta, PyPI SDK wheel, and TensorRT as an optional backend.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.14.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.14.0
```
- ›Adds --gpus flag support for CUDA Device Index in addition to GPU UUID in Model Analyzer, giving teams more flexibility when targeting specific devices.
- ›Full-featured beta of Business Logic Scripting (BLS) released for the Python backend, enabling custom inference logic within the server.
- ›Beta Java client released with initial support for Triton's inference API.
- ›Triton Client SDK wheel now available directly from PyPI for both Ubuntu and Windows, installable via pip.
- ›TensorRT backend is now optional; the compose utility can build a Triton container without it, just like all other backends.
+1 moreshow less
- ›Model Analyzer can now profile using perf_analyzer's C-API.
v2.13.0 Aug 28, 2021 · issue -411

Triton v2.13.0 adds Business Logic Scripting beta, a Container Composition Utility, and Model Analyzer support for custom ops.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.13.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.13.0
```
└──▷ TRY IT
Run Triton on Jetson with an explicit backend directory and TensorFlow 2.x selected instead of the default TF1.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
```
- ›Adds --backend-directory flag and --backend-config=tensorflow,version=2 argument to tritonserver for explicit backend path and TensorFlow version selection on Jetson.
- ›Initial beta release of Business Logic Scripting — a new set of utility functions allowing a Python model to execute inference requests on other models being served by Triton.
- ›New Container Composition Utility (docs/compose.md) for building custom Triton containers with specific backends and repository agents.
- ›Publishes two new focused NGC containers: nvcr.io/nvidia/tritonserver:21.08-tf-python-py3 (TensorFlow 2.x + Python backends) and nvcr.io/nvidia/tritonserver:21.08-pyt-python-py3 (PyTorch + Python backends).
- ›Adds Model Analyzer support for models with custom operations.
└──▷ BREAKING ON UPGRADE
- !In the Python client shared-memory support library, utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return np.object_ numpy arrays instead of np.bytes_ arrays; code depending on np.bytes_ must be updated.
v2.12.0 Jul 27, 2021 · issue -412

Triton v2.12.0 adds CPU support for RAPIDS FIL, multi-copy C API inputs, TensorRT precision/workspace config for ONNX, and Model Analyzer offline mode.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.12.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.12.0
```
- ›The tensorrt_accelerator option in the ONNX model configuration can now specify precision and workspace size for TensorRT-accelerated ONNX models.
- ›Adds CPU support to the RAPIDS FIL Backend, extending inference beyond GPU-only deployments.
- ›C API inference requests can now supply multiple copies of an input tensor in different memory locations; Triton selects the most performant copy based on execution context.
- ›Model Analyzer gains an offline mode that prioritizes throughput over latency, with a new set of reports and graphs tailored to offline inferencing use cases.
v2.11.0 Jun 25, 2021 · issue -413

Triton v2.11.0 adds FIL backend for forest models, OpenVINO on Windows, C API benchmarking, conda Python environments, and MIG GPU metrics.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.11.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.11.0
```
└──▷ TRY IT
Deploy an XGBoost or LightGBM forest model through Triton using the new FIL backend.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends
```
- ›Adds the Forest Inference Library (FIL) backend, enabling deployment of forest models trained with XGBoost, LightGBM, Scikit-Learn, and cuML.
- ›Adds perf_analyzer support for benchmarking Triton directly via the C API, in addition to HTTP/gRPC endpoints.
- ›Python backend now supports conda to create isolated execution environments per Python model.
- ›Adds passive model instance support in model configuration (via ModelInstanceGroup in model_config.proto), allowing instances to be loaded and initialized without receiving inference requests.
- ›Model repositories hosted in S3 storage can now be accessed using the HTTPS protocol.
+6 moreshow less
- ›Triton now collects GPU metrics for MIG (Multi-Instance GPU) partitions.
- ›NVDLA support added to the TensorRT backend.
- ›Windows release of Triton now includes the OpenVINO backend (OpenVINO version 2021.2).
- ›DALI backend now accepts GPU inputs and adds support for dynamic batching and ragged inputs.
- ›Model Analyzer gains an offline mode and improved detailed and summary reports.
- ›ONNX Runtime updated to version 1.8.0.
└──▷ BREAKING ON UPGRADE
- !In the Python client shared-memory support library, utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return np.object_ numpy arrays instead of np.bytes_ arrays; code depending on np.bytes_ must be updated.
v2.10.0 May 21, 2021 · issue -414

Triton v2.10.0 adds ONNX on Jetson, HTTP compression, ragged batching, and Model Analyzer CLI subcommands.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.10.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.10.0
```
└──▷ TRY IT
Run Triton on Jetson with ONNX Runtime backend and TensorFlow 2.x, explicitly setting the backend directory.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
```
- ›Adds profile, analyze, and report subcommands to the Model Analyzer CLI for structured profiling and reporting workflows.
- ›Adds --backend-directory flag to tritonserver to explicitly set the backend directory, required on Jetson deployments.
- ›Adds --backend-config=tensorflow,version=2 flag to select TensorFlow 2.x on Jetson (where TF 1.x is the default).
- ›Triton server and HTTP clients (Python and C++) now support request/response compression.
- ›Ragged batching is now supported for ONNX models.
+4 moreshow less
- ›Triton on Jetson (JetPack 4.5) now supports ONNX via the ONNX Runtime backend (version 1.7.1), including the TensorRT execution provider.
- ›Model Analyzer gains a report subcommand to generate a detailed report for any specific model configuration.
- ›Model Analyzer now supports CPU-only mode.
- ›Windows alpha release adds GRPC endpoint support for TensorRT and ONNX Runtime (CPU, CUDA, TensorRT execution providers) models.
└──▷ BREAKING ON UPGRADE
- !Legacy custom backend support is removed; all custom backends must be reimplemented using the TRITONBACKEND API.
- !Model Analyzer CLI subcommands (profile, analyze, report) require updating existing Model Analyzer config files and CLI flags.
- !The Triton client libraries have moved to a separate repository at https://github.com/triton-inference-server/client.
- !utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() in the Python client shared-memory support library now return np.object_ arrays instead of np.bytes_ arrays; code depending on np.bytes_ must be updated.
v2.9.0 Apr 27, 2021 · issue -415

Triton v2.9.0 adds GKE Marketplace deployment, GRPC compression, ragged batching for TensorFlow, and ensemble unload control.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.9.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.9.0
```
└──▷ USE IT
Load a TensorFlow SavedModel selecting a specific graph and signature_def via backend parameters in the model config.
yaml
```
parameters {
  key: "TF_GRAPH_TAG"
  value: { string_value: "serve" }
}
parameters {
  key: "TF_SIGNATURE_DEF"
  value: { string_value: "serving_default" }
}
```
- ›The Model Control API gains an option to unload all models contained within an ensemble when unloading the ensemble itself — see the model-repository extension protocol docs.
- ›GRPC client libraries now support enabling compression.
- ›Ragged batching is now supported for TensorFlow models.
- ›For TensorFlow SavedModel-format models, adds the ability to select which graph and signature_def to load via backend parameters.
- ›Triton Server is now available as a GKE Marketplace Application.
+5 moreshow less
- ›Adds a Helm Chart example for AWS deployments.
- ›Python backend performance is significantly increased.
- ›ONNX Runtime updated to version 1.7.1.
- ›Model Analyzer summary reports now include GPU power usage.
- ›Model Analyzer can now find the Top N model configurations across multiple models.
└──▷ BREAKING ON UPGRADE
- !In the Python client shared-memory support library, utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return np.object_ numpy arrays instead of np.bytes_ numpy arrays for tensors of type BYTES — code depending on np.bytes_ must be updated.
v2.8.0 Mar 27, 2021 · issue -416

Triton 2.8.0 adds Repository Agent C API, OpenVINO backend, SSL/TLS mutual auth for gRPC, and ONNX support on Windows.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.8.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.8.0
```
- ›Adds gather_kernel_buffer_threshold Model Configuration option to instruct Triton to use a CUDA kernel to gather input buffers onto the GPU, improving inference performance for some models.
- ›Introduces Repository Agent, a new extensibility C API for implementing custom authentication, decryption, conversion, or similar operations at model-load time.
- ›Adds an OpenVINO backend enabling execution of OpenVINO models on CPUs.
- ›Adds SSL/TLS Mutual Authentication support to the GRPC client library.
- ›Adds system shared-memory protocol support between clients and the Triton server on Jetson deployments.
+1 moreshow less
- ›Windows alpha release now supports ONNX models via the ONNX Runtime 1.6.0 backend, with CPU, CUDA, and TensorRT execution providers (OpenVINO execution provider not supported).
└──▷ BREAKING ON UPGRADE
- !The Python client shared-memory support library functions utils.serialize_byte_tensor() and utils.deserialize_byte_tensor() now return np.object_ numpy arrays instead of np.bytes_ arrays; code depending on np.bytes_ must be updated.
v2.7.0 Feb 27, 2021 · issue -417

Triton v2.7.0 adds Java/Scala gRPC clients, TFServing/TorchServe perf_analyzer support, and YAML-based Model Analyzer configuration.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.7.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.7.0
```
- ›Extends perf_analyzer to work with TFServing and TorchServe, enabling benchmarking across non-Triton serving backends.
- ›Model Analyzer parameters and test model configurations can now be specified via a YAML configuration file.
- ›Model Analyzer now reports performance metrics for end-to-end latency and CPU memory usage.
- ›Adds example Java and Scala clients based on the gRPC-generated API.
- ›Alpha Windows release (tritonserver2.7.0-win.zip) ships with TensorRT 7.2.2 support and a GRPC endpoint.
+1 moreshow less
- ›JetPack 4.5 release (tritonserver2.7.0-jetpack4.5.tgz) adds support for TensorFlow 2.4.0, TensorFlow 1.15.5, TensorRT 7.1, Custom backends, ensembles, and system shared memory on Jetson.
v2.6.0 Dec 18, 2020 · issue -419

Triton 2.6.0 adds alpha Windows support, Model Analyzer, and a reorganized SDK container with renamed image tags.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.6.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.6.0
```
└──▷ TRY IT
Run Triton on a Jetson device with an explicit backend directory and TensorFlow 2.x selected — required when the default backend path or TF version is not appropriate.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
```
Install the Python client library on a Jetson JetPack 4.4 system from the bundled wheel included in the release tarball.
```
$ python3 -m pip install --upgrade clients/python/tritonclient-2.6.0-py3-none-linux_aarch64.whl[all]
```
- ›Renames the *-py3-clientsdk container to *-py3-sdk; the renamed image now bundles the Model Analyzer alongside client libraries and examples.
- ›Initial release of the Model Analyzer tool, available in the Triton SDK container and as the nvidia-triton-model-analyzer PIP package on the NVIDIA Py Index.
- ›Moves the PyTorch backend to a separate repository (https://github.com/triton-inference-server/pytorch_backend), enabling it to be added or removed without rebuilding Triton via the compose workflow.
- ›Alpha release of Triton for Windows (tritonserver2.6.0-win.zip), supporting TensorRT 7.2.2 models over the GRPC endpoint (HTTP/REST, Prometheus metrics, and shared memory are not yet supported).
- ›Adds --backend-directory flag to explicitly set the backend path, enabling Jetson/JetPack deployments where the default path is not used.
+2 moreshow less
- ›Adds --backend-config=tensorflow,version=2 flag to select TensorFlow 2.x on Jetson, where TensorFlow 1.x is the default.
- ›Releases Triton for JetPack 4.4 (tritonserver2.6.0-jetpack4.4.tgz) with support for TensorFlow 2.3.1, TensorFlow 1.15.4, TensorRT 7.1, custom backends, and ensembles.
└──▷ BREAKING ON UPGRADE
- !The *-py3-clientsdk container image is renamed to *-py3-sdk; any pipelines or scripts referencing the old image name will break.
- !The ONNX Runtime OpenVINO execution provider is disabled in this release due to Ubuntu 20.04 interactions; workloads relying on it will not function until it is re-enabled in a future release.
v2.5.0 Nov 20, 2020 · issue -420

Triton v2.5.0 releases ModelAnalyzer, drops Caffe2, and moves PyTorch to a dedicated backend repo.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.5.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.5.0
```
└──▷ TRY IT
Run Triton on Jetson with an explicit backend directory and TensorFlow 2.x selected instead of the default TF 1.x.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
```
- ›Adds --backend-directory flag to explicitly set the backend directory path, required on Jetson deployments.
- ›Adds --backend-config=tensorflow,version=2 to select TensorFlow 2.x when Triton defaults to TensorFlow 1.x.
- ›Releases Triton ModelAnalyzer in the Triton SDK container and as a PIP package available in NVIDIA PyIndex.
- ›Moves the PyTorch backend to a dedicated repository at triton-inference-server/pytorch_backend.
- ›Updates ONNX Runtime backend to use ONNX Runtime 1.5.3.
+1 moreshow less
- ›Adds Jetson JetPack 4.4 support with TensorFlow 2.3.1, TensorFlow 1.15.4, TensorRT 7.1, custom backends, and ensembles (GPU metrics, GCS, and S3 storage not supported on Jetson).
└──▷ BREAKING ON UPGRADE
- !The Caffe2 backend is removed; Caffe2 models are no longer supported.
v2.4.0 Oct 27, 2020 · issue -421

Triton 2.4.0 adds Python and DALI backends, a new Model Analyzer tool, and moves backends to separate repositories.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.4.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.4.0
```
└──▷ TRY IT
Run Triton on Jetson with an explicit backend directory and TensorFlow 2.x selected, since the device requires both flags.
```
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
```
- ›New --backend-directory flag lets you explicitly set the backend directory path when running Triton, required on Jetson deployments.
- ›New --backend-config=tensorflow,version=2 argument allows selecting TensorFlow 2.x when Triton defaults to TensorFlow 1.x.
- ›New Python backend enables running arbitrary Python code as a model within Triton (see github.com/triton-inference-server/python_backend).
- ›New DALI backend enables running pre-processing and data augmentation pipelines directly within Triton (see github.com/triton-inference-server/dali_backend).
- ›New Model Analyzer project provides analysis and guidance for optimizing single or multiple models within Triton; initial release analyzes GPU memory usage (see github.com/triton-inference-server/model_analyzer).
+1 moreshow less
- ›TensorFlow, ONNX Runtime, Python, and DALI backends are now hosted in separate repositories, enabling independent versioning and distribution.
└──▷ BREAKING ON UPGRADE
- !The perf_client application is renamed to perf_analyzer; any scripts or automation invoking perf_client will break.
v2.3.0 Oct 2, 2020 · issue -421

Triton v2.3.0 ships the Python client as a pip package and drops the legacy custom backend SDK in favor of the new Triton backend API.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.3.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.3.0
```
- ›Python client library is now installable as a pip package from the NVIDIA PyPI index, replacing the previous tarball-only distribution.
- ›The legacy custom backend API (custom.h) and its SDK are no longer shipped with Triton releases; users are directed to migrate to the new Triton backend API.
└──▷ BREAKING ON UPGRADE
- !The custom backend SDK (custom.h and associated tooling) is no longer included in the Triton release package; new custom backends must use the new Triton backend API, though existing compiled legacy backends continue to work.
v2.2.0 Aug 27, 2020 · issue -423

Triton v2.2.0 adds TensorFlow 2.x support, per-request inference timeouts, and API versioning
└──▷ GET THIS VERSION
```
$ git clone --branch v2.2.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.2.0
```
└──▷ TRY IT
Launch Triton using TensorFlow 2.x instead of the TensorFlow 1.x default — useful when your saved models require TF2 semantics.
```
$ tritonserver --backend-config=tensorflow,version=2 --model-repository=/models
```
- ›Adds TensorFlow 2.x support alongside TensorFlow 1.x, selectable at launch via --backend-config=tensorflow,version=<version> (set <version> to 1 or 2; default is 1).
- ›Adds TRITON_CLIENT_SKIP_EXAMPLES CMake option to disable building client examples.
- ›Adds inference request timeout option to Python and C++ client libraries.
- ›Adds explicit major/minor versioning to TRITONSERVER and TRITONBACKEND APIs.
v2.1.0 Jul 30, 2020 · issue -424

Triton v2.1.0 adds automatic FP16 TensorFlow optimization and TorchVision support in the PyTorch backend.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.1.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.1.0
```
- ›Adds a TensorFlow optimization option that enables automatic FP16 optimization of models served by the server.
- ›Adds TorchVision operations support to the PyTorch backend.
v1.15.0 Jul 30, 2020 · issue -424

Triton v1.15.0 publishes legacy V1 HTTP/REST and GRPC support on a dedicated branch and NGC container.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.15.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.15.0
```
- ›Releases legacy V1 HTTP/REST and GRPC endpoints, plus corresponding client libraries, on GitHub branch r20.07-v1 and NGC container 20.07-v1-py3 for teams that cannot yet migrate to the V2 API.
v2.0.0 Jun 30, 2020 · issue -425

Triton Inference Server v2.0.0 brings updated KFServing HTTP/REST and gRPC protocols with new Python and C++ client libraries.
└──▷ GET THIS VERSION
```
$ git clone --branch v2.0.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.0.0
```
- ›Updates KFServing HTTP/REST and gRPC protocols with corresponding V2 Python and C++ client libraries.
- ›Adds Jetson JetPack 4.4 support via v2.0.0-jetpack4.4ga.tgz, with TensorFlow 1.15.2, TensorRT 7.1, Custom backends, and ensemble support on edge devices.
- ›Client libraries and examples for Ubuntu 18.04 are packaged in v2.0.0_ubuntu1804.clients.tar.gz and installable via the triton*.whl Python wheel.
- ›Custom backend SDK for Ubuntu 18.04 is packaged in v2.0.0_ubuntu1804.custombackend.tar.gz.
- ›Client SDK is also available as an NGC container image.
└──▷ BREAKING ON UPGRADE
- !Migration from Triton V1 to Triton V2 requires significant changes — see the 'Backwards Compatibility' and 'Roadmap' sections of the GitHub README.
v1.14.0 Jun 30, 2020 · issue -425

Triton v1.14.0 publishes legacy V1 HTTP/REST and GRPC support on a dedicated branch and NGC container.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.14.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.14.0
```
- ›Releases legacy V1 HTTP/REST and GRPC endpoints, along with corresponding client libraries, on GitHub branch r20.06-v1 and as NGC container 20.06-v1-py3.
v1.13.0 Jun 1, 2020 · issue -425

Triton v1.13.0 updates KFServing HTTP/REST and gRPC protocols and bumps gRPC to 1.24.0
└──▷ GET THIS VERSION
```
$ git clone --branch v1.13.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.13.0
```
- ›Updates KFServing HTTP/REST and gRPC protocols with corresponding Python and C++ client libraries.
- ›Upgrades gRPC dependency to version 1.24.0.
└──▷ BREAKING ON UPGRADE
- !The Caffe2 backend is deprecated; support for Caffe2 models will be removed in a future release.
v1.12.0 Mar 26, 2020 · issue -428

Triton v1.12.0 adds dynamic batching queue policies, ONNX large-model support, and an experimental GRPC inferencing API.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.12.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.12.0
```
- ›Adds --min-supported-compute-capability flag to allow Triton Server to run on older, otherwise unsupported GPUs.
- ›Adds queuing policies for the dynamic batching scheduler, configurable per model via the model configuration: maximum queue size, timeouts, and priority levels for inference requests.
- ›Supports configuring ONNX Runtime optimization level via the model configuration optimization setting.
- ›Supports large ONNX models where weights are stored in separate files.
- ›Experimental Python client and server support for the community standard GRPC inferencing API.
+1 moreshow less
- ›Experimental Triton release for JetPack 4.4 (Developer Preview), supporting TensorFlow 1.15.2, TensorRT 7.1, Custom backends, and ensembles on Jetson hardware.
v1.11.0 Feb 26, 2020 · issue -429

Triton v1.11.0 adds ragged batching for custom backends, local S3 endpoint support, and TensorRT boolean/shape tensor types.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.11.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.11.0
```
- ›Adds support for local S3 storage endpoints in model repositories, specified as s3://host:port/path/to/repository.
- ›New model configuration option enables the dynamic batcher to create 'ragged' batches for custom backend models, where input/output tensors may have different shapes across batch entries.
- ›Removes LD_LIBRARY_PATH from the inference server container in favor of RUNPATH for shared library resolution.
- ›Adds support for boolean data-type and shape tensors in TensorRT models.
- ›Significantly improves TensorRT backend performance by reducing thread contention, using pinned memory for faster CPU<->GPU transfers, and increasing compute/memory copy overlap.
+2 moreshow less
- ›Reduces memory usage of TensorRT models by sharing weights across multiple model instances.
- ›Updates the Helm chart example Kubernetes deployment to include Prometheus and Grafana support for metrics collection and visualization.
└──▷ BREAKING ON UPGRADE
- !Python 2 support has been fully removed; only Python 3 is supported.
v1.10.0 Jan 28, 2020 · issue -430

Triton v1.10.0 adds JSON-format server status via REST and ordered batching across multiple model instances.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.10.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.10.0
```
└──▷ TRY IT
Retrieve server status in JSON format for programmatic parsing or monitoring pipelines.
```
$ curl http://<server>:8000/api/status?format=json
```
- ›Adds JSON-format server status via the HTTP/REST endpoint GET /api/status?format=json.
- ›Adds a preserve-ordering option to the dynamic batcher so batched requests maintain sequence across multiple model instances (configured via model_config.proto).
v1.9.0 Dec 21, 2019 · issue -431

Triton v1.9.0 adds model warmup, a new repository index API, Oldest sequence-batcher strategy, and enhanced tracing for ensembles.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.9.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.9.0
```
- ›Adds a model warmup option to model configuration, allowing models to be pre-tuned with synthetic or realistic workloads before serving inference requests — particularly useful for frameworks like TensorFlow that optimize on initial inference.
- ›New repository index API (available via shared library API, HTTP, and GRPC) returns an index of all models visible in the model repositories, enabling discovery of models available for loading.
- ›Server status API response now includes the timestamp of the last inference request received for each model.
- ›Adds an Oldest scheduling strategy to the sequence batcher, integrating with the dynamic batcher to improve throughput for models that do not require all requests in a sequence to share the same batch slot.
- ›Adds a perf_client option to generate requests using a Poisson distribution or a user-provided distribution for more realistic load testing.
+2 moreshow less
- ›Extends tracing support to cover ensembles and their contained models.
- ›Adds a community-contributed Dockerfile for building inference server clients on CentOS.
└──▷ BREAKING ON UPGRADE
- !The beta custom backend API v2 CustomGetNextInputV2Fn_t function signature adds the memory_type_id argument — existing custom backends using this function will break.
- !The beta custom backend API v2 CustomGetOutputV2Fn_t function signature adds the memory_type_id argument — existing custom backends using this function will break.
- !The beta library API TRTSERVER_ResponseAllocatorAllocFn_t function signature and behavior has changed — see src/core/trtserver.h for the new behavior; existing callers will break.
- !The beta library API TRTSERVER_InferenceRequestProviderSetInputData function signature adds the memory_type_id argument — existing callers will break.
- !The beta library API TRTSERVER_InferenceResponseOutputData function signature adds the memory_type_id argument — existing callers will break.
v1.8.0 Nov 27, 2019 · issue -432

Triton v1.8.0 adds CUDA shared memory support and string tensor handling in perf_client
└──▷ GET THIS VERSION
```
$ git clone --branch v1.8.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.8.0
```
- ›Adds CUDA shared memory support, expanding shared-memory capabilities beyond CPU/pinned memory.
- ›Adds string tensor support to perf_client, enabling performance benchmarking of string-typed inputs and outputs.
- ›Improves perf_client command-line arguments for easier use while maintaining compatibility with existing arguments.
- ›Improves pinned-memory efficiency for ensemble models.
└──▷ BREAKING ON UPGRADE
- !In the C++ client library, the non-callback version of AsyncRun is removed.
- !In the C++ client library, GetReadyAsyncRequest is removed.
- !In the C++ client library, the signature of GetAsyncRunResults is changed — the is_ready and wait arguments are removed.
- !In the Python client library, the non-callback version of async_run is removed.
- !In the Python client library, get_ready_async_request is removed.
- !In the Python client library, the signature of get_async_run_results is changed — the wait argument is removed.
v1.7.0 Oct 30, 2019 · issue -433

Triton v1.7.0 adds TensorRT optimization for TensorFlow, ONNXRuntime TensorRT/OpenVINO providers, multi-repo support, and a Go GRPC client example.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.7.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.7.0
```
- ›Adds --strict-model-config=false support for TensorRT models with variable-sized inputs and/or outputs, enabling automatic configuration generation for those models.
- ›Multiple model repositories can now be specified on the command line, with optional flags to explicitly load specific models from each repository.
- ›TensorRT optimization can now be enabled for any TensorFlow model via the optimization section of the model configuration.
- ›The ONNXRuntime backend now includes TensorRT and OpenVINO execution providers, configurable via the optimization section of the model configuration.
- ›Ensemble models are now pruned dynamically so only models required to compute the requested outputs are executed.
+2 moreshow less
- ›A Client SDK container is now available on NGC, packaging client libraries and examples separately from the inference server container.
- ›Example clients now include a Go example using the GRPC API.
v1.6.0 Sep 27, 2019 · issue -434

Triton v1.6.0 adds TensorRT 6 dynamic shapes, shared memory tensors, S3 model repos, and a new library API.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.6.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.6.0
```
- ›Supports Amazon S3 as a remote model repository via the s3:// prefix on model repository paths.
- ›Adds shared memory support (alpha) so input and output tensors can be communicated via system (CPU) shared memory instead of over the network, reducing data-copy overhead.
- ›Introduces a libtrtserver.so library API (beta) that lets applications link the full inference server functionality directly instead of communicating over HTTP/gRPC.
- ›Adds TensorRT 6 support, including dynamic shapes.
- ›Extends perf_client to support models with variable-sized input tensors.
+3 moreshow less
- ›The gRPC endpoint now uses significantly less memory while delivering higher throughput.
- ›The ensemble scheduler now allows batching and non-batching models to be composed together.
- ›The ensemble scheduler retains tensors in GPU memory between models when possible, avoiding round-trips through system memory.
v1.5.0 Sep 3, 2019 · issue -434

Triton v1.5.0 adds a GRPC/HTTP model control API, multi-GPU TF distribution, and a C++ custom backend SDK wrapper.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.5.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.5.0
```
- ›Adds a new GRPC/HTTP model control API for dynamically loading and unloading models at runtime, enabling an execution mode where the server starts with no models loaded from the model repository.
- ›Adds a new instance-group mode for TensorFlow models to explicitly distribute inferencing across multiple GPUs within the inference server.
- ›Extends perf_client to report per-model statistics and aggregate ensemble statistics for ensemble models.
- ›Adds a C++ wrapper around the custom backend C API in the custom backend SDK to simplify authoring custom backends.
- ›Improves input/output tensor reshape to support variable-sized dimensions.
v1.4.0 Jul 30, 2019 · issue -436

Triton v1.4.0 adds PyTorch/TorchScript backend, Custom Backend SDK, and richer perf_client latency reporting.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.4.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.4.0
```
- ›Adds --strict-model-config=false improvements to automatic model configuration creation, reducing manual config overhead.
- ›New libtorch backend enables running PyTorch models decorated or traced to TorchScript directly in the inference server.
- ›Custom Backend SDK now available with improved documentation to simplify building custom backends, distributed as v1.4.0_ubuntu1604.custombackend.tar.gz and v1.4.0_ubuntu1804.custombackend.tar.gz.
- ›perf_client now accepts user-supplied input data instead of being limited to random or zero values.
- ›perf_client now reports latency at multiple confidence percentiles (p50, p90, p95, p99) plus a user-supplied percentile that also drives latency stabilization.
+3 moreshow less
- ›C++ and Python client libraries now support specifying additional HTTP headers when using the HTTP protocol.
- ›Improved AsyncRun API in C++ and Python client libraries.
- ›Build system migrated from Bazel to CMake for improved portability and modularity.
v1.3.0 Jun 28, 2019 · issue -437

Triton v1.3.0 adds ONNX Runtime integration and an independent --http-health-port flag.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.3.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.3.0
```
└──▷ TRY IT
Run the inference server with the health check on a dedicated port so load-balancer probes never share bandwidth with inference traffic.
```
$ trtserver --http-health-port=8080 --http-port=8000 --grpc-port=8001 --model-store=/models
```
- ›Adds --http-health-port flag to specify the HTTP health endpoint port independently from the inference and status HTTP port.
- ›Integrates the ONNX Runtime, enabling ONNX models to be used directly in a model repository without conversion.
v1.2.0 May 24, 2019 · issue -438

Triton v1.2.0 adds model ensembling pipelines, Kubernetes Helm chart, and new perf_client latency flags
└──▷ GET THIS VERSION
```
$ git clone --branch v1.2.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.2.0
```
└──▷ TRY IT
Benchmark a model under realistic SLO conditions by reporting p99 latency instead of average latency.
```
$ perf_client --percentile=99 -m <model_name> -u <server_url>
```
Run a zero-input benchmark to isolate server-side compute latency from data-generation overhead.
```
$ perf_client -z -m <model_name> -u <server_url>
```
- ›Adds --percentile flag to perf_client to report Nth-percentile latency (e.g. --percentile=99 for p99) instead of average latency.
- ›Adds -z flag to perf_client to use zero-valued input tensors instead of random values during benchmarking.
- ›Adds --allow-gpu-metrics option to enable or disable reporting of GPU metrics.
- ›Introduces ensemble model support: a single inference request to an ensemble triggers execution of an entire pipeline of chained models with connected input/output tensors.
- ›Adds a Helm chart for deploying the inference server into a Kubernetes cluster.
+1 moreshow less
- ›Client library builds now support both Ubuntu 16.04 and Ubuntu 18.04, with Python wheels compatible with both Python 2 and Python 3.
v1.1.0 Apr 24, 2019 · issue -439

Triton v1.1.0 adds zero-sized tensor support, TensorFlow custom C++ ops, and a standalone client build system.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.1.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.1.0
```
- ›Supports zero-sized input and output tensors for batching models, enabling models that require inputs and outputs with shape [ batch-size ].
- ›Supports variable-size tensor dimensions (indicated by -1 in model configuration) where the variable dimension has value 0.
- ›TensorFlow custom operations (C++) can now be built directly into the inference server, with an included example and documentation.
- ›Client libraries and examples now build via a separate Makefile (with an accompanying Dockerfile), decoupled from the main server build.
v1.0.0 Mar 18, 2019 · issue -440

TensorRT Inference Server 1.0.0 GA adds stateful model scheduling, GRPC streaming, and async HTTP for lower-latency inference.
└──▷ GET THIS VERSION
```
$ git clone --branch v1.0.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.0.0
```
- ›Adds the *sequence batcher* to schedule and batch stateful models that require multiple inference requests routed to the same model instance/batch slot.
- ›Adds GRPC streaming protocol support for inference requests.
- ›HTTP front-end is now asynchronous, enabling lower-latency and higher-throughput inference handling.
- ›Enhances perf_client to support stateful models and backends.
v0.11.0 Feb 28, 2019 · issue -441

Triton v0.11.0 adds variable-size tensor support, string datatype tensors, and CPU-only (non-GPU) deployment.
└──▷ GET THIS VERSION
```
$ git clone --branch v0.11.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v0.11.0
```
- ›Supports variable-size input and output tensors in model configuration by setting a dimension size of -1 for any dimension that can take on any size.
- ›Adds string datatype support for input and output tensors in TensorFlow models and custom backends.
- ›Enables inference server operation on systems without GPUs, nvidia-docker, or CUDA installed.
v0.10.0 Jan 28, 2019 · issue -442

Triton Inference Server v0.10.0 adds custom backend support for models implemented outside deep-learning frameworks.
└──▷ GET THIS VERSION
```
$ git clone --branch v0.10.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v0.10.0
```
- ›Adds custom backend support, allowing individual models to implement arbitrary logic while retaining GPU support, concurrent execution, and dynamic batching from the server.
v0.9.0 Dec 20, 2018 · issue -443

TRTIS 0.9.0 adds live model repository reloading, priority controls, and a --strict-model-config=false auto-config flag.
└──▷ GET THIS VERSION
```
$ git clone --branch v0.9.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v0.9.0
```
└──▷ TRY IT
Start the inference server without requiring a hand-authored config.pbtxt for supported model types, letting the server auto-derive configuration.
```
$ trtserver --strict-model-config=false --model-store=/models
```
- ›Adds --strict-model-config=false flag to allow automatic derivation of model configuration properties, removing the need for a config.pbtxt file for some model types.
- ›Adds a model priority parameter to model configuration, controlling CPU thread priority for all models and CUDA stream priority for TensorRT models.
- ›Live model repository monitoring now dynamically reloads models without a server restart — supports adding/removing model versions, adding/removing entire models, modifying config.pbtxt, and updating model labels at runtime.
- ›Asynchronous GRPC frontend delivers improved inference throughput.
└──▷ BREAKING ON UPGRADE
- !The GRPC API model version parameter type has changed from string to int — existing clients passing a string model version will break.

Triton Inference Server

Summary

What Triton Inference Server answers

Examples

Command line

Release history