Triton v2.21.0 adds raw binary requests, custom metrics via C API, multi-cloud model repos, and ensemble optional inputs.
$ git clone --branch v2.21.0 https://github.com/triton-inference-server/server.git # already have the repo? check out this version: $ git checkout v2.21.0
- ›Adds
--tmp-dirargument tobuild.pyto specify a custom temporary directory during container builds. - ›Supports raw binary HTTP requests to the inference endpoint, eliminating the need to specify an inference header.
- ›Adds custom metrics support via the Triton C API, allowing custom backends and applications to publish metrics to the existing Triton metrics endpoint.
- ›Supports loading models from multiple cloud storage repositories simultaneously — including multiple S3 buckets, GCS buckets, and Azure Storage containers — in a single Triton instance.
- ›Ensembles now recognize optional inputs, enabling more flexible ensemble pipeline configurations.
+2 moreshow less
- ›ONNX Runtime backend now uses available execution providers (e.g. GPU/TensorRT) when autocomplete is enabled, rather than always defaulting to the CPU execution provider.
- ›Adds PyTorch and TensorFlow 1 backend support to
build.pyandcompose.pyfor CPU-only builds.