Triton v2.15.0 adds rate limiting, response caching, string sequence IDs, ARM SBSA beta, and Windows HTTP support.
$ git clone --branch v2.15.0 https://github.com/triton-inference-server/server.git # already have the repo? check out this version: $ git checkout v2.15.0
$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends --backend-config=tensorflow,version=2
- ›Adds rate limiter that manages the rate at which requests are scheduled on model instances, configurable via
docs/rate_limiter.md. - ›Adds support for caching responses to inference requests.
- ›Sequence IDs (via the sequence extension protocol) can now accept string values in addition to integers.
- ›Container composer tool gains the ability to generate CPU-only Triton containers (see
docs/compose.md#cpu-only-container-composition). - ›Windows Triton build now supports the HTTP/REST protocol (previously GRPC only).
+3 moreshow less
- ›Beta release of Triton is available for ARM SBSA platforms.
- ›Windows alpha release now supports OpenVINO models at version 2021.2 and ONNX models via ONNX Runtime 1.9 with CPU, CUDA, and TensorRT execution providers.
- ›Jetson JetPack 4.6 release adds support for TensorFlow 2.6.0, TensorFlow 1.15.5, TensorRT 8.0.1.6, OnnxRuntime 1.8.1, and ensembles.