Triton v1.4.0 adds PyTorch/TorchScript backend, Custom Backend SDK, and richer perf_client latency reporting.
$ git clone --branch v1.4.0 https://github.com/triton-inference-server/server.git # already have the repo? check out this version: $ git checkout v1.4.0
- ›Adds
--strict-model-config=falseimprovements to automatic model configuration creation, reducing manual config overhead. - ›New
libtorchbackend enables running PyTorch models decorated or traced to TorchScript directly in the inference server. - ›Custom Backend SDK now available with improved documentation to simplify building custom backends, distributed as
v1.4.0_ubuntu1604.custombackend.tar.gzandv1.4.0_ubuntu1804.custombackend.tar.gz. - ›
perf_clientnow accepts user-supplied input data instead of being limited to random or zero values. - ›
perf_clientnow reports latency at multiple confidence percentiles (p50, p90, p95, p99) plus a user-supplied percentile that also drives latency stabilization.
+3 moreshow less
- ›C++ and Python client libraries now support specifying additional HTTP headers when using the HTTP protocol.
- ›Improved
AsyncRunAPI in C++ and Python client libraries. - ›Build system migrated from Bazel to CMake for improved portability and modularity.