The AI Toolchain — issue -438, May 24, 2019

ONNX Runtime v0.4.0 adds Intel nGraph and NVIDIA TensorRT execution providers, ONNX 1.5/Opset 10 support, and quantization operators.

$ git clone --branch v0.4.0 https://github.com/microsoft/onnxruntime.git
# already have the repo? check out this version:
$ git checkout v0.4.0

• ›Adds Intel nGraph execution provider for accelerated inference on Intel hardware.
• ›Adds NVIDIA TensorRT execution provider for GPU-accelerated inference.
• ›Supports ONNX 1.5 and Opset 10 operators, enabling use of newly added ONNX Model Zoo object detection models (YOLO v3, SSD).
• ›Adds quantization operators from ONNX 1.5.
• ›Extends the C API for Custom Operators to support allocation of outputs during compute, includes a C++ wrapper to simplify implementation, and enables custom op DLLs when ONNX Runtime is compiled statically.

Triton v1.2.0 adds model ensembling pipelines, Kubernetes Helm chart, and new perf_client latency flags

$ git clone --branch v1.2.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v1.2.0

$ perf_client --percentile=99 -m <model_name> -u <server_url>

$ perf_client -z -m <model_name> -u <server_url>

• ›Adds --percentile flag to perf_client to report Nth-percentile latency (e.g. --percentile=99 for p99) instead of average latency.
• ›Adds -z flag to perf_client to use zero-valued input tensors instead of random values during benchmarking.
• ›Adds --allow-gpu-metrics option to enable or disable reporting of GPU metrics.
• ›Introduces ensemble model support: a single inference request to an ensemble triggers execution of an entire pipeline of chained models with connected input/output tensors.
• ›Adds a Helm chart for deploying the inference server into a Kubernetes cluster.

Weaviate 0.13.0: initial stable release with GraphQL traversal, pluggable backends, and OIDC auth support

$ git clone --branch 0.13.0 https://github.com/weaviate/weaviate.git
# already have the repo? check out this version:
$ git checkout 0.13.0

• ›Supports pluggable database backends ('connectors'), defaulting to janusgraph with cassandra for storage and elasticsearch as the indexing backend
• ›Pluggable authentication and authorization providers, defaulting to anonymous_access with optional OIDC (Open ID Connect) configuration
• ›Full REST API for CRUD operations on schema and concepts (Things and Actions)
• ›Dynamic graph traversal via GraphQL, including context-based search through 'Fetch' GQL APIs
• ›Optional asynchronous analytics jobs via Spark integration