The AI Toolchain — issue -416, March 27, 2021

ONNX Runtime v1.7.0 adds audio operators, QDQ quantization, Model Building API, CUDA 11, and broader platform support.

$ git clone --branch v1.7.0 https://github.com/microsoft/onnxruntime.git
# already have the repo? check out this version:
$ git checkout v1.7.0

• ›Experimental audio operators — Fourier Transforms (DFT, IDFT, STFT), Windowing Functions (Hann, Hamming, Blackman), and MelWeightMatrix — added under the com.microsoft.experimental domain, buildable with the ms_experimental build flag.
• ›Experimental Model Building API available under the Microsoft.AI.MachineLearning.Experimental namespace (included in the Microsoft.AI.MachineLearning NuGet package) for creating dynamic models on the fly with hardware-accelerated tensor featurization.
• ›WinML ImageFeatureDescriptor runtimeclass gains a new ImageNominalPixelRange property exposing native support for pixel ranges [0..1] and [-1..1] in image models, alongside existing BitmapPixelFormat and BitmapAlphaMode properties.
• ›Quantization tool now supports QDQ (QuantizeLinear-DequantizeLinear) format models.
• ›Python transformer optimizer tool now supports fusion for BART models.

Triton 2.8.0 adds Repository Agent C API, OpenVINO backend, SSL/TLS mutual auth for gRPC, and ONNX support on Windows.

$ git clone --branch v2.8.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.8.0

• ›Adds gather_kernel_buffer_threshold Model Configuration option to instruct Triton to use a CUDA kernel to gather input buffers onto the GPU, improving inference performance for some models.
• ›Introduces Repository Agent, a new extensibility C API for implementing custom authentication, decryption, conversion, or similar operations at model-load time.
• ›Adds an OpenVINO backend enabling execution of OpenVINO models on CPUs.
• ›Adds SSL/TLS Mutual Authentication support to the GRPC client library.
• ›Adds system shared-memory protocol support between clients and the Triton server on Jetson deployments.

Milvus v1.0.0 adds stdout log output support.

$ git clone --branch v1.0.0 https://github.com/milvus-io/milvus.git
# already have the repo? check out this version:
$ git checkout v1.0.0

• ›Supports writing log output to stdout.

Weaviate v1.2.0 adds out-of-the-box transformer NLP model support via the text2vec-transformers module with GPU-friendly microservice architecture.

$ git clone --branch v1.2.0 https://github.com/weaviate/weaviate.git
# already have the repo? check out this version:
$ git checkout v1.2.0

{
  "class": "Article",
  "moduleConfig": {
    "text2vec-transformers": {
      "poolingStrategy": "cls"
    }
  }
}

• ›Adds ENABLE_MODULES=text2vec-transformers environment variable to enable transformer-based vectorization (BERT, DistilBERT, RoBERTa, etc.) without custom code.
• ›Adds DEFAULT_VECTORIZER_MODULE=text2vec-transformers environment variable to set transformers as the default vectorizer across all schema classes.
• ›Adds TRANSFORMERS_INFERENCE_API environment variable to point Weaviate at a separately hosted inference container, enabling GPU-optimized model serving independent of Weaviate's CPU-optimized core.
• ›Adds poolingStrategy class-level module config for the text2vec-transformers module, accepting masked_mean or cls to control how sentence vectors are derived from word vectors.
• ›Supports vectorizeClassName, vectorizePropertyName, and skip module-configuration fields on classes and properties for the text2vec-transformers module, mirroring the existing text2vec-contextionary API.