The AI Toolchain — issue -413, June 25, 2021

Haystack v0.9.0 adds LFQA generative QA, a Ranker node, WeaviateDocumentStore, QueryClassifier, and ONNXRuntime support.

$ git clone --branch v0.9.0 https://github.com/deepset-ai/haystack.git
# already have the repo? check out this version:
$ git checkout v0.9.0

from haystack.document_store import WeaviateDocumentStore

document_store = WeaviateDocumentStore()
document_store.write_documents(documents, duplicate_documents="overwrite")

• ›Adds WeaviateDocumentStore class (from haystack.document_store) for combined vector search and scalar filtering, using Weaviate 1.4.0.
• ›Adds FARMRanker node for document re-ranking via semantic similarity, composable with any retriever in a Pipeline.
• ›Adds Seq2SeqGenerator and RetriBERT-based retriever for Long-Form Question Answering (LFQA), generating multi-document synthesized answers.
• ›Adds QueryClassifier node to route keyword queries vs. natural-language questions to different pipeline branches.
• ›Adds use_amp parameter to the DPR retriever train() method to enable mixed-precision training.

ONNX Runtime v1.8.0 adds ONNX opset 14, dynamically loadable CUDA EP, ORT Web JS API, mobile packages, and a wave of new C APIs.

$ git clone --branch v1.8.0 https://github.com/microsoft/onnxruntime.git
# already have the repo? check out this version:
$ git checkout v1.8.0

• ›Adds --enable_type_reduction flag when creating ORT format models and --enable_reduced_operator_type_support flag for minimal builds to support per-type reduction across all operators in the ai.onnx domain.
• ›Adds new C inferencing APIs: KernelInfoGetAttributeArray_float, KernelInfoGetAttributeArray_int64, CreateArenaCfgV2, AddRunConfigEntry, CreatePrepackedWeightsContainer, PrepackedWeightsContainer, CreateSessionWithPrepackedWeightsContainer, and CreateSessionFromArrayWithPrepackedWeightsContainer.
• ›Adds SetIntraOpThreadSpinning native WinML API to toggle intra-op thread spin behavior (enabled by default), improving throughput for current workloads.
• ›Supports ONNX opset 14 with new and updated operators from the ONNX 1.9 release.
• ›Introduces a dynamically loadable CUDA execution provider, allowing a single build to serve both CPU and GPU inference (excluding Python packages).

Triton v2.11.0 adds FIL backend for forest models, OpenVINO on Windows, C API benchmarking, conda Python environments, and MIG GPU metrics.

$ git clone --branch v2.11.0 https://github.com/triton-inference-server/server.git
# already have the repo? check out this version:
$ git checkout v2.11.0

$ tritonserver --model-repository=/path/to/model_repo --backend-directory=/path/to/tritonserver/backends

• ›Adds the Forest Inference Library (FIL) backend, enabling deployment of forest models trained with XGBoost, LightGBM, Scikit-Learn, and cuML.
• ›Adds perf_analyzer support for benchmarking Triton directly via the C API, in addition to HTTP/gRPC endpoints.
• ›Python backend now supports conda to create isolated execution environments per Python model.
• ›Adds passive model instance support in model configuration (via ModelInstanceGroup in model_config.proto), allowing instances to be loaded and initialized without receiving inference requests.
• ›Model repositories hosted in S3 storage can now be accessed using the HTTPS protocol.

Zed v0.1 debuts with syntax-aware editing, Tree-sitter highlighting, multi-cursor support, and a Metal-based GPU rendering engine.

$ git clone --branch v0.1 https://github.com/zed-industries/zed.git
# already have the repo? check out this version:
$ git checkout v0.1

• ›Adds alt-cmd-[ and alt-cmd-] keybindings for indentation-based code folding and unfolding.
• ›Adds cmd-shift-l to split a selection into lines for multi-cursor editing.
• ›Adds cmd-alt-up and cmd-alt-down for columnar (column-select) editing up and down.
• ›Adds ctrl-a and ctrl-e to move to the beginning and end of a line.
• ›Adds cmd-l to select the current line.

Milvus v1.1.1 adds S3 storage support, GPU index persistence, and raises CPU search topk limit to 1M.

$ git clone --branch v1.1.1 https://github.com/milvus-io/milvus.git
# already have the repo? check out this version:
$ git checkout v1.1.1

• ›Enables S3 storage backend support for Milvus deployments.
• ›Supports keeping indexes resident in GPU memory to avoid reloading overhead between queries.
• ›Raises the CPU search topk limit from 16,384 to 1,000,000, unlocking large-scale nearest-neighbor retrieval.
• ›Raises the index_file_size limit from 4 GB to 128 GB, supporting much larger index files.

Qdrant v0.3.4 adds a scroll API endpoint for paginating over filtered point sets.

$ git clone --branch v0.3.4 https://github.com/qdrant/qdrant.git
# already have the repo? check out this version:
$ git checkout v0.3.4

$ curl -X POST 'http://localhost:6333/collections/{collection_name}/points/scroll' \
  -H 'Content-Type: application/json' \
  -d '{"filter": {"must": [{"key": "city", "match": {"value": "London"}}]}, "limit": 100}'

• ›Adds scroll_points API endpoint for paginating over collections of points with filter support.

$ git clone --branch v0.3.0 https://github.com/qdrant/qdrant.git
# already have the repo? check out this version:
$ git checkout v0.3.0

Weaviate v1.4.0 adds image vectorization via img2vec-neural, a new blob datatype, nearImage search, per-query ef tuning, and full arm64 support.

$ git clone --branch v1.4.0 https://github.com/weaviate/weaviate.git
# already have the repo? check out this version:
$ git checkout v1.4.0

{
  Get {
    MyImage(nearImage: {
      image: "/9j/4AAQSkZJRgABAgE..."
      certainty: 0.7
    }) {
      image
    }
  }
}

{
  "class": "Article",
  "vectorIndexConfig": {
    "skip": false,
    "ef": 100,
    "efConstruction": 128,
    "maxConnections": 64
  }
}

• ›Adds img2vec-neural vectorizer module with imageFields config in moduleConfig to vectorize images using neural networks; resnet50 (pytorch and keras) supported at launch, with pytorch variant supporting amd64, arm64, and CUDA.
• ›Adds nearImage GraphQL search operator to vectorize a query image at search time and retrieve results by image similarity.
• ›Adds "skip": true option in vectorIndexConfig to bypass HNSW vector indexing entirely for classes where vectorization is unnecessary (e.g. reference-only or high-duplicate classes); defaults to false.
• ›Adds ef field to vectorIndexConfig (settable at schema definition and updatable post-creation) to tune HNSW recall/performance trade-off at search time; defaults to -1 (auto).
• ›Introduces new primitive datatype blob for storing arbitrary base64-encoded binary data; blob fields are never indexed in the inverted index, so valueBlob in whereFilters is not supported.