The AI Toolchain — issue -398, September 30, 2022

Haystack v1.9.0 adds a health-check endpoint, layout-based PDF extraction, MultipleNegativesRankingLoss for retriever training, and a unified Docker image.

$ git clone --branch v1.9.0 https://github.com/deepset-ai/haystack.git
# already have the repo? check out this version:
$ git checkout v1.9.0

retriever.train(
    data_dir="training_data/",
    train_filename="train.json",
    loss_function="MultipleNegativesRankingLoss"
)

• ›Adds a health check endpoint to the REST API, enabling liveness probes and load-balancer integration.
• ›Adds MultipleNegativesRankingLoss as a training loss option for EmbeddingRetriever when using sentence-transformers.
• ›Adds public layout-based text extraction support to PDFToTextConverter, enabling structure-aware PDF parsing.
• ›Adds exponential backoff with exponentially decreasing batch size for OpenSearch and Elasticsearch clients under load.
• ›Publishes a new unified deepset/haystack Docker image with support for multiple flavors and versions via Docker tags.

Zed v0.57.0 adds anonymous telemetry with a built-in log viewer accessible via zed: open telemetry log.

$ git clone --branch v0.57.0 https://github.com/zed-industries/zed.git
# already have the repo? check out this version:
$ git checkout v0.57.0

• ›Adds zed: open telemetry log command palette command to inspect exactly what anonymous telemetry data Zed has reported.
• ›Adds anonymous telemetry reporting to help inform product development, viewable via Help > View Telemetry Log in the application menu.

$ git clone --branch v0.55.0 https://github.com/zed-industries/zed.git
# already have the repo? check out this version:
$ git checkout v0.55.0

$ git clone --branch v0.54.0 https://github.com/zed-industries/zed.git
# already have the repo? check out this version:
$ git checkout v0.54.0

{
  "terminal": {
    "option_as_meta": true
  }
}

$ git clone --branch v0.53.0 https://github.com/zed-industries/zed.git
# already have the repo? check out this version:
$ git checkout v0.53.0

Milvus 2.1.2 adds configurable SASL mechanism support for Kafka.

$ git clone --branch v2.1.2 https://github.com/milvus-io/milvus.git
# already have the repo? check out this version:
$ git checkout v2.1.2

• ›Supports configurable SASL mechanism for Kafka broker connections.

Qdrant v0.10.0 adds multiple vectors per point, batch search, and full-text filtering across all filterable APIs.

$ git clone --branch v0.10.0 https://github.com/qdrant/qdrant.git
# already have the repo? check out this version:
$ git checkout v0.10.0

$ curl -X PUT 'http://localhost:6333/collections/my_collection/points' \
  -H 'Content-Type: application/json' \
  -d '{
    "points": [
      {
        "id": 1,
        "vectors": {
          "image": [0.9, 0.1, 0.1, 0.2],
          "text": [0.4, 0.7, 0.1, 0.8, 0.1, 0.1, 0.9, 0.2]
        }
      }
    ]
  }'

$ curl -X POST 'http://localhost:6333/collections/my_collection/points/search/batch' \
  -H 'Content-Type: application/json' \
  -d '{
    "searches": [
      { "vector": [0.2, 0.1, 0.9, 0.7], "limit": 3 },
      { "vector": [0.5, 0.3, 0.2, 0.3], "limit": 3 }
    ]
  }'

• ›Adds POST /collections/{collection_name}/points/search/batch endpoint for batching multiple vector queries in a single request, reducing network overhead and sharing filter computation across queries.
• ›Extends PUT /collections/{collection_name}/points to accept multiple named vectors per point (e.g. image and text vectors) alongside the existing anonymous single-vector format, enabling multi-modal embeddings on a single record.
• ›Adds full-text filtering to all filterable APIs, supporting ngram-prefix and word tokenizers to restrict queries to records containing specified words.
• ›Multi-platform Docker images now available for both x86_64 and aarch64, enabling native runs on ARM-based CPUs.
• ›scroll API performance improved by up to 1000x for strict filters.

Weaviate v1.15 adds cloud-native backups to S3/GCS, Manhattan and Hamming distance metrics, HuggingFace and SUM-Transformers modules, and new monitoring metrics.

$ git clone --branch v1.15.0 https://github.com/weaviate/weaviate.git
# already have the repo? check out this version:
$ git checkout v1.15.0

• ›Adds backup-s3 Weaviate module for backing up and restoring to/from AWS S3; requires the target bucket to already exist.
• ›Adds backup-gcs Weaviate module for backing up and restoring to/from Google Cloud Storage; requires the target bucket to already exist.
• ›Supports backing up and restoring multiple classes in a single request.
• ›Adds monitoring for backup and restore operations via Prometheus.
• ›Enables use of GOMEMLIMIT environment variable (Go 1.19) to cap Weaviate memory usage — a significant operational lever for high-memory deployments.