Haystack v1.3.0 adds PineconeDocumentStore, BEIR benchmarking integration, YAML pipeline validation, and new RouteDocuments/JoinAnswers nodes.
$ git clone --branch v1.3.0 https://github.com/deepset-ai/haystack.git # already have the repo? check out this version: $ git checkout v1.3.0
from pathlib import Path
from haystack.pipelines.config import validate_yaml
validate_yaml(Path('rest_api/pipeline/pipelines.haystack-pipeline.yml')) import os from haystack.document_stores import PineconeDocumentStore document_store = PineconeDocumentStore(api_key=os.environ['PINECONE_API_KEY'])
from haystack.pipelines import DocumentSearchPipeline, Pipeline
from haystack.nodes import ElasticsearchRetriever
from haystack.document_stores.elasticsearch import ElasticsearchDocumentStore
document_store = ElasticsearchDocumentStore(search_fields=['content', 'name'], index='scifact_beir')
retriever = ElasticsearchRetriever(document_store=document_store, top_k=1000)
query_pipeline = DocumentSearchPipeline(retriever=retriever)
ndcg, _map, recall, precision = Pipeline.eval_beir(
index_pipeline=index_pipeline, query_pipeline=query_pipeline, dataset='scifact'
) - ›Adds validate_yaml(Path(...)) from
haystack.pipelines.configto programmatically validate pipeline YAML files, identifying erroneous components and parameters. - ›Adds
PineconeDocumentStoretohaystack.document_stores, backed by Pinecone's managed vector database for large-scale dense retrieval; requires only aPINECONE_API_KEY. - ›Adds Pipeline.eval_beir() for zero-shot benchmarking of retrieval pipelines against BEIR datasets in 17 languages; available via
pip install farm-haystack[beir]. - ›Adds
RouteDocumentsandJoinAnswerspipeline nodes tohaystack.nodes. - ›Adds deploy and undeploy support for Pipelines on Deepset Cloud.
+4 moreshow less
- ›Adds
*.haystack-pipeline.ymlfile suffix convention enabling IDE schema validation and autocompletion via SchemaStore; schema published athttps://raw.githubusercontent.com/deepset-ai/haystack/master/haystack/json-schemas/haystack-pipeline.schema.json. - ›Supports
version: 'unstable'in pipeline YAML files to bypass schema validation. - ›Reintroduces
debugas a valid global key in Pipelineparams. - ›Adds bulk insert support to SQL DocumentStores.
- !
Milvus2DocumentStorenow requirespymilvus>=2.0.0; setups using older pymilvus versions will break. - !The
deviceparameter in internal methods is now atorch.device; code passing plain strings fordevicein affected onnxruntime paths may break.