← all datasets
from the project README
◆ Evals & benchmarks
Sierra tau2-bench
MITSierra's τ²-bench — agents must follow policy while using tools with a simulated user in airline, retail and telecom domains.
Summary
🚀 τ³-bench is here! From text-only to multimodal, knowledge-aware agent evaluation. Voice full-duplex · Knowledge retrieval · 75+ task fixes τ-Voice paper · τ-Knowledge paper · Task fixes paper · Release notes How do you say $\tau^3$-bench? We just say "tau three," but you do you!
- Formats
json- License
- MIT
- Added
- 2026-09-10
Get it
From the project’s own instructions where it documents any; otherwise a plain clone.
-
$ git clone https://github.com/sierra-research/tau2-bench
-
$ cd tau2-bench && uv sync
-
$ tau2 run --domain airline --agent-llm gpt-4.1 --user-llm gpt-4.1 --num-trials 1 --num-tasks 5
Put provider keys in .env first (cp .env.example .env). Models go through LiteLLM, so any supported provider works.
Contents
1,570 files 359.8 MB repository
-
.json927 -
.py335 -
.md80 -
.png34 -
.pdf28 -
.jpg23 -
.wav23 -
.html20
Use it with
Commands are curated, not yet run by us.
jq — Stream the records one per line to see their shape.
$ jq -c . sample.json
DuckDB — Query the file in place with SQL — DuckDB detects CSV, JSON and JSONL by extension.
$ duckdb -c 'SELECT * FROM '\''sample.json'\'' LIMIT 10'