You can now configure and run Evals directly in the OpenAI Dashboard. Get started → Evals provide a framework for evaluating large language models (LLMs) or systems built using LLMs. We offer an existing registry of evals to test different dimensions of OpenAI models and the ability to write your own custom evals for use cases you care about. You can also use your data to build private evals which represent the common LLMs patterns in your workflow without exposing any of that data publicly.
from the project README
Formats
yamljsonl
License
license: see source
Added
2026-09-10
Get it
From the project’s own instructions where it documents any; otherwise a plain clone.
shell
$ git clone https://github.com/openai/evals
shell
$ cd evals && git lfs fetch --all && git lfs pull
The registry data is stored in Git LFS. Install it (git-lfs.com) first.
shell
$ pip install evals
Only if you want to run evals rather than read the data.
Contents
1,728 files6.5 MB repository
.jsonl661
.yaml524
.py353
.md42
.txt39
.sh30
(no ext)26
.png13
Use it with
Commands are curated, not yet run by us.
jq — Stream the records one per line to see their shape.
sample.jsonl
$ jq -c . sample.jsonl
DuckDB — Query the file in place with SQL — DuckDB detects CSV, JSON and JSONL by extension.
sample.jsonl
$ duckdb -c 'SELECT * FROM '\''sample.jsonl'\'' LIMIT 10'