← all datasets
from the project README
◆ Evals & benchmarks
HumanEval
MIT164 hand-written Python problems with unit tests. Old and heavily contaminated — a smoke test, not a benchmark.
Summary
This is an evaluation harness for the HumanEval problem solving dataset described in the paper "Evaluating Large Language Models Trained on Code".
- Formats
jsonl- License
- MIT
- Added
- 2026-09-10
Get it
From the project’s own instructions where it documents any; otherwise a plain clone.
-
$ git clone https://github.com/openai/human-eval
-
$ pip install -e human-eval
Python 3.7 or later.
Contents
12 files 48 KB repository
-
.py6 -
.jsonl2 -
.gz1 -
.md1 -
(no ext)1 -
.txt1
Use it with
Commands are curated, not yet run by us.
jq — Stream the records one per line to see their shape.
$ jq -c . sample.jsonl
DuckDB — Query the file in place with SQL — DuckDB detects CSV, JSON and JSONL by extension.
$ duckdb -c 'SELECT * FROM '\''sample.jsonl'\'' LIMIT 10'