SWE-bench

MIT

Real GitHub issues with tests, used to measure whether a coding agent can resolve them end to end.

source ↗ ★ 5,818 last push 2026-09-02

Summary

From the project’s own instructions where it documents any; otherwise a plain clone.

shell

$ git clone https://github.com/SWE-bench/SWE-bench.git

shell
```
$ cd SWE-bench && pip install -e .
```
Evaluation runs in Docker, so install that first. The task instances are on Hugging Face (SWE-bench, Lite, Verified, Multimodal).

679 files 78.7 MB repository

Commands are curated, not yet run by us.

jq — Stream the records one per line to see their shape.

sample.jsonl

$ jq -c . sample.jsonl

DuckDB — Query the file in place with SQL — DuckDB detects CSV, JSON and JSONL by extension.

sample.jsonl

$ duckdb -c 'SELECT * FROM '\''sample.jsonl'\'' LIMIT 10'