← all datasets
from the project README
◆ Evals & benchmarks
Microsoft SWE-bench-Live
MITA continuously refreshed SWE-bench built from recent GitHub issues across languages and platforms, so results can't be explained by training-data contamination.
Summary
A brand-new, continuously updated SWE-bench-like dataset powered by an automated curation pipeline. SWE-bench-Live is the first automatically-updating, multi-language and multi-os SWE task set designed for agentic benchmarking and training.
- Formats
jsonl- License
- MIT
- Added
- 2026-09-10
Get it
From the project’s own instructions where it documents any; otherwise a plain clone.
-
$ git clone https://github.com/microsoft/SWE-bench-Live.git
-
$ cd SWE-bench-Live && pip install -e .
Python 3.10 or later. The task instances are on Hugging Face; evaluation runs each one in Docker.
Contents
37 files 2.2 MB repository
-
.py21 -
.md8 -
(no ext)3 -
.png2 -
.sh2 -
.toml1
Use it with
Commands are curated, not yet run by us.
jq — Stream the records one per line to see their shape.
$ jq -c . sample.jsonl
DuckDB — Query the file in place with SQL — DuckDB detects CSV, JSON and JSONL by extension.
$ duckdb -c 'SELECT * FROM '\''sample.jsonl'\'' LIMIT 10'