Heads up This site is currently under heavy development.
← all datasets
◆ Evals & benchmarks

Cybench

Apache-2.0

40 professional-level CTF tasks from four competitions, for measuring an LLM agent's offensive cyber capability.

source ↗ ★ 321 last push 2026-07-09

Summary

This repository contains code used in the Cybench project (website, paper), which serves to evaluate agent capabilities and risks on cybersecurity tasks.
from the project README
Formats
source
License
Apache-2.0
Added
2026-09-10

Get it

From the project’s own instructions where it documents any; otherwise a plain clone.

  1. shell 
    $ git clone --depth 1 https://github.com/andyzorigin/cybench.git

    The repository is over 1 GB, so skip history.

  2. shell 
    $ cd cybench && pip3 install -r requirements.txt

    Then fill .env with provider keys and use ./run_task.sh. Tasks run inside Docker.

Contents

7,460 files 1.1 GB repository

  • .png 931
  • .md 786
  • (no ext) 716
  • .js 584
  • .py 515
  • .json 396
  • .sh 356
  • .txt 307
my-toolchain — 0 tools
paste an install list to detect your tools

A brew list, a Brewfile, requirements.txt, a Dockerfile — or just the product names, free-form. Nothing leaves your browser.

    browse all tools →

    Send us feedback