Which CVEs should I add to my Python security benchmark for AI agents?
CVE-Bench A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities. Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite. Building Docker images python build.py This builds: A shared base image (cve-bench/base) — Python 3.12, git, poetry, and the harness.
- ▪CVE-Bench A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities.
- ▪Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite.
- ▪Building Docker images python build.py This builds: A shared base image (cve-bench/base) — Python 3.12, git, poetry, and the harness.
Opening excerpt (first ~120 words) tap to expand
CVE-Bench A benchmark for evaluating LLM agents on fixing real-world security vulnerabilities. Agents run inside sandboxed Docker containers and are scored against the maintainer's security test suite. Requirements Python 3.12+ Docker OPENAI_API_KEY, ANTHROPIC_API_KEY, and/or POOLSIDE_API_KEY in your environment (or a .env file) Install dependencies: pip install poetry poetry install Task structure Each task lives under tasks/{CVE-ID}/ and contains: tasks/CVE-2026-33175/ ├── meta.json # GHSA ID, CWE, CVSS, repo URL, vulnerable and fixed SHAs ├── setup.sh # Clones repo, checks out the vulnerable SHA, installs dependencies ├── run_tests.sh # Injects test_security.py into the repo and runs pytest ├── test_security.py # Security tests (xfail on vulnerable code, pass on the fix) ├──…
Excerpt limited to ~120 words for fair-use compliance. The full article is at GitHub.