DeepSWE Measuring frontier coding agents
DeepSWE is a new benchmark designed to evaluate frontier coding agents on complex software engineering tasks. It offers significant improvements over existing benchmarks by ensuring task originality, high diversity, and real-world complexity. The benchmark aims to provide a more accurate reflection of how advanced coding models perform in practical scenarios.
- ▪DeepSWE tasks are created from scratch, avoiding contamination from pretraining data.
- ▪The benchmark includes tasks from 91 repositories across five programming languages.
- ▪Solutions to DeepSWE tasks require significantly more code and output tokens compared to previous benchmarks.
Opening excerpt (first ~120 words) tap to expand
@keyframes deepswe-waves-draw { from { stroke-dashoffset: 1; } to { stroke-dashoffset: 0; } } .deepswe-waves-path { stroke: currentColor; stroke-opacity: 0.32; stroke-width: 3; stroke-dasharray: 1; stroke-dashoffset: 1; animation-name: deepswe-waves-draw; animation-fill-mode: forwards; animation-timing-function: cubic-bezier(0.25, 0.1, 0.25, 1); } @media (prefers-reduced-motion: reduce) { .deepswe-waves-path { animation: none; stroke-dashoffset: 0; } } DeepSWEbyMeasuring frontier coding agents on original, long-horizon engineering tasksRead the blogRun DeepSWEToday's leading public coding benchmarks are starting to saturate at the frontier: top models cluster within a narrow score band where adjacent configurations often overlap on confidence intervals.
…
Excerpt limited to ~120 words for fair-use compliance. The full article is at DeepSWE.
Discussion
0 commentsMore from DeepSWE
