ARC-AGI
Benchmark · Text & code
A researcher found Opus 5 solves games similar to ARC-AGI-3 well but does worse than Opus 4.8 on novel-rule games, suggesting benchmark overfitting
Versions
| Version | Released | Rank |
|---|---|---|
| ARC-AGI-3LATEST | ||
| ARC-AGI-2 |
Evidence
- Sep 27Opus 5.5 not yet on the leaderboard at recording
- Sep 7GPT-6 Astra scores over 60% on the ARC-AGI-3 leaderboard, near 100% with an extra harness; other frontier models score under 10%
- Sep 6GPT-6 Astra at max effort scores above 60% while most frontier models stay under 10%; near 100% with a harness
- Sep 3Astra used 51.7% fewer actions than the median human baseline per level, beating it on 96.0% of levels (max, Provider Adapter)
- Aug 23Most frontier models score under 10% and Claude Opus 5 about 30% on ARC-AGI-3 without a special harness
Show all 6
- Jul 26Opus 5 scores over 30% versus about 7% for GPT Sol and under 1% for others; reviewer suspects familiar game types, worse than Opus 4.8 on novel ones