SYLIN

ARC-AGI

Benchmark · Text & code

A researcher found Opus 5 solves games similar to ARC-AGI-3 well but does worse than Opus 4.8 on novel-rule games, suggesting benchmark overfitting

Versions

VersionReleasedRank
ARC-AGI-3LATEST
ARC-AGI-2

Evidence

  1. Sep 27
    Opus 5.5 not yet on the leaderboard at recording
  2. Sep 7
    GPT-6 Astra scores over 60% on the ARC-AGI-3 leaderboard, near 100% with an extra harness; other frontier models score under 10%
  3. Sep 6
    GPT-6 Astra at max effort scores above 60% while most frontier models stay under 10%; near 100% with a harness
  4. Sep 3
    Astra used 51.7% fewer actions than the median human baseline per level, beating it on 96.0% of levels (max, Provider Adapter)
  5. Aug 23
    Most frontier models score under 10% and Claude Opus 5 about 30% on ARC-AGI-3 without a special harness
Show all 6
  1. Jul 26
    Opus 5 scores over 30% versus about 7% for GPT Sol and under 1% for others; reviewer suspects familiar game types, worse than Opus 4.8 on novel ones

Alternatives

dots NoteCharts show it leading its weight class on agentic and reasoning suites; on ARC-AGI-3 behind only Opus 4.8