Questions Raised Over OpenAI’s Benchmark Claims for o3 Model

In partnership with

In Today’s Issue:

• "Please, thank you" is costing your LLM millions

• Questions Raised Over OpenAI’s Benchmark Claims for o3 Model

Read time: 2 minutes

A new benchmark test has cast doubt on OpenAI’s performance claims for its o3 model. When o3 was introduced in December, OpenAI said it could solve over 25% of problems in FrontierMath, a tough math benchmark. That result was far ahead of other models, which scored below 2%.

But according to new data from Epoch AI, o3 actually scored closer to 10% — far lower than the original figure. The discrepancy likely stems from differences in compute power and test setups. OpenAI used a more powerful internal version of o3 for its original tests, while the public version released last week is optimized for speed and cost.

OpenAI has not denied the gap. A technical staff member said in a livestream that the public release was tuned for “real-world use cases,” and might show “disparities” on benchmarks.

The ARC Prize Foundation, which tested an earlier version of o3, confirmed that the current public version is “a different model,” with smaller compute tiers than the one OpenAI originally benchmarked.

Benchmark differences aren’t new in the AI world. Meta and xAI have also faced scrutiny over mismatched testing setups. For many, it’s another reminder not to take AI benchmark claims at face value — especially when they come from the companies behind the models.