Gut Feel Benchmark

Gut Score averages skill over seven deadlines from 0.25 to 16 seconds, with every late answer counted as the base rate. Skill is the share of the base rate’s squared error that a forecast removes, with no deadline. Both run to 100, and both average the poker, base-rate and Hacker News packs.

Twenty-five items per pack leave wide intervals. Jev beats Microsoft-Decision-1 in 99% of bootstrap resamples, but Microsoft-Decision-1 and Opus with reasoning off are not separated, at 81%, and neither are Sol and Gemini, at 90%.

The decision models keep nearly all their skill from half a second on, while the chat models keep none of it before two seconds. OpenAI Luna scores 24.3 at the quarter-second deadline and 63.3 at half a second, against a skill of 65.1. Opus with reasoning off scores nothing before 2 seconds, 58.0 at 4 seconds and its full 71.2 only at 16.

Each cell below is skill at one deadline, shaded more strongly the more skill it holds.

High reasoning bought Opus no skill. It used 141 reasoning tokens per answer instead of 36, scored 69.9 against 71.2 and slowed from 2.0 to 3.1 seconds, which cost it 6 points of Gut Score.

Reasoning off held for none of the chat models. Sol used 162 reasoning tokens per answer and Gemini 3.8 Flash 2,169, with no warning from the SDK, and Gemini ran past the 120-second limit on 23 of 25 poker hands, so a quarter of its items count as failures.

The exact packs separate the field and the outcome packs barely do. Every chat model and OpenAI Luna scored 100.0 on base rates, and on poker the chat models that answered scored 98.6 to 99.7, against 95.8 for Luna and 80.2 for Jev. On Hacker News no contestant passed 14.7.

Heart disease sits outside the headline, in grey, because its records have been public since 1988 and a model may remember them. Luna scored 6.4 there, while Jev scored 43.0 and Opus 68.5 to 73.1.

The run used $3.51 of AI Gateway credit. A thousand answers from Jev or Luna cost about 2 cents, and a thousand from Opus with high reasoning cost $5.02.

Recorded token counts explain $1.66 of the bill. The rest is most likely Gemini’s 25 timed-out calls, which Gateway appears to bill although they return nothing.

Each contestant answered the first 25 items of four packs. Poker hands are scored against exact equity over all 1,712,304 boards, and screening-test problems against the exact posterior by Bayes’ rule. Cleveland Clinic patients are scored against their angiograms, and Hacker News stories posted from 2 to 9 October 2026 against whether they reached 100 points.

Every item went through AI SDK 7’s experimental_decide on Vercel AI Gateway, natively for the decision models and through a structured-output adapter for the chat models. Calls ran four at a time with a 120-second limit and no retries, after one untimed warm-up call per contestant. Gateway answered a request that does no model work in 40 to 56 milliseconds, so network time is a small share of every latency.

No deadline scores above a contestant’s skill with no deadline, so a late answer can lower a score but never raise it. The intervals come from 1,000 paired bootstrap resamples of the items, each scored against the pack’s own base rate.