OenoBench

OenoBench is a wine-knowledge benchmark for large language models. The current release holds 3,266 multiple-choice questions across six domains — viticulture, winemaking, wine business, wine regions, grape varieties and producers — at four difficulty tiers calibrated to the WSET / Court of Master Sommeliers ladder. 16 frontier and open-source model configurations have been scored against it, over 52,256 evaluations, and every release ships as a versioned JSON file committed to this repository so that historical runs stay reproducible.

The results are on the leaderboard — ranked accuracy with filters by domain, difficulty tier and closed-book versus contextual slice, plus three analyses computed from the same run: reasoning-mode lift, self-preference bias and cost efficiency. Every figure on it is generated from the run file rather than drawn once, so it cannot fall out of step with the data behind it.

How the question set is built

Four stages, automated where automation is reliable and gated by a human where judgement is required.

• Data collection. Curated source material across all six domains, with explicit coverage targets so that no sub-domain dominates and every fact traces to an external authoritative source.
• Multi-model question generation. Five frontier and open-source model families (Claude, GPT, Gemini, Llama, Qwen) plus deterministic templates propose candidate questions from that material. This is the central bias-mitigation step, not a scaling one: no single family disproportionately shapes the questions it will later be scored on.
• AI validation. A nine-agent audit checks each candidate for factual accuracy, ambiguity, leakage from the generation prompt, distractor quality, country-representation balance and verbatim copying. Failing items are dropped or rewritten.
• Human review. A final pass spot-checks a sample per stratum, resolves disagreements between validators, and signs off on the release.

What the leaderboard reports

Headline accuracy is one number and the least interesting one. Four orthogonal slices sit under it — per domain, per difficulty tier, closed-book versus contextual (is the question answerable from parametric memory, or does it need reasoning over supplied context?), and reasoning-mode lift, which scores the same base model with and without extended reasoning. Self-preference bias — each evaluator’s accuracy on questions generated by its own family against questions from other families — is reported separately, because it is a property of the benchmark’s own construction rather than of the model.

A delta is called a finding only when its confidence interval excludes zero, and the figure and the card beside it take that verdict from the same function, so they cannot disagree on screen.

What is not published

There is no companion paper. This page and the versioned JSON release behind the leaderboard are the published record of the benchmark. Full prompts, generator model versions and inter-rater statistics are not released at this stage, and the histograms in the run file sum to slightly more than the question count — a known artefact of the research data, left visible rather than reconciled by hand.

This page was two until 2026-08-12, when the overview and the methodology were combined: a benchmark whose method is on a second page is a benchmark whose method is optional reading.