The standard way to compare language models is to look at a number on a leaderboard and form an opinion. The problem is that a score without documented methodology is close to meaningless, and most leaderboard platforms hide the methodology by design.
This became sharply visible in 2023 when Hugging Face researchers documented a reproducibility problem in MMLU evaluation. Three implementations of the same benchmark, run on the same dataset, produced substantially different results for LLaMA-65B: the UC Berkeley original and Stanford HELM both returned 0.637, while the EleutherAI Harness implementation used by the Open LLM Leaderboard returned 0.488. The divergence traced back to mundane but consequential implementation decisions: whether answer probabilities were computed as log-likelihoods of individual letter choices, full answer text, or complete sequences; how prompts were formatted; how tokenization was handled. These are not exotic edge cases. They are decisions every evaluator must make, and collectively they can shift a published score by nearly a third.
A score of 0.637 and a score of 0.488 are not different measurements of the same thing. They are different things that happen to share a benchmark name.
This is the problem Community Evals, published by Hugging Face in February 2026, is primarily designed to address. The framing in the title, about being done trusting black-box leaderboards, is accurate but undersells the technical nature of what is being proposed. Community Evals is not a replacement leaderboard. It is an infrastructure layer that makes evaluation results a first-class citizen of the Hub, version-controlled, auditable, and contributed through pull requests.
What the system actually looks like
The design has three moving parts. Benchmark dataset repositories can register themselves by adding an eval.yaml file to their root, which declares the benchmark name, the evaluation framework, and the list of tasks being measured. Each task gets its own aggregated leaderboard, built automatically by ingesting results from across the Hub.
Model repositories store their evaluation results in a .eval_results/ folder as individual YAML files. A minimal result file looks like this:
dataset:
name: MMLU-Pro
type: TIGER-Lab/MMLU-Pro
revision: abc123
task:
type: mmlu_pro_biology
metrics:
- type: accuracy
value: 0.712
source:
url: https://huggingface.co/spaces/org/model-eval-logsThe optional fields are where the provenance story lives. A verifyToken field is issued when an evaluation ran inside HF Jobs using Inspect AI, the evaluation framework maintained by the UK AI Security Institute. Results with a valid token receive a verified badge. Results submitted via open pull request receive a community badge before being merged. Results linked to an external source (a paper, a Space, logs) receive a source badge. Results linked to the benchmark leaderboard receive a leaderboard badge.
This badge hierarchy is doing real work. It communicates, at a glance, the epistemic status of a score: whether it was cryptographically tied to a specific execution environment, submitted by a community member pending review, or sourced from a paper that can be cross-referenced.
Why centralized leaderboards structurally fail at this
The existing model for leaderboards, whether that is the Open LLM Leaderboard or external platforms, creates a single point of failure for evaluation integrity. When the Open LLM Leaderboard quietly removed the DROP benchmark after discovering normalization bugs (numbers followed by newlines were not matched, and a stop-token bug prevented floating-point answers from ever being parsed correctly), there was no audit trail. Models had been ranked on broken infrastructure, and the correction happened without a public record of what changed and when.
Benchmark registration and scoring were coupled: the platform owned both the execution environment and the results, which meant users had no way to independently verify published scores or understand what exactly was being measured.
Community Evals decouples these. The Hub stores results and exposes them through APIs, but does not own the evaluation execution. Any user can submit results to any model repository, including models they do not own, by opening a pull request. The full Git history records when results were added, what changed across versions, and who contributed. Model authors retain control by being able to close result PRs, but the default is openness.
The four pilot benchmarks and what they tell you
The initial launch includes four benchmarks that collectively reflect the current state of the field.
MMLU-Pro is the successor to the original MMLU, with 12,032 questions across 14 disciplines and ten answer choices per question instead of four, with expert-reviewed distractors. Models that scored above 88% on original MMLU drop substantially: GPT-4o falls from 88.7% to 72.55%, Claude-3-Sonnet from 81.5% to 55.11%. The gap between benchmark performance and genuine reasoning capability becomes more visible as evaluation pressure increases.
GPQA (Graduate-Level Google-Proof Q&A) contains 448 multiple-choice questions in biology, physics, and chemistry written by domain experts with the explicit property that web search does not help. PhD-holding domain experts score around 65% on questions outside their subfield. The benchmark was developed as infrastructure for scalable oversight research, where the goal is to study how AI systems and humans can collaborate on problems neither can solve alone.
HLE (Humanity’s Last Exam) takes the difficulty ceiling further: 2,500 multimodal questions across dozens of subjects, developed by the Center for AI Safety and Scale AI. It includes a canary string to help builders filter it from training data, which is an acknowledgment, baked into the dataset itself, that contamination is a live concern.
All three benchmarks share the property that they were designed with future saturation in mind, unlike the original MMLU, which crossed 91% average accuracy before most practitioners had finished relying on it.
What this does not fix
Hugging Face is explicit about the limits. Community Evals will not solve benchmark saturation; harder benchmarks will eventually be saturated too. It will not close the gap between benchmark performance and real-world utility, which is a fundamentally harder problem tied to benchmark selection and task distribution. It will not prevent training on test sets, since the very openness of the system means anyone can see which datasets are being used for evaluation.
The stated goal is narrower and more tractable: make evaluation methodology visible. Expose what is being measured, how, by which implementation, at what version of the dataset, and by whom. This is a documentation and infrastructure problem, and it is addressable in a way that the deeper problems are not.
The broader context
There is a parallel here to what happened with software dependencies when package managers started publishing checksums and lock files. The problem was not that packages were bad; it was that “install lodash” could mean different things on different machines at different times, and there was no standard way to record which version you actually tested against. Lock files solved the metadata problem without solving the problem of bad packages.
Evaluation results have been operating without lock files. The same benchmark name has covered meaningfully different implementations, and there has been no standard format for recording the difference. The .eval_results/ schema is an attempt at that standard, and the Hub’s aggregation layer makes it useful at scale rather than just a local convention.
The full Community Evals documentation and the Inspect AI integration for LightEval are worth reading together if you are building evaluation pipelines. The broader lesson, looking back from a few weeks after the February 2026 announcement, is that the field needed this layer of infrastructure before it could have a productive conversation about benchmark quality. Without reproducible, attributed, auditable scores, arguments about which models are better are largely arguments about whose leaderboard you trust.