Why Annotation Accuracy Is Misleading

Researchers at Google interviewed 53 AI practitioners working in high-stakes domains (healthcare, finance, agriculture, public health and others) and reached a conclusion: data problems rarely surface as “a batch of labels that failed quality control.” They appear as cascades — problems planted early in data definition and collection run through the whole pipeline and only erupt together at deployment.

The numbers from that study: 92% of practitioners experienced at least one data cascade on a project, and 45.3% experienced two or more.

Source: Sambasivan et al., "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI, CHI 2021 (interviews with 53 practitioners across India, East Africa, West Africa and the United States)

Note the four triggers the study identifies; two of them have no direct relationship to “labelling quality”: the fragility of the physical world (clean training data, noisy production environments) and insufficient domain expertise (data definition done by people without domain knowledge).

In other words, most data incidents are not “labelled wrong” but “defined wrong from the start.” And the metric of “accuracy” covers only the “labelled wrong” kind, quite blind to definition-level problems.

This explains why many projects look excellent at acceptance and then break down repeatedly after launch: acceptance tests the data; what breaks is the definition.

Accuracy is defined as correctly labelled samples ÷ total samples. It has two structural flaws.

First, it measures against the samples that already exist. It describes how well this batch of data was labelled; it does not describe what the model will encounter, nor how the model will choose when it is unsure. A dataset with 99% accuracy can perfectly well leave a model performing badly on edge cases — because the share of edge cases in the dataset has nothing to do with their contribution to accuracy.

Second, it is an aggregate metric. Missed labels, misjudgements, unnormalised units, inconsistent formats, unsupported inference — the causes of these errors and the ways to fix them are entirely different, yet in an accuracy report they collapse into one number. The result: you cannot locate them, cannot regress them, and cannot price them.

The term “data apathy” comes up repeatedly in data-centric AI discussions: treating data as fuel for the model rather than as an engineering object, doing “preprocessing” once at the start of the process and rarely returning to improve it systematically. The cost of data quality problems usually shows up only at deployment.

See: Jarrahi et al., "The Principles of Data-Centric AI", Communications of the ACM, 66(8), 2023

PEN is not an abstract framework. Walk through one of the most basic tasks in chemistry — turning the structure diagrams in patents and papers back into machine-readable structures, i.e. OCSR (optical chemical structure recognition) — and the place of each of the three data categories becomes very clear.

The task matters because chemical information travels mainly as images: compounds in papers and patents are mostly drawn as structure diagrams rather than written as searchable text. One study notes that the US patent office is one of the few bodies that also supplies chemical structures as ChemDraw CDX or MDL MOL files; other sources can basically only be reconstructed by recognition.

Source: Weber, L. et al., "Comparing Optical Chemical Structure Recognition Tools", ChemRxiv

How far can automated tools go? An extraction pipeline built from open-source tools was evaluated on 25 patents and 17 papers. Stratified by document form, the results are:

Note the shape of these numbers — for the very same “recognise the structure” task, different document forms differ by nearly 20 percentage points. And the reality is that the non-table form is precisely the norm in patents and papers.

Taking it apart with PEN: where those 20 points come from

Positive: the structure diagram prints cleanly, is laid out regularly, each structure sits next to its number and identifier, a single molecule, nothing occluded. Many table-form patents fall into this class, which is why tools reach about 95% here.

Edge: amorphous structures (polymers, mixtures, undetermined crystal forms), the variable definitions of a Markush formula, structures spanning pages, captions separated from the body text, substituents occluded by the next line. These are not cases of “drawn unclearly”; rather the same drawing admits two reasonable readings and must be resolved by chemical convention or the applicant's intent.

What is special about negatives at this layer

Negative: the model may output a structure that is syntactically valid but chemically impossible, such as a pentavalent carbon. Such results throw no exception; they simply flow quietly into the dataset. And for chemical data, one wrong bond means the compound's identity changes, and property prediction, structure search and synthesis routes all fail in a chain.

So the role of Negative in OCSR is not only “error correction” but keeping the evaluation discriminating: if every evaluation sample is a regular structure the model already recognises correctly, no model swap will show any difference — which corresponds exactly to the discussion of evaluation saturation further on.

Piling up positives causes problems, a point repeatedly verified in the classic research on natural language inference (NLI): the model is often not reasoning at all but exploiting the statistical traces left by the annotation process — for instance, that samples with one label happen to be shorter or to use more specific words. Researchers call this “annotation artifacts” and “shortcut learning.”

The result: models score highly on these datasets and then collapse quickly once you move to a real distribution or to adversarial samples.

Back to our framework, it reads like this: positives teach the model “samples of this kind usually look like this,” whereas boundary samples are what teach it “why this one belongs to A rather than B.” The former is pattern recognition; the latter is decision.

This is also why adding more to a positives-only dataset yields rapidly diminishing returns — every new positive reinforces the same pattern already learned, while the part the model actually lacks (the decision basis at the boundary) gains not a single sample.

Most datasets aggregate annotation results by majority vote: wherever there is disagreement, take the majority. The approach is simple and reproducible, but it assumes “disagreement = noise.” Research of recent years has raised sustained doubts about this.

A widely cited piece of evidence comes from Stanford's Jury Learning work: in a toxicity task, on average more than a third of annotators disagreed with any “toxic” verdict, and this held even after label noise was removed; in a misinformation classification task, three professional fact-checkers agreed fully on only half of the URLs.

Source: Gordon et al., "Jury Learning: Integrating Dissenting Voices into Machine Learning Models", CHI 2022

If half of those three fact-checkers' disagreements are “noise,” what is the other half? Later research answers by splitting disagreement into two kinds:

• Noise-type disagreement — coming from annotator error or process defects, and it should be eliminated through guidelines and training;
• Signal-type disagreement — coming from the ambiguity of the task itself, reflecting a decision criterion that has not yet been written down. Such disagreement should be captured, not flattened.

See: Kraus & Kroll, "Maximizing Signal in Human-Model Preference Alignment", 2025

This kind of disagreement does not appear only in subjective tasks such as sentiment or toxicity. One study states plainly that even in relatively objective domains such as law and medical decision-making, genuine disagreement persists between annotators, and the information it contains is thrown away by the majority-vote process.

See: Leveraging Annotator Disagreement for Text Classification, ICNLSP 2024

More strikingly, preserving disagreement does not drag down the accuracy of the main label; it may actually help. One study compared two approaches on an alignment task and concluded that keeping annotator disagreement not only improved the fit to the human judgement distribution but also boosted main-label prediction performance; on the ChaosNLI dataset, conventional majority-label supervision pushed the JS divergence from the human judgement distribution up from 0.374 to 0.477.

See: SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs, arXiv:2606.05376

The problems at the dataset level replay in another form at the evaluation stage. A systematic analysis of 60 mainstream LLM benchmarks found that nearly half the benchmarks are already saturated — the score gaps between leading models can no longer be distinguished statistically — and that the saturation rate rises with the age of the benchmark.

Two further findings deserve particular attention: hiding the test set (turning a public benchmark private) does not help delay saturation; and benchmarks built by experts are markedly more resistant to saturation than crowd-sourced ones.

Source: Akhtar & Reuel et al., "When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation", arXiv:2602.16763

The second finding has a direct engineering implication: the value of an evaluation set lies not in how hard it is but in whether it discriminates. If the evaluation cases skew easy, scores sit near full marks for a long time, and then any before-and-after comparison of a change returns “no significant difference” — the evaluation system looks like it is running while it has in fact lost its ability to tell things apart.

There is corresponding methodological research on this: one work points out that saturation is not only a property of the task distribution but also a result of insufficient evaluation resolution; the same batch of model outputs can regain discrimination under a different evaluation protocol.

See: SEAL: Can Saturated Benchmarks Be Revived by LLM-as-a-Meta-Judge?, arXiv:2605.30104

Quantitatively, discrimination can be improved by engineering means. One study compared different metrics on the same set of questions: on ARC-Challenge, exact match could distinguish only 17% of model differences, whereas a metric weighted by reasoning-chain correctness reached 46%.

See: Etzine et al., "Revitalizing Saturated Benchmarks", arXiv:2503.05551

Gathering these problems together, one study groups them into three structural defects: low discrimination and score compression, a ceiling effect that exhausts the remaining headroom, and popularity decoupled from rigour — tests with low transfer cost dominate for long stretches, while the genuinely hard problems, where headroom remains, get marginalised.

See: Zhu et al., "Benchmark Health Index", arXiv:2602.11674

We do not give a universal ratio. Any statement that directly hands out “X percent for each of the three categories” deserves scepticism — because the proportions depend on two variables: how contested the business itself is, and the cost of a single wrong decision.

The denser the disagreement, the higher the cost of error, and the longer the iteration cycle (foundation models turn over, standards get updated), the greater the weight Edge and Negative should carry; conversely, the less ambiguous the decision, the larger the natural share of positives.

Scenarios that barely need PEN: tasks with a clear objective, unambiguous decisions, and a one-off delivery that ends there. Producing to a target count of positives is enough for these; forcing in boundaries and negatives markedly raises cost without returning corresponding value.

The test is simple: if an error on this task would set your team arguing about “who is actually right,” then it needs PEN; if not, it does not.