The evaluation challenge at the heart of scientific AI

Hand someone an AI system and ask a deceptively simple question: is it any good? The LLM-based systems being put to work across science and industry are intended to answer open-ended questions, weigh evidence, and work through steps toward a conclusion. A fluent, confident response is easy to produce, but knowing whether it is correct and if it will hold up the next time you ask is another matter. 

Machine learning models have been used and tested for years. What has changed? Traditional ML models are used to performing tasks such as classifying an email as spam, predicting whether a loan applicant will default, or identifying a tumor in a radiology image. Each problem is specific, and the outcomes are deterministic: for a given input there is a single output. There are decades of established practice on how to check their work. Simply, you set aside examples that the model never sees during training, run them against the trained model, and measure how often it gets the answers correct. 

That framework is rigorous, interpretable, and trusted. It is also insufficient for the kinds of AI systems that organizations are now deploying. The shift reflects a fundamental change in what AI is being asked to do. Today's language models and the agentic systems that act on their reasoning do not map inputs to a fixed set of labels. Instead, they generate open-ended language and take actions, and in doing so they break many of the assumptions the classical approaches use. The same question, asked twice, can produce different answers. A response can be fluent, confident, and wrong. For the hardest and most valuable questions that we want AI systems to answer, there may be no settled answer to check against at all. 

Nowhere does that gap matter more than in science, where the questions worth asking are precisely the ones no one has answered yet. Further, a wrong answer is not just an incorrect prediction - it can send a research program down a months-long dead end.  

What follows traces this change: why the classical framework worked so well, why it falls short for modern AI, and the new approaches that are emerging to close the distance. 

The comfortable world of classical ML evaluation 

The assumptions these newer systems break are rarely spelled out because, in classical machine learning, they almost never need to be. Traditional evaluation rests on three of them, all easy to overlook because they are usually satisfied:  

• First, that there is a ground truth: a definitive correct answer against which model outputs can be compared.  

• Second, that the task is bounded such that the space of possible outputs is finite and well-defined.  

• Third, that correctness is unambiguous in that a prediction is either right or wrong, and a human (or a labeled dataset) can tell the difference. 

These assumptions hold well for the problems that first made machine learning famous:  

• Spam detection is a binary classification problem.  

• Image recognition maps an input to a fixed set of labels.  

• Fraud detection scores a transaction against a learned threshold.  

In each case, evaluation works the same way: compare the model’s output against examples whose correct answers are already known and measure how often it gets them right. 

There are standard metrics that emerge from this framework:  

• Accuracy. The share of all predictions a model gets right; intuitive but misleading when one outcome is far more common than another. 

• Precision. Of everything the model flagged as positive, the fraction that actually was; a measure of how trustworthy its alerts are. 

• Recall. Of all the true positives that existed, the fraction the model caught; a measure of how little it misses. 

• F1 score. The harmonic mean of precision and recall; useful when you need a single number that penalizes models that sacrifice one for the other. 

• Area under the ROC curve. A model's confidence score can be tuned up or down to trade between catching more true positives and triggering more false alarms. The ROC curve plots that trade off across every possible threshold, and the area underneath it summarizes overall discriminative power in a single number — 1.0 meaning the model perfectly separates the two classes, 0.5 meaning it performs no better than a coin flip. 

Each captures something real about model behavior, and these measures have become so foundational that they are the first thing most practitioners reach for regardless of the problem at hand. The trouble begins when the problem no longer satisfies the underlying assumptions. 

Why language models break the old framework 

For a language model, each of those assumptions gives way. It does not pick from a predefined set of answers.  Instead, it generates open-ended text, so the space of possible outputs is unbounded. For most of the questions worth asking, there is no single ground truth to compare against, and one passage can be accurate in one respect and wrong in another, relevant but vague, well-reasoned but only superficially, sometimes all at once. Whether an output is correct becomes a judgment rather than a simple comparison to a fixed answer. 

The challenge becomes sharper when you consider that agentic systems act rather than just respond. A research agent might retrieve information, synthesize it across sources, reason through intermediate steps, and produce a conclusion that no human reviewed along the way. Evaluating such a system means evaluating not just the final output but the chain of decisions that produced it. A correct conclusion reached through flawed reasoning is not the same as a correct conclusion reached through sound reasoning, particularly in environments where the reasoning itself will inform subsequent actions. 

Underneath these shifts are several properties of language model outputs that make evaluation difficult: 

• Reproducibility is no longer guaranteed. Because the same prompt can return different answers on different runs, a fixed input no longer yields a fixed, checkable result. This unsettles a cornerstone that scientific practice and classical ML evaluation share. 

• Failure leaves no signal. Language models are trained to produce fluent and coherent text, but fluency is not the same as correctness. A model can generate a confidently worded, internally consistent response that is factually wrong, and will often do so without any signal that something has gone awry.  

• Correctness depends on context. Whether a model response is good often depends on who is asking, why, and what they plan to do with the answer. A summary that satisfies a general reader may mislead a specialist. An answer that is technically accurate may be useless if it omits the right caveats. 

The scientific domain makes it harder still 

These general challenges are amplified in scientific applications, where precision is not a preference but a professional and practical requirement. 

Scientific questions are rarely fully bounded. Asking an AI system to identify promising drug candidates, propose a synthetic route, or surface cross-domain connections in the literature is not like asking it to classify an image. The answer space is vast, the criteria for a satisfactory answer are multidimensional, and an error can lead to a misdirected experiment, cause months of lost time, or prevent the ability to replicate results. 

This is compounded by a deeper problem: in the domains where AI could add the most value, a definitive ground truth often does not yet exist. A system that surfaces a novel hypothesis connecting two previously unrelated bodies of literature cannot be benchmarked against known answers. If the answer were known, the hypothesis would not be novel. A model that scores well on internal benchmarks but breaks down in real experimental conditions can misallocate resources, delay programs, and erode the trust of the researchers whose adoption determines whether AI earns a lasting place in their scientific practice.  

Existing text metrics 

Machine-generated text did not arrive with large language models, and neither did the problem of scoring it. Machine translation and summarization systems were producing text that needed evaluating at scale long before today's models, and the field built a family of automated metrics specifically to do it. Three remain in wide use: 

• BLEU. Designed for machine translation, it measures how much of the model’s output overlaps with one or more reference texts at the level of short word sequences with a penalty for answers that are too short (Papineni et al., 2002). 

• ROUGE. Developed for summarization, it takes the complementary view, measuring how much of the reference text is recovered in the model’s output (Lin, 2004). 

• METEOR. A refinement that tries to credit meaning rather than surface form, matching not only identical words but stems and synonyms, and balancing precision against recall (Banerjee and Lavie, 2005). 

These metrics are a measure of accuracy when a task has a narrow band of acceptable answers. However, they inherit the assumption that breaks down for the systems in question: they require a reference text to compare against and they measure how closely an output matches that reference, not whether it is correct, well-reasoned, or useful. A correct answer phrased differently from the reference can score poorly, while a fluent answer that echoes the reference wording can score well even when it is wrong. For open-ended synthesis, where many phrasings are valid and the most valuable answers may have no reference at all, resemblance to a fixed target is a weak proxy for quality. 

Emerging evaluation approaches 

In response to these challenges, several evaluation approaches have emerged. These are not replacements for classical metrics, but necessary complements to them. Used together, they address different failure modes that a single method would miss. 

LLM-as-a-judge uses a separate language model to evaluate the outputs of the model under assessment. This approach scales in a way that human review cannot. A well-designed judge model can assess thousands of outputs in the time it would take a domain expert to review dozens. Can language models perform at least as accurately as a human? Yes, and the key to making it work is specificity.  

As an example, take a panel of domain experts independently reviewing the same text. Each of the reviewers may reach different conclusions about its quality not because one is wrong, but because each is implicitly weighing different unspoken criteria. One prioritizes factual precision while another weights completeness or appropriate hedging of uncertainty. Without explicit, shared metrics, "good" means whatever the reviewer privately decides it means.  

This variation in human judgment is well-documented. Zheng et al. (2023) found that human expert annotators evaluating open-ended LLM responses agreed with each other only 80% of the time on a raw pairwise basis, and that figure represents a favorable condition, as annotators were working from the same explicit rubric on the same task. The picture is more nuanced when evaluation is decomposed by skill type. Ye et al. (2024) measured human inter-annotator agreement across 12 distinct evaluation dimensions using Krippendorff's alpha and found an overall human-human alpha of just 0.488 which indicates only moderate agreement. 

A well-designed evaluation prompt forces that implicit reasoning into the open, asking the judge model to assess specific, separable dimensions (factual consistency, relevance, completeness, appropriate expression of uncertainty) rather than rendering a single holistic verdict (Liu et al., 2023; Ye et al., 2024). This produces assessments that are more consistent, auditable, and actionable, whether the judge is a language model or a human expert. 

Human-in-the-loop evaluation remains indispensable for the cases where automated methods are weakest, and the logic for when to deploy it mirrors a principle well-established in large-scale manufacturing. In a production environment, not every unit receives the same level of scrutiny. Components destined for medical devices or safety-critical systems receive direct, individual human inspection. Standard outputs are covered by automated testing with human reviewers brought in to investigate failures, validate edge cases, or conduct periodic batch audits. These principles are reflected in the U.S. Food and Drug Administration's Quality Management System Regulation (21 CFR Part 820) and numerous quality standards such as AS9100D and ISO 13485:2016. 

Scientific AI evaluation follows the same logic. Novel outputs, high-stakes conclusions, and the calibration of automated evaluation systems warrant direct expert review. Routine outputs at scale are handled systematically, with human judgment available as an escalation path rather than a bottleneck. Domain experts bring something automated systems cannot: the ability to recognize a subtly wrong answer that is nonetheless internally coherent to identify when a model is reasoning from an outdated understanding of a field or to flag when a result, while technically correct, would mislead a practitioner interpreting it without context. The goal is not to maximize human involvement but to direct it where it has the most leverage. 

Retrospective benchmarking addresses one of the more fundamental tensions in scientific AI evaluation: how do you measure a system on questions for which you do not already know the answer?  

Consider two distinct use cases. The first: a researcher asks an AI system to look up a specific piece of information such as a melting point, a regulatory classification, or a documented reaction yield. This is straightforward to evaluate. Either the system retrieved the correct answer or it did not. The second use case is harder: a researcher asks the system to synthesize information across multiple sources and produce an insight that no single document contains. This is what researchers care about most, and it is far harder to evaluate because there is no pre-existing answer that the AI system can use to validate its result. These open-world tasks are also the ones most prone to drift. Both the expected reasoning approach and the acceptable answer can shift as scientific understanding evolves, which means evaluation must be revisited rather than treated as a one-time exercise. 

The instinct might be to test only what can be verified. However, that would mean evaluating a system only on the tasks it was least designed to do. A better approach is to ask the system to answer questions that have already been answered through synthesis rather than retrieval alone. Findings that were once novel, once required connecting disparate bodies of literature, and that are now part of the established scientific record make ideal test cases. If the information was available at the time of the discovery, a capable system should be able to reconstruct the reasoning.  

This is retrospective benchmarking: using the history of scientific discovery as a proxy evaluation set for the kind of open-ended synthesis that cannot otherwise be directly tested. The approach has precedent in the growing field of literature-based discovery. Swanson, (1986) identified two bodies of literature that were logically connected but noninteractive and showed that the connections between them implied a hypothesis (that dietary fish oil might benefit Raynaud's patients) that had not been explicitly published. Later, Spangler et al. (2014) extended this logic computationally, mining the scientific literature to generate hypotheses about which kinases phosphorylate the tumor suppressor p53, then using retrospective analysis to validate the accuracy of the approach before any new experiments were run. In both cases, the value of the retrospective framing is the same: the inputs are fixed to a specific moment in time, so the evaluation does not go stale in the way open-world benchmarks do.  

The approach still has limits as it tests for connections that have already been made, not the full range of future capability, and it requires careful engineering to constrain the information available to the system at evaluation time. However, as a validity check on whether a system can genuinely synthesize across sources rather than merely retrieve from them, it is one of the few rigorous tools available. 

Semantic stability testing addresses a failure mode that static tests commonly miss. Language models can be sensitive to how a question is constructed: rephrasing the same query without changing its meaning can produce a different answer. For a scientific assistant, this is a significant reliability concern. A researcher should be able to trust that the system's response reflects the substance of their question, not the particular words they happened to use. Several scenarios illustrate this issue. 

The first is synonym substitution. Using invariance tests (perturbations that preserve meaning while varying surface form), Ribeiro et al. (2020) identified critical failures that had gone undetected in extensively tested commercial sentiment analysis models. Swapping a word for a near-synonym such as 'film' for 'movie' or 'determine' for 'find out' should not alter the answer a scientific assistant gives, but in practice it frequently does. 

The second is negation. Systematic evaluation of models across many negation benchmarks found consistent limitations: insensitivity to the presence of negation, an inability to capture the lexical semantics of negation, and a failure to reason correctly under negation (Truong et al., 2023). This matters in scientific contexts where a few words can shift the factuality of a statement: 'A causes B', 'A may cause B', 'suggests that A causes B', 'does not support the claim that A causes B', and so on.   

The third is formality. Studies have shown that variations in grammar, tone, or formality can significantly alter the quality and direction of a model's output, with some models producing lower-quality answers for users who phrase queries less formally (e.g. "What's the mechanism?" vs. "Could you explain the mechanistic basis..."). A researcher who writes in plain language rather than technical prose should receive the same substantive response as one who does not. Recent work formalizes this concern: Lior et al. (2025) argue for stochastic evaluation over the space of meaning-preserving prompt perturbations and find that even top-performing models such as GPT-4o and Claude 3.7 Sonnet exhibit substantial prompt sensitivity. 

Given the evidence from various studies, an evaluation that assesses only one phrasing per question will not surface this problem. Therefore, varying language across otherwise equivalent queries and checking whether responses remain consistent should be part of any serious evaluation discipline. 

Reasoning evaluation is important for agentic systems and scientific applications. When a research assistant reasons through a problem, retrieves sources, and produces a conclusion, scientists are not only interested in whether the conclusion is correct, but they need to understand how it was reached. This reflects the foundational scientific expectation that results should be reproducible and methods should be transparent.  

Chain-of-thought prompting, in which models are prompted to produce intermediate reasoning steps before a final answer, was shown to significantly improve performance on complex reasoning tasks. Its adoption was partly motivated by the interpretability benefit of making reasoning legible (Wei et al., 2022). However, the visibility of reasoning steps is not a guarantee of trustworthiness. Turpin et al. (2023) showed that this legibility can be illusory: chain-of-thought explanations can systematically misrepresent the true basis for a model's prediction. In their key experiment, a subtle bias was added to model inputs by reordering multiple-choice options so the correct answer was always "A". This reliably shifted model outputs, yet models consistently failed to mention this influence in their stated reasoning. A system that produces a correct answer via a flawed or opaque reasoning chain is a latent risk, because the next question it answers may not be correct, and there will be no way to tell. 

The question of how to train models to reason correctly, rather than merely produce correct answers, has begun to receive direct attention. Lightman et al. (2023) illustrate the stakes by comparing outcome supervision, which rewards models based on whether the final answer was correct with process supervision, which assigns credit to each individual reasoning step. They found that models trained with outcome supervision regularly used incorrect reasoning to reach correct final answers, and that process supervision produced substantially more reliable systems. The training insight translates into an evaluation principle: if correct steps can be defined precisely enough to train against, they can also be defined precisely enough to audit. 

Evaluating the reasoning process rather than just the output therefore requires a different kind of test: not "is this answer right?" but "did the system follow the appropriate steps to arrive at an answer?" This is itself a rubric-based exercise, but one applied to process rather than product. It is more demanding to design and requires expert input to define what the right steps look like, but it is precisely the kind of evaluation that builds the justified trust scientists need before they will act on an AI system's conclusions. 

An emerging direction in agentic evaluation suggests that observable actions (the tools a system calls, the searches it executes, the order in which it retrieves information) may provide a more reliable signal of reasoning quality than stated rationales alone. Unlike chain-of-thought text, which is generated by the same process as any other model output and is therefore subject to the same hallucination risks, tool calls and retrieval steps leave an audit trail and have consequences that constrain subsequent behavior. Early work on agent trajectory evaluation in benchmarks designed to assess multi-step tool use has begun scoring these intermediate actions rather than task outcomes alone. If this approach matures, it may offer scientific users something closer to a genuine methods audit. 

Blending systematic and human-mediated approaches 

No single evaluation method is sufficient. The organizations making the most progress on scientific AI evaluation are those treating it as a portfolio problem: deploying automated techniques at scale for coverage and consistency and reserving human expert review for calibration, validation of edge cases, and the kinds of judgments that require genuine scientific understanding. The best evaluation frameworks make neither subordinate to the other, leveraging the complementary strengths of each. 

In building AI solutions for scientific research, CAS applies this layered approach. Automated evaluation handles scale and consistency across large output sets; structured rubrics ensure that assessment is multidimensional rather than collapsed into a single number; and domain scientists, the same experts whose curation work underlies the CAS Content Collection™, provide the human judgment that determines whether a system is  technically performant and scientifically sound. This includes evaluating not just what CAS Newton℠ concludes, but how it reasons: whether the steps it takes to reach an answer are ones a scientist would recognize as appropriate, traceable, and reproducible. The goal is not to minimize human involvement in the name of efficiency. It is to direct human judgment where it has the most leverage. 

What good evaluation looks like 

Evaluation is not a step that happens after an AI system is built. It is a design constraint that should shape how systems are designed from the beginning. The questions worth asking early: 

•  “What does success look like?” 

• “How would we know if the system was wrong?” 

• “Who has the expertise to make that judgment?” 

These are not just evaluation questions. They are the questions that determine whether an AI system will earn the trust of the scientists and researchers who are being asked to act on its outputs. 

The classical ML toolkit will not be retired. Precision, recall, and benchmark performance still matter, and for many components of even complex AI systems, they remain the right measures. However, for systems that operate at the frontier of what AI can do — reasoning across scientific literatures, surfacing latent connections, proposing hypotheses in domains where ground truth is uncertain — rigorous evaluation requires more than what the old approaches provide. Developing that rigor is itself an ongoing scientific challenge, and one that deserves the same care as the AI systems being evaluated.