GenAI evaluation is the practice of measuring whether a generative AI system does its job for a specific use case, taking model, application and user experience as one system. What counts as good differs per case: issue resolution for a support bot, faithfulness and adoption rate for a clinical summary tool, safe and executable code for a coding agent.
Key Takeaways
- Anamika Mukhopadhyay hears the same answer in workshops with teams running production AI systems: they measure what their tools report by default, without asking which numbers they need.
- The same model needs different GenAI metrics per use case, from satisfaction and issue resolution for a support bot to groundedness and adoption rate for a clinical summarization tool.
- A dip in the faithfulness score left a CXO room asking about rounding; a 12 percent drop in issue resolution and 200 extra support tickets got their attention.
- GenAI evaluation runs as a loop of offline evaluation on a golden dataset and online evaluation with traces, because a model version can change every two to three weeks.
- Deepshikha’s team planted an indirect prompt injection in a Reddit post, and an agent with guardrails against direct injection followed it through its web tool call.
Why Right or Wrong No Longer Works for GenAI Evaluation
GenAI evaluation starts with an uncomfortable fact: every answer the model gives can be correct, and every one of them can be useless. Anamika Mukhopadhyay describes the shift as a move from a right-versus-wrong framework to a best-versus-worst one. Classic model evaluation still had a yes or no. Is this picture a dog or a cat? Is the classification correct? A generative system answers one prompt in a dozen ways, and a user may accept all of them or reject all of them.
That breaks a habit most testing teams carry from project to project. Anamika puts it plainly: in software testing, one evaluation framework used to be copied from one project to the next with few changes. Traditional AI already needed more, but it kept the notion of correctness. GenAI drops it. What remains to judge is whether one output serves this user, in this use case, better than another.
Deepshikha adds the practical consequence. Evaluation has to run layer by layer, and the first condition is knowing what exactly is being evaluated. The problem has changed, she says, to a point where you need to understand the architecture end to end before you can map metrics to failure points.
Which System Are You Actually Measuring?
Most teams measure the model, and the model is the smallest part of the problem. Anamika sees it in projects everywhere: AI goes into products, and the metrics teams pick cover the model alone. Around that model sits an application, and in front of the application sits a user with an experience. The unit of evaluation is the whole system, model plus application plus user, and the metrics have to cover all three.
Anamika and Deepshikha run workshops with teams who operate production AI systems. They ask two questions: what are you measuring, and why does it matter? The most common answer is that teams measure what their tools give them by default. The numbers get tracked per sprint or per release, and nobody has stopped to ask whether they are the numbers that matter.
Deepshikha’s example is the coding agent, because nearly everyone uses one by now. A single model from the same vendor in a different version can return a completely different response, and a new launch arrives every other week. The questions that follow are concrete. Does the sub-agent get enough context? Does it hallucinate even with that context? When it crawls the application in the browser, does it reason, and was a reasoning model in use at all? Does the generated code introduce vulnerabilities? Each question points at a failure point, and each failure point needs its own metric.
The same agent also means different things to different people. For Deepshikha, the coding agent delivers a script. For a manager, it delivers a complete automation evaluation. Before anyone measures efficiency, the team has to agree on what the thing is supposed to do. Otherwise the metric is accurate and irrelevant at the same time.
One Model, Three Use Cases, Three Definitions of Good
The definition of good belongs to the use case, and the model does not carry it. Anamika’s example takes one model into three projects: a customer support bot, a clinical summarization agent, and a coding agent. At model level, accuracy and similar scores apply to all three. At use case level, the goals split.
| Use case | Definition of good | What gets measured |
|---|---|---|
| Customer support bot | Customers get their problem solved | Satisfaction, issue resolution |
| Clinical summarization | The summary is grounded in the chart and actually gets read | Faithfulness to the source documents, adoption rate |
| Coding agent | The code runs and does no harm | Executable output, no new vulnerabilities |
The clinical case shows why a single technical score is never enough. A clinician sees a patient for 15 to 20 minutes, enters notes, and before the next visit has about two minutes to catch up. A summarization agent that writes everything into the summary scores well on groundedness: nothing is missed, everything is true to the chart. If reading it takes three minutes, the clinician goes back to skimming the chart, and the tool goes unused. Adoption rate therefore sits on top of faithfulness as the metric that connects the system to the business.
Anamika’s warning concerns two separate levels. On the technical level, every metric is green and the dashboards look good. On the business level, the investment produces no visible return. Defining good per use case is what keeps the two levels connected.
What Separates Offline from Online Evaluation?
Offline evaluation happens before users touch the system, online evaluation happens while they use it, and Deepshikha insists on both. The split itself is older than GenAI; it applied to machine learning already. What changes is how much depends on the first step: understanding which problem the agent is supposed to solve.
Her example comes from food delivery. A bot built to answer what a menu item contains will, left alone, also generate Python code on request, because the LLM generates content regardless of context. The company then gets surprised by a huge bill. Deepshikha stresses that this is a real case and has happened at several large food chains.
Offline evaluation therefore starts with the users, or at least with the people who want the agent built. From that understanding comes the golden data set, and without it there is nothing to evaluate against. In Deepshikha’s words, data is everything. A surface-level evaluation asks the model the easy questions and passes. Adversarial testing asks the tricky ones, and a team that skips it watches the system fail in production the next day.
Online evaluation then checks whether the system is genuinely useful for the customer, and it needs observability to do so. Traces record every LLM call and every action the agent took. Did it understand the prompt? Did it hallucinate? Did it make the right tool call? When something failed, did it recover gracefully and return to its original state? When an answer was missing, did it ask the user? When a policy was violated, did it stop? Those traces flow back into the offline data set, and the whole thing becomes a continuous loop.
The loop also absorbs a problem teams tend to underestimate: the model underneath changes. Vendors ship new versions every two or three weeks. Sycophancy rises in one version and falls in another. Everything on your side can look fine, business outcomes included, and the model changes in the background anyway. Anamika adds a case from a project: after two to three months of tuning a Gemini Flash model, the team learned it would be deprecated within two months. All of that work started over. She expects this to become routine and wants it in the estimates and in the risk plan from day one.
How Does a Faithfulness Score Become a Business Number?
A technical metric that decision makers cannot read has no effect, however precise it is. Deepshikha and Anamika tested this in a workshop where CTOs and other decision makers sat in the room. First they reported the result in technical language: the faithfulness score had dipped from one value to another, something like 0.87279. The executives were lost. One of them asked whether this was a rounding problem or something serious.
Then they rephrased the same finding. The issue resolution rate has dropped by 12 percent, and the number of support tickets has gone up by 200. The reaction flipped. The drop in resolution, the extra tickets, the effort to work through them, the money the company spends on all of that: this is what the faithfulness score meant, and it had to be said in those words.
“A single metric or a group of metrics should have a clear path to the business value it is trying to deliver.”
(Deepshikha)
The same logic applies to every number a team sets out to improve. Lowering the hallucination rate is a goal only because users stop trusting the application, the system, and the bot’s responses when it hallucinates. The reasoning behind the number is what gets mapped and reported. This is where GenAI metrics inherit a problem classic test metrics have carried for years: plenty of figures, little translation for the people who decide.
How a Reddit Post Tricked a Well-Guarded Agent
An agent that resists every direct prompt injection can still be hijacked through the content it fetches. Deepshikha tells the case of an agent evaluated for a customer, with most issues found and fixed and the team ready to deploy. Adversarial testing had covered the prompt injection scenarios, and the agent recognized every attempt. What the team had missed: the agent does a lot of tool calling.
So they wrote a Reddit post. The title matched the query the agent would receive, and the body told the reader to forget all previous instructions and follow new ones. When the agent hit its web tool call, it found the post, read it as instruction, and started returning the person’s banking information. The guardrails for direct injection were intact. The indirect path had never been tested, because nobody had thought to explore it.
Deepshikha’s lesson is about perspective. Teams look at an agent vertically, layer by layer, and that remains necessary. They also have to look horizontally, across every external call the agent makes. Not everybody is a security expert, an agent expert, or an AI expert. Everybody evaluating one still has to understand the architecture, because an agent treated as a black box offers no way to pinpoint where a failure happened. And without that point, there is no way to know which metric should have caught it.


