When a PM says an AI feature is "94 percent accurate," a good hiring manager or a sharp engineer will ask three questions the number cannot answer. Accurate on what distribution of inputs? Accurate in what way, since a summary can be factually correct and still useless? And at what cost and speed, since a perfect answer that costs a dollar and takes ten seconds may be unshippable? Accuracy is not wrong. It is thin. It compresses a multi-dimensional feature into one number and throws away most of what determines whether the product works.
This is the measurement stack an AI PM should actually carry. It has four layers: quality, cost and latency, trust and safety, and business outcome. A number from each, defended together, is what a real launch review looks like.
Layer one: quality, unpacked
"Accuracy" usually means "the answer was right." But right along which axis? For most AI features, quality is several dimensions, and a single feature can be strong on one and failing on another.
- Factual correctness: is the content true. The axis people mean by accuracy.
- Completeness: did it include what mattered, or leave out the key point. A summary can be entirely true and still omit the one fact that mattered.
- Faithfulness: for anything grounded in sources, does the answer stay true to the source or invent beyond it. Distinct from correctness.
- Format and structure: did it produce the shape the product needs, every time. A correct answer in the wrong format breaks downstream.
- Tone and appropriateness: is it in the register the product requires. Often the difference between shippable and not for user-facing features.
The practical move is to score the two or three dimensions that matter for your feature separately, not to collapse them into one accuracy figure. A support answer that is 95 percent factually correct but 60 percent complete has a completeness problem the accuracy number completely hid. This is what an eval suite is for, and building AI eval suites as a PM is the mechanics.
Layer two: cost and latency, the ship-or-not numbers
These are not engineering footnotes. They decide whether the feature can exist at scale, and they are the PM's to own.
- Cost per unit of value. Not cost per call, cost per the thing the user cares about: per resolved ticket, per summary, per completed task. This is what you compare against the revenue that task supports.
- Cost at projected scale. A feature that costs pennies at a hundred users can break the unit economics at a hundred thousand. Model the curve, not the current point. The method is in AI cost modeling for PMs.
- Latency, at the percentile that matters. Average latency lies. Users feel the slow tail. Measure the 95th percentile, because the answer that takes eight seconds one time in twenty is the one that shapes the perception of the whole feature.
A quality number without a cost-per-value number and a tail-latency number is an incomplete picture. Plenty of high-accuracy features never ship because they are too expensive or too slow, and a PM who only measured accuracy never saw it coming.
Layer three: trust and containment
These metrics predict whether users keep using the feature after the novelty fades, and they are specific to AI in a way traditional product metrics are not.
- Containment or deflection rate. For assistants, what fraction of interactions the AI handled without a human having to step in. This is often the real business metric, and it is downstream of quality but not identical to it.
- Escalation quality. When the feature does hand off to a human, was that the right call. A feature that escalates too rarely erodes trust with confident wrong answers; one that escalates too often provides no value.
- Confidently-wrong rate. The single most trust-destroying failure. Not how often it is wrong, but how often it is wrong while sounding certain. One confident fabrication costs more trust than several honest "I am not sure" responses. Track it separately.
- Known-unknown behavior. How often the feature correctly says it does not know versus fabricating. A high honest-refusal rate on genuinely unanswerable questions is a feature, not a failure.
The reason these matter: retention on an AI feature is gated by trust, and trust is destroyed asymmetrically. A handful of confident wrong answers can lose a user who would have tolerated many honest gaps. The guardrail design behind these numbers is covered in hallucinations and guardrails for PMs.
Layer four: the business outcome
Finally, the metric the feature exists to move. This is the same discipline as any product work, but two cautions are specific to AI.
First, connect the AI quality metrics to the business metric explicitly, so you can reason about the chain. "Higher completeness raised containment, which cut support cost per ticket" is a defensible story. A pile of disconnected eval scores is not.
Second, watch for the metric that improves for the wrong reason. Containment can rise because the feature got better, or because it got more willing to confidently answer things it should have escalated. The business number moved, but you bought it with trust you will pay back later. Pair every business metric with a guardrail metric so you can tell the difference.
Assembling the scorecard
A defensible launch scorecard has one number from each layer:
- Quality: the two or three dimensions that matter, scored separately on a representative eval set.
- Cost and latency: cost per unit of value at projected scale, and 95th-percentile latency.
- Trust: containment, and the confidently-wrong rate, tracked separately.
- Business: the target metric, connected to the quality numbers by an explicit chain.
That is what you bring to a launch review instead of a single accuracy figure and a demo. It is also what an interviewer is probing for when they ask how you would measure an AI feature, which is why this shows up in AI PM interview prep.
TL;DR
- Accuracy is thin. It hides which quality dimension you mean, on what input distribution, and at what cost and speed.
- Measure quality as separate dimensions: factual correctness, completeness, faithfulness, format, tone. A feature can pass one and fail another.
- Cost per unit of value at projected scale and 95th-percentile latency decide whether it ships. They are the PM's to own.
- Trust metrics predict retention: containment, escalation quality, and especially the confidently-wrong rate, because trust is lost asymmetrically.
- Pair every business metric with a guardrail metric, because containment can rise for the wrong reason.
- A defensible scorecard carries one number from each layer, connected by an explicit chain to the business outcome.
The reason this stack is hard to fake is that assembling it forces you to have actually measured the feature, not just tried it. In ShipSet, you build the eval suite and the cost model on a real feature, so the scorecard above is something you produce rather than something you describe, and by Day 90 it is a portfolio artifact you can defend line by line.