A traditional PM designs the happy path and a handful of error states. An AI PM has an extra job that has no equivalent in deterministic software: designing the behavior of being wrong. AI features hallucinate. They produce fluent, confident, entirely fabricated answers, and they will do this in production no matter how good the model is. The feature being occasionally wrong is not a bug you can fully fix. It is a property you have to design around.
The difference between a trustworthy AI product and a dangerous one is not whether it ever hallucinates. It is whether the failure was designed or discovered. This is the playbook for designing it.
Why hallucinations happen, in plain terms
A language model generates text by predicting plausible continuations. It is optimized to produce fluent, likely-sounding output, not to be true. When it does not have the right information, it does not stop. It produces the most plausible-sounding answer anyway, and plausible-sounding is not the same as correct. The result is a confident, well-written, false answer, which is the most dangerous kind because it does not look like an error.
This matters for the PM because it means you cannot rely on the model to know when it does not know. The safeguards have to be built around it, in the product. That is what guardrails are.
The guardrail patterns that actually contain it
There is no single fix. There is a stack of patterns, and you choose the ones that fit the risk of your feature.
Ground it in sources. The most effective single move for factual features is to make the model answer from retrieved documents rather than its own memory, and to bind the answer tightly to those sources. This is RAG, and it turns "what does the model believe" into "what do our documents say." Covered in RAG explained for PMs. Grounding does not eliminate hallucination, the model can still drift from the sources, but it sharply reduces it and makes the failures traceable.
Design the refusal. The feature must be able to say "I do not know" or "I could not find this," and it must be willing to. A feature that always answers will always fabricate on questions it cannot handle. The willingness to refuse is a design choice, and often the single highest-trust one you make. Covered more below.
Verify before showing. For high-stakes outputs, add a check between generation and display: a second pass that confirms the answer against the sources, a rules check on format or claims, or a confidence threshold below which the answer is withheld or flagged. A cheaper model plus a verification step frequently beats a single expensive call on both cost and safety.
Escalate to a human. For anything where a wrong answer is costly, design the handoff: when confidence is low or the topic is sensitive, route to a person rather than answer. The quality of this escalation, catching the right cases, is itself something to measure.
Cite sources to the user. Showing where an answer came from lets the user verify it and shifts the product from "trust me" to "here is my evidence." It both reduces the damage of a wrong answer and builds trust in the right ones.
Designing the behavior of not knowing
This deserves its own section because it is the most underrated guardrail and the most PM-owned. Most hallucination damage happens on questions the feature should never have tried to answer. The fix is not a better model, it is a product that knows how to decline.
Three decisions are yours:
- When should the feature refuse? Define the conditions: no relevant source found, confidence below a threshold, a topic flagged as sensitive or out of scope. This is a product spec, not a model setting.
- How should the refusal read? "I could not find this in our documentation, here is how to reach a human" preserves trust. A blunt "I cannot help with that" erodes it. The wording is UX work.
- What is the fallback? When the feature declines, what happens next. A dead end frustrates; a graceful handoff to search, a human, or a related answer keeps the user moving.
A feature that answers everything is a feature that fabricates on the questions it should have declined. The willingness and grace to say "I do not know" is, counterintuitively, one of the strongest trust features you can ship.
Matching guardrails to risk
Do not put the same guardrails on every feature. Match the stack to what a wrong answer costs.
- Low stakes (a brainstorming assistant, a draft generator with a human reviewing every output): light guardrails. The human is the safety net. Grounding and refusal are enough.
- Medium stakes (a support assistant, an internal knowledge tool): grounding, refusal design, source citation, and monitoring. A wrong answer misleads a user but is recoverable.
- High stakes (anything touching money, legal, medical, or compliance): the full stack plus human-in-the-loop. Verification before display, mandatory escalation on sensitive topics, and a bias toward refusing over guessing. Here, a confident wrong answer is a real-world harm, and the design should prefer saying nothing to saying something wrong.
The PM's job is to place the feature on this spectrum honestly and resist pressure to ship a high-stakes feature with low-stakes guardrails because it demos well.
The metrics that tell you it is working
Guardrails are only real if you measure them. The key numbers:
- Confidently-wrong rate. How often the feature is wrong while sounding certain. The most trust-destroying failure, tracked separately from overall error rate.
- Honest-refusal rate on unanswerable questions. How often it correctly declines when it should. A high number here is good, it means the refusal design works.
- Over-refusal rate. How often it declines questions it could have answered. The failure mode of guardrails that are too aggressive. You are balancing this against the confidently-wrong rate.
- Escalation precision. When it hands off to a human, was that the right call.
These come from an eval set and from production monitoring, covered in AI product metrics beyond accuracy and monitoring AI features in production. Without them, "we added guardrails" is a claim, not a fact.
TL;DR
- AI features hallucinate because models are built to produce plausible text, not true text, and they do not reliably know when they do not know.
- You cannot eliminate hallucination. You design around it. The difference between a trustworthy and a dangerous product is whether the failure was designed or discovered.
- Guardrail stack: ground answers in sources, design the refusal, verify before showing, escalate to humans, and cite sources.
- Designing the behavior of not knowing is the most PM-owned guardrail: when to refuse, how the refusal reads, and what the fallback is.
- Match guardrails to stakes. Do not ship a high-stakes feature with low-stakes safeguards because it demos well.
- Measure it: confidently-wrong rate, honest-refusal rate, over-refusal rate, and escalation precision. Guardrails you do not measure are guardrails you do not have.
The reason this is hard to learn from reading is that the failure modes only become vivid when you watch your own feature confidently make something up. In ShipSet, you build a real feature, design its guardrails, and measure the confidently-wrong rate against an eval set, so the playbook above is something you have operated rather than a checklist you have memorized.