The AI PM take-home is where most transitioning candidates lose the role, and they lose it in a specific, avoidable way: they answer it like a traditional PM take-home. They write a crisp problem statement, a tidy set of user stories, a prioritized roadmap, and a metrics section that says "we will track engagement." It is a competent PM document. It scores in the middle of the stack and gets passed over, because it never demonstrates the three things the take-home was actually built to test.
This guide is about what the graders are really reading for, and how to structure a response that surfaces it. If you have an assignment in hand or expect one, this is the difference between a mid-stack submission and one that gets the interview.
What the take-home is actually testing
A traditional PM take-home tests problem framing, prioritization, and communication. An AI PM take-home tests those too, but it layers three AI-specific things on top, and those three are where the decision is made:
- Do you think in evaluation? Can you define what "good" means for a probabilistic feature and how you would measure it, when there is no obvious correct answer.
- Do you think in cost and latency? Do you treat the per-call price and speed as product constraints, or ignore them the way you could with traditional software.
- Do you design for failure? Do you plan for the feature being wrong, hallucinating, or hitting inputs it cannot handle, or do you only design the happy path.
A submission that nails classic PM craft but is silent on these three reads as "a good PM who has not internalized AI." A submission that covers them, even at a company where the traditional craft is slightly rough, reads as "someone who gets it." The three AI dimensions are the differentiator.
The structure that scores
Use a structure that forces the three signals to the surface instead of burying them. A version that works:
- Problem and users. Short. Standard PM framing. Do not over-invest here; it is table stakes, not the differentiator.
- The AI approach and why. What kind of AI solution, and critically, whether AI is even the right tool versus deterministic software. Naming that you considered not using AI is itself a signal of judgment.
- What good looks like, and how you would measure it. The evaluation section. This is where you separate from the pack. Define the quality dimensions, describe an eval set, state a bar for shipping.
- Cost and latency. A back-of-envelope estimate of cost per interaction and the latency budget, and what that implies for the design. Even a rough number beats silence.
- Failure modes and guardrails. How the feature behaves when it is wrong or does not know. Refusal behavior, escalation, fallback.
- Roadmap and metrics. Standard PM close, but tie the metrics back to the eval and guardrail sections so it is coherent.
Notice the middle three sections are the AI-specific ones, and they should carry the most weight. Most candidates spend their effort on problem framing and roadmap, the table-stakes bookends, and rush the middle. Invert that.
The section most candidates skip
If there is one section that decides the outcome, it is "what good looks like, and how you would measure it." Almost every mediocre submission hand-waves it: "we will measure accuracy and user satisfaction." That sentence tells the grader nothing and signals that you have not thought about evaluation at all.
A strong version does four concrete things:
- Names the specific quality dimensions for this feature, not generic accuracy. For a summarizer: factual correctness, completeness, and faithfulness to the source.
- Describes how you would build an eval set: how many cases, how representative, how you would get labels.
- States a shipping bar: what score on which dimension is good enough to launch, and why.
- Acknowledges the trade-off: where you would accept lower quality to hit cost or latency, and where you would not.
Writing that section well demonstrates all three AI signals at once, because a real eval plan implies you understand the feature is probabilistic, that quality has to be defined, and that trade-offs against cost exist. The mechanics of building one are in building AI eval suites as a PM, and the metrics to name are in AI product metrics beyond accuracy.
A worked fragment of a strong response
Suppose the prompt is: "Design an AI feature that drafts replies to customer support emails for agents to review."
A weak evaluation section: "We will measure reply quality and agent satisfaction, and iterate based on feedback."
A strong one: "Good means the draft is factually correct against our help docs, complete enough that the agent edits rather than rewrites, and in our support tone. I would build a 60-case eval set sampled from real historical tickets across our top intents, labeled by two senior agents for correctness, completeness, and tone on a three-point scale. Shipping bar: 90 percent correctness and 80 percent 'edit not rewrite' rate, because a draft the agent rewrites saves no time and a factually wrong draft that an agent misses is worse than no draft. On cost, at roughly the input of the ticket plus our help context and a short draft output, I estimate a few cents per draft, well under the labor it saves, so cost is not the binding constraint here; latency under three seconds is, because agents work in a queue. For failure: when retrieval finds no relevant policy, the feature should surface 'no confident draft, handle manually' rather than guess, because a confident wrong draft in a support context is a trust and compliance risk."
That fragment demonstrates evaluation thinking, cost awareness, latency awareness, and failure design in one paragraph. That is what a top submission reads like.
Common mistakes that sink take-homes
- Answering it as a traditional PM. Great problem framing and roadmap, silent on eval, cost, and failure. Reads as not-yet-AI.
- Generic metrics. "We will track accuracy and satisfaction" with no dimensions, no eval set, no bar. The single most common tell.
- No cost or latency at all. Designing as if the feature were free and instant. Even a rough estimate beats silence.
- Only the happy path. No plan for the feature being wrong or not knowing. In AI, the behavior of being wrong is part of the design.
- Over-engineering the framing. Pages of problem discovery and market sizing, rushing the AI-specific middle. Invert the effort.
TL;DR
- The AI PM take-home tests three things on top of classic PM craft: evaluation thinking, cost and latency awareness, and failure design.
- Structure it so those three sit in the middle and carry the most weight, not buried under problem framing and roadmap.
- The section that decides the outcome is "what good looks like and how you would measure it." Hand-waving it is the most common way to lose.
- A strong eval section names specific quality dimensions, describes an eval set, states a shipping bar, and acknowledges the cost trade-off.
- Include a rough cost-per-interaction and a latency budget. Even a back-of-envelope number beats silence.
- Design the behavior of being wrong: refusal, escalation, fallback. The happy path alone is a fail.
The reason this take-home is hard to fake is that a convincing evaluation and cost section is much easier to write when you have actually built one. In ShipSet, you produce a real eval suite, a cost model, and a guardrail spec on a shipped feature, so when the take-home asks for exactly those, you are describing work you have done, not improvising. The interview that follows is covered in AI PM interview prep.