When an AI feature is not doing what you want, there are three levers to reach for: change the prompt, give the model your data at answer time (RAG), or retrain the model on examples (fine-tuning). Engineering teams, left alone, often reach for fine-tuning first, because it sounds the most serious. It is usually the wrong first move. It is the slowest, the most expensive, and the hardest to change later.
A PM does not need to implement any of these. A PM needs to know what each one is for, so the team spends effort in the right order and does not fine-tune its way around a problem a better prompt would have solved in an afternoon. This is the decision guide.
What each one actually does
Prompting is changing the instructions you send the model. You are not changing the model at all, just what you ask and how. Better instructions, examples inside the prompt, a clearer format request, a role. It is instant, free beyond the normal call cost, and endlessly adjustable.
RAG gives the model relevant information at answer time by searching your documents and putting the results in front of it. You are not changing the model, you are changing what it knows for this one answer. It is how you make a general model answer from your specific, current data. The full mechanics are in RAG explained for PMs.
Fine-tuning actually retrains the model on a set of your examples so it internalizes a pattern, a style, a format, or a behavior. You are changing the model itself. It is the heaviest lever: it needs labeled training data, a training run, and a new version to maintain, and you cannot tweak it as easily afterward.
The one-line distinction that resolves most confusion: prompting changes the instructions, RAG changes the knowledge, fine-tuning changes the model.
The question that picks the tool
Before comparing cost or effort, ask one thing: what kind of gap are you closing.
- If the model does not know something specific to you (your docs, your data, today's facts), that is a knowledge gap. RAG.
- If the model knows enough but is not behaving the way you want (wrong format, wrong tone, ignoring an instruction, inconsistent structure), that is an instruction gap. Prompting first.
- If you need the model to consistently produce a specialized style or format that you cannot reliably get from instructions no matter how you word them, and you have many examples of the target, that is a behavior gap deep enough for fine-tuning.
Most product problems are knowledge gaps or instruction gaps. Genuine fine-tuning problems are the minority, which is exactly why fine-tuning-first is usually a mistake.
The order to try them in
Run these in order of cost and reversibility, cheapest and most flexible first:
1. Prompting. Always start here. It is free, instant, and you learn what the model can and cannot do before spending on anything heavier. A surprising share of "we need fine-tuning" turns out to be "our prompt was vague." Get the prompt right and measure it. The patterns worth knowing are in prompting as a PM tool.
2. RAG, if the gap is that the model lacks your specific or current information. Adding retrieval is a real build, but it is far cheaper and more maintainable than retraining, and it keeps your knowledge editable (fix a document, fix the answer).
3. Fine-tuning, only when prompting cannot get the behavior and the problem is genuinely about internalized style or format, and you have the labeled examples to train on. Even then, many teams combine a fine-tuned model with RAG rather than choosing one.
The reason the order matters is that each step teaches you something the next one needs. You cannot write good training data for fine-tuning until you have discovered, through prompting, what good output even looks like.
What each costs, in PM terms
Cost is not only dollars, it is time and future flexibility.
- Prompting: minutes to iterate, essentially free, infinitely adjustable. If you change your mind, you change a sentence.
- RAG: a real engineering build (retrieval, a knowledge base, wiring), ongoing cost per call for the retrieval and the larger prompt, but the knowledge stays editable. Change a document and the behavior changes with no retraining.
- Fine-tuning: upfront cost to gather and label training data (often the biggest hidden expense), a training run, and a version to maintain. Least flexible: to change the behavior you retrain. It also does not solve knowledge freshness, a fine-tuned model is still frozen at training time.
That last point is the most common expensive mistake: teams fine-tune to make a model "know" their data, then discover the data is now stale and the model has no way to learn the new facts. Knowledge freshness is a RAG problem, not a fine-tuning problem. Fine-tuning is for behavior, not for facts.
A worked decision
Say you are building a support assistant for your product. Walk the framework:
- Does it need to answer from your specific, changing help docs? Yes, that is a knowledge gap. RAG is in.
- Is the tone or format off in a way instructions can fix? Probably, so prompt the format and tone first, on top of RAG.
- Do you need it to internalize a specialized style that instructions genuinely cannot produce, with hundreds of labeled examples on hand? Almost certainly not for a support assistant.
Result: RAG plus careful prompting, no fine-tuning. That combination covers the large majority of AI product features you will ship. Fine-tuning enters for narrower cases: a consistent proprietary format, a domain style that resists instruction, a latency or cost optimization where a smaller fine-tuned model replaces a larger prompted one.
The mistakes this framework prevents
- Fine-tuning first. Reaching for the heaviest, least flexible tool before trying the prompt. Expensive, slow, and often unnecessary.
- Fine-tuning for freshness. Trying to teach a model facts it needs to keep current. Those facts go stale and the model cannot update. That is a RAG job.
- Over-prompting a knowledge gap. Endlessly tweaking instructions to make a model produce facts it was never given. No prompt can conjure information the model does not have. That is a RAG job.
- Treating these as mutually exclusive. The real answer is often prompting plus RAG, or fine-tuning plus RAG. They stack.
TL;DR
- Prompting changes the instructions, RAG changes the knowledge, fine-tuning changes the model.
- Ask what gap you are closing: a knowledge gap points to RAG, an instruction gap to prompting, a deep behavior or style gap to fine-tuning.
- Try them in order of cost and reversibility: prompting first, then RAG, then fine-tuning only if needed.
- Fine-tuning does not solve knowledge freshness. A fine-tuned model is still frozen. Use RAG for facts, fine-tuning for behavior.
- Most shipped AI features are RAG plus careful prompting. Genuine fine-tuning cases are the minority.
The reason teams waste months here is that the wrong choice is invisible until the bill or the staleness shows up. In ShipSet, you make this call on a real feature and measure the result, so the framework above becomes a decision you have defended with evidence rather than a diagram. The companion piece on AI cost modeling for PMs shows how to price each option before you commit.