How to Write a PRD for an AI Feature (Lessons From Shipping One That Judged Itself)

The first version of Draftic's quality score shipped exactly to spec, and that turned out to be the problem.

Draftic has a feature called multi-model validation: after one AI model generates your PRD, a second model reviews it and gives you a quality score before you share it with your team. The PRD I wrote for that feature was, I thought, pretty tight. "The validator returns a score from 0–100 with section-level feedback. Score displays on the document header. Feedback appears as inline annotations."

Engineering built exactly that. It worked in every demo. Then I ran the same PRD through the validator twice and got a 71 and an 84.

Nothing was broken. The model was doing what large language models do: sampling from a distribution. But my spec had assumed a deterministic world where the same input produces the same output, and the feature I'd shipped was confidently telling users two different things about the same document. A user who gets a 71, changes one sentence, and gets an 84 doesn't learn their document improved. They learn the score is noise.

I've now rewritten that spec three times, and each rewrite taught me something a standard PRD template never would have. Here are the four places where AI features break the traditional PRD, and what to write instead.

Why your PRD template fails on AI features

A traditional PRD is a contract built on a deterministic promise:

1. When the user does X, the system does Y.
2. Acceptance criteria are binary.
3. QA verifies against the spec.
4. The feature works or it doesn't.

AI features void that contract. The same input can produce different outputs. "Correct" isn't a single value, it's a distribution. And the system's behavior can shift under your feet when the model provider ships an update you didn't ask for.

There's a popular response to this problem right now: throw out the PRD and replace it with evals. I think that's half right, and the wrong half is expensive. Evals are a great answer to "how do we verify behavior we can't fully specify?" They are a bad answer to "why are we building this, for whom, and what tradeoffs did we accept?" An eval suite can tell you the model passes 94% of test cases. It cannot tell a new engineer why you chose an 85% threshold instead of 95%, why you picked the smaller model, or what happens to the user during the other 6%. That reasoning has to live somewhere people can read and review. In other words, a document.

So don't replace the PRD. Renovate it. Four sections need rebuilding.

1. Replace acceptance criteria with behavior contracts

The deterministic instinct is to spec outputs: "the summary must include the top three risks." For an AI feature, you can't guarantee any single output. What you can do is define the range of acceptable behavior, along with the unacceptable.

A behavior contract has three parts:

The floor. What must always be true, on every run? For our validator, the score must always come with section-level reasons. A bare number is a spec violation even if the number is accurate, because a score without an explanation gives the user nothing to act on.

The ceiling. What must never happen? This is where most AI PRDs are silent, and it's the most important part. Our ceiling included the validator must never rewrite the user's content while reviewing it, and must never score above 90 if a required template section is empty. Defining the bad output is usually easier and more valuable than defining the good one, because bad outputs are what users remember.

The variance budget. How much can outputs differ across runs before users stop trusting the feature? This is the section that would have saved me. Our eventual answer: repeated scores on an unchanged document must land within ±3 points. That single sentence forced the engineering conversation that mattered. We moved from a free-form "rate this PRD" prompt to a rubric where the model scores 12 specific, mostly verifiable criteria, and the final number is computed from those judgments rather than generated directly. Score variance dropped from ±13 to ±2. What fixed it wasn't a clever prompt. It was that variance had finally become a written requirement.

2. Put the eval plan in the PRD, not the QA plan

For deterministic features, testing comes after building. For AI features, that order is fatal. If you haven't defined how quality is measured, you can't know whether what you built is any good, can't compare model options, and can't detect regression when the underlying model changes.

Your PRD should answer four questions before engineering starts:

What's the golden set? A fixed collection of real inputs with known-good expected behavior. Ours is a set of PRDs, some excellent and some deliberately broken (missing success metrics, contradictory requirements, vague user stories), each with the score range and specific issues the validator should catch. State in the PRD where this set comes from. "We'll make some up" is a warning sign; sourcing from real usage, support tickets, or your own document archive is the answer.

What's the rubric? Humans need to grade outputs during development, and they'll disagree wildly without written criteria. Write the rubric in the PRD. If you can't articulate on paper what makes an output good, no model will figure it out for you.

What's the threshold? What pass rate ships? 100% is not available, and pretending otherwise just means the threshold gets decided implicitly, late, and under pressure. Put a number in the document and defend it.

What triggers re-evaluation? Model provider updates, prompt changes, template changes. The eval suite is not a launch gate you pass once. It stays on after launch.

One clarification, because the "evals are the new PRD" crowd will object: yes, the eval suite eventually becomes the living, executable expression of the behavior contract. The PRD is where you decided what the contract should be and recorded why. Tests without recorded intent are how a team ends up enforcing thresholds nobody remembers choosing.

3. Treat cost and latency as product requirements

Traditional software has near-zero marginal cost per action. AI features bill you by the token, per use, forever. That changes product decisions, so it belongs in the product spec.

Write a cost-per-action budget: the maximum you'll spend per meaningful user action, derived from your business model. Draftic is free, so validation costs come straight out of my pocket. My spec caps cost per validation, which immediately ruled out running the largest model on every request and pushed us to a tiered design: a fast, cheap model for the structural checks, the stronger model only for the judgment-heavy criteria. That architecture decision came from the PRD, not from an infrastructure review after the bill arrived.

Do the same for latency, and spec the experience of waiting, not just the number. "P95 under 20 seconds" is incomplete. The real requirement is what the user sees during those 20 seconds. We show which section the validator is currently reviewing. Same wait, completely different perceived wait. Streaming, staged progress, or a plain spinner are product choices someone has to make deliberately, and the PRD is where.

Include a line about model swap tolerance: if you later change the underlying model to save money, what quality bar (per your eval suite in section 2) must the cheaper model clear? This turns future cost negotiations from arguments into lookups.

4. Spec the failure UX like it's the main flow

Every AI feature has a failure budget. The model will refuse, time out, or return something malformed a small but nonzero percent of the time. Users don't judge your feature by the median run. They judge it by the worst run they personally hit, and that's the one they tell people about.

Your PRD needs a failure modes table. Four columns: the failure, how it's detected, what the user sees, what gets logged. Some rows from ours:

• Validator times out → detected by a hard deadline → user gets partial results clearly labeled ("Structural checks complete; deep review didn't finish — retry?") → logged with input size for pattern analysis. Never an infinite spinner, never a blank screen.
• Validator returns malformed output → schema check fails → automatic single retry, then a plain "couldn't complete" state → logged with the raw output. The user never sees JSON debris.
• Confidence is low on a criterion → the score for that criterion carries an "uncertain" flag → the UI shows it as something to review, not a fact. A hedge is honest. False certainty is what actually destroys trust.

Notice that these are all product decisions that look like engineering decisions. Whether a timeout shows partial results or an error, whether low confidence is hidden or surfaced: leave these to whoever happens to be implementing that code path, and your most trust-sensitive UX gets designed by accident, in a Slack thread, at 6pm before launch.

The checklist: 10 sections your AI feature PRD needs

The standard sections (problem, users, goals, non-goals) still apply. Add these:

1. Why AI at all — what this approach enables that deterministic logic can't, and what non-determinism costs you in exchange
2. Behavior floor — what's always true of every output
3. Behavior ceiling — what must never appear
4. Variance budget — how much run-to-run difference users will tolerate
5. Golden set — source, size, and coverage of your eval examples
6. Quality rubric and threshold — how outputs get graded and what pass rate ships
7. Cost-per-action budget — with the model tiering it implies
8. Latency budget and waiting UX — the number and what the user sees during it
9. Failure modes table — failure, detection, user experience, logging
10. Re-evaluation triggers — what events force the eval suite to run again

If your PRD has all ten, you've specced a probabilistic system. If it has none of them and your feature calls a language model, you've written a spec for a different feature than the one you're shipping. I know, because I did. And once my validator actually worked, it would have caught it.

Draftic generates structured PRDs from a plain-text description — and runs every one past a second model before you share it. Generate your first PRD free →