What I mean by "production AI"
A demo and a production system differ by about six things, and none of them are the model. A short list of what actually has to exist before an AI feature is safe to ship.
Every AI project I have joined had a working demo before it had a working system. That is not a criticism of the demo — a demo is the cheapest way to find out whether an idea has any substance at all. The problem is what happens next, when the demo is mistaken for a head start.
The gap is not the model
A demo proves that a model can produce a good answer once, for an input someone chose. Production means it produces an acceptable answer for inputs nobody chose, at a cost somebody signed off, with a trace that survives an audit.
Concretely, the things that were missing every time:
- A definition of "good" that is written down and that two people would score the same way.
- A test set drawn from real traffic rather than the examples used while building.
- Regression on every prompt change, because prompts are code and nobody treats them that way.
- Cost and latency budgets per request, enforced rather than reported after the fact.
- An access boundary — who may call this, with whose data, and what the model may reach.
- Someone on call who can explain a bad output without reading the prompt for the first time.
None of that list is about model selection. Model selection is the part teams enjoy arguing about, which is why it is usually the only part that is finished.
Start with the evaluation
The most useful thing I do on a new engagement is refuse to touch the prompt for the first week. Instead we collect a hundred real inputs, argue about what a correct response looks like, and turn that argument into a scoring function — sometimes a rubric run by an LLM judge, sometimes something as dull as a regex.
Once that exists, everything downstream gets easier. Prompt changes become measurable. Model upgrades become a decision rather than a leap of faith. And the conversation with the compliance team turns from "we think it works" into a number with a history behind it.
If you cannot measure the feature, you cannot improve it and you cannot defend it.
The unglamorous conclusion
Production AI is mostly ordinary software engineering applied to an unusually non-deterministic dependency. The teams that ship it are not the ones with the best prompt. They are the ones that treated the model like any other unreliable upstream service and built the scaffolding around it.