What Three Years of Production AI Actually Taught Me
I started building with LLM APIs in March 2022, when GPT-3.5 became accessible. Not as a side experiment — as the core of real products with real users.
Three years later, I have a set of beliefs about production AI development that I didn't have going in, and a few things I believed early that turned out to be wrong. Here's what actually stuck.
The Demo Problem is Real and It's Structural
Every LLM integration looks good in a demo. The inputs are clean, the prompt is tuned for the exact use case being shown, and nobody is asking it the weird question that breaks the pattern.
Production is different. Real users give you malformed input, contradictory context, and requests the system was never designed to handle. The failure modes you didn't anticipate become support tickets within days of launch.
The fix isn't prompt engineering. It's architecture. Specifically: assume your AI component will fail on some percentage of inputs and design the system to handle that gracefully — flag it, route it to a human review queue, log it with enough context to diagnose. The systems I've built that held up best in production all had a human-in-the-loop fallback that felt like a feature, not an admission of failure.
Context Windows Are a Product Design Problem, Not a Technical Problem
Early on I spent a lot of time on the technical side of context management — chunking strategies, embedding models, retrieval tuning. That stuff matters. But the more important decisions are product decisions.
What does the system actually need to remember? For how long? With what fidelity? A tenant management system that handles maintenance requests doesn't need to remember the full history of every tenant interaction — it needs to remember the last open request and the maintenance vendor preferences for that unit. That's a product decision that makes the technical problem tractable.
Most context window problems I've seen are really scope creep in disguise. "Make it smarter" usually means "make it remember more things" — and that question, answered without product discipline, leads to expensive, fragile systems.
Vendor Lock-In Is a Real Risk and It's Easy to Avoid
I built my first several AI integrations directly against OpenAI's API. When Anthropic released Claude and it handled certain tasks measurably better, migrating those integrations was painful — not because the APIs are that different, but because the prompt design had assumptions baked in about how the model would respond.
Every production AI system I build now has an abstraction layer between the application logic and the model provider. Model selection, prompt routing, and fallback logic are configuration, not code. It costs maybe two hours to set up correctly. It has saved many more hours when I wanted to swap a model or when a provider had an outage.
The models are commoditizing faster than anyone expected. Vendor neutrality is cheap insurance.
Accuracy Targets Are Negotiating Positions
"We need 95% accuracy" is almost always a starting position, not a requirement. The real question is: what happens to the 5% that's wrong?
For the AI intake pipeline I built for automotive shop operations — license plate recognition feeding into VIN lookup and customer record hydration — we targeted 85-90% accuracy at launch and built the admin correction workflow at the same time. Corrections fed back into extraction quality over time. The 10-15% that required human review was handled in a queue that took seconds per record, not minutes.
The system was useful at 85%. It would have taken months longer to reach 95%, and the correction workflow was better than a perfect system with no fallback anyway — because edge cases always exist.
Define what happens to the failures before you define the accuracy target.
The Orchestration Layer Is Where Things Actually Break
Multi-step agentic workflows look clean on a whiteboard. An agent decides what to do, calls a tool, gets a result, decides what to do next. Simple.
In production, this is where I've seen the most failures. The model loops. It calls a tool with malformed parameters. It interprets a tool result incorrectly and takes an action downstream that's hard to reverse. It finishes successfully from its own perspective but produces output that doesn't match what the application expected.
The patterns that fixed these problems: explicit state management at every step so the system can recover from failures without starting over, loop detection with hard limits, structured output contracts with validation and retry logic on malformed responses, and an audit trail that logs every decision point with enough context to diagnose failures after the fact.
None of these are exciting to build. All of them are necessary.
The Model Isn't the Hard Part
This is the thing I'd go back and tell myself in early 2022: the model is the easy part. OpenAI, Anthropic, Google — they've solved the hard problem of making a capable model available via API. Your job is to build a system around it that's reliable, recoverable, and honest about what it doesn't know.
The hard parts are the same hard parts as any other software: clean data models, graceful failure handling, good UX for the edge cases, an architecture that can evolve as the underlying models change.
The builders who treat AI as a magic component they're wrapping will have brittle systems. The builders who treat it as a powerful but fallible component in a larger system will build things that last.
I've been on the right side of that distinction for three years. It's why the platforms I build still work when things don't go according to plan.
If you're building something AI-powered and want to talk through the architecture, let's talk.