The model is not the system
Many AI initiatives begin with a useful demonstration: a person enters a question, an application sends context to a model, and a convincing answer appears. That proves that a capability may be valuable. It does not yet show that the capability can be operated safely, reliably, or economically inside an enterprise.
A production design must explain more than which model it uses. It must show where identity is established, how data is selected, which actions are permitted, where policy is enforced, what happens when dependencies fail, and how an operator can reconstruct an outcome. Those decisions belong to the architecture around the model.
Five runtime layers
Start with one representative request and trace it through five layers. The names matter less than whether every responsibility has an explicit home.
- Experience layer: the user or system entry point, the interaction contract, accessibility, disclosure, and the place where the intended job is made clear.
- AI application and orchestration layer: prompt and context assembly, workflow state, routing, deterministic business rules, fallbacks, and response shaping.
- Model, retrieval, and tool layer: model gateways, retrieval services, approved tools, safety boundaries, and provider-specific adapters.
- Enterprise data and API layer: systems of record, document repositories, business APIs, event streams, ownership, classification, and authoritative access decisions.
- Runtime platform layer: compute, networking, secrets, deployment, scaling, resilience, and the operational services on which the other layers depend.
The layers force a useful distinction between reasoning and authority. A model may propose a search, classification, or action. The enterprise systems and deterministic controls around it decide whether that operation is allowed and how it is performed. This prevents a fluent output from silently becoming an authorization decision.
Three control planes cross every layer
Layer diagrams show where work happens. Control planes show how the system remains governable from end to end. If a control appears in only one box, ask what happens before and after that box.
- Security and policy plane: identity, authorization, data classification, trust boundaries, input and output handling, tool permissions, secrets, and incident containment.
- Evaluation and release plane: acceptance criteria, offline test sets, adversarial cases, human review, rollout gates, model or prompt change controls, and rollback decisions.
- Observability and economics plane: request traces, quality signals, dependency health, latency, token and tool cost, business outcomes, alerts, and accountable ownership.
These planes should follow the same correlation from the original request to the final answer or effect. A dashboard that reports only model latency cannot explain which evidence was retrieved. A security log that records only an API call cannot show which user authority caused it. A cost report that cannot connect spend to a useful outcome cannot support an operating decision.
Draw the request sequence, not only the boxes
A static component diagram can hide the most consequential behaviour. Add a sequence for one normal request, one denied request, and one degraded request. Show when identity and policy are evaluated; which context crosses each boundary; where untrusted text enters; which system owns the final decision; and what evidence is retained.
For a retrieval-assisted answer, for example, the sequence should show the authenticated subject, entitlement filters, retrieval query, selected passages, source references, model request, output checks, and response. The companion article on secure RAG architecture examines this path in detail.
Also draw failure branches. What does the user see when retrieval is unavailable? Can the system distinguish no evidence from no permission? If the model provider times out after receiving the request, may the application retry? A production architecture is defined as much by these branches as by its successful path.
The gaps that usually appear after a prototype
When a proof of concept reaches architecture review, the missing pieces are often ordinary enterprise concerns that the demonstration never needed to resolve.
- The use case has no measurable outcome, prohibited scope, or named owner.
- The application trusts model output where a deterministic policy decision is required.
- Retrieved data has no lineage or user-specific authorization filter.
- Prompts, models, tools, and indexes can change without an evaluation and release gate.
- Logs reveal latency and errors but cannot reconstruct evidence, decisions, or downstream effects.
- Fallback behaviour, retention, cost limits, and incident containment are implied rather than designed.
None of these gaps is solved by choosing a more capable model. They require system boundaries, ownership, controls, and evidence. Finding them early is useful: each gap can become a concrete architecture decision instead of a late production surprise.
A compact review for the first architecture
Before discussing products, ask the following questions against one end-to-end request:
- What user or business decision does the system improve, and how will we know?
- What is explicitly out of scope, and which actions must never be delegated to a model?
- Where are identity, authorization, and data classification enforced?
- What evidence may the model use, and can every important claim be traced to an allowed source?
- Which model, retrieval, prompt, or tool changes require re-evaluation before release?
- How do we detect poor quality, policy violation, dependency failure, and uncontrolled cost?
- Who owns the service, the model risk, the data, and the decision to stop or roll back?
For a fuller production conversation, use the 12-question AI architecture review. The output should not be a perfect diagram. It should be a set of reviewable decisions, open risks, owners, and evidence for the next release gate.