How to use this review
Take one concrete use case and one representative end-to-end request. Bring the architecture and sequence, not only a vendor slide. For each question, record the decision, evidence, assumptions, unresolved risk, and owner. ‘We will decide later’ is a valid answer only when it has an owner and a release gate.
1–2. Value and prohibited scope
1. What decision or job improves, and what evidence will demonstrate that improvement? A production use case needs a measurable outcome connected to a real user or process. Model quality metrics may support that outcome, but they are not a substitute for it.
2. What is explicitly prohibited? Name the users, decisions, data, actions, and contexts that are outside the release. Prohibited scope is particularly important when a general-purpose model appears capable of doing more than the approved service. The application should enforce the boundary instead of relying on a disclaimer in a prompt.
- Bring: use-case contract, success measures, prohibited actions, intended users, and accountable business owner.
5–6. System boundaries, identity, and authorization
5. Can the team trace one request across every trust boundary? Show the user entry point, orchestration, model and retrieval providers, tools, enterprise APIs, stores, and operational services. Mark where untrusted content enters and where data or effects leave organisational control. The five-layer enterprise AI architecture is one way to make that path visible.
6. Which identity authorizes each data access and effect? User identity, workload identity, delegated authority, and provider credentials should not collapse into one broad service account. Policy decisions belong in deterministic enforcement points. Model output can supply proposed parameters; it must not grant itself a permission.
- Bring: context and sequence diagrams, data flows, trust boundaries, identity propagation, policy decision points, and least-privilege grants.
7–8. Data, retrieval, and evidence
7. What data enters the model path, and who owns its quality, permission, retention, and deletion? Include prompts, retrieved content, tool output, conversation state, feedback, evaluation records, and operational logs. State what providers may retain and how confidential or personal data is minimized.
8. When the system uses RAG, can every material claim be connected to current, permitted evidence? Apply authorization before context construction, preserve source lineage, surface conflicts, and define abstention. Review the full secure RAG architecture rather than treating a vector database as the control boundary.
- Bring: data inventory, classification, lineage, access model, retention decisions, evidence contract, and retrieval evaluation set.
9–10. Effects, tools, memory, and semantic supply chain
9. What happens if an effect succeeds but the response is lost? Define idempotency, reconciliation, retries, compensation, and the state used when an outcome is unknown. Bind human approval to the exact action and revalidate it at execution. A model's guess about provider state is never confirmation.
10. Which mutable inputs can change behaviour? Models, prompts, retrieval indexes, tool descriptions, external agents, packages, policies, and memory form a semantic supply chain. Inventory them, constrain their sources, version material changes, and decide which change requires security review and re-evaluation before release.
- Bring: tool contracts, canonical action schema, idempotency keys, approval record, memory model, dependency inventory, change process, and rollback plan.
11. How will the system be evaluated and operated?
Define offline evaluation before release and online signals after release. Cover task success, grounding, authorization, harmful outputs, unsafe effects, abstention, latency, resilience, and cost. Include ordinary, edge, and adversarial cases that reflect the intended environment. A single aggregate score can hide a critical failure class.
Connect telemetry from the initiating goal to evidence, decisions, model calls, tools, effects, and outcome. Name service levels and alerts that lead to an action. Plan how operators disable a model or tool, revoke authority, quarantine a source or memory, stop active work, roll back, and reconstruct an incident without exposing unnecessary sensitive content.
- Bring: evaluation catalogue and thresholds, release gates, production indicators, service objectives, cost limits, dashboards, alerts, incident runbook, and rollback evidence.
12. Who owns the decision and residual risk?
Architecture is incomplete when every risk belongs to ‘the AI team.’ Name the business owner, service owner, model or AI-risk owner, data owners, security owner, and people authorized to release, pause, and retire the system. Record which residual risks are accepted, by whom, for how long, and what evidence would reopen the decision.
Ownership should survive organisational and provider boundaries. Outsourcing a model, vector store, or platform does not outsource the outcome. The team still needs the evidence and operational access required to meet its responsibilities.
- Bring: decision log, architecture decision records, risk owners, residual-risk acceptance, review date, and production go/no-go authority.
The useful output of a review
A good review produces a short, prioritized decision record: what is approved now, what must change before release, what evidence is missing, which risks are accepted, and who owns every next step. It does not need to remove all uncertainty. It needs to make uncertainty bounded and visible.
Independent review can help when the team is committed to one design, when authority or data paths are difficult to see, or when a consequential release needs a clear challenge. Ardevant's architecture advisory focuses on that kind of bounded decision or design review; private teams can also work through the method together in private training.