Architecture Gallery

How to Read These Architectures

Each reference architecture below includes a diagram (zoom with +/−), a component table naming every layer and its responsibility, a numbered step-by-step procedure for implementation and operations, plus technical and enterprise deployment context.

Use the sidebar to jump between architectures. Procedures are written for AI engineers, architects, and auditors who need repeatable, production-grade workflows.

Prompt Engineering Workflow

Architecture Overview

The enterprise prompt lifecycle treats prompts as versioned, testable software artifacts—not ad-hoc strings. Every change flows through design, evaluation, staging, and governed production release.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Requirements & Task Specification

   Document the business objective, input/output schema (JSON Schema), allowed tools, latency budget, and compliance constraints (PII, retention). Map to measurable KPIs: accuracy, refusal rate, citation fidelity.

2. 2

   Step 2 — Prompt Design

   Author system message (role, constraints), user template (variables), few-shot exemplars if needed, and output format instructions. Separate trusted system channel from untrusted user content to mitigate injection.

3. 3

   Step 3 — Offline Evaluation

   Execute eval suite against frozen model version: unit tests for format, semantic similarity to references, task-specific scorers (exact match, F1, LLM judge). Record baseline metrics in eval dashboard.

4. 4

   Step 4 — Red-Team & Security Review

   Run adversarial prompts (jailbreaks, instruction override via documents). Validate guardrails: PII redaction, topic refusal, tool allowlists. Security signs off before promotion.

5. 5

   Step 5 — Staging Deployment

   Deploy prompt hash vN+1 to staging environment with shadow traffic or internal dogfood. Compare side-by-side with production prompt on held-out queries.

6. 6

   Step 6 — Canary & Production Release

   Route 5% → 25% → 100% traffic via feature flag. Monitor token cost, latency p99, error rate, human escalation rate. Rollback automatically if SLO breach.

7. 7

   Step 7 — Continuous Monitoring

   Weekly regression on golden set; alert on metric drift. Quarterly review with SMEs when model provider upgrades base weights.

LLM Architecture (Production Stack)

Architecture Overview

A production LLM stack decouples user-facing applications from model inference through gateways, routers, and scalable GPU serving—enabling cost control, failover, and observability.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Request Ingestion

   Client sends HTTPS POST with API key/OAuth JWT. Gateway validates schema: messages[], max_tokens, response_format, tools[]. Assign correlation ID.

2. 2

   Step 2 — Authentication & Quota

   Map API key to tenant quota (RPM, TPM). Reject or throttle if exceeded. Attach billing metadata.

3. 3

   Step 3 — Routing Decision

   Router scores candidate models: task type (chat vs embed), sensitivity (on-prem vs cloud), budget. May cascade: try small model first, escalate if confidence low.

4. 4

   Step 4 — Prompt Assembly

   Merge system prompt, conversation history (trimmed to context window), RAG context if applicable. Apply chat template (ChatML, Llama format).

5. 5

   Step 5 — Tokenization & Prefill

   Tokenizer converts text to input_ids. GPU executes prefill pass—processes entire prompt in parallel—building KV cache.

6. 6

   Step 6 — Autoregressive Decode

   Sample next token (greedy, top-p, or temperature). Append to sequence; update KV cache per layer. Repeat until EOS or max_tokens.

7. 7

   Step 7 — Post-Processing

   Parse structured output if JSON mode. Run safety classifiers on completion. Stream tokens via SSE if streaming enabled.

8. 8

   Step 8 — Response & Telemetry

   Return completion + usage {prompt_tokens, completion_tokens}. Emit metrics to Prometheus/Datadog; store trace for debugging.

Transformer Architecture

Architecture Overview

The Transformer is an encoder-decoder (or decoder-only) neural architecture where self-attention replaces recurrence, enabling parallel training and long-range dependency modeling at scale.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Tokenization

   Input text → subword tokens via BPE/SentencePiece. Special tokens: BOS, EOS, PAD. Max length truncation or sliding window for long docs.

2. 2

   Step 2 — Embedding Lookup

   token_id → vector x_i ∈ ℝ^d. Scale by √d_model per original paper.

3. 3

   Step 3 — Positional Information

   Add positional encodings or apply RoPE to Q,K in each layer so model distinguishes order.

4. 4

   Step 4 — Encoder Layer (×N)

   For each layer: (a) Multi-head self-attention with residual + LayerNorm, (b) FFN with residual + LayerNorm. Output: contextualized representations H.

5. 5

   Step 5 — Decoder Layer (×N) [seq2seq]

   Masked self-attention (causal) + cross-attention to encoder output + FFN. Mask prevents attending to future tokens during training.

6. 6

   Step 6 — Decoder-Only [GPT-style]

   Stack of causal self-attention layers only. Training: predict token t+1 given tokens 1..t (teacher forcing).

7. 7

   Step 7 — Output Projection

   Final hidden state h_t linearly mapped to logits over vocabulary. Softmax → probability distribution.

8. 8

   Step 8 — Training Objective

   Minimize cross-entropy loss over next-token predictions. Backprop through all layers; AdamW optimizer; learning rate warmup + cosine decay.

9. 9

   Step 9 — Inference

   Autoregressive generation: sample/argmax next token, append, repeat. KV cache optimizes by caching past K,V tensors.

RAG Pipeline Architecture

Architecture Overview

Retrieval-Augmented Generation (RAG) grounds LLM outputs in externally retrieved documents, reducing hallucination and enabling answers over private, fresh enterprise data.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Document Ingestion

   Connectors pull PDFs, wikis, tickets from SharePoint, Confluence, S3. Extract text (OCR if needed). Attach metadata: source, ACL, timestamp, version.

2. 2

   Step 2 — Chunking

   Split into overlapping segments (512–1024 tokens, 10–20% overlap). Preserve headings hierarchy; optional parent-child chunk linking.

3. 3

   Step 3 — Embedding & Indexing

   Batch-embed chunks; upsert into vector DB with metadata filters. Build sparse index (Elasticsearch) in parallel for hybrid retrieval.

4. 4

   Step 4 — Query Processing

   User question → optional query expansion/reformulation → embed query vector q.

5. 5

   Step 5 — Retrieval

   ANN search top-k (e.g., k=20); apply metadata filters (tenant, classification). Hybrid: fuse dense scores with BM25 via RRF. Re-rank top-20 → top-5 with cross-encoder.

6. 6

   Step 6 — Context Assembly

   Pack chunks into context window budget. Order by relevance; deduplicate near-duplicates; inject citation markers [1][2].

7. 7

   Step 7 — Grounded Generation

   Prompt LLM: system (cite sources only), context blocks, user question. Constrain: 'If not in context, say unknown.'

8. 8

   Step 8 — Post-Validation

   Check citations exist in retrieved set; NLI model verifies answer entailed by context; block if hallucination score high.

9. 9

   Step 9 — Feedback Loop

   Log query, chunks, answer, user thumbs. Hard negatives retrain reranker; stale docs trigger re-ingestion.

Agent Pipeline Architecture

Architecture Overview

An AI agent closes the loop between language reasoning and environment action: it observes state, plans, executes tools, reflects on outcomes, and iterates until the goal is satisfied or limits are hit.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Goal Initialization

   Receive user objective + constraints (budget, time, allowed tools). Load session memory and user profile from long-term store.

2. 2

   Step 2 — State Assessment

   LLM summarizes current knowledge gaps. Decide: answer directly, retrieve knowledge, or invoke tool.

3. 3

   Step 3 — Planning

   Emit plan: ordered subtasks with success criteria. For ReAct: interleave Thought → Action → Observation traces.

4. 4

   Step 4 — Tool Selection

   Model outputs structured tool call: {name, arguments} per JSON schema. Validator checks types, ranges, auth scope.

5. 5

   Step 5 — Tool Execution

   Run API call / SQL query / code in sandbox. Capture result or error. Timeout and retry with backoff on transient failures.

6. 6

   Step 6 — Observation Integration

   Append tool output to conversation state. Truncate/summarize if exceeds token budget.

7. 7

   Step 7 — Reflection

   Critique: Did subtask succeed? Update plan. Detect loops (same action repeated) → break or escalate to human.

8. 8

   Step 8 — Termination

   Stop when goal met, max iterations (e.g., 10), cost cap, or user approval. Synthesize final response with audit trail of actions.

Multi-Agent Ecosystem Architecture

Architecture Overview

Multi-agent systems distribute complex work across specialized agents coordinated by a supervisor, message bus, or shared state graph—mirroring organizational division of labor.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Task Decomposition

   Supervisor LLM analyzes request; assigns subtasks to agents by capability matrix (who can use which tools).

2. 2

   Step 2 — Parallel Dispatch

   Independent subtasks run concurrently (e.g., research + data pull). Dependent tasks queued.

3. 3

   Step 3 — Intra-Agent Loop

   Each worker runs full perceive-plan-act cycle on its subtask. Reports status: running, blocked, done.

4. 4

   Step 4 — Inter-Agent Messaging

   Agents pass artifacts: research summary → writer; code patch → reviewer. Schema: {from, to, payload, message_type}.

5. 5

   Step 5 — Conflict Resolution

   Supervisor detects contradictory conclusions; triggers debate round or defers to human.

6. 6

   Step 6 — Aggregation

   Merge partial outputs; deduplicate; ensure consistent tone/format. Final QA agent checks completeness.

7. 7

   Step 7 — Delivery & Trace

   Return unified deliverable. Distributed trace links each agent's tool calls for debugging.

Model Context Protocol (MCP) Architecture

Architecture Overview

MCP standardizes how AI applications discover and invoke tools and resources from external servers—replacing bespoke integrations with a uniform client-host-server protocol.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Server Registration

   Admin configures MCP server executable + env vars in host config. Host spawns server process or connects to remote endpoint.

2. 2

   Step 2 — Capability Discovery

   Client sends initialize → server returns ServerInfo, capability flags (tools, resources, prompts). Client caches tool schemas.

3. 3

   Step 3 — Resource Listing

   Client requests resources/list → server returns URIs (e.g., file://, db://schema). Model can read resource contents on demand.

4. 4

   Step 4 — Tool Invocation Request

   LLM selects tool; host sends tools/call with arguments. Server validates against JSON Schema.

5. 5

   Step 5 — Server Execution

   Server runs query/API/action in isolated context. Returns structured content or error {code, message}.

6. 6

   Step 6 — Result to Model

   Observation injected into conversation. Model continues reasoning or chains additional tool calls.

7. 7

   Step 7 — Lifecycle Management

   Health checks, reconnection on failure, permission scopes per server (read-only vs write).

Enterprise AI Platform Architecture

Architecture Overview

An enterprise AI platform unifies experience channels, agent orchestration, model serving, data planes, and governance into a single governed stack deployed on private cloud or hybrid infrastructure.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Identity & Access

   User/service authenticates via SSO (OIDC). RBAC maps roles to models, datasets, tools. Attribute-based access on document collections.

2. 2

   Step 2 — Request Routing

   Traffic enters WAF → API gateway → platform router. Classify workload: chat, embed, batch, agent.

3. 3

   Step 3 — Orchestration

   Agent framework loads workflow definition. Retrieves context from data plane; calls model gateway for LLM steps.

4. 4

   Step 4 — Model Inference

   Gateway selects endpoint (on-prem Llama vs Azure OpenAI). Applies rate limits, prompt templates from registry.

5. 5

   Step 5 — Data Access

   Vector search and SQL tools enforce row-level security. Query logs attributed to user and use case.

6. 6

   Step 6 — Governance Hooks

   Every response logged: inputs hash, output, model version, retrieved doc IDs. High-risk flows queue for human approval.

7. 7

   Step 7 — Observability & Cost

   Metrics per tenant: tokens, GPU hours, $/request. Chargeback to business units. Dashboards for SRE.

8. 8

   Step 8 — Lifecycle Management

   Model/prompt promotion through dev → staging → prod. Disaster recovery: multi-region failover for gateway.

AI Governance Platform Architecture

Architecture Overview

AI governance platforms operationalize responsible AI: inventory models, assess risk, enforce policies, maintain audit trails, and coordinate human oversight across all production AI workloads.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Use Case Registration

   Team submits AI initiative: purpose, data types, affected users, automation level. Governance assigns risk tier and required controls.

2. 2

   Step 2 — Impact Assessment

   Complete DPIA/FRIA where required. Document limitations, bias testing plan, fallback when AI unavailable.

3. 3

   Step 3 — Model & Data Approval

   Only registered model IDs allowed in production namespace. Training data provenance documented; prohibited sources blocked.

4. 4

   Step 4 — Policy Deployment

   OPA policies enforce: no PII to public APIs, EU data stays in EU region, financial advice requires disclaimer + human review.

5. 5

   Step 5 — Runtime Enforcement

   Gateway intercepts requests; checks policy; blocks or routes to approved path. Tool calls validated against allowlist.

6. 6

   Step 6 — Continuous Monitoring

   Drift detection on input distribution and output quality. Incident playbooks for model failure, prompt injection spike.

7. 7

   Step 7 — Periodic Audit

   Quarterly sample of traces reviewed by compliance. Evidence package for regulators: policies, logs, test results.

AgentOps Framework Architecture

Architecture Overview

AgentOps extends MLOps/LLMOps to autonomous agents: distributed tracing of multi-step reasoning, evaluation of trajectories (not just final text), cost tracking, and regression gates before release.

Core Components

Step-by-Step Procedure

End-to-end flow from initiation to production operation. Follow these steps when designing, implementing, or auditing this architecture.

1. 1

   Step 1 — Instrumentation

   Wrap every LLM invoke, tool call, retrieval with span: parent_trace_id, span_id, latency, tokens, status. Propagate context across async agents.

2. 2

   Step 2 — Trace Collection

   Export spans to collector. Build waterfall view: which tool caused failure, token hotspots.

3. 3

   Step 3 — Define Eval Suites

   Task-specific datasets with expected outcomes: final answer match, required tool sequence, max steps, max cost.

4. 4

   Step 4 — Offline Regression

   On each PR: run agent against eval suite in sandbox. Compare to baseline: success rate, avg steps, P95 latency, $/run.

5. 5

   Step 5 — Canary in Production

   Shadow 10% traffic; compare live metrics to baseline. Auto-rollback agent graph version on regression.

6. 6

   Step 6 — Human Feedback Loop

   Thumbs-down triggers trace review; add failure to golden set; fix prompt/tool; re-run CI.

7. 7

   Step 7 — Cost & Capacity Planning

   Dashboard: $/successful task by use case. Right-size models; cache frequent retrievals.