Everything in this series reduces to one principle I kept coming back to: local execution still needs a control plane.
Models can run on a laptop, an edge server, Azure Local, a cloud endpoint, or specialized local hardware. That flexibility is only useful when applications can use it without hardcoding placement decisions and when operators can see, secure, and govern what happened. This post is the reference architecture for the project - the stable one to bookmark or cite while the individual demo posts keep moving around it.
Why Local AI Needs a Control Plane
Enterprises want local AI for good reasons:
- Keep private data near where it is generated.
- Reduce latency.
- Continue operating during network disruption.
- Control inference costs.
- Use local accelerators and edge hardware.
- Avoid sending every operational event to a cloud model.
I share every one of those motivations; they are why this project exists at all. The risk is unmanaged local AI: invisible endpoints, inconsistent model APIs, unclear fallback, missing telemetry, ad hoc secrets, and no proof that restricted data stayed local.
The architecture solves that by separating application intent from model placement.
Components and Request Flow
The high-level components are:
| Component | Responsibility |
|---|---|
| Application | Sends AI requests to one gateway. |
| AI on the Edge Gateway | Exposes OpenAI-compatible chat, embeddings, models, health, and admin endpoints. |
| Router | Selects eligible backends by policy, health, requested model family, latency, classification, and fallback rules. |
| Policy engine | Enforces LocalOnly, PreferLocal, CloudAllowed, AcceleratorPreferred, FallbackDisabled, and NoPromptLogging. |
| Backend registry | Stores local, edge, cloud, Azure Local, Tenstorrent, and mock targets. |
| Knowledge Assistant | Ingests documents, embeds locally, retrieves chunks, cites sources, and routes answer generation. |
| Operations Assistant | Turns camera events and runbooks into incident summaries and recommended actions. |
| Control Dashboard | Shows backend health, route traces, privacy posture, latency, incidents, and failure controls. |
| Telemetry | Emits route decisions, metrics, logs, and KQL-friendly events. |
| Azure operations layer | Arc, GitOps, Monitor, Managed Prometheus, Grafana, Key Vault, Policy, and optional AIO. |
The request flow is deliberately short:
- The application calls the gateway.
- The gateway classifies the request or accepts classification metadata.
- The router filters backends by policy, then by which ones serve the requested model family.
- The router selects a backend or denies the request.
- The gateway calls the selected backend through an adapter.
- The gateway streams or returns the response.
- Telemetry records the route decision and privacy posture.
- Dashboards and KQL queries make the behavior visible.
Step three is where applications stop caring about model names. A request asks for a family - chat or embeddings - and each backend’s registry entry lists the families it serves alongside the model name it uses for them, so the router does the translation and the application never learns that local-chat and cloud-chat are the same request in different places.
Eight steps, one place to look whenever anyone asks what happened to a request.
Deployment Lanes
The same app should run across multiple lanes:
| Lane | Role |
|---|---|
| Laptop local | Conference-safe local path with Foundry Local or mocks. |
| Private cloud edge | Kubernetes or k3s deployment in the local lab. |
| Azure-governed edge | Arc-connected cluster with GitOps, Monitor, Prometheus, Grafana, Key Vault, and policy. |
| Azure Local | Enterprise edge inference path with Foundry Local on Azure Local where preview access and environment readiness exist. |
| Cloud fallback | Azure AI Foundry model endpoint for workloads that are allowed to leave the edge. |
| Accelerator lane | Tenstorrent or mock accelerator backend exposed through the gateway. |
The lanes are not interchangeable, and choosing between them is mostly a trade between evidence and independence. Laptop local is the one lane that never needs anything outside the machine, which is why it is the conference path, and the price is that none of the Azure-side evidence exists there - what you can prove is limited to what the local dashboard shows. Private cloud edge, described in the Tenstorrent buildout, buys the real deployment shape: multiple nodes, real networking, real failure modes, and telemetry that still stops at the cluster boundary.
Azure-governed edge is where the operating model becomes visible to people who are not in the room, and it is also the lane with a hard dependency: Arc and Azure Monitor need outbound connectivity and prior setup, so local application behavior survives a link failure while live management and cloud telemetry do not. Azure Local is the enterprise version of that same lane, gated on preview access rather than on effort. Cloud fallback is the only lane that is a policy decision before it is a deployment decision - a workload reaches it because its classification allowed it to, never because it was the healthiest option available. And the accelerator lane trades a capability question for a performance one: the gateway has to discover what the hardware actually serves before it can prefer it, which is why every accelerator claim in this series is hedged and every accelerator demo has a mock behind it.
Foundry Local is the on-device application runtime. Foundry Local on Azure Local is the enterprise edge inference lane and, as of September 2026, is treated as preview/request-access - re-check the linked overview before repeating that status, since it is exactly the kind of claim that ages fast. Tenstorrent is an accelerator backend, not a separate application architecture. Azure IoT Operations supplies the edge data-plane option for the camera workload.
Six Policies That Do the Governing
The policy vocabulary is intentionally small - six values carrying the whole boundary story. LocalOnly and PreferLocal decide how hard the router tries to stay on the edge, CloudAllowed opens the cloud lane, AcceleratorPreferred biases toward specialized hardware when it is healthy and capable, FallbackDisabled stops the router from trying a second backend after the first one fails, and NoPromptLogging keeps bodies out of telemetry. The row-by-row behavior of each one is tabulated in One App, Many Places to Run AI, and I would rather point at that table than print a second version of it that can drift.
This is not enough for every production system, and I would not pretend otherwise. It is enough to demonstrate the most important boundary: restricted data does not leave the edge just because the cloud is available.
The Camera Fleet Workload
The camera fleet is the reference workload because it has real edge characteristics (and because anyone who has read this blog knows I have a soft spot for cameras that outlive their vendors). The three-part Azure IoT Operations series owns the details - the camera control plane, the MQTT broker and camera fleet, and data flows with the ONVIF connector - and what makes that workload useful here is the shape of it:
- Distributed sites.
- Outbound-only device connectivity.
- MQTT topic contracts.
- Operational events and incidents.
- Local runbooks and private configuration.
- Azure IoT Operations as an optional broker and data-flow layer.
The operations assistant consumes the existing cameras/# stream, normalizes events, builds incidents, retrieves runbook snippets, and asks the gateway for a local summary. The camera control plane remains the owner of camera identity, commands, network model, and topic contract.
Operating It Like Real Infrastructure
The operations model has to prove that edge AI can be run like real infrastructure:
- Git is the source of desired state.
- Arc brings edge Kubernetes into Azure management.
- Metrics show backend selection, latency, fallback, and denials.
- Logs show route decisions and incident summaries.
- Key Vault stores secrets in the governed path.
- Policy events explain why requests were allowed or denied.
- Reset and smoke tests keep demos repeatable, because local execution is not an excuse for local-only operations.
The Case Where It Refuses
Every claim above collapses into one scenario. A Restricted RAG request arrives under LocalOnly, every device and edge backend is down, and a cloud backend is sitting there healthy. The gateway denies the request and the dashboard and logs carry the reason.
That is the architecture doing its job: it refused to answer, because answering would have violated policy. I would much rather stage that refusal during a demo than meet it for the first time in an audit.
The Target an Implementation Has to Meet
The build guide specifies the implementation, and the status framing there applies to how much of it exists today. What that implementation is aiming at comes down to three things. Placement has to be invisible to application code, so moving a workload from a laptop runtime to an edge cluster to an accelerator changes configuration and nothing else. Every routing decision has to be explainable after the fact, with restricted requests structurally unable to reach a cloud backend and both assistants citing the local material their answers came from - source documents for RAG, structured events and runbooks for camera incidents.
The third is operational. Azure has to be able to observe routing, failures, policy denials, and backend health from outside the cluster, the same system has to come up in laptop-only mode and on an edge cluster without divergent code paths, and seed, reset, and smoke tests have to behave the same way every time. Accelerator hardware stays optional throughout, with a mock backend standing in for it, because an architecture that only works when a specific board is awake is a much smaller claim than the one this series is making.
Related Posts
Use this post as the evergreen architecture link. Use Building the AI on the Edge Demo System as the implementation guide. Use the earlier posts for the individual demo slices - each one is the story of how a piece of this diagram earned its place. The two prerequisite series are the Tenstorrent private cloud buildout for the environment and the Azure IoT Operations camera control plane for the workload.