Years ago a vendor shut down the cloud service behind my home security cameras and left me holding hardware I owned but could no longer use. I salvaged what I could and kept the lesson: anything I actually depend on should run where I can reach it. That instinct is most of why local AI appeals to me - privacy, latency, cost control, disconnected operation, and the simple fact that some data already lives at the edge. The hard part was never getting one model to answer one prompt on one box. The hard part is making local AI behave like a platform instead of a pile of model servers hiding under desks.
That is the point of AI on the Edge. The project is a governable edge AI system where applications call one API, policy decides where inference is allowed to run, Azure provides the management plane, and the local environment keeps working when the network is not perfect.
The tagline is not decoration. It is the operating model:
Cloud-governed, locally executed.
Why local AI goes sideways
Model sprawl never announces itself. A developer points an app at a local runtime. An operations team deploys a different endpoint on a Kubernetes node. A hardware experiment bolts on an accelerator-specific API. A cloud team wants a managed endpoint for approved workloads. Every one of those decisions is reasonable on its own. Stacked together, they become an unmanaged surface:
- Applications hardcode model endpoints.
- Sensitive prompts can fall back to cloud by accident.
- Local model failures are invisible to operations teams.
- Hardware-specific demos turn into one-off branches.
- Edge environments have no consistent reset, smoke test, or dashboard story.
AI on the Edge treats those as platform problems. The model runtime matters, but the project is really about routing, policy, observability, secrets, fallback, repeatability, and deployment.
One gateway, every backend
The center of the system is a reusable .NET gateway that exposes an OpenAI-compatible API to the application. Behind that gateway sit a backend registry and a policy-driven router. Depending on the request, inference can land on a laptop-local runtime, an Azure Local deployment, an Azure AI Foundry model endpoint, a Tenstorrent-backed endpoint, or a deterministic mock backend that exists purely for conference safety.
The important part is that the application never picks the backend directly. It sends the request with metadata, and that metadata is the contract: the workload (chat, embeddings, RAG answer, incident summary), the classification (public, internal, restricted, secret), the policy (local only, prefer local, cloud allowed, accelerator preferred), and the operational context (latency target, streaming requirement, fallback rules).
For every request, the gateway emits a route decision event. That event records the selected backend, the denied backends, the policy, the classification, the fallback behavior, latency, token counts, and whether prompt bodies were logged, redacted, or suppressed. When somebody asks why an answer came from where it did, the answer lives in the telemetry, not in my memory.
Azure provides the control plane wrapped around that local execution:
- Azure Arc brings the edge Kubernetes cluster into Azure management.
- GitOps applies the desired state.
- Azure Monitor, Managed Prometheus, and Grafana make behavior visible.
- Key Vault handles secrets and certificates where the Azure-governed path is active.
- Azure Policy and application policy events make denied routes explicit.
- Azure IoT Operations gives the camera workload a real edge data plane.
How the demos stack up
The series maps straight onto how I am building the talk:
| Post | Demo | Milestone | Audience moment |
|---|---|---|---|
| One App, Many Places to Run AI | Blog Demo 1 | Gateway and router | One prompt routes to different backends without app changes. |
| Private RAG That Cannot Leave the Edge | Blog Demo 2 | Local documents and citations | Restricted data fails closed instead of falling back to cloud. |
| From Camera Events to Operator Guidance | Blog Demo 3 | IoT operations assistant | Raw camera events become an incident summary and first actions. |
| Edge AI You Can Actually Operate | Blog Demo 4 | Azure governance | Routing, failures, and policy decisions show up in Azure-backed dashboards. |
| When the Edge Has to Stand Alone | Blog Demo 5 | Failure lab | Backend failures and cloud blocks produce visible, correct behavior. |
| Specialized Hardware Without an App Rewrite | Blog Demo 6 | Accelerator lane | Tenstorrent or mock accelerator is just another governed backend. |
The final two posts turn the demo system into an implementation guide and an evergreen reference architecture.
Standing on work I already published
The private cloud and physical lab are already covered in Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos. That post owns the hardware story: basement studio, private cloud, Tenstorrent paths, NVIDIA systems, Proxmox, TrueNAS, Kubernetes, and multi-cloud demo intent.
The camera control plane is already covered in the three-part Azure IoT Operations series. Those posts own the camera details: outbound MQTT, the cameras/<site>/<camera>/<channel> topic tree, TLS and X.509 on the AIO MQTT broker, data flows to Event Hubs, and the ONVIF connector bridge.
AI on the Edge builds on both instead of repeating either. The lab is the environment, the cameras are the workload, and the new thing is the AI platform that sits between them.
What I am actually building here
The demo system is the new work:
AiOnTheEdge.Gatewayfor the OpenAI-compatible facade.AiOnTheEdge.Routingfor backend selection, fallback, and policy.AiOnTheEdge.KnowledgeAssistantfor private RAG over local documents.AiOnTheEdge.OperationsAssistantfor camera event triage.AiOnTheEdge.ControlDashboardfor health, route traces, privacy posture, and failure controls.AiOnTheEdge.DemoScenariosfor deterministic seed, reset, replay, and smoke tests.- Azure and Kubernetes deployment assets for the governed edge path.
One status note, stated here once so the whole series inherits it: these components are part of my private reference implementation for the talk. The build is in progress and the repo is not published, so the posts that follow present API surfaces, registries, and acceptance criteria as the design the demos target - not as downloadable software. Nobody will be cloning their way into this series, and pretending otherwise would just waste your afternoon.
The first version must run in laptop mode with mock or local backends. Azure-governed mode is the richer path, not the live-demo dependency.
The failure that proves the design
The failure this project protects against is quiet policy drift. A local backend goes down, a cloud endpoint happens to be healthy, and a restricted prompt silently leaves the edge because the only trick the application knows is retry.
That is unacceptable behavior for this architecture, full stop. Restricted data must fail closed. Cloud fallback must be a policy decision, not a retry side effect. And the audience should see the denial, the reason, and the telemetry - not take my word for any of it.
What done looks like
By the end of the series, all of this had better be true:
- A reader can understand why local AI still needs governance.
- One application can call one gateway and reach multiple backend types.
- Restricted RAG content never routes to cloud.
- Camera telemetry can drive an operator assistant without reworking the existing MQTT contract.
- Azure operations can see routing, failures, policy denials, and backend health.
- The demo runs in laptop mode with no cloud dependency yet can still be reset, seeded, and smoke-tested before a session.
Related Posts
Start with the Tenstorrent buildout if you want the hardware context. Start with the Azure IoT Operations camera series if you want the workload context. Start here if you want the AI platform and presentation system. Everything after this post either goes through the gateway or fails because of it.