Years ago a vendor shut down the cloud service behind my home security cameras and left me holding hardware I owned but could no longer use. I salvaged what I could and kept the lesson: anything I actually depend on should run where I can reach it. That instinct is most of why local AI appeals to me - privacy, latency, cost control, disconnected operation, and the simple fact that some data already lives at the edge. The hard part was never getting one model to answer one prompt on one box. The hard part is making local AI behave like a platform instead of a pile of model servers hiding under desks.
That is the point of AI on the Edge. The project is a governable edge AI system where applications call one API, policy decides where inference is allowed to run, Azure provides the management plane, and the local environment keeps working when the network is not perfect.
The tagline is not decoration. It is the operating model:
Cloud-governed, locally executed.
Why Local AI Goes Sideways
Model sprawl never announces itself. A developer points an app at a local runtime. An operations team deploys a different endpoint on a Kubernetes node. A hardware experiment bolts on an accelerator-specific API. A cloud team wants a managed endpoint for approved workloads. Every one of those decisions is reasonable on its own. Stacked together, they become an unmanaged surface:
- Applications hardcode model endpoints.
- Sensitive prompts can fall back to cloud by accident.
- Local model failures are invisible to operations teams.
- Hardware-specific demos turn into one-off branches.
- Edge environments have no consistent reset, smoke test, or dashboard story.
AI on the Edge treats those as platform problems. The model runtime matters, but the project is really about routing, policy, observability, secrets, fallback, repeatability, and deployment.
One Gateway, Every Backend
The center of the system is a reusable .NET gateway that exposes an OpenAI-compatible API to the application. Behind that gateway sit a backend registry and a policy-driven router. Depending on the request, inference can land on a laptop-local runtime, an Azure Local deployment, an Azure AI Foundry model endpoint, a Tenstorrent-backed endpoint, or a deterministic mock backend that exists purely for conference safety.
The important part is that the application never picks the backend directly. It sends the request with metadata, and that metadata is the contract: the workload (chat, embeddings, RAG answer, incident summary), the classification (public, internal, restricted, secret), the policy (local only, prefer local, cloud allowed, accelerator preferred), and the operational context (latency target, streaming requirement, fallback rules).
For every request, the gateway emits a route decision event. That event records the selected backend, the denied backends, the policy, the classification, the fallback behavior, latency, token counts, and whether prompt bodies were logged, redacted, or suppressed. When somebody asks why an answer came from where it did, the answer lives in the telemetry, not in my memory.
Azure provides the control plane wrapped around that local execution:
- Azure Arc brings the edge Kubernetes cluster into Azure management.
- GitOps applies the desired state.
- Azure Monitor, Managed Prometheus, and Grafana make behavior visible.
- Key Vault handles secrets and certificates where the Azure-governed path is active.
- Azure Policy and application policy events make denied routes explicit.
- Azure IoT Operations gives the camera workload a real edge data plane.
How the Demos Stack Up
The series maps straight onto how I am building the talk:
| Post | Milestone | Audience moment |
|---|---|---|
| Demo 1: One App, Many Places to Run AI | Gateway routing | One prompt routes to different backends without app changes. |
| Demo 2: Private RAG That Cannot Leave the Edge | Private RAG | Restricted data fails closed instead of falling back to cloud. |
| Demo 3: From Camera Events to Operator Guidance | Camera operations | Raw camera events become an incident summary and first actions. |
| Demo 4: Edge AI You Can Actually Operate | Azure governance | Routing, failures, and policy decisions show up in Azure-backed dashboards. |
| Demo 5: When the Edge Has to Stand Alone | Failure lab | Backend failures and cloud blocks produce visible, correct behavior. |
| Demo 6: Specialized Hardware Without an App Rewrite | Accelerator lane | Tenstorrent or mock accelerator is just another governed backend. |
The final two posts turn the demo system into an implementation guide and an evergreen reference architecture.
Standing on Work I Already Published
The private cloud and physical lab are already covered in Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos. That post owns the hardware story: basement studio, private cloud, Tenstorrent paths, NVIDIA systems, Proxmox, TrueNAS, Kubernetes, and multi-cloud demo intent.
The camera control plane is already covered in the three-part Azure IoT Operations series. Those posts own the camera details: outbound MQTT, the cameras/<site>/<camera>/<channel> topic tree, TLS and X.509 on the AIO MQTT broker, data flows to Event Hubs, and the ONVIF connector bridge.
AI on the Edge builds on both instead of repeating either. The lab is the environment, the cameras are the workload, and the new thing is the AI platform that sits between them.
What I Am Actually Building Here
The demo system is the new work:
AiOnTheEdge.Gatewayfor the OpenAI-compatible facade.AiOnTheEdge.Routingfor backend selection, fallback, and policy.AiOnTheEdge.KnowledgeAssistantfor private RAG over local documents.AiOnTheEdge.OperationsAssistantfor camera event triage.AiOnTheEdge.ControlDashboardfor health, route traces, privacy posture, and failure controls.AiOnTheEdge.Telemetryfor route decision events, metrics, and log export.AiOnTheEdge.DemoScenariosfor deterministic seed, reset, replay, and smoke tests.- Azure and Kubernetes deployment assets for the governed edge path.
One status note, stated here once so the whole series inherits it: these components are part of my private reference implementation for the talk. The build is in progress and the repo is not published, so the posts that follow present API surfaces, registries, and acceptance criteria as the design the demos target - not as downloadable software. Nobody will be cloning their way into this series, and pretending otherwise would just waste your afternoon.
The first version has to run in laptop mode on mock or local backends. Azure-governed mode is the richer path, and I am deliberately keeping it off the critical path for a live session.
Quiet Policy Drift
Quiet policy drift is the failure mode this whole project is arranged against. A local backend goes down, a cloud endpoint happens to be healthy, and a restricted prompt leaves the edge because retry is the only trick the application knows.
So restricted data fails closed here. Cloud fallback happens because a policy allowed it, and the audience gets to watch the denial, the reason, and the telemetry rather than take my word for any of it.
What the Series Has to Earn
By the last post, a few claims have to hold up on stage rather than on paper. One application should reach several kinds of backend through one gateway with no code change, and restricted content should stay on the edge even when a healthy cloud endpoint is sitting right there waiting to answer. Camera telemetry should drive an operator assistant on top of the MQTT contract the camera series already published, without renegotiating that contract to make the AI work.
The operations half matters just as much. Azure has to be able to see routing, failures, policy denials, and backend health from outside the cluster, because a system nobody can observe is a system nobody can run. And all of it has to come up in laptop mode with no cloud dependency, then reset, seed, and smoke-test itself before a session starts, since a conference network is the one piece of infrastructure I never get to choose.
Related Posts
Start with Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos if you want the hardware context. Start with the Azure IoT Operations camera series - the camera control plane, the MQTT broker and camera fleet, and data flows with the ONVIF connector - if you want the workload context. Start here if you want the AI platform and the presentation system, then go straight to One App, Many Places to Run AI for the gateway and the router. Everything after this post either goes through that gateway or fails because of it.