AI on the Edge series

This series builds the AI on the Edge demo system: cloud-governed, locally executed AI using a .NET gateway, policy-driven model routing, private RAG, camera fleet telemetry, Azure Arc operations, and accelerator-backed endpoints.

  1. AI on the Edge: Local AI Without Local Chaos
  2. One App, Many Places to Run AI
  3. Private RAG That Cannot Leave the Edge
  4. From Camera Events to Operator Guidance
  5. Edge AI You Can Actually Operate
  6. When the Edge Has to Stand Alone
  7. Specialized Hardware Without an App Rewrite
  8. Building the AI on the Edge Demo System
  9. The AI on the Edge Reference Architecture

Background posts:

Local AI is useful for obvious reasons: privacy, latency, cost control, disconnected operation, and the simple fact that some data already lives at the edge. The harder problem is making local AI behave like a platform instead of a pile of model servers hiding under desks.

That is the point of AI on the Edge. The project is not about proving that one model can answer one prompt on one box. It is about building a governable edge AI system where applications call one API, policy decides where inference is allowed to run, Azure provides the management plane, and the local environment keeps working when the network is not perfect.

The tagline is the operating model:

Cloud-governed, locally executed.

Problem

Local inference can decay into model sprawl quickly. A developer points an app at a local runtime. An operations team deploys a different endpoint on a Kubernetes node. A hardware experiment adds an accelerator-specific API. A cloud team wants a managed endpoint for approved workloads. None of those are wrong in isolation, but together they create an unmanaged surface:

  • Applications hardcode model endpoints.
  • Sensitive prompts can fall back to cloud by accident.
  • Local model failures are invisible to operations teams.
  • Hardware-specific demos turn into one-off branches.
  • Edge environments have no consistent reset, smoke test, or dashboard story.

AI on the Edge treats those as platform problems. The model runtime matters, but the project is really about routing, policy, observability, secrets, fallback, repeatability, and deployment.

Architecture Slice

AI on the Edge architecture: applications and workloads call an OpenAI-compatible gateway; routing and policy select local, Azure Local, cloud, accelerator, or mock execution targets while Azure Arc, Monitor, Grafana, Key Vault, and IoT Operations provide control-plane services

The center of the system is a reusable .NET gateway that exposes an OpenAI-compatible API to the application. Behind that gateway is a backend registry and a policy-driven router. A request can target a laptop-local runtime, an Azure Local deployment, an Azure AI Foundry model endpoint, a Tenstorrent-backed endpoint, or a deterministic mock backend for conference safety.

The important part is that the application does not pick the backend directly. It sends the request with metadata:

  • Workload: chat, embeddings, RAG answer, incident summary.
  • Classification: public, internal, restricted, secret.
  • Policy: local only, prefer local, cloud allowed, accelerator preferred.
  • Operational context: latency target, streaming requirement, fallback rules.

The gateway emits a route decision event for every request. That event records the selected backend, denied backends, policy, classification, fallback behavior, latency, token counts, and whether prompt bodies were logged, redacted, or suppressed.

Azure provides the control plane around that local execution model:

  • Azure Arc brings the edge Kubernetes cluster into Azure management.
  • GitOps applies the desired state.
  • Azure Monitor, Managed Prometheus, and Grafana make behavior visible.
  • Key Vault handles secrets and certificates where the Azure-governed path is active.
  • Azure Policy and application policy events make denied routes explicit.
  • Azure IoT Operations gives the camera workload a real edge data plane.

Demo Roadmap

The series maps directly to the presentation build:

AI on the Edge roadmap: prerequisite Tenstorrent lab and Azure IoT Operations camera posts feed gateway routing, private RAG, camera operations, Azure governance, failure lab, and accelerator milestones, which become the build guide, reference architecture, and 45-minute presentation flow
Post Demo milestone Audience moment
One App, Many Places to Run AI Gateway and router One prompt routes to different backends without app changes.
Private RAG That Cannot Leave the Edge Local documents and citations Restricted data fails closed instead of falling back to cloud.
From Camera Events to Operator Guidance IoT operations assistant Raw camera events become an incident summary and first actions.
Edge AI You Can Actually Operate Azure governance Routing, failures, and policy decisions show up in Azure-backed dashboards.
When the Edge Has to Stand Alone Failure lab Backend failures and cloud blocks produce visible, correct behavior.
Specialized Hardware Without an App Rewrite Accelerator lane Tenstorrent or mock accelerator is just another governed backend.

The final two posts turn the demo system into an implementation guide and an evergreen reference architecture.

What Already Exists

The private cloud and physical lab are already covered in Home Lab - Tenstorrent Buildout for Multi-Cloud Edge Demos. That post owns the hardware story: basement studio, private cloud, Tenstorrent paths, NVIDIA systems, Proxmox, TrueNAS, Kubernetes, and multi-cloud demo intent.

The camera control plane is already covered in the three-part Azure IoT Operations series. Those posts own the camera details: outbound MQTT, the cameras/<site>/<camera>/<channel> topic tree, TLS and X.509 on the AIO MQTT broker, data flows to Event Hubs, and the ONVIF connector bridge.

AI on the Edge builds on both. It does not repeat the private cloud buildout or the camera control-plane implementation. It uses them as the workload and environment for an edge AI platform.

What Is New

The new work is the demo system:

  • AiOnTheEdge.Gateway for the OpenAI-compatible facade.
  • AiOnTheEdge.Routing for backend selection, fallback, and policy.
  • AiOnTheEdge.KnowledgeAssistant for private RAG over local documents.
  • AiOnTheEdge.OperationsAssistant for camera event triage.
  • AiOnTheEdge.ControlDashboard for health, route traces, privacy posture, and failure controls.
  • AiOnTheEdge.DemoScenarios for deterministic seed, reset, replay, and smoke tests.
  • Azure and Kubernetes deployment assets for the governed edge path.

The first version must run in laptop mode with mock or local backends. Azure-governed mode is the richer path, not the live-demo dependency.

Failure Mode

The failure mode this project protects against is quiet policy drift. A local backend goes down, a cloud endpoint is healthy, and a restricted prompt silently leaves the edge because the application only knows how to retry.

That is unacceptable behavior for this architecture. Restricted data must fail closed. Cloud fallback must be a policy decision, not a retry side effect. The audience should see the denial, the reason, and the telemetry.

Acceptance Criteria

The full project is useful when these are true:

  1. A reader can understand why local AI still needs governance.
  2. One application can call one gateway and reach multiple backend types.
  3. Restricted RAG content never routes to cloud.
  4. Camera telemetry can drive an operator assistant without reworking the existing MQTT contract.
  5. Azure operations can see routing, failures, policy denials, and backend health.
  6. The demo can run in laptop mode with no cloud dependency.
  7. The demo can be reset, seeded, and smoke-tested before a session.

Start with the Tenstorrent buildout if you want the hardware context. Start with the Azure IoT Operations camera series if you want the workload context. Start here if you want the AI platform and presentation system.

References