Answering prompts is not the same as operating a system, a distinction I did not fully appreciate until the demos started stacking up. Teams also have to deploy it, secure it, observe it, rotate its secrets, understand its failures, and prove policy decisions after the fact. That is what Demo 4 sets up with Azure. The model may run locally, but the estate should still be visible and governable.
Local AI Goes Invisible
Local AI can become invisible infrastructure, and it happens without anyone deciding anything. A model server starts on a developer machine. A Kubernetes deployment gets copied to an edge node. A gateway ends up with API keys in a config file. Logs stay local. Metrics are whatever the process prints. Nobody can tell which requests went where. I have watched perfectly serious systems drift into exactly that state one shortcut at a time, and invisibility is not an operating model an enterprise can run on.
For AI on the Edge, every local execution path needs an operations path:
- How was it deployed?
- Which version is running?
- Which backends are healthy?
- Which requests fell back?
- Which requests were denied?
- Where are secrets stored?
- Which policies are being enforced?
The Control Plane Around Local Execution
The Azure-governed mode uses Azure as the control plane around local execution. Each layer gets exactly one job:
| Layer | Role |
|---|---|
| Azure Arc-enabled Kubernetes | Brings the edge cluster into Azure inventory and management. |
| GitOps with Flux | Reconciles the cluster from Git. |
| Azure Monitor and Log Analytics | Centralizes routing events, incident events, and service logs. |
| Managed Prometheus | Scrapes service and Kubernetes metrics. |
| Azure Managed Grafana | Presents routing, latency, privacy, and incident dashboards. |
| Key Vault | Stores backend API keys, certificates, and demo secrets. |
| Policy | Enforces approved endpoints, local-only classifications, tags, and secret-source rules. |
The Kubernetes deployment should be conventional:
infra/
azure/
kubernetes/
arc/
dashboards/
policies/
Cluster resources should include:
| Resource | Purpose |
|---|---|
| Gateway deployment | OpenAI-compatible facade and route policy. |
| Knowledge Assistant deployment | Private RAG service. |
| Operations Assistant deployment | IoT incident assistant. |
| Dashboard deployment | Control UI. |
| ServiceMonitor or PodMonitor | Prometheus scraping. |
| SecretProviderClass | Key Vault-backed secret mounting where enabled. |
| Ingress | TLS endpoint for gateway and dashboard. |
| Demo namespace | Isolated presentation environment. |
The dashboards should make the architecture measurable:
- Requests by backend.
- Fallback count.
- Denied count.
- P50/P95/P99 latency.
- Prompt body logging posture.
- Local-only request count.
- Backend health.
- Camera events by site and type.
- Active incidents and summaries.
If a number is not on a dashboard, it does not exist during an outage.
Evidence in Three Places
Policy evidence has to land in three places at once: the app response, the dashboards, and the query history. Any one of them alone is a story; together they are proof.
So this segment starts from the outside and works in. The app is running at the edge, the cluster shows up Arc-connected in the portal, and GitOps reports what it last reconciled - three screens that establish the estate exists before anything interesting happens to it. Then requests go through the gateway, Grafana fills in with backend selection, latency, fallback and denial counts, and a KQL query over routing events shows the same activity from the log side.
The turn is a deliberate LocalOnly denial, and the point is watching one refusal appear in all three surfaces: the response the app got, the counter on the dashboard, and the row in Log Analytics. If a secret rotation is wired up by then, it closes the segment with a config reload or a rollout, which is the least glamorous and most reassuring thing in the talk.
One caveat before anyone pastes these into their own workspace: the queries below target the planned Log Analytics schema. AiRoutingEvents, AiInferenceRequests, and CameraEvents exist once the ingestion pipeline ships, so read them as the observability contract rather than as queries against a live workspace:
AiRoutingEvents
| summarize count() by selectedBackend, policy, decision
AiRoutingEvents
| where policy == "LocalOnly" and decision == "Denied"
AiInferenceRequests
| summarize p95Latency=percentile(latencyMs, 95) by backendId, bin(timestamp, 5m)
CameraEvents
| summarize count() by siteId, cameraId, eventType, bin(timestamp, 5m)
Signals That Already Exist
The earlier demos already emit the signals worth collecting: gateway routing decisions, RAG retrieval and privacy decisions, camera incident summaries, and backend health and failures. The private cloud buildout already frames Arc and GitOps as the management overlay for the local Kubernetes environment. Demo 4 turns those signals into operational evidence instead of log lines nobody reads.
What Had to Be Built
The new work is infrastructure and observability - none of it glamorous, all of it load-bearing:
- Bicep or Terraform for the Azure resources.
- Helm or Kustomize for Kubernetes deployment.
- Managed Prometheus scrape config.
- Grafana dashboard JSON.
- KQL saved queries.
- Key Vault secret integration for the Azure-governed path.
- Application policy events that match dashboard and query fields.
Policy examples, split by the layer that actually enforces them. This split matters more than it looks, because people assume Azure Policy reaches inside applications, and it does not:
| Policy | Enforced by | Demo behavior |
|---|---|---|
| Approved model endpoints only | Application | Unknown endpoint cannot be enabled. |
| Local-only classification | Application | Restricted workload cannot route to cloud. |
| No prompt body logging | Application | Restricted prompts emit metadata-only telemetry. |
| Required tags and labels | Azure Policy | Manifests carry app, scenario, owner, data-classification. |
| Required secret source | Azure Policy | Production manifests cannot use raw API keys. |
The three application rows land in three different components: the gateway admin surface refuses the unknown endpoint, the routing engine refuses the cloud hop, and the telemetry pipeline drops the prompt body. To say the split plainly: Azure Policy governs Azure resources such as tags and secret sources, and it cannot block a gateway from enabling an unknown model backend. The routing boundaries in this demo are application-level policy events, surfaced through the same evidence flow as the Azure-side rules.
The Brittle Demo Trap
Installing cloud operations live during a talk is how this demo breaks, and the fix is to refuse the temptation. Preflight the Azure path, keep a recording or a screenshot for the portal views, and save the live running for the local dashboard.
Arc and Azure Monitor need outbound connectivity and prior setup, so a disconnected demo shows local application behavior and local telemetry with no cloud forwarding while the link is down. Pretending Azure is live while the edge is offline buys nothing: nobody in the audience can see your resource group anyway, and everybody can see a stalled terminal.
When the Governance Demo Is Real
The governance story holds up when the Azure side is reproducible rather than hand-built: make infra-plan and make infra-apply produce the same resources twice, the gateway, assistants, and dashboard deploy through Helm or Kustomize, and the Grafana dashboards import themselves instead of being rebuilt from memory an hour before the talk. Metrics show up locally either way, and in Managed Prometheus when Azure is connected.
The claim that actually matters is the denial one. A single LocalOnly refusal has to be findable in the app logs, on the dashboard, and in a KQL query over Log Analytics, because governance you can only see from one angle is governance nobody will trust. And make teardown has to remove the demo resources afterward, since leaving resources behind is how a demo quietly becomes a bill.
Related Posts
This post makes the prior demos operational. It depends on the gateway routing events from One App, Many Places to Run AI and the camera incident events from From Camera Events to Operator Guidance.