The first demo has to make the whole architecture obvious. One application asks one question, sends one request, and gets a useful answer. The interesting part is that the application does not know whether the model ran on the laptop, an edge cluster, a cloud endpoint, a Tenstorrent-backed server, or a mock backend.

That abstraction is the AI on the Edge Gateway.

Problem

Hardcoding model endpoints is fine for a prototype and bad for a platform. Once an application knows too much about the model runtime, every placement decision becomes an app change:

  • Moving from a laptop runtime to an edge server changes code.
  • Adding a cloud fallback changes code.
  • Testing a specialized accelerator changes code.
  • Denying cloud fallback for restricted data becomes app-specific retry logic.

The app should express what it needs. The platform should decide where that request is allowed to run.

Architecture Slice

Model routing flow: an app request enters the gateway, classification and policy metadata are evaluated, eligible backends are filtered by policy, capability, and health, then a backend is selected or the request is denied fail closed with route trace and metrics evidence

The gateway exposes the API surface the app already understands:

POST /v1/chat/completions
POST /v1/embeddings
GET  /v1/models
GET  /healthz
GET  /readyz
GET  /metrics
POST /admin/route/explain
POST /admin/backends/{backendId}/enable
POST /admin/backends/{backendId}/disable
POST /admin/demo/faults

Behind those endpoints is a backend registry:

{
  "Backends": [
    {
      "id": "foundry-local",
      "kind": "FoundryLocal",
      "baseUrl": "http://localhost:5273/v1",
      "models": ["local-chat", "local-embed"],
      "location": "device",
      "capabilities": ["chat", "embeddings", "streaming"],
      "tags": ["local", "offline", "private"],
      "priority": 10
    },
    {
      "id": "azure-foundry",
      "kind": "OpenAICompatible",
      "baseUrl": "https://example.services.ai.azure.com/openai/v1",
      "apiKeySecretName": "azure-foundry-key",
      "models": ["cloud-chat"],
      "location": "cloud",
      "capabilities": ["chat", "streaming"],
      "tags": ["cloud", "managed"],
      "priority": 50
    },
    {
      "id": "mock-accelerator",
      "kind": "OpenAICompatible",
      "baseUrl": "http://localhost:5199/v1",
      "models": ["accelerator-chat"],
      "location": "edge",
      "capabilities": ["chat", "streaming"],
      "tags": ["local", "accelerator", "specialized-hardware"],
      "priority": 20
    }
  ]
}

Every backend is normalized into the same internal shape: health, supported model aliases, capabilities, location, tags, priority, observed latency, and current fault state.

The router evaluates policy before it evaluates convenience:

Routing policy decision matrix: request classification, requested policy, and backend health feed policy evaluation, which chooses a local route, permitted cloud route, or denied fail-closed result
Policy Behavior
LocalOnly Only device or edge backends are eligible. Cloud fallback is denied.
PreferLocal Prefer local, then Azure Local or edge, then cloud if allowed.
CloudAllowed Any healthy backend is eligible.
AcceleratorPreferred Prefer a backend tagged accelerator when the model family is supported.
FallbackDisabled Fail closed if the selected backend is unavailable.
NoPromptLogging Emit metadata only and suppress prompt and response bodies.

Foundry Local belongs in this first demo as the device-local runtime. Microsoft positions it as an on-device AI runtime and SDK, with an optional local server for development and integration scenarios. It is not the shared multi-user server inference layer for the whole edge estate. For server-style edge inference, the demo keeps Azure Local, other OpenAI-compatible services, and accelerator endpoints behind the same registry.

Demo Script

The visible audience moment is route explanation.

  1. Open the AI on the Edge Console.
  2. Ask: Summarize the current edge site health and recommend the next action.
  3. Show the response streaming through the gateway.
  4. Open the route trace and show selectedBackend: foundry-local.
  5. Disable foundry-local.
  6. Ask again with CloudAllowed.
  7. Show fallback to azure-foundry or mock-cloud.
  8. Ask again with LocalOnly.
  9. Show the request is denied instead of silently falling back to cloud.
  10. Re-enable the local backend and show the metrics panel.

The key line for the audience is simple: same app, same API, different placement decision.

What Already Exists

The private cloud lab already supplies the edge environment and the Tenstorrent hardware lane. The camera series already supplies an operational workload. Neither one needs to be rebuilt to prove the gateway. Demo 1 can run entirely with a local runtime and deterministic mock backends.

What Is New

The new implementation is the gateway foundation:

  • OpenAI-compatible request and response normalization.
  • Backend registry and health checks.
  • Route explanation endpoint.
  • Policy evaluation before fallback.
  • Prometheus metrics.
  • Route decision events.
  • Admin controls for enable, disable, and fault injection.
  • A dashboard panel for backend health, latency, selected backend, and denial reasons.

The implementation should include a mock local backend, mock cloud backend, and mock accelerator backend before it depends on real cloud or hardware. Deterministic mocks are not a shortcut; they are what make the conference demo reliable.

Failure Mode

The first failure to show is local backend loss:

Situation Correct behavior
Local backend down, policy CloudAllowed Route to an allowed cloud or mock-cloud backend.
Local backend down, policy LocalOnly Deny with a clear reason.
Accelerator saturated, local backend healthy Route to another eligible edge backend.
Backend returns malformed OpenAI-compatible payload Record adapter error and fallback only before streaming begins.

The gateway must never treat “cloud is healthy” as permission to send restricted content there.

Acceptance Criteria

Demo 1 is complete when:

  1. One app can call /v1/chat/completions on the gateway.
  2. The gateway can route to local, cloud/mock-cloud, and accelerator/mock-accelerator backends.
  3. The dashboard shows selected backend, policy, denial reason, fallback reason, and latency.
  4. Disabling a backend changes the routing decision.
  5. LocalOnly prevents cloud fallback.
  6. Metrics include request count, latency, selected backend, fallback count, and denied count.
  7. make demo-laptop runs the whole demo without a cloud dependency.

This post is the platform foundation for the private RAG, camera operations assistant, failure lab, and accelerator lane that follow. The existing Tenstorrent buildout is the hardware background; the existing camera control-plane post is the workload background.

References