One application asks one question and gets one useful answer. Where the model actually ran - the laptop, an edge cluster, a cloud endpoint, a Tenstorrent-backed server, or a mock backend - is none of the application’s business. Keeping that placement choice invisible to the app is the entire job of the AI on the Edge Gateway, and this first demo exists to make that abstraction obvious end to end.

Why hardcoded endpoints stop scaling

Hardcoding model endpoints is fine for a prototype and bad for a platform. Once an application knows too much about the model runtime, every placement decision turns into an app change:

  • Moving from a laptop runtime to an edge server changes code.
  • Adding a cloud fallback changes code.
  • Testing a specialized accelerator changes code.
  • Denying cloud fallback for restricted data becomes app-specific retry logic.

The app should express what it needs. The platform should decide where that request is allowed to run.

What the gateway looks like

Model routing flow: an app request enters the gateway, classification and policy metadata are evaluated, eligible backends are filtered by policy, capability, and health, then a backend is selected or the request is denied fail closed with route trace and metrics evidence

On the outside, the API surface is deliberately boring - it is the surface applications already speak:

POST /v1/chat/completions
POST /v1/embeddings
GET  /v1/models
GET  /healthz
GET  /readyz
GET  /metrics
POST /admin/route/explain
POST /admin/backends/{backendId}/enable
POST /admin/backends/{backendId}/disable
POST /admin/demo/faults

Routing those endpoints is a backend registry:

{
  "Backends": [
    {
      "id": "foundry-local",
      "kind": "FoundryLocal",
      "baseUrl": "http://localhost:5273/v1",
      "models": ["local-chat", "local-embed"],
      "location": "device",
      "capabilities": ["chat", "embeddings", "streaming"],
      "tags": ["local", "offline", "private"],
      "priority": 10
    },
    {
      "id": "azure-foundry",
      "kind": "OpenAICompatible",
      "baseUrl": "https://example.services.ai.azure.com/openai/v1",
      "apiKeySecretName": "azure-foundry-key",
      "models": ["cloud-chat"],
      "location": "cloud",
      "capabilities": ["chat", "streaming"],
      "tags": ["cloud", "managed"],
      "priority": 50
    },
    {
      "id": "mock-accelerator",
      "kind": "OpenAICompatible",
      "baseUrl": "http://localhost:5199/v1",
      "models": ["accelerator-chat"],
      "location": "edge",
      "capabilities": ["chat", "streaming"],
      "tags": ["local", "accelerator", "specialized-hardware"],
      "priority": 20
    },
    {
      "id": "mock-cloud",
      "kind": "OpenAICompatible",
      "baseUrl": "http://localhost:5188/v1",
      "models": ["cloud-chat"],
      "location": "cloud",
      "capabilities": ["chat", "streaming"],
      "tags": ["cloud", "mock"],
      "priority": 40
    }
  ]
}

Every backend gets normalized into the same internal shape: health, supported model aliases, capabilities, location, tags, priority, observed latency, and current fault state. From there on, the router does not care whether a backend is a laptop runtime or a rack in the basement.

Then comes my favorite line in the whole codebase: the router evaluates policy before it evaluates convenience.

Routing policy decision matrix: request classification, requested policy, and backend health feed policy evaluation, which chooses a local route, permitted cloud route, or denied fail-closed result

The policies are small enough to keep in your head:

Policy Behavior
LocalOnly Only device or edge backends are eligible. Cloud fallback is denied.
PreferLocal Prefer local, then Azure Local or edge, then cloud if allowed.
CloudAllowed Any healthy backend is eligible.
AcceleratorPreferred Prefer a backend tagged accelerator when the model family is supported.
FallbackDisabled Fail closed if the selected backend is unavailable.
NoPromptLogging Emit metadata only and suppress prompt and response bodies.

Foundry Local belongs in this first demo as the device-local runtime. Microsoft positions it as an on-device AI runtime and SDK, with an optional local server aimed at development and integration scenarios - which is a different job from being the shared multi-user server inference layer for the whole edge estate. For server-style edge inference, the demo keeps Azure Local, other OpenAI-compatible services, and accelerator endpoints behind the same registry.

Walking through the demo

Route explanation carries this demo. This is the sequence I rehearse until it gets boring:

  1. Open the AI on the Edge Console.
  2. Ask: Summarize the current edge site health and recommend the next action.
  3. Show the response streaming through the gateway.
  4. Open the route trace and show selectedBackend: foundry-local.
  5. Disable foundry-local.
  6. Ask again with CloudAllowed.
  7. Show fallback to azure-foundry or mock-cloud.
  8. Ask again with LocalOnly.
  9. Show the request is denied instead of silently falling back to cloud.
  10. Re-enable the local backend and show the metrics panel.

The line I want the audience walking out with is simple: same app, same API, different placement decision.

Borrowing the lab instead of rebuilding it

The private cloud lab already supplies the edge environment and the Tenstorrent hardware lane, and the camera series already supplies an operational workload. Neither one needs rebuilding just to prove a gateway. Demo 1 can run entirely with a local runtime and deterministic mock backends.

The new implementation is the gateway foundation:

  • OpenAI-compatible request and response normalization.
  • Backend registry and health checks.
  • Route explanation endpoint.
  • Policy evaluation before fallback.
  • Prometheus metrics.
  • Route decision events.
  • Admin controls for enable, disable, and fault injection.
  • A dashboard panel for backend health, latency, selected backend, and denial reasons.

Before any of this depends on real cloud or real hardware, the implementation should include a mock local backend, the mock-cloud backend from the registry above, and a mock accelerator. Deterministic mocks are not a shortcut; they are what make the conference demo reliable.

When the local box dies

The first failure worth showing is local backend loss:

Situation Correct behavior
Local backend down, policy CloudAllowed Route to an allowed cloud or mock-cloud backend.
Local backend down, policy LocalOnly Deny with a clear reason.
Accelerator saturated, local backend healthy Route to another eligible edge backend.
Backend returns malformed OpenAI-compatible payload Record adapter error and fallback only before streaming begins.

And the rule underneath every row of that table: the gateway must never treat “cloud is healthy” as permission to send restricted content there.

What done looks like

Demo 1 earns its number when:

  1. One app can call /v1/chat/completions on the gateway.
  2. The gateway can route to local, cloud/mock-cloud, and accelerator/mock-accelerator backends.
  3. The dashboard shows selected backend, policy, denial reason, fallback reason, and latency.
  4. Disabling a backend changes the routing decision.
  5. LocalOnly prevents cloud fallback.
  6. Metrics include request count, latency, selected backend, fallback count, and denied count.
  7. make demo-laptop runs the whole demo without a cloud dependency.

This post is the platform foundation for the private RAG demo, the camera operations assistant, the failure lab, and the accelerator lane that follow. The existing Tenstorrent buildout is the hardware background; the existing camera control-plane post is the workload background. Neither is required reading, but both are why this post stayed short.

References