The edge has to stand alone sometimes. Internet links fail. Cloud services throttle. A local model crashes mid-request, an accelerator endpoint saturates, and sooner or later a response adapter meets a payload that is almost, but not quite, OpenAI-compatible. None of that is hypothetical; it is just a Tuesday.

Demo 5 turns all of that into a failure lab: a controlled set of ways to break the system on demand, so the architecture can be shown failing in visible, policy-correct ways. Failure behavior is part of the product here, and I would rather show it on purpose than meet it for the first time on stage.

Why Build a Failure Lab

Happy-path AI demos are easy to fake. The questions a real system has to answer are harder:

  • What happens when the cloud is unavailable?
  • What happens when the local model is unavailable?
  • What happens when the best backend is slow or saturated?
  • What happens when a backend returns malformed output?
  • What happens when a restricted request has no eligible backend?
  • Can the presenter reset the demo without debugging state live?

Every one of those has an answer in this architecture. The lab exists so the answers can be demonstrated instead of asserted.

Inside the Failure Lab

Failure lab architecture: demo controls inject backend, cloud, and LocalOnly faults; the gateway records fault state, reevaluates healthy eligible backends, then falls back, denies, or continues local operation while reset and smoke tests verify behavior

In the design, AiOnTheEdge.DemoScenarios owns deterministic failures and reset controls behind the shared demo endpoints. Fault types are posted to the single faults endpoint, and reset is the shared endpoint every service exposes:

POST /admin/demo/faults   { "fault": "backend-down",      "backendId": "<id>" }
POST /admin/demo/faults   { "fault": "backend-slow",      "backendId": "<id>" }
POST /admin/demo/faults   { "fault": "backend-malformed", "backendId": "<id>" }
POST /admin/demo/faults   { "fault": "network-cloud-blocked" }
POST /admin/demo/faults   { "fault": "policy-local-only" }
POST /admin/demo/reset    clears active faults and restores the seeded state

The dashboard needs a Failure Lab page:

Control Result
Toggle backend down Backend health changes and route decisions adapt.
Force restricted prompt Policy changes to LocalOnly.
Block cloud Cloud backends become unavailable.
Return malformed response Adapter error is recorded and fallback is evaluated safely.
Run smoke test Pass/fail appears for all primary demo scenarios.
Reset Stable state returns without manual cleanup.

Faults should be simple and explicit - every one maps to something boring that can actually happen:

Fault Implementation
Backend down Disable backend or point to a dead URL.
Backend slow Add delay in mock backend.
Malformed response Return invalid OpenAI-compatible payload.
Cloud blocked Mark cloud backends unhealthy in demo mode.
Local unavailable Stop or disable local adapter.
Accelerator saturated Return HTTP 429 or a configured saturation signal.

Staging a Fail-Closed Denial

A fail-closed denial is the moment worth staging. The sequence I use:

  1. Start with all backends healthy.
  2. Ask a normal public question and show a valid route.
  3. Mark the prompt restricted and show LocalOnly.
  4. Disable foundry-local, then mock-accelerator, so no device or edge backend is left.
  5. Ask again and show the denial: cloud is healthy but ineligible, and nothing eligible is healthy.
  6. Change policy to CloudAllowed for nonrestricted content.
  7. Show fallback to an allowed backend.
  8. Block cloud.
  9. Show local RAG and camera incident summaries still work with cached models and local data.
  10. Run make demo-smoke-test.
  11. Reset the demo.

The edge does not have to answer every request. It has to answer the requests it is allowed to answer and refuse the rest clearly. That sentence is the whole design review, and everything else in this post is machinery for proving it.

Parts That Can Already Break

The earlier demos already provide the components that can fail: gateway route selection, RAG retrieval and generation, camera event replay, Azure-backed telemetry, and accelerator backend registration. Demo 5 turns those components into test cases instead of ad hoc failure stories.

The New Work Is the Harness

The new build is the failure harness itself:

  • Central fault state.
  • Reset and seed endpoints.
  • Smoke-test runner.
  • Failure Lab UI.
  • Deterministic mock backend behaviors.
  • Local telemetry when Azure is unavailable.
  • Dashboard panels for denials, fallbacks, adapter errors, and reset status.

The smoke test should validate the contract the presenter needs:

make demo-reset
make demo-seed
make demo-smoke-test

That test should cover local route, cloud-allowed fallback, local-only denial, malformed response handling, camera replay, RAG citation, and reset. When it passes, I stop worrying about whatever the venue’s Wi-Fi is doing.

Too Many Faults Ruin the Show

The failure lab can become too noisy. The audience should see two or three failures live, not every possible fault; a parade of toggles is its own kind of confusion. The best live sequence is:

  1. Local backend down.
  2. Restricted prompt denied.
  3. Cloud blocked but local RAG still answers.

Everything else is useful for validation and backup, but not every control needs to be shown in a 45-minute talk.

It pays to be precise about what offline actually means here: local application behavior can continue after warmup, while live Azure management and cloud telemetry depend on connectivity and prior setup. Anything vaguer and someone walks away convinced the whole stack runs air-gapped forever.

Signing Off on the Failure Lab

The lab is finished when every fault can be turned on and cleared from both the API and the UI, and when each failure produces a reason a person can read and a reason a query can find. Those two audiences are different: the user-facing text has to say what happened, and the telemetry has to say why the router decided what it decided. The rule underneath all of them stays fixed - a restricted prompt never reaches cloud, whatever combination of faults is active.

The other half is recoverability. make demo-smoke-test covers every primary failure path, at least one complete demo path runs with the network unplugged after warmup, and reset is designed to return the system to a known state in under a minute, which is roughly the length of a question from the audience. A demo you cannot recover from between segments is a demo you only get to run once.

This is the credibility test for everything before it: the gateway from One App, Many Places to Run AI, the assistants from Private RAG That Cannot Leave the Edge and From Camera Events to Operator Guidance, and the operations layer from Edge AI You Can Actually Operate. Build it before depending on real hardware or live cloud services in a session, because credibility is much cheaper to install up front.

References