Private data is the easiest reason to care about edge AI. If the data cannot leave the site, then the answer cannot depend on a cloud fallback that nobody noticed.

Demo 2 extends the platform with a Knowledge Assistant. In the design, it ingests local documents, builds embeddings locally when available, retrieves relevant chunks, generates grounded answers, and sends every generation request through the same routing policy engine from Demo 1. The RAG part is nearly conventional. The part I actually care about is that the privacy promise gets enforced by routing instead of by hope.

Where Most RAG Demos Leak

RAG demos tend to blur the privacy boundary in the same comfortable way: the documents are local, but answer generation quietly calls a cloud model. For public docs, nobody gets hurt. For camera runbooks, incident notes, network diagrams, customer data, or site-specific operational history, that silent hop is the whole ballgame.

So the privacy promise has to be testable:

  • What classification was assigned to the document?
  • Which chunks were retrieved?
  • Was the prompt body logged?
  • Which backend generated the answer?
  • What would happen if the local backend failed?

Each of those answers has to be visible in the demo itself, or the demo is proving nothing.

Four Jobs and a Deliberately Boring Vector Store

Private RAG local-only architecture: private documents are chunked, embedded, stored in a vector store, retrieved for a restricted question, grounded into a prompt, and routed through a LocalOnly gateway policy that allows local generation while denying cloud fallback

AiOnTheEdge.KnowledgeAssistant has four jobs:

  1. Ingest documents with metadata and classification.
  2. Chunk and embed the documents.
  3. Retrieve relevant context for a question.
  4. Ask the gateway for an answer under an explicit policy.

The first version should keep storage boring: one local vector store, wrapped behind an interface, with the retrieval trace visible. The citations come from retrieval metadata, not from model-generated prose - if the model is writing its own bibliography, the citations are fiction with good formatting.

A seeded document looks like this:

{
  "documentId": "garage-camera-runbook",
  "title": "Garage Camera Runbook, Remote Property 1",
  "sourcePath": "samples/documents/garage-camera-runbook.md",
  "classification": "Restricted",
  "tags": ["iot", "cameras", "runbook"],
  "createdUtc": "2026-05-11T09:00:00Z"
}

One honesty note before anyone asks for the dataset: the documents, queries, and responses in this post are synthetic samples rather than captured traffic. Site ids follow the camera series naming - remote1 and remote2 are the remote properties in the Part 1 network model, and garage-east is the Class B agent camera configured on remote1 in Part 2.

The query shape makes policy explicit:

{
  "question": "What should I check first if the remote1 garage camera is offline?",
  "classification": "Restricted",
  "policy": "LocalOnly",
  "topK": 5,
  "includeCitations": true
}

The response carries everything those five testability questions need: the answer, citations, route decision, privacy decision, and retrieval trace:

{
  "answer": "Check the remote1 site network path before rebooting cameras...",
  "citations": [
    {
      "documentId": "garage-camera-runbook",
      "heading": "Offline camera checklist",
      "score": 0.86
    }
  ],
  "routeDecision": {
    "policy": "LocalOnly",
    "selectedBackend": "foundry-local",
    "fallbackUsed": false
  },
  "privacyDecision": {
    "promptBodyLogged": false,
    "reason": "Restricted content suppresses prompt body logging."
  }
}

Foundry Local is a good fit for the laptop version because Microsoft documents local embedding generation and RAG-style workflows that run on device. The implementation still supports mock embeddings, because the demo has to run even before every model is cached.

Running the Denial on Purpose

What the audience watches for is a denied fallback.

  1. Show the document library: runbooks, deployment notes, and incident notes classified Restricted, plus the published architecture posts sitting alongside them as background.
  2. Ask: Which cameras are offline, what probably caused it, and what should the operator check first?
  3. Show retrieved chunks and citation metadata, all of them from the restricted set.
  4. Show the grounded answer.
  5. Open the route trace: classification Restricted, policy LocalOnly, selected local backend.
  6. Disable foundry-local and mock-accelerator, so no device or edge backend is left eligible.
  7. Ask again.
  8. Show the request fail closed, because the only healthy backends left are cloud.
  9. Change the classification to Internal.
  10. Ask again with a policy that allows fallback.
  11. Show cloud or mock-cloud fallback arriving only after the policy changed.

That last step carries the demo. The privacy boundary moved because somebody changed a policy on stage, which is the only way it is ever supposed to move.

Seed Documents I Already Own

The existing posts pull double duty as seed documents:

  • The Tenstorrent private cloud buildout gives hardware and environment context.
  • The Azure IoT Operations camera posts give camera topology, MQTT topics, broker security, data-flow routing, and connector notes.

Those posts get ingested as samples, but this demo does not retell them. It proves that private operational knowledge can be queried locally with citations.

The new build is AiOnTheEdge.KnowledgeAssistant:

POST /documents/ingest
GET  /documents
POST /query
POST /query/explain
POST /admin/reindex

Those are additions. Underneath them the service exposes the same operational endpoints every service in the system exposes - /healthz, /readyz, /metrics, /admin/demo/reset, /admin/demo/seed, and /admin/demo/faults - so seeding the corpus goes through the shared /admin/demo/seed, and /admin/reindex is the only genuinely RAG-specific control.

Implementation requirements:

  • Ingest Markdown, PDF text, JSON, CSV, and plain text.
  • Chunk by heading, paragraph, and token budget.
  • Preserve source filename, heading path, and classification.
  • Use Foundry Local embeddings where available.
  • Support local or mock embeddings for fallback.
  • Generate citations from retrieval metadata.
  • Send answer generation through the AI on the Edge Gateway.
  • Evaluate known questions with expected citations.

The Outage That Proves the Boundary

Everything above assumes the local model answers. The case worth rehearsing is the one where it does not and the data is still restricted.

Private RAG fail-closed sequence: a restricted operator question retrieves local chunks and builds a grounded prompt, but when the local backend is unavailable and LocalOnly is active, cloud eligibility is rejected, the request is denied, and local retrieval trace and route telemetry remain visible

Cloud fallback is not a recovery path for restricted RAG. The system returns a clear denial instead:

{
  "decision": "Denied",
  "policy": "LocalOnly",
  "reason": "Restricted content stays on edge; no eligible edge backend."
}

The UI shows the retrieval trace and the route denial side by side, which makes the behavior legible: the system found useful context and correctly refused to send it to an ineligible backend.

The Bar for Demo 2

The demo works when sample documents ingest with their metadata and classification intact, queries come back grounded with citations that point at real chunks, and restricted content only ever reaches a device or edge backend. The denial has to be visible rather than inferred: LocalOnly with nothing local left standing produces a refusal on screen, with the retrieval trace beside it showing the context it declined to send anywhere.

Two supporting pieces make the claim checkable rather than theatrical. An evaluation pass runs ten known questions against expected citations and produces pass or fail, which is the only way I can tell whether a chunking change made retrieval worse. And once the models and packages are cached, the whole thing runs with the network unplugged, which is the shortest possible proof that nothing was quietly reaching out.

This demo ingests the existing camera and lab posts as background context around a restricted corpus, and it depends on the gateway from One App, Many Places to Run AI, because the RAG assistant should not own fallback policy itself. The moment a component owns fallback policy, it starts making exceptions.

References