Private data is the easiest reason to care about edge AI. If the data cannot leave the site, then the answer cannot depend on a cloud fallback that nobody noticed.

Demo 2 extends the platform with a Knowledge Assistant. In the design, it ingests local documents, builds embeddings locally when available, retrieves relevant chunks, generates grounded answers, and sends every generation request through the same routing policy engine from Demo 1. The RAG part is nearly conventional. The part I actually care about is that the privacy promise gets enforced by routing instead of by hope.

Where most RAG demos leak

RAG demos tend to blur the privacy boundary in the same comfortable way: the documents are local, but answer generation quietly calls a cloud model. For public docs, nobody gets hurt. For camera runbooks, incident notes, network diagrams, customer data, or site-specific operational history, that silent hop is the whole ballgame.

So the privacy promise has to be testable:

  • What classification was assigned to the document?
  • Which chunks were retrieved?
  • Was the prompt body logged?
  • Which backend generated the answer?
  • What would happen if the local backend failed?

Each of those answers has to be visible in the demo itself, or the demo is proving nothing.

Four jobs and a deliberately boring vector store

Private RAG local-only architecture: private documents are chunked, embedded, stored in a vector store, retrieved for a restricted question, grounded into a prompt, and routed through a LocalOnly gateway policy that allows local generation while denying cloud fallback

AiOnTheEdge.KnowledgeAssistant has four jobs:

  1. Ingest documents with metadata and classification.
  2. Chunk and embed the documents.
  3. Retrieve relevant context for a question.
  4. Ask the gateway for an answer under an explicit policy.

The first version should keep storage boring: one local vector store, wrapped behind an interface, with the retrieval trace visible. The citations come from retrieval metadata, not from model-generated prose - if the model is writing its own bibliography, the citations are fiction with good formatting.

A seeded document looks like this:

{
  "documentId": "garage-camera-runbook",
  "title": "Garage Camera Runbook, Remote Property 1",
  "sourcePath": "samples/documents/garage-camera-runbook.md",
  "classification": "Restricted",
  "tags": ["iot", "cameras", "runbook"],
  "createdUtc": "2026-05-11T09:00:00Z"
}

One honesty note before anyone asks for the dataset: the documents, queries, and responses in this post are synthetic samples rather than captured traffic. Site ids follow the camera series naming - remote1 and remote2 are the remote properties in the Part 1 network model, and garage-east is the Class B agent camera configured on remote1 in Part 2.

The query shape makes policy explicit:

{
  "question": "What should I check first if the remote1 garage camera is offline?",
  "classification": "Restricted",
  "policy": "LocalOnly",
  "topK": 5,
  "includeCitations": true
}

The response carries everything those five testability questions need: the answer, citations, route decision, privacy decision, and retrieval trace:

{
  "answer": "Start with the remote1 site network path before rebooting individual cameras...",
  "citations": [
    {
      "documentId": "garage-camera-runbook",
      "heading": "Offline camera checklist",
      "score": 0.86
    }
  ],
  "routeDecision": {
    "policy": "LocalOnly",
    "selectedBackend": "foundry-local",
    "fallbackUsed": false
  },
  "privacyDecision": {
    "promptBodyLogged": false,
    "reason": "Restricted content suppresses prompt body logging."
  }
}

Foundry Local is a good fit for the laptop version because Microsoft documents local embedding generation and RAG-style workflows that run on device. The implementation still supports mock embeddings, because the demo has to run even before every model is cached.

Running the denial on purpose

What the audience watches for is a denied fallback.

  1. Show the document library: camera runbooks, deployment notes, incident notes, and the existing camera architecture posts.
  2. Ask: Which cameras are offline, what probably caused it, and what should the operator check first?
  3. Show retrieved chunks and citation metadata.
  4. Show the grounded answer.
  5. Open the route trace: classification Restricted, policy LocalOnly, selected local backend.
  6. Disable the local backend.
  7. Ask again.
  8. Show the request fails closed because cloud fallback is not allowed.
  9. Change the classification to Internal.
  10. Ask again with a policy that allows fallback.
  11. Show cloud or mock-cloud fallback only after the policy changes.

That last step is the point of the whole demo: the privacy boundary is enforced by the platform, not by hope.

Seed documents I already own

The existing posts pull double duty as seed documents:

  • The Tenstorrent private cloud buildout gives hardware and environment context.
  • The Azure IoT Operations camera posts give camera topology, MQTT topics, broker security, data-flow routing, and connector notes.

Those posts get ingested as samples, but this demo does not retell them. It proves that private operational knowledge can be queried locally with citations.

The new build is AiOnTheEdge.KnowledgeAssistant:

POST /documents/ingest
GET  /documents
POST /query
POST /query/explain
POST /admin/reindex
POST /admin/demo/seed-rag

Implementation requirements:

  • Ingest Markdown, PDF text, JSON, CSV, and plain text.
  • Chunk by heading, paragraph, and token budget.
  • Preserve source filename, heading path, and classification.
  • Use Foundry Local embeddings where available.
  • Support local or mock embeddings for fallback.
  • Generate citations from retrieval metadata.
  • Send answer generation through the AI on the Edge Gateway.
  • Evaluate known questions with expected citations.

The outage that proves the boundary

The critical failure for this design is a local model outage while the data is restricted.

Private RAG fail-closed sequence: a restricted operator question retrieves local chunks and builds a grounded prompt, but when the local backend is unavailable and LocalOnly is active, cloud eligibility is rejected, the request is denied, and local retrieval trace and route telemetry remain visible

Cloud fallback is not a recovery path for restricted RAG. The system returns a clear denial instead:

{
  "decision": "Denied",
  "policy": "LocalOnly",
  "reason": "Restricted content must stay on edge and no eligible local backend is healthy."
}

The UI shows the retrieval trace and the route denial side by side, which makes the behavior legible: the system found useful context and correctly refused to send it to an ineligible backend.

What done looks like

Demo 2 earns its number when:

  1. Sample documents can be ingested with metadata and classification.
  2. Queries return grounded answers with deterministic citations.
  3. Restricted content routes only to local or edge backends.
  4. Cloud fallback is visibly denied for LocalOnly.
  5. Retrieval traces are visible in the dashboard.
  6. An evaluation script can run 10 known questions and produce pass/fail results.
  7. The demo runs without internet after models and packages are pre-cached.

This demo consumes the existing camera and lab posts as private knowledge sources, and it depends on the gateway from One App, Many Places to Run AI, because the RAG assistant should not own fallback policy itself. The moment a component owns fallback policy, it starts making exceptions.

References