Private data is the easiest reason to care about edge AI. If the data cannot leave the site, the answer cannot depend on a cloud fallback that nobody noticed.

The second demo adds a Knowledge Assistant to the gateway. It ingests local documents, builds embeddings locally when available, retrieves relevant chunks, generates grounded answers, and sends the generation request through the same routing policy engine from Demo 1.

Problem

RAG demos often blur the privacy boundary. The documents are local, but the answer generation quietly calls a cloud model. That may be fine for public docs. It is not fine for camera runbooks, incident notes, network diagrams, customer data, or site-specific operational history.

The privacy promise has to be testable:

  • What classification was assigned to the document?
  • Which chunks were retrieved?
  • Was the prompt body logged?
  • Which backend generated the answer?
  • What would happen if the local backend failed?

If those answers are not visible, the demo is just trust with a search box.

Architecture Slice

Private RAG local-only architecture: private documents are chunked, embedded, stored in a vector store, retrieved for a restricted question, grounded into a prompt, and routed through a LocalOnly gateway policy that allows local generation while denying cloud fallback

AiOnTheEdge.KnowledgeAssistant has four jobs:

  1. Ingest documents with metadata and classification.
  2. Chunk and embed the documents.
  3. Retrieve relevant context for a question.
  4. Ask the gateway for an answer under an explicit policy.

The first version should keep the storage boring. Use one local vector store, wrap it behind an interface, and make the retrieval trace visible. The citations should come from retrieval metadata, not from model-generated prose.

{
  "documentId": "camera-control-plane",
  "title": "Cameras on Azure IoT Operations, Part 1",
  "sourcePath": "samples/documents/camera-control-plane.md",
  "classification": "Restricted",
  "tags": ["iot", "aio", "mqtt", "camera-fleet"],
  "createdUtc": "2026-06-19T12:00:00Z"
}

The query shape makes policy explicit:

{
  "question": "What should I check first if the remote2 garage camera is offline?",
  "classification": "Restricted",
  "policy": "LocalOnly",
  "topK": 5,
  "includeCitations": true
}

The response should include the answer, citations, route decision, privacy decision, and retrieval trace:

{
  "answer": "Start with the remote2 site network path before rebooting individual cameras...",
  "citations": [
    {
      "documentId": "camera-control-plane",
      "heading": "The network model",
      "score": 0.86
    }
  ],
  "routeDecision": {
    "policy": "LocalOnly",
    "selectedBackend": "foundry-local",
    "fallbackUsed": false
  },
  "privacyDecision": {
    "promptBodyLogged": false,
    "reason": "Restricted content suppresses prompt body logging."
  }
}

Foundry Local is a good fit for the laptop version because Microsoft documents local embedding generation and RAG-style workflows that run on device. The implementation should still support mock embeddings so the demo can run before every model is cached.

Demo Script

The visible audience moment is a denied fallback.

  1. Show the document library: camera runbooks, deployment notes, incident notes, and the existing camera architecture posts.
  2. Ask: Which cameras are offline, what probably caused it, and what should the operator check first?
  3. Show retrieved chunks and citation metadata.
  4. Show the grounded answer.
  5. Open the route trace: classification Restricted, policy LocalOnly, selected local backend.
  6. Disable the local backend.
  7. Ask again.
  8. Show the request fails closed because cloud fallback is not allowed.
  9. Change the classification to Internal.
  10. Ask again with a policy that allows fallback.
  11. Show cloud or mock-cloud fallback only after the policy changes.

The audience should see that the privacy boundary is enforced by the platform, not by hope.

What Already Exists

The existing posts provide useful seed documents:

  • The Tenstorrent private cloud buildout gives hardware and environment context.
  • The Azure IoT Operations camera posts give camera topology, MQTT topics, broker security, data-flow routing, and connector notes.

Those posts should be ingested as sample documents, but this demo should not retell them. It should prove that private operational knowledge can be queried locally with citations.

What Is New

The new build is AiOnTheEdge.KnowledgeAssistant:

POST /documents/ingest
GET  /documents
POST /query
POST /query/explain
POST /admin/reindex
POST /admin/demo/seed-rag

Implementation requirements:

  • Ingest Markdown, PDF text, JSON, CSV, and plain text.
  • Chunk by heading, paragraph, and token budget.
  • Preserve source filename, heading path, and classification.
  • Use Foundry Local embeddings where available.
  • Support local or mock embeddings for fallback.
  • Generate citations from retrieval metadata.
  • Send answer generation through the AI on the Edge Gateway.
  • Evaluate known questions with expected citations.

Failure Mode

The critical failure is local model outage with restricted data.

Private RAG fail-closed sequence: a restricted operator question retrieves local chunks and builds a grounded prompt, but when the local backend is unavailable and LocalOnly is active, cloud eligibility is rejected, the request is denied, and local retrieval trace and route telemetry remain visible

Cloud fallback is not a recovery path for restricted RAG. The system should return a clear denial:

{
  "decision": "Denied",
  "policy": "LocalOnly",
  "reason": "Restricted content must stay on edge and no eligible local backend is healthy."
}

The UI should show both the retrieval trace and the route denial. That makes it clear the system found useful context but correctly refused to send it to an ineligible backend.

Acceptance Criteria

Demo 2 is complete when:

  1. Sample documents can be ingested with metadata and classification.
  2. Queries return grounded answers with deterministic citations.
  3. Restricted content routes only to local or edge backends.
  4. Cloud fallback is visibly denied for LocalOnly.
  5. Retrieval traces are visible in the dashboard.
  6. An evaluation script can run 10 known questions and produce pass/fail results.
  7. The demo runs without internet after models and packages are pre-cached.

This demo consumes the existing camera and lab posts as private knowledge sources. It also depends on the gateway from One App, Many Places to Run AI, because the RAG assistant should not own fallback policy itself.

References