Documentation & Risk Register
Complete online documentation — including this site — plus a deliberately unflattering engineering document: what our high-availability design actually survives, what it only appears to survive, and what remains untested.
What was asked
Create complete online documentation via GitHub Pages and present via a poster plus a live demonstration. No slide presentations or traditional PDF project reports.
Documentation map
| Document | What it covers |
|---|---|
docs/architecture.md | system architecture, local-to-Pi mapping, ~30 key Kubernetes/networking concepts explained |
docs/storage.md | how persistence works, the two storage variants considered, the SPOF inventory |
docs/flow.md | end-to-end threat-detection flow as a rendered Mermaid diagram |
docs/weaknesses.md | the risk register this page summarizes below |
cluster/docs/*.md | a build-step-by-step runbook: local cluster setup, operations, incident log, worker-onboarding runbook, and one doc per migration step (backend/storage, frontend, Telegram, HA/Postgres, distributed storage, MQTT, Pi migration) |
The risk register — why it exists
docs/weaknesses.md was written before the demo. Its stated scope: failure mode → blast
radius → why the existing mitigation doesn't cover it → where the fix is tracked.
The central finding: the diskless twist
The eight Pi 3 workers are diskless — they PXE-boot and mount their entire root filesystem
over NFS from the Pi 5 (Task 1). Kubernetes' default
local-path storage provisioner satisfies every volume claim by creating a
directory on "the node's local disk" — which, for an NFS-rooted worker, is physically a
path on the Pi 5's single SSD. So all four MinIO erasure shards and all three PostgreSQL
copies (Task 7) currently land on one physical
device, even though the Kubernetes layer reports them as spread across four and
three separate nodes respectively.
The redundancy is not fake — it genuinely survives a worker crashing, rebooting, being power-cycled, or being replaced by the identical board (same serial ⇒ same NFS export ⇒ data comes back automatically). What it does not survive is loss of the Pi 5's disk itself, which backs every worker's OS, every MinIO shard, and every Postgres copy at once.
What was actually measured on hardware
Not all analysis in the register is theoretical — a meaningful share was tested on the physical cluster (2026-08-13/14):
| Test | Result |
|---|---|
| 4 MinIO pods on 4 distinct Pi 3 workers | ✅ /minio/health/cluster returns 200 |
| Quorum split, 3 of 4 drives online | ✅ read and write both succeed |
| Quorum split, 2 of 4 drives online | ✅ read OK, write correctly fails (502) |
| A returning drive heals | ✅ observed catching up from 5 objects to its peers' 7 |
| CloudNativePG failover | ✅ standby promoted, single 5-second gap, zero data loss |
| Real node power-off | ✅ survived; node went NotReady after ~70s, writes kept succeeding at 3 of 4 drives |
| NFS root-drive rejection gate | ✅ confirmed via stat on both paths — measured, not just read from source |
| O_DIRECT support over the NFS mount | ✅ confirmed — this was the dangerous one; a failure would have killed distributed MinIO on NFS outright |
What's explicitly not yet verified: permanent (not just temporary) node loss recovery, bucket versioning behavior, the background scanner and MRF queue timing, behavior under sustained load, and any capacity claim — nothing has been load-tested, and the largest object stored so far is a test JPEG.
Full SPOF inventory
| Layer | State | If it fails |
|---|---|---|
| Backend | 🟢 3 replicas, Active/Active | another replica serves; no outage |
| PostgreSQL | 🟢 CNPG, 1 primary + 2 standbys | standby auto-promoted, data intact |
| MinIO | 🟢 distributed, 4 nodes, EC:2 | object rebuilt from surviving shards |
| Storage volumes | 🟡 replicated across Kubernetes nodes, but collapsed onto one physical device | survives a worker loss; does not survive the Pi 5's disk |
| The Pi 5's SSD (physical) | 🔴 backs every replica — all worker filesystems, all shards, all DB copies | total data loss |
| Control plane (Pi 5) | 🔴 single node, also runs the ingress | cluster management and ingress down together |
| Ingress (Traefik) | 🔴 single replica on the Pi 5 | no external traffic enters the cluster |
| Backups | 🔴 none yet | replication ≠ backup — doesn't protect against accidental deletion |
| PXE/DHCP/TFTP/NFS, registry, k3s datastore | 🔴 each a single service on the Pi 5 | no worker can boot; image pulls fail; cluster state is unrecoverable |
Other documented findings worth naming
- NFS export security. Worker root exports use
sec=sys(no cryptographic auth) and a wildcard host list — required for diskless boot to work at all, but wider than necessary. Reported, not fixed, since it belongs to the PXE subtask. - Health probes lie by omission. MinIO's readiness probe returns
200even with zero usable drives; only/minio/health/clusterreports quorum honestly. A backend liveness probe was once observed killing a healthy pod during a MinIO outage because its check made two network round trips against a 1-second timeout. - Stateless vs. stateful scheduling. MinIO and Postgres use required node/anti-affinity (never co-locate, never land on the Pi 5); the backend and frontend use preferred (may roam freely) — deliberately asymmetric, because a stateful pod's volume gets welded to whatever node it first lands on.
- "What we'd do differently with more hardware" — a nine-item, impact-ordered list in the register, topped by "give every worker a real disk," which is the single change that would make the Shared-Nothing design physically true rather than only logically true.
This documentation site
This multi-page site — served from /pages — is the GitHub Pages deliverable
for Task 10. It's generated from the same source-of-truth READMEs and design docs linked
throughout, reorganized by task rather than by directory so a reader can follow the grading
rubric directly.
The complete, unabridged risk register — including the MinIO shard-quorum math, the
step-by-step worker-failure walkthrough, and 15 tracked open questions each paired with the
experiment that would settle it — lives in
docs/weaknesses.md in the repository root.