Backend
A FastAPI service running three replicas on k3s, backed by replicated PostgreSQL and distributed MinIO. Every high-availability claim on this page was measured on the physical Raspberry Pi cluster — including the ones that turned out to have limits we did not expect.
What was asked
Develop a backend to manage the sensor nodes and the collected data. Deploy Docker containers on a Raspberry Pi Kubernetes cluster with k3s, including a distributed file system (Ceph, SeaweedFS) or a storage service (MinIO).
What actually runs
edge-monitoring14 podsAPI
| Method | Path | Purpose |
|---|---|---|
| GET | /healthz | health probe |
| POST | /api/events | multipart: type, source, message, detections JSON, optional image |
| GET | /api/events | list events |
| GET | /api/events/{id} | single event |
| GET | /api/events/{id}/image | evidence image |
Four modules behind it: app/main.py (routes), app/db.py
(PostgreSQL via psycopg and a connection pool), app/storage.py
(the MinIO object store), and app/bus.py (the MQTT publisher).
Two stores, not one
MinIO holds the large binary evidence images as objects, reachable over the
network by any backend pod on any node. PostgreSQL holds the small,
structured, queryable metadata — type, source, timestamp, detections. "The 50 newest THREAT
events, by time" is a query an object store cannot answer natively. Each image is written
under a date-prefixed object key (2026/06/02/<uuid>.jpg) and only that key
is stored in the metadata row.
The metadata store began as SQLite — one local file, therefore one writer, therefore exactly
one backend replica. Multi-replica HA needs a network-shared, concurrently writable
database, and that is what drove the move to PostgreSQL. The change stayed contained in
backend/app/db.py.
High availability, by layer
| Layer | Design | Failure model | Measured |
|---|---|---|---|
| Backend | 3 stateless replicas behind a Service | Active/Active — every replica serves; no PVC, so a pod can reschedule anywhere | killing one pod: no request lost |
| PostgreSQL | CloudNativePG, 1 primary + 2 standbys, asynchronous WAL streaming | Active/Passive — one writer; CNPG promotes a standby and repoints the pg-rw Service |
one ~5 s gap, no data loss |
| MinIO | 4-node StatefulSet, Reed–Solomon erasure coding, EC:2 (2 data + 2 parity) | Active/Active peers — any pod rebuilds any object from 2 surviving shards | 3 of 4 drives: read + write 2 of 4 drives: read only |
Active/Active means all replicas serve traffic at once. Active/Passive means one serves and a standby is promoted on failure. The asymmetry in the MinIO row is the detail most write-ups miss: read quorum is 2 but write quorum is 3, because parity is half the set. Losing a second drive does not halve availability — it silently turns the object store read-only, and the backend surfaces that as failed uploads while reads keep working.
This is Shared Nothing replication — each node holds its own storage, data is replicated between nodes — chosen deliberately over a single Shared Disk, on the principle that shared storage is always a single point of failure. At the Kubernetes layer, that is exactly what we built. At the physical layer, it is not.
The eight Pi 3 workers have no disks of their own (Task 1). They netboot over PXE and mount their entire root filesystem over NFS from the Pi 5, so every "local" volume Kubernetes provisions on a worker is really a directory on the Pi 5's single 458 GB SSD. Erasure coding and streaming replication still buy real, demonstrable protection against a worker crashing, rebooting or being power-cycled — that was tested on hardware and it works. What they do not protect against is the loss of that one disk, which would take every shard and every replica at the same instant. Full analysis in Task 10 — the risk register.
Fitting a distributed stack into 476 MiB
A Raspberry Pi 3 reports 906 MiB of capacity. After reserving 350 MiB for the system and setting a hard eviction threshold, the scheduler is left with 476 MiB allocatable per worker — without that reservation it would happily overfill a node until the kernel started killing processes.
That budget is why the PostgreSQL request is 192Mi and not 256Mi. MinIO
requests 256Mi, and both carry required anti-affinity, so at 256Mi each a worker
cannot hold one of each and instances would sit Pending forever. It is also the
reason MinIO was chosen over Ceph, which does not fit on a 1 GB node at all.
Stateless and stateful workloads are pinned differently on purpose. Backend and frontend use
topologySpreadConstraints with ScheduleAnyway — they may roam,
because they own no data. MinIO and PostgreSQL use required node and anti-affinity
rules, because the local-path provisioner welds a hard nodeAffinity
onto every PersistentVolume it creates: a pod whose data lives on worker 3 can only ever run
on worker 3.
Three failures worth documenting
On an NFS root, MinIO compares st_dev of its data directory against
st_dev of /. Both measure the same device, so its root-drive
guard rejected all four drives — and the pods still reported Ready, because
/minio/health/ready does not check quorum. A healthy-looking deployment with
no usable storage. Fixed by setting
MINIO_ROOTDRIVE_THRESHOLD_SIZE=1MiB, which replaces the device check with a
size check. The semantics are inverted from what you would guess: 0 or an
empty value does not disable the guard, only a positive value does.
A backend pod crash-looped 176 times, roughly 43 s per attempt, ending each run with a
misleading error. The retry loop reused a single psycopg_pool
ConnectionPool: open(wait=True) calls close() on
timeout, and a closed pool cannot be reopened, so every retry after the first failed
instantly for a different reason than the original. The fix was to construct the pool
inside the retry loop — three lines, found only by reading why attempt 2 failed
differently from attempt 1.
All eight workers went NotReady at once while still answering
ping. The cause was not in Task 7: a DaemonSet belonging to another subtask
ran mount 10.0.0.1:/shared/mpi on every node — including the Pi 5,
which exports that directory. A server mounting its own export deadlocks NFS, and the same
nfsd threads serve every worker's root filesystem, so the whole cluster froze
together. No data was lost. It is documented here because it is the sharpest illustration
of the point above: the Pi 5 is not just a control plane, and a fault anywhere in it stops
being local very quickly.
Single points of failure we accepted
| Component | State |
|---|---|
| The Pi 5 | 🔴 control plane, NFS server for every worker root, image registry, DHCP/TFTP boot server, NAT gateway and the only physical disk — all on one machine |
| Mosquitto | 🔴 one broker replica. Live push stops; the frontend falls back to REST polling, so it degrades rather than breaks |
| Off-node backups | 🔴 none — replication protects against node loss, not against deletion or site loss |
Real control-plane HA needs three or more k3s servers with embedded etcd. With one Pi 5 this is a known, documented limitation of a seminar-scale cluster — recorded rather than hidden.
Run and verify it
On the Pi cluster the images are already built and pre-pulled onto the workers, so deploy without building. The Makefile enforces this: every building target refuses to run against a non-local cluster, because rebuilding would invalidate the pre-pulled images.
cd cluster
make deploy-storage deploy-postgres deploy-mqtt deploy-backend deploy-frontend
make status # nodes and all pods
make test-backend # post an event through the ingress, then list eventsThe high-availability demonstrations are make targets of their own:
make ha-test # kill a backend pod, confirm the data survives
make pg-failover # kill the Postgres primary, watch CNPG promote a standby
make minio-status # which of the 4 MinIO instances runs on which node
make postgres-status # which instance is currently primaryDeeper reading
This page is the summary. The working documents behind it live in the repository:
| Document | What it covers |
|---|---|
| docs/storage.md | where data lives, how the two stores replicate, and the Shared Nothing vs. Shared Disk argument |
| docs/weaknesses.md | the risk register: erasure-coding maths, the failure walk-through, and what is still untested |
| docs/architecture.md | the concept reference — Services, StatefulSets, PVCs, probes, ingress, replication |
| cluster/docs/step-7-pi-migration.md | the migration from the local development cluster onto the physical Pis, phase by phase |
| cluster/docs/migration-issues.md | the incident catalogue — every problem hit during that migration, with cause and fix |
| cluster/docs/operations.md | the operations runbook: bring-up, shutdown order, and the failure tests |
| cluster/docs/adding-a-worker.md | adding a diskless worker, and powering the cluster down without freezing one |