Architecture Decision Records (ADRs)
Formal engineering records documenting key architectural decisions made during the design, deployment, and operational hardening of Cloud Lab.
Existing Prometheus Server as Authoritative Telemetry Engine
The cluster requires continuous health, CPU, memory, load, and container resource metrics across 4 physical nodes. We considered installing multiple custom telemetry collectors or querying K3s Metrics Server directly.
Use the existing Prometheus server (monitoring/prometheus-server) as the single authoritative telemetry source. A lightweight FastAPI adapter issues targeted PromQL queries with in-memory TTL caching.
Eliminates redundant monitoring agents, protects node CPU/disk I/O, and provides high-precision PromQL rates while decoupling UI clients from raw PromQL complexity.
CloudPanel Non-Invasive Monitoring Strategy
CloudPanel hosts ~30 commercial virtual hosts on Node 1 (node-1-control). We needed full availability, SSL expiry countdown, and error rate tracking without interfering with production PHP/Node processes.
Combine Uptime Kuma external probes for availability and SSL certificate countdowns with a high-performance Python log analyzer parsing /home/*/logs/nginx/access.log directly.
Zero agent footprint on host virtual hosts. Strips synthetic healthcheck pings automatically, providing organic traffic figures and SSL expiry alerts.
Unified Server-Side Log Analytics vs. Client-Side Analytics
We needed accurate traffic analytics across both CloudPanel websites and Kubernetes microservices without privacy violations, cookies, or ad-blocker drop-offs.
Implement server-side log analysis as the primary traffic tracking engine, supplemented by Umami where client-side journey tracking is explicitly required.
Captures 100% of API consumers, mobile clients, and web visits accurately with zero browser performance overhead and strict IP hashing.
Homepage Declarative Widget Boundary
Homepage supports native integrations for tools like Prometheus, ArgoCD, and Gitea, but lacks built-in cards for multi-node hardware grids, cross-platform traffic, and live sports.
Strictly use Homepage native widgets where mature integrations exist (Prometheus, Uptime Kuma, ArgoCD, Gitea). Custom Python endpoints via FastAPI are exclusively used for multi-node vitals, unified traffic tables, sports tickers, and curated RSS news.
Minimizes custom code to under 500 lines of Python while maximizing dashboard density and reliability.
Public Showcase Air-Gapped SSG Deployment Model
The public showcase (lab.techarvest.co.zw) must showcase live platform engineering achievements to hiring managers and recruiters without creating security risks for internal cluster networks.
Deploy the showcase as a Next.js Static Site Generation (SSG) application served by standard NGINX. A scheduled snapshot pipeline exports sanitized data from FastAPI into static assets, with zero runtime cluster network connectivity.
Complete air-gap security guarantee. Even under massive public DDoS or complete web container compromise, internal cluster networks and databases are completely unreachable.