Kubernetes Platform Engineering & Linux Networking

Rather than relying on an external database or single SQLite instance, Cloud Lab runs a true distributed 3-node embedded etcd cluster (node-1-control, node-2-ingress, node-3-workloads) with automatic snapshotting. Any single control-plane node can be taken offline for maintenance without service interruption.

In enterprise hybrid environments, host-level firewalls (UFW on Ubuntu, firewalld on Rocky) often conflict with Kubernetes overlay networks (Flannel VXLAN). When nodes restart or firewall rules re-apply, pod-to-pod traffic across nodes drops. To permanently eliminate this failure mode, we engineered a dedicated Kubernetes DaemonSet that enforces iptables and UFW whitelist rules across all nodes:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fix-ufw-ds
  namespace: kube-system
spec:
  selector:
    matchLabels:
      name: fix-ufw
  template:
    metadata:
      labels:
        name: fix-ufw
    spec:
      hostNetwork: true
      containers:
      - name: fix-ufw
        image: alpine:latest
        securityContext:
          privileged: true
        command: ["/bin/sh", "-c"]
        args:
          - |
            while true; do
              ufw allow from 10.42.0.0/16 comment 'Allow K3s Pod CIDR' 2>/dev/null || true
              ufw allow from 10.43.0.0/16 comment 'Allow K3s Service CIDR' 2>/dev/null || true
              sleep 3600
            done

A key architectural triumph of Cloud Lab is running bare-metal CloudPanel NGINX (serving ~30 production PHP/Node sites) and a K3s control-plane node on the exact same physical server (node-1-control) without port contention:

• Port Decoupling: Host NGINX binds public 80/443 on IP [Control Plane IP]. K3s Ingress-NGINX controller runs on Node 2 ([Ingress Gateway IP]), preventing port binding collisions.
• Unified Log Pipeline: FastAPI dynamically discovers all 26+ Unix site directories (/home/*/logs/nginx/access.log), stripping bot heartbeats and calculating 24h unique visitors and error rates.
• Observability Unification: Host-level Node Exporter (port 9100) feeds Prometheus, giving identical visibility into both bare-metal and containerized workloads.