Back to tags
#kubernetes
pods that won't stay up, configs that won't render, and the kubectl command you'll wish you'd typed sooner.
9 blog posts · 2 TILs.
Posts on Kubernetes as it actually behaves at 3am. Six come from the GPUs in production series: the dozen layers under a GPU pod, scaling past one box, scale-to-zero, inference routing, the probe that kills a healthy model server, and multi-tenancy on a shared card. The rest are the debugging tricks that saved a production cluster, a benchmarked VictoriaLogs against Loki comparison, and two short kubectl notes — JSONPath queries and kubectl neat. Exit codes, probe configs, scheduler behaviour, and the YAML that explains why.
Blog posts
CrashLoopBackOff is a symptom with five common causes. The exit code and the last event narrow it to one in under a minute, before you open the…
A Kubernetes namespace isolates the API, not the silicon. Under time-slicing two teams share a physical GPU with no memory wall. Quotas, isolation, and…
A model server that takes five minutes to load and a liveness probe that gives it ten seconds is a crash loop waiting to happen. Probes, drains, and safe…
Same GPUs, same model, same replica count. Swap round-robin for prefix-cache-aware routing and the fleet gets 2.3x faster. The router was throwing the…
Idle GPUs at six dollars an hour are a bonfire. Scaling to zero saves the money, but the first user back waits minutes unless you kill the cold start.
Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.
A GPU pod sits on a dozen layers from silicon to scheduler, and each one fails its own way. Drivers, the container toolkit, MIG, DCGM, and the metrics…
VictoriaLogs vs Loki, benchmarked on 500 GB of logs over 7 days on identical nodes: 94% lower query latency, 37% less storage, under half the CPU and RAM.
Five Kubernetes debugging tricks from real 3 AM pages: kubectl logs --previous, ephemeral debug containers, scheduler events, and fixes that saved…
Today I learned
kubectl can return exactly the field you want without a grep-awk-sed pipeline. JSONPath queries that replaced my entire stash of one-liners.
kubectl get -o yaml dumps 200 lines of generated noise. kubectl neat strips it down to what you actually wrote. Two commands, no more copy-paste cleanup.