Back to tags
#infrastructure
the parts of the stack you only think about when they break.
5 blog posts.
Blog posts
Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.
Eight GPUs in one server behave like a small network. NVLink vs PCIe, reading nvidia-smi topo -m, NCCL transports, the ACS trap, and fitting a 70B model.
A GPU pod sits on a dozen layers from silicon to scheduler, and each one fails its own way. Drivers, the container toolkit, MIG, DCGM, and the metrics…
Our AWS bill hit $50k/month. Here's exactly how EC2 rightsizing, Reserved Instances, S3 lifecycle rules, and CloudWatch retention cut it to $20k in six…
Terraform mistakes I made so you don't have to: hardcoded AMIs, lost local state, a 2000-line main.tf, an unpinned provider, and a stray terraform destroy.