Back to tags
#infrastructure
the parts of the stack you only think about when they break.
5 blog posts.
Five posts about the layer you only think about when it breaks. The AWS cost post walks a fifty-thousand-dollar monthly bill down, CloudWatch ingestion levers included. The Terraform post is a list of infrastructure-as-code mistakes paid for in pages and money. The other three are the opening chapters of the GPUs in production series — the dozen layers under a GPU pod, eight GPUs and the wires between them, and what changes the moment a model no longer fits inside one box.
Blog posts
Past one node the network becomes the machine. InfiniBand vs RoCE, gang scheduling, FSDP and Megatron, and why a 16k-GPU cluster fails every three hours.
Eight GPUs in one server behave like a small network. NVLink vs PCIe, reading nvidia-smi topo -m, NCCL transports, the ACS trap, and fitting a 70B model.
A GPU pod sits on a dozen layers from silicon to scheduler, and each one fails its own way. Drivers, the container toolkit, MIG, DCGM, and the metrics…
Our AWS bill hit $50k/month. Here's exactly how EC2 rightsizing, Reserved Instances, S3 lifecycle rules, and CloudWatch retention cut it to $20k in six…
Terraform mistakes I made so you don't have to: hardcoded AMIs, lost local state, a 2000-line main.tf, an unpinned provider, and a stray terraform destroy.