Scaling Inference: Latency, Throughput, Cost