The Inference Engineering Masterclass

The inference engineering masterclass is an in-depth conversation exploring how artificial intelligence is transforming after a model has finished training. Hosted on a podcast with guests Philip Kiely and Ali Taha from Baseten, the discussion covers the complex systems required to turn raw model weights into fast, reliable, and affordable production-ready APIs at scale.

The piece highlights inference engineering as a critical and distinct discipline within AI. Rather than focusing on training, inference engineering tackles optimization problems like speed, reliability, and cost. Key technical topics include cache-aware routing to reuse precomputed inputs, disaggregated prefill and decode where separate sets of GPUs handle different stages of generation, speculative decoding using smaller draft models to accelerate larger ones, and quantization errors canceling each other out to improve throughput without sacrificing quality. The conversation also touches on tool calling, structured outputs, model parallelism, GPU kernels, and the enormous hardware demands of long-form video generation and large models like Kimi K3.

This matters because efficient inference directly determines whether AI products are economically viable and responsive for end users. As models grow larger and context windows expand to hundreds of thousands of tokens, sophisticated infrastructure and specialized engineering are essential to reduce latency, manage compute barriers, and make AI scalable in real-world applications.