Three hands holding Molly Moon's ice creams: a waffle cone, a cup and a caramel-coloured scoop in a waffle bowl

In May I headed out to MLSys 2026 in Bellevue, Washington, with my Fractile colleagues Carlo and Stelios.

There were more papers and talks than I care to list here, but it was good to validate the direction things had been going in. The pace of innovation is clearly outstripping the lead times of conference reviews: a lot of what was presented has already been built on, or landed in production frameworks, since it was submitted.

It was interesting to see how prevalent agent hill-climbing for kernel optimisation was. By my rough count, about a third of the papers touched on agentic optimisation of kernels or mappings. My time in the hand-written kernel mines in years gone by is clearly an artifact of a bygone era.

A few that stood out, from my rough notes taken during the talks:

Keynotes

  • Beyond Model Serving: Cross-Stack Co-design for Agentic Systems (Esha Choukse, Microsoft Azure Research). We still treat agentic workloads as tools wrapped around model calls, and so optimise the wrong thing. She argued against three illusions: that the model is the system, that more compute solves accuracy, and that correctness is binary. Case studies were Sherlock and StreamWise.
  • Rethinking Pretraining: Data and Architecture (Luke Zettlemoyer). A meaty one on data efficiency, controllable test-time compute, getting rid of the tokeniser, and modular or opt-out training.
  • LMCache (Yuhan Liu). “KV-cache is the new AI-native data.” LMCache runs as a separate service that vLLM and SGLang connect to, decoupling KV-cache management from the inference engine. See also CacheGen and CacheBlend.

Inference and serving

  • BLASST: training-free block-sparse attention that reuses FlashAttention’s online softmax statistics to skip negligible blocks. Roughly 1.6x prefill and 1.5x decode speedups at ~74% sparsity, and it has already landed in TensorRT-LLM.
  • SparseSpec: self-speculative decoding where the draft is the same model with sparse attention, aimed at long chain-of-thought reasoning where decode is memory-bound. Up to 2.13x throughput, lossless.
  • Demystifying the Mixture of Experts Serving Tax: MoE models run 2–3x worse latency than FLOP-equivalent dense models. Prefill suffers from load imbalance; decode from expert activations.
  • Event Tensor: a compiler abstraction for dynamic megakernels, encoding tile-level dependencies so that shape- and data-dependent dynamism can live inside one persistent kernel.
  • Breaking the Ice: a breakdown of vLLM cold-start latency into six stages, from framework bootstrap to CUDA graph capture, with a predictor for each.
  • Optimizing Deployment Configurations for LLM Inference: Meta’s retrospective on serving Llama 3. By their own admission nothing surprising, but large-scale validation of what the community already suspected.

Tooling and hardware design

  • XProf: Google’s ML accelerator profiler, with the full story of how it integrates with frameworks, the compiler and the PJRT runtime.
  • ExecuTorch: Meta’s on-device runtime, lowering from torch.export through composable backend delegates. The speaker opened with a live demo on his Ray-Ban Meta glasses.
  • VeriMoA: a training-free mixture-of-agents for generating HDL from a spec, using C++ and Python as intermediate languages and a quality-ranked cache across layers.
  • RocketPPA: one fine-tuned Llama 3.1 model predicting power, performance and area straight from RTL, skipping synthesis for early design-space exploration.

Seattle

We got an evening across the water in Seattle, checked oot the original Starbucks at Pike Place Market.

The green shopfront of the original Starbucks at 1912 Pike Place, with the old brown logo in the window
The original Starbucks, 1912 Pike Place.
A lit sign reading CALL (800) DUI-AWAY beside the Bradley Johnson Lawyers building on Stewart St
Stewart St, Seattle..

Afterwards, Carlo and I swung by San Francisco to meet a variety of interesting folk.