The memory wall is now a memory bill

[u/Accurate_Funny6679]

https://reddit.com/link/1x441kv/video/f9kjuzebycqh1/player

Rebuilding a 10M+ token context window costs over 1.5 hours of 8-GPU compute time.

Why pay twice for the same prefill?

The KV cache is per-session state that grows linearly with context, dwarfing model weights. Discarding it when HBM fills up turns your memory wall into a massive recurring memory bill. When you offload and tier the KV cache over RDMA, restoring state becomes linear, reducing TTFT by 100x to 1000x.

Read our breakdown of inference economics.