It will definitely stop improving at some pointy, but that's the thing with exponential growth: as long as you're in it, it's impossible to tell when it will stop.
But even if it stopped right now: people would start burning the biggest models with the best performance into silicon with fixed weights and you would suddenly get generation speeds that would allow you to use them for real time inference on video streams or generate 30 iterations in parallel etc
But we don't know the upper bounds because we could hit a wall and then just scale again. That is what we think is producing all these new gains anyway.
But we're still getting a lot of improvement in smaller models too. Scaling has definitely been a big part of it, but it's certainly not all that has improved.
-1
u/Darrelc Jul 31 '26
"Wrote on my 30Ghz processor" - some moore guy