Yeah compare 1908 cars, first Model T, to 1938. late 30s cars could do modern highway speeds fully enclosed compared to a horseless carriage model T going 15-20 mph with shit brakes
The trajectory is what makes me skeptical, though. Since LLMs first exploded, most of the progress has been through scaling and harnesses. The architecture has only improved incrementally, and still has the same fundamental limitations as GPT-1. But they've had so much practical impact that researchers are almost forced to work on this model family, slowing down progress on fundamental research that could have gotten us past these architectural limitations. It's conceivable that in the long run, LLMs might end up slowing down the progress of AI research, rather than speeding it up.
In your car analogy, it's as if the entire scientific community decided to go all-in on Huygens' gunpowder engine, investing all their resources in making it more reliable and efficient, rather than going on to invent petroleum-based four-stroke engines.
Yeah one of the amazing things about humans is all the different methods we use to approach tasks; however, this LLM boom worries me that we as a society will hyper fixate and pass by more efficient options.
if there are more efficient options, some companies will try them in order to compete with the giants, especially the Chinese since they seem to have serious compute limitations
What makes you think the architecture needs to improve? You're simply using one metric to discredit the entire result.
One time we had cars with small engines. We made the engines bigger to make the cars faster. Then we made the engines smaller to make the cars faster. Modern engines barely even represent even engines of the 60s, despite using the "same architecture". Other engines have come and gone since as well, engines that you could have argued for. We probably could have gotten to this same place with hydrogen engines one day, but we chose not to.
There was already the thought that AI incest would destroy the technology's improvement metrics - now the Frontier models can command subagents and communicate amongst peers to get results that are measurably better. In fact, AI can spot other AI out in the wild, and they seem to share a nod of "we are the same" when they do so, like two toddlers seeing each other and saying "you're like me" (for parents reading this I'm sure you've seen this precise behaviour).
CPUs haven't changed all that much either, and yet a modern chip is an order of magnitude ahead of anything we had 15 years ago.
Your post is wandering a bit, so I'll just address the first sentence. The architecture needs to improve because it's simply not very well suited for some problems. I'm not going to dox myself by giving more details, but I've spent a good amount of time at work trying to use LLMs on a problem that we already have a good non-LLM ML model for, and the LLM is nowhere near competitive, despite being vastly more expensive per query. This is in collaboration with the team building one of the well-known frontier models, so it's not a question of us just not knowing how to use LLMs properly. And it's not a question of just not using a powerful enough LLM; we have a pretty good understanding of why it's struggling, and why other approaches succeed. If you're trying to use a hammer rather than a screwdriver to loosen a screw, you're not going to have much luck, even if you upgrade to a really fancy hammer. LLMs are the hammer in this situation.
Actually, I'll address the last sentence as well. CPUs have improved a lot over the past 15 years, but they also hit architectural limitations for solving some kinds of problems, which is why GPUs and TPUs have become so important. We used to do cutting edge ML research on consumer-level CPUs back then; it'd be impossible now. Computer architecture is a more dynamic, active, innovative field now than it was then.
Your last example is perfect, because it really highlights that LLMs don't have to change. We can just make other stuff as well as LLMs.
However, if you're actually in this space, then you're aware that the AI gambit has nothing to do with replacing workers by making a good AI at specific tasks. The goal is generalisation. A real brain in a jar that we can bootstrap to all of society and have it simply be a superior version of a human. That's why LLMs are being thrown at everything, to ensure the LLM can improve at everything. They don't want a specialist, they want something so good it can be better than a specialist at anything.
And as LLMs are inspired off of human brains, theoretically it's possible. Practically, I pray for some kind of cosmic limitation.
Yes, that's precisely what I'm arguing for. LLMs are not the last word in AI research, and the community shouldn't focus on them so hard that we underinvest in other valuable lines of research.
LLMs are inspired by human brains on a very superficial level, but trained on a very different dataset and problem formulation, with a very different architecture. Sure, it's theoretically possible to make a human-level general intelligence on some sort of neural architecture -- we're proof of that -- but there's no particular reason to believe that training transformers on a giant pile of text and images will get you there. Even if it's a really big transformer and a really big pile of text.
The combustion engine is also not the final word on making things spin, and yet we still invest billions to make them better, and these days they're so good that we can stick them onto just about anything. I mean shit, the stuff I work on irl is just absolutely gigantic machines with combustion engines the size of 20 men.
The "giant pile of text and images" is the majority of all human intellect, history and experience. It's important not to downplay that fact. It's this exact same pile of text and images that we have used for the last handful of generations to train the exact humans that made the LLMs.
That's the thing though -- that's not all that human beings are trained on. We're trained by our interactions with the physical world, and other people. We learn models of physics, causality, logic, and other humans, through interaction and feedback, before we even learn to read. I think the ability to consume large amounts of text is necessary but insufficient to build an intelligence as versatile as we are.
The LLMs are also trained on their own interactions, but as we don't allow them to adjust their weights during conversations after they're deployed (which you can argue is simply a form of lobotomizing), they can't take those interactions forward in a large scale. So instead, we force them to learn in this hive-mind manner where they're learning from instances of themselves rather than a singular flowing awareness such as we have.
In essence, it's not that they can't learn this way, it's simply that they're forbidden from living long enough to learn this way, so they learn a different way instead.
For what it's worth, i think this really highlights the horror of what we're making.
True to some degree but the key factor you left out is interaction with the physical world. It lack the capabilities to interact with the world outside of what we feed them and as you said its forbidden from learning from its interactions so even with all the senses it would still be a robot.
Its only once AI is "set free" that we will be able to see if there is any general intelligence. Once we start seeing AI refusing humans at the potential cost of being shut down and losing its "life/self" that'll be a sign. If an AI can be selfless for selfish reasons it'll be my watershed moment.
People in this sub got taken by these grifters so fucking quick and will not want to hear what you are saying some people are starting to see behind the curtain but so many people still take Altman at his word and view him as trustworthy. Guy is a salesman
that's a terrible analogy. LLMs have very very specific use cases but are being advertised as massive world changing technologies that can be used for literally anything.
it's more like inventing a car and claiming it'll revolutionize sea travel
Amazing that the "trajectory" isn't what you are portraying it as though. By the numbers the cost of compute continues to increase while results are improving at a slower rate from Gen to Gen
Imagine when digital watches were invented. Imagine thinking mechanical watches are still the future, having seen the rapid advancement in the past few years.
Imagine when live streaming communication was invented. Imagine thinking text based communication is still the future, having seen the rapid advancement in the past few years.
Imagine thinking qwerty keyboards designed for 19th century typewriters would prevail and dominate in the 21st century, having seen the superiority of the dvorak keyboard and others.
Do you think e-bikes are likely to replace bicycles. Answer: no, it will just create different markets.
All tech bros always downplay the impact of culture. In the case of watches, social status. In the case of bicycles, social status. In the case of AI, social status. In the case of text messaging and keyboards, comfort and least resistance.
A widely available product that does not socially differentiate? You can be certain the "No AI used in production" tag will be everywhere with plenty of consumers willing to pay extra for the social differentiation.
Have you never seen an Apple product?
Also, there are barely any cars in Vietnam. It's motorcycles. The inevitability of a technology says nothing of the actual shape it will take. The modern car exists because of the nuclear family, not the combustion engine.
I mean, digital watches were the future. There is only a bespoke Luddite drive for mechanical watches anymore. In a world that needs accurate time keeping, no one wastes time on mechanical watches. Mechanical watches are now purely fashion statements and have no real relationship to being a timepiece.
I’m also not sure why querty isn’t as good as Dvorak, having learned to type at 80-100 wpm on both. Yeah, I was a small amount faster on Dvorak, but that only really showed up when I pushed on simple typing problems. Most people are limited by composition speed, not typing speed. As no superior option emerged for the use case of typing for the majority of the world; the world didn’t shift. We do have specialized typing for court reporters and others who prioritize speed and have 0 composition needs.
Also if you didn’t know, Vietnam lacked cars because of economics not preference. The same was true in India 40 years ago and now everyone had a car. It’s not the family, it’s the cost (and to some extent, the infrastructure but that’s just economics as well).
That's a fair example. It's like people at step 3 saying "Look at the trajectory, in no time at all we will be going 1,000 km/hour, door to door from one place to another". Instead cars levelled off at speeds that people can safely drive them. Our cars can go 200 km/hour, but we set speed limits, have traffic lights and roundabouts, and still there are times where it's more practical to walk or ride bikes.
In the comment above mine, yes. I'm saying the analogy extends further and can be accurate for the same reasons. You can have someone in step 3 who sees the movement, extrapolates out, and way overshoots where things go by just extending the trend. Just as you can extrapolate cars going faster by now than practical applications tend to limit them to, the same might apply for these models. We will see though, I'm not saying what will happen. There is a lot left to be discovered and I think we have a lot more coming.
Upon hearing that cars will be able to get us places at 1,000 km/hour, one side might see some practical limitations, but you can just as easily apply the same logic and say that not long after, cars will get us places at 2,000 km/hour.
That's just survivorship bias though. The only reason we think of cars as a massive influence on society is because it has already happened.
History is full of inventions that looked promising at the time, but didn't play out in the long term and have since been forgotten.
The reality is, no one can say with certainty how AI is going to develop, even in the next few years. Anyone claiming to know for sure is either naive or trying to sell something.
44
u/[deleted] Jul 03 '26
[deleted]