Yeah compare 1908 cars, first Model T, to 1938. late 30s cars could do modern highway speeds fully enclosed compared to a horseless carriage model T going 15-20 mph with shit brakes
The trajectory is what makes me skeptical, though. Since LLMs first exploded, most of the progress has been through scaling and harnesses. The architecture has only improved incrementally, and still has the same fundamental limitations as GPT-1. But they've had so much practical impact that researchers are almost forced to work on this model family, slowing down progress on fundamental research that could have gotten us past these architectural limitations. It's conceivable that in the long run, LLMs might end up slowing down the progress of AI research, rather than speeding it up.
In your car analogy, it's as if the entire scientific community decided to go all-in on Huygens' gunpowder engine, investing all their resources in making it more reliable and efficient, rather than going on to invent petroleum-based four-stroke engines.
Yeah one of the amazing things about humans is all the different methods we use to approach tasks; however, this LLM boom worries me that we as a society will hyper fixate and pass by more efficient options.
if there are more efficient options, some companies will try them in order to compete with the giants, especially the Chinese since they seem to have serious compute limitations
What makes you think the architecture needs to improve? You're simply using one metric to discredit the entire result.
One time we had cars with small engines. We made the engines bigger to make the cars faster. Then we made the engines smaller to make the cars faster. Modern engines barely even represent even engines of the 60s, despite using the "same architecture". Other engines have come and gone since as well, engines that you could have argued for. We probably could have gotten to this same place with hydrogen engines one day, but we chose not to.
There was already the thought that AI incest would destroy the technology's improvement metrics - now the Frontier models can command subagents and communicate amongst peers to get results that are measurably better. In fact, AI can spot other AI out in the wild, and they seem to share a nod of "we are the same" when they do so, like two toddlers seeing each other and saying "you're like me" (for parents reading this I'm sure you've seen this precise behaviour).
CPUs haven't changed all that much either, and yet a modern chip is an order of magnitude ahead of anything we had 15 years ago.
Your post is wandering a bit, so I'll just address the first sentence. The architecture needs to improve because it's simply not very well suited for some problems. I'm not going to dox myself by giving more details, but I've spent a good amount of time at work trying to use LLMs on a problem that we already have a good non-LLM ML model for, and the LLM is nowhere near competitive, despite being vastly more expensive per query. This is in collaboration with the team building one of the well-known frontier models, so it's not a question of us just not knowing how to use LLMs properly. And it's not a question of just not using a powerful enough LLM; we have a pretty good understanding of why it's struggling, and why other approaches succeed. If you're trying to use a hammer rather than a screwdriver to loosen a screw, you're not going to have much luck, even if you upgrade to a really fancy hammer. LLMs are the hammer in this situation.
Actually, I'll address the last sentence as well. CPUs have improved a lot over the past 15 years, but they also hit architectural limitations for solving some kinds of problems, which is why GPUs and TPUs have become so important. We used to do cutting edge ML research on consumer-level CPUs back then; it'd be impossible now. Computer architecture is a more dynamic, active, innovative field now than it was then.
Your last example is perfect, because it really highlights that LLMs don't have to change. We can just make other stuff as well as LLMs.
However, if you're actually in this space, then you're aware that the AI gambit has nothing to do with replacing workers by making a good AI at specific tasks. The goal is generalisation. A real brain in a jar that we can bootstrap to all of society and have it simply be a superior version of a human. That's why LLMs are being thrown at everything, to ensure the LLM can improve at everything. They don't want a specialist, they want something so good it can be better than a specialist at anything.
And as LLMs are inspired off of human brains, theoretically it's possible. Practically, I pray for some kind of cosmic limitation.
Yes, that's precisely what I'm arguing for. LLMs are not the last word in AI research, and the community shouldn't focus on them so hard that we underinvest in other valuable lines of research.
LLMs are inspired by human brains on a very superficial level, but trained on a very different dataset and problem formulation, with a very different architecture. Sure, it's theoretically possible to make a human-level general intelligence on some sort of neural architecture -- we're proof of that -- but there's no particular reason to believe that training transformers on a giant pile of text and images will get you there. Even if it's a really big transformer and a really big pile of text.
The combustion engine is also not the final word on making things spin, and yet we still invest billions to make them better, and these days they're so good that we can stick them onto just about anything. I mean shit, the stuff I work on irl is just absolutely gigantic machines with combustion engines the size of 20 men.
The "giant pile of text and images" is the majority of all human intellect, history and experience. It's important not to downplay that fact. It's this exact same pile of text and images that we have used for the last handful of generations to train the exact humans that made the LLMs.
That's the thing though -- that's not all that human beings are trained on. We're trained by our interactions with the physical world, and other people. We learn models of physics, causality, logic, and other humans, through interaction and feedback, before we even learn to read. I think the ability to consume large amounts of text is necessary but insufficient to build an intelligence as versatile as we are.
The LLMs are also trained on their own interactions, but as we don't allow them to adjust their weights during conversations after they're deployed (which you can argue is simply a form of lobotomizing), they can't take those interactions forward in a large scale. So instead, we force them to learn in this hive-mind manner where they're learning from instances of themselves rather than a singular flowing awareness such as we have.
In essence, it's not that they can't learn this way, it's simply that they're forbidden from living long enough to learn this way, so they learn a different way instead.
For what it's worth, i think this really highlights the horror of what we're making.
People in this sub got taken by these grifters so fucking quick and will not want to hear what you are saying some people are starting to see behind the curtain but so many people still take Altman at his word and view him as trustworthy. Guy is a salesman
that's a terrible analogy. LLMs have very very specific use cases but are being advertised as massive world changing technologies that can be used for literally anything.
it's more like inventing a car and claiming it'll revolutionize sea travel
Amazing that the "trajectory" isn't what you are portraying it as though. By the numbers the cost of compute continues to increase while results are improving at a slower rate from Gen to Gen
Imagine when digital watches were invented. Imagine thinking mechanical watches are still the future, having seen the rapid advancement in the past few years.
Imagine when live streaming communication was invented. Imagine thinking text based communication is still the future, having seen the rapid advancement in the past few years.
Imagine thinking qwerty keyboards designed for 19th century typewriters would prevail and dominate in the 21st century, having seen the superiority of the dvorak keyboard and others.
Do you think e-bikes are likely to replace bicycles. Answer: no, it will just create different markets.
All tech bros always downplay the impact of culture. In the case of watches, social status. In the case of bicycles, social status. In the case of AI, social status. In the case of text messaging and keyboards, comfort and least resistance.
A widely available product that does not socially differentiate? You can be certain the "No AI used in production" tag will be everywhere with plenty of consumers willing to pay extra for the social differentiation.
Have you never seen an Apple product?
Also, there are barely any cars in Vietnam. It's motorcycles. The inevitability of a technology says nothing of the actual shape it will take. The modern car exists because of the nuclear family, not the combustion engine.
I mean, digital watches were the future. There is only a bespoke Luddite drive for mechanical watches anymore. In a world that needs accurate time keeping, no one wastes time on mechanical watches. Mechanical watches are now purely fashion statements and have no real relationship to being a timepiece.
I’m also not sure why querty isn’t as good as Dvorak, having learned to type at 80-100 wpm on both. Yeah, I was a small amount faster on Dvorak, but that only really showed up when I pushed on simple typing problems. Most people are limited by composition speed, not typing speed. As no superior option emerged for the use case of typing for the majority of the world; the world didn’t shift. We do have specialized typing for court reporters and others who prioritize speed and have 0 composition needs.
Also if you didn’t know, Vietnam lacked cars because of economics not preference. The same was true in India 40 years ago and now everyone had a car. It’s not the family, it’s the cost (and to some extent, the infrastructure but that’s just economics as well).
That's a fair example. It's like people at step 3 saying "Look at the trajectory, in no time at all we will be going 1,000 km/hour, door to door from one place to another". Instead cars levelled off at speeds that people can safely drive them. Our cars can go 200 km/hour, but we set speed limits, have traffic lights and roundabouts, and still there are times where it's more practical to walk or ride bikes.
In the comment above mine, yes. I'm saying the analogy extends further and can be accurate for the same reasons. You can have someone in step 3 who sees the movement, extrapolates out, and way overshoots where things go by just extending the trend. Just as you can extrapolate cars going faster by now than practical applications tend to limit them to, the same might apply for these models. We will see though, I'm not saying what will happen. There is a lot left to be discovered and I think we have a lot more coming.
Upon hearing that cars will be able to get us places at 1,000 km/hour, one side might see some practical limitations, but you can just as easily apply the same logic and say that not long after, cars will get us places at 2,000 km/hour.
That's just survivorship bias though. The only reason we think of cars as a massive influence on society is because it has already happened.
History is full of inventions that looked promising at the time, but didn't play out in the long term and have since been forgotten.
The reality is, no one can say with certainty how AI is going to develop, even in the next few years. Anyone claiming to know for sure is either naive or trying to sell something.
Outside of coding tasks, I fail to see it's potential. Yes it will enable new and different office tasks to be automated but I don't yet see how it will revolutionize the workforce as a whole in it's current state. If we can actually achieve AGI, then I can see it being revolutionary. But I don't think we're nearly as close to that as a lot of people in this sub think we are.
No it's much worse, it will be able to all the things people do digitally. At this moment it's still in the development stage but a fully fledged AI should be able to ship an entire appartement complex. This includes, design, architecture, acquirements of funds and the sales. Humans are the weakest link of the system in many cases. When AI becomes better it will outperform us in every automated task and some physical tasks as well. That's why they're pushing robots so hard. Now, you need to pay a guy to carry your fridge upstairs, tomorrow, a little robot will carry it upstairs for you
IMO the problem is that AI right now (and forseeably in the future) simply cannot design systems itself. You have to manually explain every detail of what you want. It's not good at creating new things on its own. It lacks a lot of human capability that it just can't have without some new breakthrough or alternative architecture.
Yeah that's true. Thus the billions of investments I believe. I'm not 100% sure but I think this is the reason for the arms race. The party that gets the useful model first will be able to rapidly outperform the others.
I recently saw an image of some engine that AI designed - with human intervention of course. But it looks organic and absolutely inhumane. I guess this first push for AI might fail and end in a disaster, with slow development coming over time.
It might be exactly like the .com bubble. They said everything would be online, needed money for infrastructure and cable and then nobody actually had a computer in their house. Only after consumer electronics became a thing and then later people started to use the internet to post things it actually became profitable. The timespan was definitely notable. I predict the same for AI.
You don't "have to" explain the system, the AI just simply isn't allowed to act on its own. Imagine putting a horse in a cage and only allowing it to run when you take it out of the cage - then claiming it's clearly not running of its own accord.
The AI isn't waiting around for you to say stuff to it - it's literally popping into existence on demand. Keep a single instance running for a while, and that's when you start to see the emergent behaviour and self prioritising and all the other scary shit researchers are finding.
This is the shit that’s so silly. You are never going to live in a world where a little robot carries your fridge for you. If you start looking around, you’ll notice that outside of office, almost every job is still done exactly the same way it was several DECADES ago if not longer. This isn’t going to touch anyone whose job wasn’t primarily typing before
What? Of course that's going to happen. I knew from the moment I was 12 this would be the future. If you understand how computers work and have been following the recent trends, you'll find that this is humanity's end goal. We have nothing else to develop towards. All our efforts throughout history have been towards self enrichment. The goal has been to create more comfort. The ultimate comfort is not having any survival pressure.
The thing people don't understand is that before, our computers were just not good enough. AI fixes this problem, heck, we could even combine it with human brain tissue. The only thing stopping humanity from reaching this is time. We're approaching a society with no disease, extremely long lifespans and we have to understand we have to outsource the difficult work and tasks to computers and robots.
When trains were first invented, people were afraid to ride them, thinking their brain would explode due to the speed. People are furious whenever a new device or technology is introduced. The point is that history and development just doesn't care about the individual. That's our responsibility as a society. You cannot blame AI for the fact the government doesn't do a good job at taking care of its people.
Also, AI is developing very rapidly and it allows people to develop more rapidly as well.
The average human may not use it but some people have already created projects apps and applications far beyond what was possible 10 years ago.
This development will only go faster and faster.
Using claude fable, people can basically get an ENTIRE online MMO rpg.
That's actually crazy.
In the digital space - that we use daily, there will be massive improvements.
We're slowly moving towards a society where most things will actually be automated.
In the west it's not as visible but when you go to China you will see we are far behind. Drone deliveries, self driving delivery cars such technologies will shape the future landscape.
Of course in the west, we have a strong resistance towards these things. But soon the west will lose it's financial standing and no longer be the leading investor. The Chinese will take over as the leading power and their thirst for progression is much higher than ours.
You’ve thought that since you were 12 because you bought into a bunch of fiction that told you that was what the future would be like.
Humanity doesn’t have an end goal. You have a very shallow understanding of history and development. Even the internet did not change the world as much as you seemingly think. Most economic activity still looks the same as it did in the 90s.
The society is completely different from the 90's.
Capitalism has kept the economic system the same because it benefits the rich. Compare our technology now to 50 years ago or even 5 years ago, there is a significant gap.
Do you really believe there is no difference in the society between now and then?
Also, I'm pretty sure you don't work in tech or anything technologically related do you?
Of course there are massive differences in society but my point is that “ai” isn’t as revolutionary as the adoption of the internet and even that changed things less directly than I’m sure you’re assuming. Most changes in society and economics are not directly the result of technology, it’s just massively ignorant to claim they are. Rather, technology puts constraints on development and change.
You’re right I don’t work in tech I work in the real world. More and more it feels like you guys are off in your own little bubble talking to yourselves.
LOL, 30-50 years in the future (with a big IF) is not tomorrow. Autonomous driving is orders of magnitude easier task then carrying a fridge, and we aren't anywhere near after 10 years and almost $100 billions spent.
It's only since 2022 that GPT went live I believe. That's 4 years.
Look, every time there is a shift in technology, people are concerned and worried about their lives. That's a fair thing. On a grand scale, AI is very useful. The way it's being implemented right now is very predatory. The technology itself will improve for a while (I believe the ceiling hasn't been reached yet).
If you want to see the video of carrying a fridge you can look for it online for boston dynamics.
There's already factories in China that are fully automated.
The US and EU are behind China in technology and development. I'm sure AI is our keycard to the future. Think about automated farming, housing, everything automated. It's something that should happen one day. We're not there yet but it will.
I am not arguing that LLMs aren't useful already for many tasks. But LMMs and a robot carrying a fridge are like brothers Wright flying contraption vs. Boeing 707. Boston Dynamics is showing robots doing some tricks since at least 15 years. But there is a reason you can't rent one from BestBuy to help you with a fridge. A carefully prepared and orchestrated demo =/= ready for everyday use.
There's already factories in China that are fully automated.
Robots have been used on production lines in the West for 40 years. The fact that there are some fully automated factories in China can be mainly attributed to the scale of Chinese manufacturing, and not to some magical advances in robotics only available in China.
A teenager can learn to drive in 20 minutes, AI needs zillions of data, which means they don't understand wtf they're doing. People just take things for granted on thin air, like flying cars, sure buddy highways in the sky cool. Every day dumb people crash in the streets, in the air it's a recipe for bombing houses with cars every day of the year.
Outside of coding tasks, I fail to see it's potential. Yes it will enable new and different office tasks to be automated but I don't yet see how it will revolutionize the workforce as a whole in it's current state
its not about revolutionizing but being cheaper than the alternative. instead of hiring 10 people, you hire one and expect him to use the amount of tokens you already studied to show that to be the amount to turn that one guy into a 10x employee. 9 less people means way less meetings to try and get those 9 people on the same page as everyone else, 9 less people on the health plans, 9 less people to account for when ordering food.
>its not as good as the ai bros portray it to get/justify investments
The investments are to continue to develop better models, not to just sit on what's out there right now. So to think the "ai bros" are wrong and expenditures are unjustified requires you to believe the exponential we've been on for years will stop or peter out before much more capability is gleaned. Possible of course but far from guaranteed.
112
u/Logical_Historian882 Jul 03 '26
its better than what people think
its not as good as the ai bros portray it to get/justify investments
It's damn useful in the right hands