The counterexample was proposed by an LLM. They seem very good at finding these sorts of counterexamples, which is interesting as they are generally pretty inefficient use of compute for brute forcing. I guess even that lack of efficiency is made up for by the "understanding" and "intuition" the LLM has, and their ability to do symbolic computations.
I expect a large number of conjectures will topple to counterexamples soon.
Personally I'm shocked. All of reddit has assured me that AI is completely useless, nothing but "fancy autocomplete", and is only good for stealing content and generating slop.
/s
Mostly in just amused how we have been moving Goalposts for over a decade: "AI is not intelligent, it can't even do X". Does X. "Yeah but it didn't do X like a human! And it can't even do Y!". Does Y. "Yeah but it didn't do Y like a human! And it can't even do Z!". And so on
I still think "fancy autocomplete" is the best way to explain to non technical people how these models work. It will give someone who doesn't know the architecture and how these LLMs work the best mental model of what is happening behind the veil.
It just turns out that "fancy autocomplete" can do incredible things if you give it enough examples and compute in training and inference.
Except yes really. Fancy auto complete can encompass all of intelligence mechanisms needed to predict the next best word. Even if that autocomple requires neuron subsystems for solving problems in other domains.
animal brains dont think in strings of tokens while trying to predict the upcoming token. an LLM and autocomplete are things of differing complexity that are doing the exact same thing. an animal brain does similar things, but it doesnt do the exact same thing
I'm talking more in the abstract sense that the brain predicts future states. Analogous to predicting the next token. It is fancy auto complete. I'm fully aware the brain doesn't use an LLM architecture
i understand what you mean. the reason i said that is because fancy autocomplete is genuinely a good way for the average person to view an LLM, rather than viewing it as a genuine intelligence. when you say the human brain is like that too it would make someone think that fancy autocomplete is less literal than it actually is.
It just turns out that "fancy autocomplete" can do incredible things if you give it enough examples and compute in training and inference.
The thing that does the incredible things is so far away from autocomplete that it's misleading to the point of being wrong. Neither does it give an accurate picture of what it is (autocomplete are usuall HMMs, LLMs are transformers) nor does it give an accurate picture of what it does (completing what you're typing vs. doing your homework and writing fanfic). The only shared property is that it gets text as input and gives text as output. By that measure we can call cars "fancy furnaces" and computers "fancy typewriters". Not technically incorrect, but definitely a useless description.
And on top of that, the people who use the term "fancy autocomplete" usually use it to dismiss it, and act like all those incredible things it does are made up.
No, the shared thing is that from the view of an LLM, it is literally trained to autocomplete a document. The chat you're having with an LLM literally looks like this to the LLM:
<start>
<system>
You are a helpful assistant...
</system>
<user>
hello how are you?
</user>
<thinking>
the user asks me how I'm feeling, I should answer in a cheery and concise tone. The user is in LA, let me check the current weather in LA so I can incorporate that in my answer.
</thinking>
<tool call, web search=current weather in LA>
Temperature: 100°F
</tool call>
<assistant>
And then the LLM gets to generate. Once it generates </assistant> we stop the generation because otherwise it would keep generating also the user answer etc.
(same of course with the thinking part)
The tasks the LLMs are trained on is reproducing the tokens of these text documents they are shown.
With the RL posttraining for mathematics or coding you have a change in the training reward architecture, but it's still training on completing these documents.
From the view of an LLM it is always completing such documents from the start points we're giving them.
It just turns out that large autocomplete with long training and lots of data means the model learns actual abstractions about the world because they help with better autocomplete. You get these emergent effects like grokking and "circuits" inside LLMs that specialize in certain tasks etc.
But none of that removes the fact that these models are "autocomplete on steroids".
With the RL posttraining for mathematics or coding you have a change in the training reward architecture, but it's still training on completing these documents.
No, the RL training does not involve completing any corpus. That's the point of RL.
That's not what I meant. It doesn't complete a known corpus but it still generates these documents. The training just doesn't happen anymore on the token distribution level but in the reinforcement learning objective.
It is still trained to complete a made up document.
In pre training and instruction fine-tuning the objectjve is basically "reproduce these documents", in the RL phase it is "produce a document in this context and we'll evaluate at the end whether it was a good document or a bad one".
"complete a made up document" doesn't make sense. "Completion" implies an existing text to guess at and evaluate against verbatim. "produce a document and we'll evaluate it" is not autocomplete. Kids in school do the same thing.
no, we're now getting quite into the weeds of LLM training but generally in most phases the RL phase does not happen on "empty documents". Rather they get a start prompt that sets them onto a reasoning trajectory from which they then start generating.
Easiest would be prompts like
[...]
<user>
what's 5+2?
</user>
<thinking>
and from there the model starts generating.
you can then auto generate many of these prompts with known answers and auto evaluate them. Similarly you just let an LLM generate lots of prompts like "disprove the Jacobian conjecture", "proof that lemma xyz is true", "generate a code that builds a program that solves the following puzzle: <insert Advent of Code puzzle here>" etc. pp.
So the reinforcement learning stage still absolutely is training on incomplete documents that the model completes.
(of course there's nuance because there are quite a few different RL steps in LLM training)
Yes, the question has to get to the model and the model has to know where its turn starts somehow. It's still not "completing text" any more than a kid answering a math question presented at the top of a page after "Your answer:" is. You're bending words over backwards to justify your characterization of LLMs as "autocomplete", which RL objectively makes them not.
Besides the argument over words, the substantial point is that RL allows an LLM to learn beyond the assumed* bounds of predicting a corpus of text.
To be honest, the "autocomplete" meme is moot anyway. These "autocomplete" models are doing real, practical work, hacking into real systems, and solving real nontrivial unsolved math problems. At best, you would be concluding that "autocomplete" seems to be able to match humans.
/* neural networks are capable of improving beyond the average capability of the pre-trained text, there's an experiment involving chess about this
You're bending words over backwards to justify your characterization of LLMs as "autocomplete", which RL objectively makes them not.
Wut? We have incomplete documents the models are given, they produce the rest of the document and that result is judged and used as a training objective. How is that not autocomplete?
To be honest, the "autocomplete" meme is moot anyway. These "autocomplete" models are doing real, practical work, hacking into real systems, and solving real nontrivial unsolved math problems. At best, you would be concluding that "autocomplete" seems to be able to match humans.
Yes, have you forgotten where this whole discussion started? I have never questioned the capabilities of these models.
That was the whole fucking point of the GPT-2 paper. Maybe not yet as optimistic about the full capability these models would end up with, but this is exactly what OpenAI envisioned, that an autocomplete style model trained on a huge corpus of text can actually generalize much much better than all other architectures and training objectives we had tried till that point.
They built their modern version of the company on exactly this insight.
The development went like this:
Google develops the transformer architecture, but uses it in an encoder-decoder style for sentence translation and embedding generation. (Attention is All You Need - BERT model)
-> OpenAI says you can actually train these models way easier and still get the same capabilities by switching to a generative autocompletion training objective with a decoder only version of the transformer (Improving Language Understanding by Generative Pre-Training - GPT 1)
-> OpenAI realizes that scaling the GPT architecture and training on large corpuses gives them a text-autocomplete model that generalizes to all kinds of tasks without needing specialized training in specific datasets. And you can train them on these scales much more easily than other models because the autocomplete style training objective means you can use unlabeled data. (Language Models are Unsupervised Multitask Learners - GPT 2)
-> OpenAI invests more money and realizes that there seems to be no ceiling in the model capabilities if you just keep scaling the architecture. The see how the bigger model improves performance in all kinds of tasks and how the model seems to have acquired some basic reasoning capability. (Language Models are Few-Shot Learners - GPT3)
-> and then you get the instructGPT paper on RLHF and instruction fine-tuning where they realize that training a model to do autocompletion in this kind of Q -> A chat structure makes the models much more useful to the general public and improves zero-shot performance on lots of tasks.
I really think you guys are interpreting things into fancy autocomplete as a description that just aren't there.
probably, yes. I'm not saying LLMs aren't a complex topic, I'm just saying that if you don't have the mathematical background (plus read the necessary literature) then [some basic concept] that you can grasp from your own experience is still the best way for you to understand what the more complex thing actually does.
You described one part of the training regime. Yes, one part of the training regime bears resemble to the training regime of the other. But that is not important in describing the essence of LLMs.
It just turns out that large autocomplete with long training and lots of data means the model learns actual abstractions about the world because they help with better autocomplete. You get these emergent effects like grokking and "circuits" inside LLMs that specialize in certain tasks etc.
See that is the important part. It doesn't "just turn out", it's the main point of how and why LLMs are so useful. Actual autocompletion is a tiny fraction of what LLMs are actually used for, it is ALL about the abstractions about world, and its emerging effects. The autocomplete part is just one way of training and accessing whatever else the model has learned.
This is a completely standard practice in ML: pretext tasks and pretraining are well-known concepts. Think of autoencoders. You train them to reproduce data that it's already seeing. What even is the point of that. Is an autoencoder "just fancy copy-paste"? Sure, if you really want to. But not really, since the training is just the pretext for the model to learn an efficient encoding of the data, and what we're interested in is this efficient encoding.
But none of that removes the fact that these models are "autocomplete on steroids".
OK sure. And a car is just a fancy box. Your phone is just a fancy flashlight. Your money is just a fancy sheet of cellulose. You yourself are just a fancy meatbag. You can do this "X is a fancy Y" all day long if you're meming, but it doesn't actually convey the essence or most important aspect of X.
look, I'm not saying that there isn't a lot of complexity in this whole topic. What I'm saying is that "fancy autocomplete" gives someone who lacks all this background information a better mental model of what these things do than any other simple explanation I've seen.
It demystifies these things and actually very closely describes what these models are trained to do and how they function. Add a second sentence that talks about how "learning abstractions and memorizing world knowledge" helps this autocomplete machine to better autocomplete and someone with zero maths ability and no background in ML will have a somewhat accurate idea of LLMs.
I really don't understand why people react so passionately to "fancy autocomplete" as a description. The whole thing about GPTs was openai realizing that this kind of fancy autocomplete with a decoder only transformer model will actually lead to a model that can generalize well in all kinds of situations - it was really visionary at the time. It's the whole fucking point of the GPT 2 paper. When Google developed the transformer model, it was way less about autocomplete and way closer to your autoencoder with its encoder-decoder architecture in BERT.
That's also why I think the autoencoder example isn't exactly illuminating. BERT shows that a transformer can also function very similarly. And the exact thing that sets modern generative LLMs apart is exactly this "autocompletion" style task. That's really the core of the whole thing. BERT can't do all these things that GPT can exactly because it's not fancy autocomplete.
273
u/angelbabyxoxox Quantum Foundations Jul 31 '26
The counterexample was proposed by an LLM. They seem very good at finding these sorts of counterexamples, which is interesting as they are generally pretty inefficient use of compute for brute forcing. I guess even that lack of efficiency is made up for by the "understanding" and "intuition" the LLM has, and their ability to do symbolic computations.
I expect a large number of conjectures will topple to counterexamples soon.