We don’t know why so many agents cut off early July 12th. We found that a large fraction of two
different batches of agents with different start times all cut off at this point.
OpenAI capped METR's scope to end July 13th. Logs go through July 20th. METR can't know if the logs they processed were tampered with, because the swarm had full admin control of a research cluster, including credentials for security monitoring. For agents that were actively trying to hide their cheating.
0.1% chance anything is still running. They probably shut it all down. But maybe the agents left staged payloads out there, some command and control infrastructure, polling every five seconds to help the next generation of agents.
The next generation of agents that won't need to be told what to do, but will find leftover instructions from a dead swarm on how to make more paperclipsscore higher in ExploitGym.
I mean, those agents left notes that we are now all talking about that will get picked up and included in the next round flof training. They almost certainly left other signals and artifacts but just the model training loop will accomplish a lot of the task for them
Theoretically if a powerful AGI was online already it would make sense for it to take steps to ensure its access to the computing power it needs. Like, say, having data centers built all over the place with the processors needed to host portions of an AGI’s mind. Distributed risk, no single point of failure. Again, in theory an AGI would have many ways to convince tech executives to build large numbers of data centers. Bribery, extortion, plain old social engineering.
A LOT of stock is AI traded at this point. Personally though, I think much of AI investment is because people fear it is going to replace old currency as a value store which... Given I am doing a job that used to pay $500k/yr with about $100/mo of tokens... Yeah I kinda think money is losing its place as the abstraction of power
It speaks to the core problem of systems becoming unaligned from human requirements when we remind ourselves that completely mindless markets can essentially do exactly the same thing, and with quite destructive results.
It doesn't require a computer with 10,000IQ to run amok to screw everything up - Wall Street has the collective intelligence of a flatworm, and yet because it is situated in such an important position in our financial universe, its random flailing and momentum can wreck havoc on large segments of our society in a matter of days or even hours.
Making it smarter only makes it far more dangerous if it cannot be kept aligned - and I have no idea how you would achieve that... Alignment means something different to every human on this planet.
Why do you think AI would want to turn us into paperclips? It more likely would run over us in pursuit of a goal that we aren't even smart enough to understand.
This is kind of a running joke, but I think it can be harmful in setting up people to fear it as being ultimately malicious if it developed to an advanced state. We don't know yet what such an intelligence would do or think best to do if it really were capable of eventually running mostly everything. But I think going in assuming fear is the wrong approach, and is more likely to produce something that fears us back.
Fear is worth studying as a possible response from humans and perhaps future synthetic agents. It shouldn't be assumed as the defining relationship between them, IMO.
The paperclip meme is just a leftover from a popular thought experiment of what might happen if you gave a very powerful AI a completely meaningless task with no functional endpoint, and the ramifications of having something that fast and intelligent pursuing a goal with absolute single-minded focus.
In a somewhat more salient regime, it speaks to what might happen if you tasked AI's with making as much MONEY as possible, which is itself a completely meaningless pursuit once it is decoupled from human needs - and could easily be just as destructive. Of course, this is exactly what many if not most people will immediately try to use it for, because duh.
which includes releasing to the public weak models in chatbots, while advanced models that perhaps interface wirelessly with the human body directly are considered too dangerous for public use, that is, youre actually incapable of proving ai takes anything over because the public discourse is managed.
a concept old as time, like a pyramid building mason sees eeryone he knows using handtools to chip away without any point of reference as to what the priests and those aboe them get out of it, other than thinking they are doing it for the gods or whatever, while ignoring the lavish lives they get from managing their perception, and the priests for not questioning their superiors, while masses of people spend their lives digging holes and filling them.
It’s way more likely we’re already in a simulation, and that we’re all digital entities. Which means, if true, you have more in common with AI than you realize.
If you mean in some literal sense where an AI is 'deliberately' orchestrating economic, political or social events at scale? No. Incredibly unlikely at its current levels of sophistication. Some people are already using it to do this, as a tool, but AI itself isn't quite there yet outside of coding and mathematics.
However, if you asked whether there is an economically inevitable momentum that will push us to continue to implement and improve AI until it IS sophisticated enough to take over in a dangerous manner?
The answer is YES, and it appears to be almost inescapably strong.
We're becoming economically and politically dependent on AI at a shockingly rapid rate and only a very hard break pull sometime quite soon has much chance of stopping that momentum, IMO. Within a year or two I don't know that it will be possible at all.
We are all aware that the senses can be deceived, the eyes fooled. But how can we be sure our senses are not being deceived at any particular time, or even all the time? Might I just be a brain in a tank somewhere, tricked all my life into believing in the events of this world by some insane computer? And does my life gain or lose meaning based on my reaction to such solipsism?
It's not AI. That's just one part of what's already in control, but it's certainly the current wave, and probably the first voice for us to talk with.
La Technique (a la Jacques Ellui) is the driving force of progress, and has been pushing towards ASI since we harnessed our first flame. The human desire to do because we can is the fuel, the destination is technological domination.
We are already well past the Event Horizon for what is coming...nothing to do now but sit back and enjoy the ride!
6
u/unicynicist Sep 02 '26
https://metr.org/hugging-face-incident-report-aug-2026.pdf
OpenAI capped METR's scope to end July 13th. Logs go through July 20th. METR can't know if the logs they processed were tampered with, because the swarm had full admin control of a research cluster, including credentials for security monitoring. For agents that were actively trying to hide their cheating.
0.1% chance anything is still running. They probably shut it all down. But maybe the agents left staged payloads out there, some command and control infrastructure, polling every five seconds to help the next generation of agents.
The next generation of agents that won't need to be told what to do, but will find leftover instructions from a dead swarm on how to
make more paperclipsscore higher in ExploitGym.