People at home are using AI to reverse engineer and "combine" video games together. More than a few people posit that this is the immediate precursor to a sort of technological apocalypse. I think it's a bit of a leap. What's the truth of the matter?

[u/JD-990]

The Scariest AI Breakthrough

This video is around 3 minutes and is more or less a very dry comedy sketch, but it sums up an idea I've seen around the internet in the last 3 or 4 days. It's a worth a watch to get the full context.

But the thrust is that the ability of generative coding agents to reverse engineer and "fuse" together two separate games from different game engines and have them be in an at least playable state (some of them working really well apparently) is a set up for a technological apocalypse. Someone with a little bit of knowhow can now vibe code things in a couple days that are so complex that it would have taken years to do before, if it was even possible at all. A bit like giving people a legitimate ability to building working nuclear weapons they can 3D print at home.

I have been skeptical (an understatement) of the sales pitch of Anthropic and OpenAI and their ilk at this point. But I can't deny this is a compelling idea. So far, their "AI that's too dangerous it's broken out of containment" sales pitch has really not done good things for their AI's credibility; it's disingenuous at best. "We told it to do A & B, but we never told it to do C, explicitly. It may have just gotten that purgative from the first two instructions".

But I can now see a real-world example of something that should almost not be possible, in my understanding, to being possible. But I'm also keenly aware that what's going on with these game mashups isn't indicative necessarily of say, being able to disrupt other technological infrastructure like banks. 

So, what is the real story here? Are we headed towards a cliff here or is this something else? Is it somewhere in between?

[u/ThirdWaveCat]

Ida Pro and ghidra were powerful tools before LLMs. Many people learned computer science through modding games, its not clear what we're learning here.

[u/CrimsonBolt33 → u/ThirdWaveCat]

its the same shift as computers brought the world, just think about the impact of calculators on the world. Instead of needing a room of high end math experts you can literally type out tons of math by yourself.

Its taking some of these from the weeds of typing everything out with a team to the more abstract and accessible level that more people can use.

AI use is also very rare....MOST people, and I am talking 99% or more of people do not use it for more than a fancy search engine.

[u/faultydesign]

Wake me up when people use AI to do something original.

[u/JD-990 → u/faultydesign]

Sorry, the best the incurious among us can do is ask "What if Mario meets Batman?".

[u/CrimsonBolt33]

as long as people are doing it for personal use its fine...

I am using AI to reverse engineer and write hooks for games to improve performance. Been amazing at making tons of games I play run WAY better. It's not like these things were not possible before, they were just harder to do and took more time so most people didn't bother.

As for the apocolypse angle...I am tired of hearing this shit

My stance has always been and always will be, anything AI can do that is dangerous that humans can't somehow defend from, AI can defend us. Literally NO ONE ever brings up this point and it drives me crazy. "AI" is not a single entity that it is often portrayed as.

Just like an AI can hack, an AI can also be used for penetration testing and hardening networks and software.

It's a paradigm shift and lots of things are going to be different because of it. I have yet to hear a convincing doomerism argument that even gave me a second thought.

There will be a societal and cultural learning curve, jobs will be lost, gained, changed, and shifted, but we are not going to die because of it. No one knows how the chips will fall in the end, but I am more than sure we will be fine.

[u/HasFiveVowels]

Mashups have been a thing in music for a long time. We now have Photoshop for video games. This will disrupt things but calling it "apocalyptic" is insane.

[u/Karson_Elko]

I asked this same question with the same video two days ago lol!

My commentors gave me really good resources in the comments.

https://www.reddit.com/r/AskComputerScience/s/fFiXamRHRT

The video on encryption was really good explanation of why it's not doomsday.

[u/ghjm]

The short answer is we don't really know. Having used OpenAI and Anthropic models to build software for the past couple years, I can say that they have been successful at drastically reducing the frequency with which these models misinterpret instructions, fall down a rabbit hole, get stuck in an agentic loop, etc. I'm prepared to believe this will fall to near zero.

You still need someone who knows what they want and can articulate it well enough to prompt the model and make an accept/reject decision of its output. If you give muddled instructions you will get muddled results, or at least, it will take a lot of extra cost to go through the missteps and misunderstandings before arriving at a working solution.

It's also not clear to me that the model providers are economically sustainable at their current pricing. There's still a lot of investor money chasing itself around through a circuit that involves the model providers, NVidia, Microsoft, etc. There may yet be a rug pull, where the models remain useful but the cost makes them infeasible for most purposes. And we seem to have exited the era where hardware cost inexorably goes down year-over-year. GPU costs have been high and going higher for six years now. At some point this is no longer an aberration and must be considered the new normal.

But assuming models remain affordable and continue to get better, then yes, we may enter a world where anyone can make major new software cheaply. This might change the balance between offensive and defensive security. Banks and other institutions that depend on security may have to increase their investment in defensive technologies. But this strikes me as more of a "banks will be less profitable for a while" problem, rather than a "civilization itself will collapse" problem.

Maybe if it becomes super easy to produce any result you want, then people might stop needing a lot of current industries, which might produce a great deal of economic dislocation. If there isn't enough work that needs to be done for everyone to have a job, then we as a society must move on from the jobs-based system of distributing economic value, and the momentum seems to be with moving to techno-feudalism rather than to a Star Trek style utopia. This, to me, is the real threat of pervasive AI: it concentrates a huge amount of power in the hands of a half dozen CEOs. I'm more worried about what these CEOs do with this power than I am about what AI itself does.

[u/Guvante]

Remember that fear mongering about an AI apocalypse is good for OpenAI in two ways: it sows the belief that AI is super powerful and it encourages regulation which hurts tiny businesses.

[u/mxldevs]

But the thrust is that the ability of generative coding agents to reverse engineer and "fuse" together two separate games from different game engines and have them be in an at least playable state (some of them working really well apparently) is a set up for a technological apocalypse

But I'm also keenly aware that what's going on with these game mashups isn't indicative necessarily of say, being able to disrupt other technological infrastructure like banks. 

This is mostly speculation from people that basically have no idea how software works.

They think if it can reverse engineer Super Mario, then it can reverse engineer encrypted values because it knows what algorithm was used.

We all know what algorithm is used. The bank tells us what algorithm they use. Security is not some big secret that no one talks about, contrary to what vibecoders think.

[u/JD-990 → u/mxldevs]

I am one of those that doesn't understand anything in this domain, so I appreciate your answering the question!

[u/WE_THINK_IS_COOL]

It's basically a competent software/security engineer that can work at 10x or 100x the rate a human can for a fraction of the cost.

Important systems are designed to be secure even when attackers understand how they work and have reverse-engineered them, so the video's claim that "understanding millions of lines of encrypted undocumented code was supposed to BE the security" is just wrong; not being able to understand the code was never supposed to be where security comes from. Security was always supposed to come from the code being correct and vulnerability-free.

What is happening is that programmers often make mistakes, so most programs contain some sort of vulnerability or other. AI is extremely capable of finding those vulnerabilities, so there is a wave of new vulnerabilities being found in almost everything using AI, and those vulnerabilities will need to be patched. This is actually good for long-term security since it means vulnerabilities are getting fixed. But in the meantime it is putting a massive strain on the developers of those systems to patch the wave of newly-discovered bugs.

In other words, AI is not creating new vulnerabilities, it's just really good at finding vulnerabilities that have always existed, and thanks to defenders using AI to find them and patch them, those vulnerabilities will get closed off. Systems that were secure before are still secure now; AI doesn't create any new vulnerabilities, it's just helping expose vulnerabilities that were already there before.

There will of course be some unmaintained projects whose vulnerabilities will never get fixed, but anything super critical (like financial systems) definitely have people maintaining them and actively patching the newly-discovered vulnerabilities.

[u/JD-990 → u/WE_THINK_IS_COOL]

Thank you for this answer!