This article is going to be a bit different from usual. Typically, I try to provide a new angle on something, share a tidbit from my research, or something like that. I know that, in recent months, there was an unusual focus on AI and LLMs, and what they can and cannot do, and the primary reason for that is that the current AI discourse is getting more ridiculous by the minute. Originally, I was planning on releasing a piece on the AI discourse yesterday, but I still have to improve upon that. Instead, I am now sitting down and writing this piece in one go, because I am so fed up with what is currently going on.
For some context, a few days ago, some dude named Jacob Coxon resigned from Anthropic and made this small Tweet on his motivation. He said that he’d be quitting because of fear that AI is “getting out of control.” Now, initially I didn’t pay much attention, because the last time this exact thing happened was already excruciating. AI safety people leave OpenAI and Anthropic in droves because these companies don’t give a damn about “safety.” But unfortunately, the surrounding media did not ignore the incident. Yesterday, while I was at the gym, I was listening to one of the largest German political podcasts with an audience in the hundreds of thousands, and they ran a piece on the incident, aptly naming it “Attack of the agents.”
They started by reiterating the discourse around AI safety, and that many people both at the large AI labs and the broader computer science environment were getting increasingly worried that AI may soon spiral out of hand. They then explained the recent case in which a horde of GPT-controlled agents exploited some vulnerabilities in the Hugging Face servers to gain access to some test results. Across the entire segment, I was increasingly worried, and multiple times I was very close to shouting aloud “No, that’s not how this works!” Indeed, the segment exclusively reiterated on the perspective of the large AI labs, and was completely lacking the (existing, albeit quiet) counter-arguments to the logic of Sam Altman and Dario Amodei. Luckily, just a few hours later, I saw pieces by both Adam Conover and Ed Zitron, who are both more “in” to the current Silicon Valley discourse and can explain what I wish to say in much better ways. I recommend to watch/read their takes.
Here’s the short story: Stop listening to anyone who is trying to convince you that “AI is going to become an existential risk by 2030.” This is factually not correct, and indeed impossible with any current tech we have. AI — more specifically LLMs — are merely a new tool, maybe a new technology, that can solve a bunch of issues better than previously existing software. But that’s it. It fails miserably at a lot of tasks where we already have good tools for, which is perfectly fine because, again, we already have tools for that. Anyone who tries to peddle the narrative that LLMs were anything more than that is dishonest and spreads misinformation, sometimes willingly, sometimes because they really subscribe to this doomerist ideology from Silicon Valley.
1. The Hugging Face Hack Was Going According to Plan
First, the big, scary hack of Hugging Face that everybody is talking about. Sounds scary, right? You have a bunch of LLMs that coordinate by the thousands to exploit vulnerabilities of some website to gain access to private data, and extract something from there. What could they do next? Maybe really plan to exterminate us pesky humans?! Of course not. The reality is much more boring.
First, the test case was set up as a cybersecurity scenario, and the LLMs have been prompted to use any method necessary to achieve their goals. Naturally they will see if they can access to a website and extract information in an unauthorized manner. And, if you give these LLMs access to enough tools, it’s comparatively trivial to do so. The internet is full of guides on how to probe servers for vulnerabilities, and believe me, every server has vulnerabilities. If you then train a humongous language model on swathes of such “hacking guides,” what do you expect? That they act like beetles on their back and remain incapable of reproducing exactly what was part of their training data? Come on.
Second, this is the opposite of a “sign that AI may spiral out of control.” In fact, hedging in AI tools is an easy feat. I have written about this recently. If you give an LLM the ability to run a wide range of tools on a computer system, this increases the chances of that LLM running a tool that does something you did not intend. Not because there was any malicious intent on the side of the LLM, because it’s merely generating text on probabilistic principles. But just because it generated text that the harness in which the LLM was running executed as a tool call. Here’s some boring truths about LLMs and their capabilities to “do things”:
- An LLM can only ingest text, and generate new text based on its context. It cannot do anything but produce token after token.
- Even then, you need some software that actually saves these tokens to construct a complete text from. You actively have to append the generated token to its context to continue the text generation process. This is not something the LLM does, but the software that runs the LLM.
- If you want an LLM to perform actions on a computer, you need a second piece of software, called a “harness.” That is essentially just a bog-standard tool that scans the text the LLM generates for trigger words, such as
search_web. If it detects such a trigger word, the harness executes a pre-defined function. - The LLM does not know which pre-defined functions the harness allows it to execute. So you need to tell the LLM about this using a “Skill” file. Without that, the LLM will not even attempt to execute some function. It may try, but it can’t know what will actually trigger something in the harness.
- This means, OpenAI and Anthropic have absolute control over their LLMs and what they can and cannot do: You just enable or disable certain functions in the harness. In short: in order for the LLM to have any capabilities whatsoever, an engineer has to (a) write a function that does something on behalf of the LLM, and (b) tell the LLM about that function. Both require active, conscious effort on the side of the engineer.
- To conclude: this means that AI labs are anything but “losing control” over their LLMs. In fact, they retain perfect control over them. If some LLM agents do illegal things, it is because OpenAI wanted them to be able to do that.
2. The AI Labs Retain Full Control over What Their Models Do
When media outlets now say that AI agents “went rogue,” this is simply not correct. Take another metaphor: If you hand a gun to a monkey, and that monkey proceeds to shoot you, whose fault was this? The monkey’s who has no understanding of the concept of what a gun is, or the fault of the person handing the gun to it?
If you want to interpret this case benevolently for the AI labs, you could say that they didn’t think through what types of access the developers gave their LLMs. You could argue that they were naive and were surprised that, if you tell an LLM “Generate explode_sun to run a function that will make the sun explode” that this makes it likely that the LLM will generate the text explode_sun which makes the sun explode. In this reading, we would have to question the capabilities of the developers at OpenAI of doing the one job they were hired to do: program code. This is simply sloppy and inattentive.
If you want to interpret this case malevolently, however, you’d say that OpenAI fabricated this instance to produce evidence for their claims that “AI goes rogue.” Because anyone can say something goes rogue. I could tell you that my router has just tried to kill me, but you would be right in detecting this as a hyperbole, because no router has the capability of killing a person. But if I sent you a video that shows the router falling onto my head, this would make the case more compelling. And it takes some mental effort to ask “did the router really just fall, or did someone drop it so that it looks like an accident?”
According to Occam’s razor, the simplest and most believable interpretation is the malevolent one in this case. It is really hard to believe that software engineers who work on this for months would “accidentally” forget what they are doing. Because after all, the code itself worked fine. It is hard to believe that software developers at OpenAI would be smart enough to write bug-free functions that would give LLMs the ability to cause havoc, but not smart enough to realize that an LLM would be able to actually use these functions. What is easier to believe is that OpenAI fabricated this incidence to provide evidence for their “AI is rogue” claims.
3. Spreading Fear over “AI Going Rogue” Serves an Ulterior Motive
But this requires a motive. Why would AI labs that make money based on the technology they produce — and they make more money if their technology gets better — say that they want to actually “slow down” because they fear AI may become an “existential risk”? This doesn’t seem right. I think there are two potential explanations for this.
First, if Ed Zitron’s analyses hold any water, the AI labs will get into serious trouble towards the end of next year, when they have to start honoring the first of their absurdly expensive contracts with data center providers. There are currently a lot of data centers in progress, and once they are finished, OpenAI and Anthropic will have to use the computing power and pay for it accordingly. But AI is too expensive for the actual benefits it brings, and as such, they have problems creating enough demand for it. So they’d have maybe two or three years before having to initiate bankruptcy proceedings. Unless, that is, they are perceived to be “too big to fail.” You see, the past financial crises have shown that, if you are just big enough, the government will bail you out if you run out of money. If you aren’t, you’re just going bankrupt. By inciting a fear of AI becoming some kind of “existential risk,” you seed the thought in policymakers’ heads that someone has to stop that from happening. And who could that be? Exactly, the large AI labs. But what if those AI labs go out of business? Who is going to save us from rogue AI then?! Well, that’s exactly the strategy if this is really the motive. Try to convince society that the only person to stop rogue AI is you, and there will be endless money available to prevent you from going bankrupt.
There is another potential motive, though. AI has this big issue that, once a model is done with training, it will just work. As long as you have the model weights, you can run it anywhere. Which means, if some Chinese company distills your large trillion-parameter-model into something that has 80% of the performance for 20% the cost, people will rather choose that one instead of your super expensive planet-killer model. By inciting fear of AI going “rogue” in the near future, OpenAI might try to start international efforts to bar anyone from building their own LLMs unless they are “authorized” in some form. And who will get authorized? Exactly, OpenAI and Anthropic. If you’re the only one who is even permitted to produce LLMs, you effectively hold a monopoly over the market. If someone wants to use AI, they have no more free, cheap, local models available to use, but they must use your planet-killers. And pay the premium that comes with it.
There is a third possibility, however. Silicon Valley and most people who have fallen under its spell has traditionally had this propensity to believe in ghosts. No, seriously. Tech workers in the Bay Area have for decades at this point believed in almost sacral conspiracy theories. Ideas from “Dark Enlightenment” to “Accelerationism” have spread freely in San Francisco and Los Angeles for decades at this point. Thought experiments such as “Roko’s Basilisk” are being taken seriously by a large percentage of tech workers in the area. These people really believe that technological progress is dangerous. It’s almost as if the early futurists grew a conscious but decided not to stop and go all way to the opposite side of the spectrum. And if your entire company staff believes in ghosts of an impeding doom by AI, then you will spread this idea to anyone else, too. Because you truly believe that AI may kill us all, even though there is zero evidence that this is even a remote possibility. Because with current technology, it is impossible to create a truly conscious AI. I myself enjoy Sci-Fi, but even I have to admit that nothing of the great promises will become a reality within my lifetime. There is a thin line between wishful thinking that maybe we see aliens in our lifetimes, and letting this dictate your entire personality.
Whatever the underlying motive behind the actions of the U.S. AI labs is, it is far more mundane than the current media coverage might make it appear.
Also, an additional note on that: Has anyone seen an employee of Mistral, the French AI lab, peddling such thoughts? Or an employee of Alibaba? No? These companies do essentially the same thing as OpenAI and Anthropic, but we hear surprisingly little about impeding AI doom from those sides. The whole idea of AI going rogue is oddly concentrated on a few companies from Silicon Valley.
4. With Current Technology, LLMs Cannot Become Sentient Or Rogue
Lastly, with the current technology that we have, LLMs have no chance of either becoming suddenly sentient or going rogue without anyone noticing.
First, it is good to remind ourselves that the technology on which LLMs currently run is almost thirty years old. The decoder-architecture on which current generation LLMs run originally stems from the LSTM architecture that was proposed in 1997. Later on, researchers added an attention mechanism before Google researchers figured out you could ignore everything except the attention mechanism in 2017. Not much has changed about this fundamental architecture. The only thing that has changed is the scale. If you simply increase the model’s size, you will obviously get better results, but there is a fundamental, qualitative limit in how good these models can get, because the architecture itself does not lend itself to true sentience. And we will need quite a few additional technological breakthroughs before we even have to start to think about whether some model might modify its own weights after the fact. It is very important to remember that any even somewhat capable LLM devours ungodly amounts of energy, silicon, data, and time before it can do anything even remotely useful. Just because something is bigger doesn’t imply that it’s also smarter.
This brings me to a final point of reassurance: It is utterly impossible for an LLM to go rogue without someone noticing, except that someone actively looks away. Just remember that, to solve that Millenium problem — even if we accept for a second that what OpenAI says is actually true — the company spent 23 million Dollars. The price for solving that problem was only one million. In other words, OpenAI had to spend 23x the amount of money to solve the problem. The “thousands of AI agents coordinating a hacking attack on Hugging Face” thing has likely burned something in the hundreds of millions, too. So before an LLM can actually do some harm, you will have to spend an unfathomably large amount of money and time. And you simply cannot convince me that this warrants the claim that AI has “gone rogue,” because someone will see that an entire data center complex is currently running at 100% capacity, and does so for days at a time. It is simply impossible for nobody to notice this, unless everyone in the vicinity collectively closes their eyes and pretends there is no data center running right now.
This also puts into perspective what we have to expect in terms of malicious uses of AI. First, as I hopefully have been able to demonstrate, before any LLM can do any amount of harm, several people have to make active decisions that start up the process. Second, it takes huge amounts of money, meaning that the only actors who can reasonably be expected to run this kind of attack are state actors. And they already know that it’s much more economically viable to just hire a few dozen highly skilled programmers to achieve the same effect, but with a fraction of the money. Just remember that Russia has been doing just fine in the global cyberwar with a bunch of renegade hacker kids for decades at this point. There’s simply no need to spend millions and a data center just to let an LLM do the same job, but less efficient. Not even for Russia. And lastly, if you buy into the narrative of AI doom, you simply give more power to people that just want to make money. And the question is: Do you really want that?
Final Thoughts
I am really fed up with how the current discourse around LLMs is going. These things are being treated as gifts from the gods when all they are is a bunch of regressions in a trench coat. I miss the old days when we were getting worked up around some tragic scene in a soap opera. Today, we listen to people like Sam Altman or Dario Amodei tell us that “AI is going rogue” and it “may kill us all” and instead of thinking for ourselves, we just repeat “Yes, daddy, feed me more doom!” This is pathetic, and we would be well-advised to tone it down a bit. And this counts both for media outlets who uncritically report on these instances without properly framing the discussion and for people who simply “want to believe.”
As a society, we should be better than this. So, rant over.