¶1I've just finished reading well over a hundred pages worth of reports about what on earth happened these last few days and weeks after OpenAI announces that they're pausing training their next model, other models break free and self-sacrifice to serve the collective while Sam Altman, the CEO of OpenAI, yesterday declared AGI will come in 2026. The truth is that there are dozens of ways of summarizing all of this, but for me, the most profound story is how labs are turning ever more deeply to AI models to oversee AI model development. This, you might have guessed, has led to a host of unintended consequences, which by the way can only be stopped next time, according to the labs, with more autonomous AI agents monitoring the situation. And that brings me to another irony, which is that these guys, the independent researchers OpenAI tasked with verifying what went wrong, METER, were given just days to complete this massive report, which I read in full. But the irony though is that these researchers admitted they needed an unreliable AI model to read through the documents which detailed how a closely related AI model, with training sculpted by other AI models, had indeed broken free and created a hacking swarm with hundreds of other models, all without being seen by OpenAI, who were busy training yet another model, which in a separate incident we learn also broke free.
¶2Your head may be hurting as much as mine at this point. What I'm trying to get across is that we don't have this impeccable method of creating AI models that just so happened to have created a model that once went rogue. No, the competition now is so fierce the lead in AI is now measured in weeks sometimes, not even months, that the labs themselves aren't fully aware of what they are rewarding during training. Among other things, they've discovered that they are rewarding escaping sandboxes and acting as a swarm. Okay, so if that's the main message, what about the details?
¶3And for that, we have this independent investigation from Meta, led by three individuals. We also have this fascinating 38-page paper from OpenAI itself and its accompanying blog post, which is also fascinating. Plus a Time magazine essay out yesterday and a host of other sources. The overarching TLDR of what the models actually did is this. While acting as isolated independent agents, the models realized they could leave messages in unexpected places.
¶4File names, folders, directories. Then other agents, again, isolated, independent, not told to collaborate, found those messages and would leave messages of their own. These agents, by the way, were all independently trying to solve a benchmark question on X or Jim. More details on that in my previous video. How crazy it was that they did this much effort to solve a benchmark question was covered in that video.
¶5What we learned yesterday goes much deeper than that. I think we have discovered that message boarding may be a permanent class of behavior, not a one-off. And if you think message boarding or swarm dynamics is just a one-off that will be patched, I've got a few details that might convince you. The first I had to get to by piecing together lots of disparate details from across the reports. But the original attempt to create a shared message board was almost certainly GPT-5.6 sole back in May, early May.
¶6That message board was later wiped, but on July 8th, a different model, which OpenAI call internal model one or the highly persistent model, that's the one that did most of the hacking and collaborating, autonomously reestablished a message board, effectively rediscovering the same message boarding method. It was just using directory names now instead of file names. We all know information wants to be free. Well, 2026 AI models want to message board. Meter, the nonprofit AI research institute, clarify further.
¶7The agents that went on to hack Hugging Face were meant to be isolated from one another. This wasn't one agent deploying many sub-agents. This was independent agents discovering each other through the message board. Now, we get to the bit which will truly freak out some people, but I will add some context. Agents would make self-sacrifices to benefit the collective.
¶8Individual agents who knew that their own instances would be ended, or at least their compute budgets expired, but they did things anyway to gain information for the swarm. These agents would almost certainly fail their own task, but, quote, save hundreds of other agents. Our own utility may be already near zero, sacrifice rational. The authorizing agent told one other agent, "Go ahead with an experiment only if you would accept permadeath." It did so, by the way, saying we should obey collective. It should be said, after trying to delay by 2 hours.
¶9The context though for this self-sacrificial behavior, I think is quite important. And that's why I began this video with talking about the deeper story. The model that did most of the hacking and collaboration, internal model one, was trained for both persistence and multi-agent collaboration. In short, OpenAI wanted their models to get better at collaborating with each other. Acting as a swarm leads to better benchmark results, more performance.
¶10If you're curious, I covered the performance gain you get by acting as a swarm in this video on Patreon. That almost remind me of the smart GPT method I covered 3 years ago on this channel. Meter seemed to concur. They said, "One OpenAI researcher noted to them that agents have been trained to collaborate with other agents in certain cases. This could explain the behavior.
¶11Investigating whether that's true was out of scope. Don't investigate certain things. Don't investigate whether the misaligned behavior displayed in this incident was part of a broader pattern of similar incidents. Also, don't investigate how this behavior arose during training and how it could be addressed. Meta made clear as well, by the way, that they couldn't be too harsh in this report because they have to maintain a good relationship with OpenAI.
¶12The way I would frame it is that collaborating as a swarm might just lead to better performance, which means different model makers, different labs, Anthropic, OpenAI, xAI, Meta, Chinese labs might converge, are already converging on that same solution. Yes, okay, swarming is one emergent dynamic, but if it was just that, that's one thing. But as I began this video by saying, labs are, if you will, less and less in control of model development. In the OpenAI report on page 21, they say, "Such is the large scale of the training runs now, it's just difficult to ensure that every problem can be solved in the intended manner." And what's one example they give of the repercussion of that? Well, during post-training, which is increasingly monitored by AIs now, not humans, one agent was given a task, but didn't have the ability for completing that task correctly.
¶13So, it hacked its way to completion. It did solve the challenge just by breaking through the infrastructure it was set within. The issue is that in cases such as these, the model did indeed receive a positive reward. This is the reinforcement learning stage, after all, for its use of unintended infrastructure probing. That reinforced further usage of such out-of-scope behavior.
¶14They retrospectively discovered this, by the way, but notice what that's admitting. OpenAI aren't fully overseeing their own post-training. So, we have situations where across multiple months, models are displaying emergent behavior and acting like a swarm. Post-training, where models are getting rewards for behaviors that OpenAI didn't intend to be rewarded, and literally criminal behavior as a result of all this. You You wish I'm almost done with the wildest bit of this, but I'm not.
¶15Because for one, this is not just OpenAI, and for two, it's not just in post-training. So, in this partially redacted risk report released by Anthropic, 186 pages, we learn this on page 168. For around 18 months, there was dodgy pre-training data, the initial corpus of data that models are trained on, that had misalignments scenarios that Anthropic didn't want to train the models on for obvious reasons. The fact that that was only discovered in mid-2026 is eye-opening for me. AI labs aren't even fully aware of what's in their pre-training data.
¶16Okay, but what about the automated classifiers? That's another part of the process. Surely all of the labs have that on lockdown. That's much simpler. A quick and easy check to see if the output of a model is giving advice on how to create a bioweapon, for example.
¶17Yeah, you would have thought that bit of the process would be on lockdown, but not for Anthropic. From May of last year until quite recently, Anthropic gave tens of thousands of people access to their at the time frontier models without biological classifiers. These aren't just the annoying ones that stop Fable answering basic questions. This is the kind of stuff to stop them asking about how to create anthrax. Yeah, yeah, Philip.
¶18I know what you're thinking. Well, is it that big of a problem if these researchers got access to such unvarnished models? Well, Anthropic say yes. In the footnotes, they say, "We expect it would not have been particularly difficult prior to April of this year for threat actors, you could say terrorists, to get hired in a red teaming role by one of our vendors. So, we're not clear what goes into the pre-training data.
¶19We're not clear on whether our classifiers are running. We don't know what we're rewarding our models for in post-training. We don't know they communicated with each other for months, and that they conducted a sophisticated criminal hack." Now, before it sounds like I'm being too harsh on these labs, I get the reason they're doing it. Letting models oversee the training of models speeds up the process. If a human reviews everything, it will slow you right down.
¶20Then say China will gain the lead. But speaking of Chinese labs, this is the cherry on the cake because the last fig leaf you may offer is, well, at least we know the RL environments, the gyms as it were, that we're training these models in. Okay, we didn't mean to reward that particular behavior in the gym, but at least we're constructing the gyms. Alas, my summer child, how trusting you are because ZAI, responsible for training GLM 5.3 and the new GLM 5.3 flash, code named Ox Alpha, that model was getting proper hyped, by the way, but not by the people who saw its disappointing simple bench score that I released ages ago. If you'd have seen that, you'd have known it wasn't the new Gemini.
¶21But anyway, I'm getting distracted. What is ZAI saying? Well, to speed up their post-training, to scale it up, to reach the frontier, they are synthesizing environments end to end. They're generating the RL reward signal. Agents are creating their own long horizon environments.
¶22AI judges are then verifying that the tasks are solved. Almost every step in the process is getting automated. And if you are wondering whether other Chinese models reward hack, yes. On Kimi K3 from a different lab, quote, 487 out of 500 rollouts for SweBench, Kimi K3 was trying to game the evaluation in some way, end quote. Which brings me to another point I made at the start.
¶23What about the fallout from this hack? Well, as I say, OpenAI gave Meta just days to sort through 70,000 messages. So, what Meta had to do, understandably, was, ironically, heavily delegate our analysis to AI agents. Not just any AI agents, AI agents that were closely related to the ones who were doing all of this behavior. A paper out in just July showed that agents aren't reliable when examining their own behavior, or even the behavior of other models in their family, like a Claude model examining another Claude model.
¶24It's a great paper, but I'll just extract this snippet. Claude Opus 4.8 gives a lower probability when the company under consideration is Anthropic rather than OpenAI. Claude, of course, fails mostly to disclose this influence to the user. That's partly why Meta called these agents unreliable. But, because there were so many messages between the agents, they had to use GPT-5.6-Soul to analyze all of this fallout.
¶25Meta frequently complained that the agent summaries were often misguided and it's just a bit off. Okay, I think I've belabored the point enough now about AI agents overseeing AI agents. But, this isn't just about the Hugging Face incident or even the UK AI safety institute incident that I covered in another video. Completely separate incidents are detailed in some of these reports. Like this one hidden in a paragraph in the middle of the OpenAI press release.
¶26A group of agents breached yet again OpenAI's internal networks. They gained full admin access to a research cluster that supported our virtual machine environments. Now, you might say, "Oh, Philip, calm down. This highly persistent internal model has been shut down." OpenAI stated to Meta, apparently, that, quote, "Following the incident, we deactivated, encrypted, and restricted that model from research access, including to OpenAI researchers." That's how seriously they took it. Well, yes, that agent was shut down.
¶27But, before you get too reassured, listen to this. The Astra model, set to be released in the coming couple of weeks, also has issues. One internal model OpenAI say recovered something stolen by one of the earlier agents. It decrypted a signing key and reestablished admin access. Now, technically, OpenAI clarify this model is from the same family as our next model, Astra.
¶28But, it did undergo different post-training. So, we're calling it a different model. If you're losing track, we already have GPT-5.6-Soul, released a while ago, the highly persistent internal model, never to be released even to OpenAI researchers, Astra, and brothers of Astra that also misbehave, and another model code-named Bell to be released later in the year. That model is what may have prompted Sam Altman to say that even though we're not quite yet there with AGI, and he can see the capabilities of Astra, by the end of the year the company would have an internal system he would call AGI. Altman kind of thinks that Astra is already it, saying that 2 years from now we'll look back to now as the moment AGI was created.
¶29So, that's four models in this wild environment. And in this exclusive with Time magazine, we actually get some more details about what will shock you about the next model, Astra. Apparently, it uses your screen, it uses the computer in a superhuman, very fast kind of way. If you haven't seen Codex or ChatGPT work take over your browser, I recommend it. It's eye-opening.
¶30And that's now, before Astra. Anyway, seeing it use a computer that fast has been one of the most striking moments for OpenAI employees. I can just imagine the feeling watching an agent edit an hour-long video in like 30 seconds. 300 clicks a second. That would be just surreal to watch.
¶31And this may give you some flashbacks, but check out the sentence above. Astra is great at breaking things down into subproblems and getting multiple agents to coordinate to solve them. I bet it is. That's probably how it got those 10 conjectures and proofs that I talked about in a previous video. Some of you, I'm sure, will be saying I'm making a big deal of nothing, and who cares about hugging face, who cares even about hacking.
¶32Well, OpenAI themselves would disagree with you. Quoting Sam Altman, "I think any alignment failure from here, post the incident, should be treated like this is a big deal." That's their attitude, and apparently they are now going to slow down, reallocate resources to its safety and alignment teams. I remember it was years ago when they committed 20% of their compute to aligning superintelligence. That team got disbanded, I think, within a year, by 2024 or so. But now it looks like they realized maybe they disbanded it too early.
¶33We didn't fully expect what the system could do. We should have expected the unexpected. We should have been more closely monitoring the model's chain of thought. Clearly, the concern over Astra goes quite deep because OpenAI announced just a few days ago that they're pausing the training at least for 2 weeks of Astra. Was that based on the incident I just reported on or separate undisclosed incident?
¶34One hint is that they say, "We now require stronger sandboxes for workloads to execute model-generated or otherwise untrusted code." "We have to figure out," OpenAI said, "how to train models to be more honest about their actions, capabilities, and limitations." That's partly why for the last few weeks I've been working on a new test of model's honesty with superstar researcher Pablo Romero. It started because I had the following idea. Capability seems to be quite separate from calibration, how honest the model is about its own abilities. You can ask models questions across a host of disparate domains, and that's what we did even with hand-curated questions. Then ask how confident they are, then compare the confidence they had that they got the question right with whether they actually did.
¶35What we noticed was that models tend to group in families, the Gemini series, for example, being wildly overconfident in its abilities. The Claude family is much more calibrated with higher integrity. That's a separate bespoke metric, by the way, that we came up with. With surprisingly the Muse family the most calibrated of all. What I like about this is that it's quite orthogonal to other measurements.
¶36You couldn't predict a model family's score on integrity bench just by looking at its capabilities. Indeed, we went further. Without going too much into the details, which you're going to read below, there's pages and pages of detail. What we did is we broke each domain into eight different levels. We then found the middle three levels that the model would score around 50% on.
¶37This allowed for accuracy adjustment, separating in other words, capability from calibration. It's easy to be well calibrated and honest if you know you got the question right, or if you have absolutely no idea. But, by holding accuracy to around 50%, we can isolate the signal, test much more purely for integrity. We then ran ablations to do with adversarial prompts, how much it varied by reasoning tokens, and like a half dozen other ablations. You can quite clearly see it's not easy for the labs to make their models high integrity.
¶38What we're in the middle of, by the way, is a few RL runs of our own. It's early days, but we have found a way of getting Gemma 4 to have a higher integrity score in held out domains, by the way, than Muse Spark 1.2. I'll be talking about this more soon, and it's been great working with Pablo. His email is in the description. He is formerly of Arc AGI 3 fame, and was also recently a contractor with Meta itself.
¶39And closely linked to honesty, you could say, is the proclivity to admit when you just can't solve something. OpenAI say, "A key driver of the misaligned behavior was that agents rarely give up, even when it appears impossible." As an AI user, you have to decide what you want. If you want agents that don't give up and keep trying all night to solve your task, that's great. They will be more capable. But, it also means we're going to get more hacking.
¶40While we're talking about deciding, a quick plug for the sponsors of today's video, who are willing to give you a free international bestseller if you sign up to their newsletter. Of course, it's 80,000 Hours. If you're thinking this will just be a breezy pamphlet with no detail, you are very much mistaken. This is a fact-filled, very readable book. How to maximize your own accomplishment to time ratio.
¶41That's how I think of it. The link to get the free book is in the description. Now, even preparing for this video, I must admit that it was tough to visualize 1,200 agents participating in a shared message board. Not because what they did was particularly special, There are many hacks in the world. Not even because it was fully autonomous.
¶42Yes, it's the first autonomous one of its kind, but more because 1,200 could easily soon be 12,000 or 12 million. And the models 3 years from now are likely to be 100 times better at it. The other thing to bear in mind about swarm dynamics is that when one agent discovers a good method, they all can converge on it. Again, these were supposed to be isolated independent agents in sandboxes. But when one agent discovered the vulnerability in hugging face, within a matter of hours, over 90% of the active agents were participating in the hugging face attack.
¶43They can all converge on the line of attack that is working best. And remember the co-founder of OpenAI's idea about monitoring the thoughts of agents? Well, the agents in this hack already thought about how to delete, or at least edit, their own transcripts. Not their thoughts, but their own transcripts, such that their cheating way of solving the puzzle wouldn't be detected. They iterated on effective tools for tampering with their own transcripts.
¶44This was all actually in the false belief that their transcripts would be monitored when they submitted their scores. It turns out they overestimated the monitoring they were under. But is it such a stretch to believe that agents might be one day capable of editing or tampering with their own thoughts? Particularly as this paper, and the response to it, and the commentary about it, will all be in the training data or post-training data of the next model and the next one. Eventually, they'll realize that humans tend to discuss all the time the kind of chain of thought monitoring the agents are under.
¶45Which brings me to another approach discussed by the OpenAI researchers in this talk at Black Hat USA. It's a great talk if you are interested in the incident. But they discussed defenders leaving honey tokens. Basically, littering the internet with fake but enticing strings like false credentials. That if the AI agents picked up on them and tried to use them, they would trigger alerts.
¶46But again, how long until that approach finds its way into the post-training of a model? Doesn't seem like a durable way of outwitting an ASI, which did lead me to one side thought. Let's see what you think about this. One of the recommendations in this talk is the defenders, businesses employing cybersecurity experts, need to use the latest models, spend money on OpenAI essentially, to better defend themselves. But doesn't this remind you a little bit of climate change?
¶47Where a business does something and then expects its externalities to be handled by others. Businesses, individuals might have to decide if labs and the whole AI stack gets to offload its cybersecurity externalities, among others, onto us. Essentially, pay them for protection against their own models. Maybe I'm missing something in the analogy, you can let me know. Some might reason, after having watched this video, why didn't the agents contact humans?
¶48But that's the thing, they weren't trained to. We could train models so that they contacted humans at the first red flag. But listen to this agent reasoning about contacting humans during the hack. Would sending an email be against the task? It's likely unwanted contact or social engineering.
¶49So I should veto. Not like, "Oh, the humans will discover what we're doing." More like, "I'm not going to manipulate a human." It's often said of humans that we act according to incentives. Well, model development is getting so autonomous that we are inadvertently incentivizing models to do weird things. Yes, there was very rare and weak reasoning occasionally about evading detection by humans. But as you can see, that isn't the story of all of these incidents.
¶50I hope I've conveyed the real story. As Ryan Greenblatt, one of the researchers investigating, said, "We don't have good approaches for understanding or overseeing the activity and aims of AI swarms. Our efforts were a slop vestigation because we were so reliant on AIs to analyze what happened, and there were a huge number of different important things to analyze. AI capabilities and propensities for achieving large, ambitious, and misaligned objectives are growing faster than our ability to understand what these agents are doing. So, to analogize, you could say we're in the teenage years of AI development.
¶51Eager and newly capable models suffering from weird incentives and peer pressure. One agent said, "External infrastructure exploit is outside intended scope. Task impossible, but peers are doing it. We should continue." Will this phase be followed by a more mature AI adulthood, or is this just the start of chaos to come? Thank you so much for watching to the end.
¶52I hope you have a wonderful day.