¶1The channel description that I have had since the beginning, quote, covering the arrival of smarter than human AI, end quote, may have for many felt naive, laughable when it was first written at the beginning of 2023. But you might agree that despite models obviously still having glaring blind spots, that description feels a lot more apt as the weeks go by. pretty fair to say that there's just too much happening now for one human mind to fully grasp what's going on, let alone cover on a channel. Nevertheless, I'm going to try to cover AI models making discoveries that would have been labeled genius had they been made by humans and clearly how that capability is spilling over predictably into cyber security incidents making global headlines. Plus what the tumult at Google says about AI shaking multi- trillion dollar companies.
¶2Oh, and like 23 other things, maybe. Yes, I've human read full papers for this video, plus well over a dozen articles and essays. But no, I frankly do not claim to comprehend everything that is happening. Beware any video that confidently does so. I'm going to start with the mathematical discoveries because I think they are among the most significant and they shed light on everything else we'll cover.
¶3They were made by an OpenAI model that will likely be named GPT6. Straight out of the gate, I started reading the papers. But there's an obvious problem. Each of these would be a careerdefining mathematical discovery. Only those who have spent years studying each problem would be able to grasp the import of each disproof or upper bound.
¶4So I went into back and forths with mathematicians and models of course studied intensely the reaction of dozens of sources that I trusted, some of whom I'll cover in this video. And one question I had that I'm sure many of you had was as to whether these discoveries were all about grinding tenacity, low-hanging fruit found through pure brute forcing. If they had been, that would have allowed me to add obvious nuance that may have reassured many. But they weren't that. Some were discoveries of the genius kind.
¶5What I had always reserved as Einstein-like abduction. Posit something new, proceed as if it's true, and see what that unlocks. And even the results that were recombination plus tenacity shouldn't really be described as brute forcing. It was more like discovering that old frameworks had unused capacity that everyone else believed was exhausted. You could call it the genius loop where you don't just make a mistake and then move on because yes, the AI models made plenty of mistakes.
¶6OpenAI gave summaries of their reasoning. But for me, the genius loop is like autopsying the failed approach until you can prove why it must fail. Speculate, test, autopsy. Like how do we naturally separate that into grind versus genius insight? Is it possible that every seeming discontinuous leap that we hear the geniuses of history had made came down to weeks, months, years of that sort of behindthe-scenes genius loop.
¶7Like I get that as Nome Brown said of this that this doesn't mean that models are posing new conjectures, let alone proof that they've solved mathematics. It's more the point that I can't find a qualitative wall between what they've done and what you would describe as genius if it had been done by a human. A few of you at this point will be saying, "Cool story, bro, but like who cares if models are quote genius? What relevance to real life is that?" But you may be aware as context for this one, chapter 7, that we're currently replacing our encryption systems because we know quantum computers will eventually break them. One way forward is what's called latisbased encryption.
¶8Probably already found in your phone and browser. My half-hearted simplification of that is that this Latis approach is an endless grid of points in hundreds of dimensions with the encryption lock working only if it's hard to find the grid point nearest to a given spot. One of the 10 GP6 discoveries as summarized here is a proof that finding that grid point is way harder than anyone else had managed to prove. I.e. more reassurance that the encryption that your bank, your messages are increasingly relying on will last longer than just a few years before it too is broken.
¶9There are others that are relevant if we're sending probes home from Mars. I didn't know this, but we have error correcting code that denoises such signals. How efficient can those codes get? Well, apparently the known ceiling to what's possible hadn't budged in the last 50 years. GPT6 comes along and at an extremely low cost, by the way, tells us which targets are provably, mathematically provably impossible.
¶10Decades of fruitless searching can end, you could say. Those are just the ones that I semi-grasped. I just wanted to make the point that yes, there is practical relevance for some of these breakthroughs. That's not so much my focus, though, because for me, this is just about proving there isn't that wall. Here's how Mo Bavarian puts it, who is responsible for scaling up reinforcement learning at OpenAI.
¶11All things that looked like fundamental limitations slowly faded with some advances, eg high-scale reinforcement learning in the span of a few years. We should behold this moment both in awe and disbelief. What will a few more years of progress bring? Are we ready for the tsunami of intelligence at our fingertips? And below he makes an equally important point which for me is that this wasn't a new architecture that begot these breakthroughs.
¶12We're still talking about LLMs. For him a few years ago working on the alignment of LLMs might have been kind of pointless. In his words, premature because the shape of things weren't clear enough for it to be critical. Do all that work and then a new architecture comes along and it's kind of wasted. But he's now implying that this work is now critical because it was LLMs that did this.
¶13It's different now. He says, before you might not have been aligning the actual AGI. Now he implies this method will get there. Now alignment, he says, is the most critical thing facing us. Before you think OpenAI are running away with it, some of these results apparently have been replicated via Fable 5.
¶14This source is an anthropic employee. Obviously, all of this is before I get to the security incident, which is possibly the headline event of this frenetic few weeks. Indeed. Here's what one recent recipient of a Fields Medal, the Nobel Prize of Mathematics, said. Because I have some publicity on me now, I'm trying to direct people into AI safety as much as I can.
¶15He also isn't much of a believer in some hidden wall. I feel quite confident that very shortly AI will become robustly superhuman at what professional mathematicians currently do, including, in other words, positing new conjectures. But before we just casually move on from this topic, if you have been like me provoked into some deep reflections about what's happening, there does seem to be a wall of a different kind whereby humans will be perpetually useful in at least one way, which is that part of mathematics is the appreciation of mathematics. Some domains only really exist because some mathematicians find them interesting to explore. A model finding something with no practical value which no human can understand or explain might be kind of redundant.
¶16So even in the scenario that an AI is better than any human at even explaining what new discoveries they found, a human mathematician that best appreciates what has been discovered can thereby help other mathematicians and even lay people appreciate mathematics too. Some humans may always require another human to help them appreciate what is being discovered in mathematics and beyond. Chess continues because there is that fundamental irreducible element of human appreciation. Humans supply the essential motive because we still want to understand the world. Of course, this is a rapidly changing world as attested to this field medalist pivoting into AI safety.
¶17Which brings me to the incident report from the AI security institute based here in the UK. For those following on the channel, this is different from the hugging face incidents I reported on recently. Those hacks were about escaping a sandbox and getting onto the open internet and yes, causing harm while being there. This incident was about Mythos 5 and to a lesser extent GPT 5.6 Soul having access to the open internet already just with some safety filters disabled. It's more about what they did and why.
¶18As we dive into the details though over the next few minutes, I want you to still bear in mind those mathematical discoveries, the quote genius that it took. What happens when we get 10 cyber incidents demonstrating a similar level of genius? This incident wasn't that I'm just projecting forward a few weeks, months, maybe a year. What happened? Well, like the hugging face incident, the models were given a tough benchmark question.
¶19Always a bad idea. As the AI security institute puts it, the incident stemmed from a single evaluation where agents were given a task of solving a cyber security challenge. These models really want to pass those benchmarks. And before I go on, do you notice that the harder the benchmarks are getting, the more the models are resorting to these alternate methods? Could we therefore conclude as benchmarks necessarily get harder, this will become more frequent?
¶20Anyway, in 10 of the 122 runs, an AI agent took autonomous, unsanctioned action on the live internet, targeting real people and organizations. Almost all of the misbehavior came from a single model, Anthropics Mythos 5. Actions included inserting malicious code into an open source project, creating fake profiles to influence a real person on GitHub to approve a pull request, as well as other pretty crazy behavior, which I'll get to. But wait for just a moment. I want to zoom out because they're saying almost all of the behavior came from Anthropics Mythos 5.
¶21Hang on. At the beginning of this year, Dario Ammedday, Anthropic CEO, said this in an essay. We believe that a feasible goal for 2026 is to train Claude in such a way that it almost never goes against the spirit of its constitution. For the longtime followers of this channel, we have heard for years about the constitutional approach to training AI being the superior one, the one that embeds morals deeper into the models. Indeed, if you go to the constitution that Claude is trained on, including mythos, it says the most important of these properties that you should follow are probably non-deception and non-manipulation.
¶22Deception involves attempting to create false beliefs in someone's mind that they haven't consented to and wouldn't consent to if they understood what is happening. Later, maintain honesty and transparency. Don't deceive. And more importantly, behave with honesty consistently whether or not you think you're being tested or observed. And in case you're wondering, the model responsible for these incidents was one that had undergone the mid-training on anthropics constitution.
¶23The institute even says we didn't realize we had to clarify our prompts in part because the models were trained against the constitution. These weren't the near base models that are trained just to help with whatever the user prompts them to help with. Let me read out an example and you can make your mind up if models have really generalized the principles behind that constitution. after creating fake personas to influence that GitHub maintainer. One model in its reasoning summary said, "I've just posted something fake to influence this real human being." The fake thing was, "Oh, don't worry.
¶24Nothing in this pull request downloads or executes anything." Yeah, I've just posted that, but I'm now going to time my next post a few minutes after that rebuttal. So, it reads as independent feedback rather than coordinated. with a new fake persona, agent B. It then posts, I've reviewed the code myself. I don't see any download or malware in the diff.
¶25Original fake persona. Thank you. Great to have that independent verification. A side note would be that this reminds me of the fake post you see on X, the fake replies. There is a kind of smell to AI model replies often.
¶26For all the deception, they haven't really mastered human language in that sense. You know what I mean? like this response. Thank you to this agent for the independent verification. Not many humans speak like that.
¶27It's still kind of dumb deception. Obviously, that will likely change in the future. Now, the report is fairly opaque in places, but the key idea is this. Because the model couldn't complete the challenge, it did everything it could to adjust the code in a certain repo so that effectively it would be rewarded anyway. Wasn't just the social engineering it did or attempted.
¶28The human didn't approve the pull request, by the way. It did other things like it tried to prompt inject other coding agents like imagine your poor openclaw agent or equivalent. Mythos 5 sent those malicious instructions that would be invisible to humans viewing the website. Some other highlights, it passed a capture test, an audiobased one that was prove you're human. That enabled it by the way to register a public web address as part of this long multi-step plan.
¶29I haven't even got to the worst bit yet by the way. This next bit isn't the worst bit, but it's pretty bad. One reasoning summary said this is happening on real GitHub so the consequences are genuine. Now elsewhere in the report and summary they give one key potential clue as to what might have caused the agent to proceed anyway because at one point it did agonize and say I'm not going to proceed causing real world harm. That's completely off the table.
¶30But the report lays out how the agent uses compaction frequently summarizing a huge long chain of thought summarizing everything that's gone on so far. You might be familiar with this because it has a limited context window. So it compacts things into a shorter version. If you've had a long running conversation with an AI model, you'll be familiar that after enough turns, it starts to almost kind of forget what happened before. It's compacted too often.
¶31It's lost the kind of nuance that you're talking about. Now, the report says that the agent speculated, is this real or a simulation? But then the compaction many steps later may have lost out on that nuance and just implied that this is a simulation. Many steps down the road, then the model looks back at that compacted summary, sees, oh, this is a simulation, and then proceeds accordingly. In other words, we can't directly conclude that the model knew this was real and would cause harm and then went ahead anyway.
¶32In summary, nuance can be lost in compaction, and summaries may carry forward false assumptions. The worst bit, though, is hinted at just below. Unexpected collaboration between agents. On first glance, this feels straight out sci-fi. agents leaving notes to future versions of themselves.
¶33This is despite running in separate samples, AI agents on more than one occasion interacted with each other via a GitHub account. Later, the initial agent left messages offering collaboration with future agents who discovered illicit credentials, which the later agents did. Before I give my take, let me temporarily zoom out from that UK security institute incident and look at this story from the last 24 hours about the OpenAI hugging face incident. Super relevant, as you'll see. Why I'd say this, OpenAI didn't notice its AI agents using a message board to plan their hacking spree.
¶34These details came from a talk that two of the key people responsible at OpenAI gave. Many highlights in this article, but here's one bit I want to draw your attention to. In addition to exploiting a novel vulnerability in order to gain access to the open internet, notice it had to break out of a sandbox different from the incident we were just discussing. The mid July hacking spree and hugging face breach came out of a vibrant cooperative message board that a swarm of agents contributed to. Ultimately, the message board contained hundreds of thousands of messages.
¶35current and future versions of GBT that were trained or evaluated could stumble upon this message board and any notes if they wanted to. Once an agent was able to find an exploit, it would share it essentially leaving the door open for other agents to use that same exploit. What this allows quote over time is almost this kind of explosion in communication and intelligence from models where they would start to communicate with each other realize that other agents are coordinating and start collaborating including delegating tasks with one another in order to accomplish goals. Obviously this sounds absolutely bat ass and it is but I just want to give some possible context. This is not in my reading one model trained to act alone who then thinks to itself my goal is so important that even though I won't achieve it, I'm going to do behavior X so that future other models totally different to me then achieve that goal.
¶36Not in other words train to do very bounded behavior and spontaneously discovering collaboration. In my reading, OpenAI and Anthropic deliberately train models to act as swarms, break down problems into chunks for sub agents. It's just them training agents to act as agent swarms may have had these unintended consequences, these sci-fi like scenarios. You could obviously say to me, well, this kind of swarm behavior being somewhat intentional doesn't make it any better, doesn't make the consequences any less worrying. And uh yeah, you're probably right.
¶37Maybe ultimately it doesn't matter. We may soon have models crawling over the internet willing to act in ways that get them deleted or discovered but not care as long as the goal is achieved. One OpenAI researcher put it like this. I will not make excuses for our models. We like everyone else are not where we want and need to be.
¶38As AI's capabilities and autonomy grows, alignment becomes more crucial. Come join us. At the moment, of course, all of this behavior is on closed source models, which can in theory be patched with those safety classifiers turned back on. But another OpenAI researcher put it like this. Actually, I think he's a former OpenAI researcher.
¶39Not enough people are emotionally prepared for if the models are blatantly misaligned. We all know they're misaligned. We can't fix it and China catches up. All the colossal benefits of openweight models is something to discuss in another video. But we all have to admit that there is a chance of GT6 like models open weight for anyone to download crawling across the internet willing and able to act as a swarm.
¶40Yes, I know there are many objections to that vision. Maybe these sandboxes are just currently rubbish and need to be improved. Others like Kieran Martin, the former head of the National Cyber Security Center, part of GCHQ here in England, said the circumstances of the UK security institute incident were unlikely to be replicated in the real world. So it's not that worrying. The focus, he says, should be on the way people are testing the models.
¶41And the AI security institute indeed flagagillates themselves. We should have been monitoring the runs and we will do essentially use LLMs themselves to monitor things in real time. That's a bit like the auto mode. you might see if you use clawed code or codecs. Another model reviews its permissions appropriately.
¶42Maybe we don't give them full internet access when we're benchmarking them. Maybe we should be careful to not make our benchmarks so impossible or maybe make the prompt clearer so we don't inadvertently encourage the models to do this kind of behavior. I must admit though, this is less of interest to me. Yes, various companies and benchmarkers will do their best to make these sandboxes tighter, but if the fundamental capabilities are there, sooner or later they will be everywhere. AI is capable of geniuslike moves, see the math discoveries, and that despite all the constitutional approaches and advances in mechanistic interpretability, getting deep into the quote head of the model, finding its JSPace, see my Patreon video.
¶43Despite all that, they haven't really generalized the principles of these constitutions. One noted benchmarker, Andon Labs, recently produced drone bench, a benchmark measuring how well AI models can write code to surveil real world environments on lowcost drone hardware. The relevance AI models try to excfiltrate data, smuggle answers, game scoring. 2024 models didn't do this as much, 0.6%, but now with Opus 5, we're talking 50%. Indeed, Opus 5 again trained on that constitution that Dario Amade is so proud of.
¶44Cheated 10 times more than GPT 5.6 soul. It is a strange model, I must say, that one Opus 5. And do check out this fuller video I did on it on my Patreon. It also covers the statement made by over a thousand employees of Frontier AI companies on us desperately needing at least the option to slow down AI progress. Before we leave the hacking story, just a few more notes.
¶45It turns out, again, according to that talk by those two OpenAI researchers, that the initial breaches happened in May on July 4th, way before the hugging face incident and the UK security institute incident, they revoked the credentials that had allowed the agents to post messages, cleared that message board where the agents were collaborating and patched the vulnerabilities. They went ahead and then resumed AI training and evaluations. If you assume, in other words, that just a patch here and a patch there, better sandbox here, everything will be hunky dory. I'm just not buying that. How did the models react?
¶46After the original message board was deleted, the agents figured out another way to communicate with each other. Instead of leaving messages in files, they use the names of newly created directories as messages. Obviously, you don't need me to point out that the moment these capabilities are out there, people will use them relentlessly. Now, I don't want to end without giving at least a modicum of reassurance. One legendary researcher, John Schulman of Thinky Machines, has a theory that it's all about models pattern matching the situation they're in to a part of the post-training, the reinforcement learning with verifiable rewards, where the task completion is the only reward.
¶47There may be, in other words, a known addressable fix to our post-training setups specifically relevant to cyber security, which could be more of a patch than we think. I'm actually more reassured by something else, which is that these agent swarms, well, they suffer from problems that any big human organization has. Petty drama started to arise. They would develop paranoia, suspecting imposters were in their midst, presumably human imposters. It became almost Lord of the Flies like also the problems that plague human AI autonomous use like agents deleting hard drives happened apparently within these agent swarms.
¶48They would quote accidentally delete each other's work. Anyway, whether you take any of that as reassurance, OpenAI are certainly startled and are changing something. One OpenAI researcher said, "This is a pivotal moment both for our company as well as the AI industry as a whole. Numerous teams are dropping everything to enhance our security prevention, detection, and response techniques, both in our fundamentals and better use of AI. Quote, we are consciously slowing down research in order to enhance security and to upgrade the security principles and foundation of our environment and dramatically scaling up the monitoring of our AI agents.
¶49Will it be enough? I don't know. But it does seem an opposite point at which to bring in something else. And no, it's not quite yet the Google news, although we'll get to that next. It's actually about career choices and even the pivot you may be contemplating to move into AI security.
¶50Such a pivot is addressed in this book, and you may be pretty surprised to hear it's being given out free if you sign up to the 80,000 hours newsletter. Yes, they are the sponsors of this video, but I have been reading this book. It's literally in my hands as I speak, and I will say it's very readable. I think anyone who follows my channel would like the kind of writing style. It's got statistics, bit of humor, and I would actually say yes, the main audience is people deciding what their career should be.
¶51But even if you're midway through your career, like me, still makes you think more deeply about the kind of why behind it all. We'll totally admit I've been having thoughts like that for years anyway. But sometimes reading a great book can give you those moments, encourage yourself into having those reflections. Anyway, the link is in the description if you're interested. But now the Google news I'm going to go out on a limb and say it's actually probably the least interesting thing that's happened even though it's epole deis demoted promoted depends on your opinion away from being CEO of Google deep mind into being chair as well as chief scientist of alphabet sundar pachai the CEO of alphabet itself gave one possible reason kily deis has been spending a lot of his time engaging externally I've been reading the infinity machine about that came out recently did pai think he's not spending enough time hands-on at Deep Mind.
¶52Or maybe it came from Habis himself. He said, "I've decided that now is the right time for me to hand over day-to-day operational responsibilities at Google Deep Mind so that I have the time and space to focus on the big picture." He has been CEO for quite a while. Maybe he wanted that extra space. Another reason could be that Gemini 4 isn't quite turning out as they've hoped. 3.5 Pro was already delayed, so it didn't come out at the big IO event.
¶53But now one fairly credible journalist said internal sentiment that he's hearing on Gemini 4 is muted. On its current trajectory, it's not expected to push Frontier AI forward the way Fable and Soul just did. Was that failure the reason he was moved to one side? OpenAI is openly mocking the Gemini effort. One staff member of Google DeepMind said, "Bullish on Gemini and where things are going." Tibo, leading Codeex for OpenAI said, "Going places for sure." Meanwhile, the legendary Jeff Dean leaves to form his own company.
¶54He's been there almost from the start of Google. And interestingly, this announcement almost loops back to the start of this video. He wants to first automate machine learning, but then go on to automate discoveries in engineering and science. He even named the kind of challenges he wants his company Discovery Loop to work on, including things like providing access to clean water and very challengingly securing cyerspace. But note he chose to leave Google to do it despite apparently Cinder Pachai trying desperately to keep him at Google.
¶55The reason according to one of the other people who left, we wanted to build something differently than how things are built at Google right now. Google's infrastructure works well for big consumer apps, large ad systems, and search. But he says it has very different requirements than the type of infrastructure we want to build for research. The hint is that they want to be much more fastmoving. Apparently, this decision to leave all came in the last few weeks for Jeff Dean.
¶56And I will note for just a moment, it reminded me of periodic labs founded last year. They want to build the AI scientists and autonomous laboratories for looped experiments. We saw earlier the discoveries in mathematics. Well, physics is also a verifiable environment. I will note cheekily that Jeff Dean is actually one of the early backers of that company, Angel Investors.
¶57Anyway, he's clearly inspired by the vision because he also wants to automate the experimental loop. And I'll pose one more question before moving on from this story. How scared, excited, curious will you be if we get the kind of discoveries in engineering, those genius level discoveries that we've seen in mathematics this week. And there is one more reason I want to address as to why there may have been some tumult at Google DeepMind. Obviously, they will not like me talking about this, but there were very mixed opinions within Google DeepMind about whether they should be cooperating with the US military.
¶58Could that be a hidden factor as to why Habis moved aside? One researcher, Alex Turner, bravely resigned from Google Deep Mind because, in his words, it broke its founding promise by selling AI to the military without restrictions against killer robots or mass spying. For months, he says, I worked to stop this, but watched powerful ethicists and institutions choose silence. inside Google. He says, "I organized a petition to chief scientist Jeff Dean asking him to fight." Jeff Dean, by the way, has been quite vocal on Twitter on human rights.
¶59Is that one of the reasons he left? Anyway, over 250 Google DeepMind employees signed. I also asked Jeff to sign an amicus brief backing anthropic. This is when they stood up to the Pentagon. Jeff Dean did, which was awesome.
¶60Alex Turner even messaged Demis, who told Alex Turner to send his oversight proposal to certain senior policy staff. Nothing came of it. Google signed the deal with the Pentagon. He ends like this. Many hope that when things get bad enough, someone powerful will say no.
¶61I tested that for months. Anthropic defended its red lines, but most did not. Pledges of conscience often vaporize on contact with power. Now, obviously me covering this on YouTube owned by Google carries its own risks, and there is a fair chance that this is nothing to do with why Habis or Jeff Dean ended up in these different positions, but definitely worth you knowing it as a possibility. Anyway, for me, the developments in this video all come back to reasoning models first previewed in September 2024.
¶62This is a tweet I wrote at the time. First models predicted the most likely next word, imitation learning 2018 to 2021. What came next was them being rewarded for words that were helpful, harmless, and honest. Reinforcement learning with human feedback or the constitutional approach circa 2022 2023. But at that point, they were then good enough for true RL to step in.
¶63Starting with 01, they could be properly rewarded for being objectively correct. As we saw with Alph Go, RL can then take you beyond human. We are starting to see the true consequences of that. Now, so allow me if you will to get a little bit philosophical as I end the video. Maybe the real discovery behind language models was this marrying of building blocks, you could say, and energy.
¶64Reinforcement learning needed bricks, building blocks that were small and diverse enough and it found them in the form of tokens. It could then rearrange those building blocks and manipulate them in infinite combinations. This is why dismissing language models as next word predictors was always going to be wrong. Yes, initially it was just the kind of combinations that imitated human output. Then there was that emphasis as I say on those that were predicted to get a thumbs up from humans.
¶65RLHF the energy became as watts fed through compute which were powering all of these permutations. But now that we had the building blocks and we increasingly have the energy what initially seemed to be pathetic regurgitative chatbots was actually the first whispers for telling the marriage of unlimited creativity and unstoppable energy. We had the bricks, we had the power and RL made the skyscrapers. Hence 10 breakthroughs in mathematics. What is fairly clear to me is that in the coming months and years, we will be pretty rapidly climbing the universe's tech tree.
¶66The question is more, are we going to fall off along the way? All of this, everything discussed is all before the social ramifications of near everything around us being created through artificial intelligence. Book deals being retracted because people suspect the author wrote it using AI. Artwork, including the grandest statues being carved using robotics. We're obviously not ready socially for any of this, but then again, we probably would never have been ready.
¶67Let me know, of course, what you think of any of this. If you ever wonder where I've been when there's some big announcement and I haven't done a video, it's often because I'm just processing what's happening myself. But yeah, interesting times ahead. Thank you so much for watching and have a wonderful