Yeah, I think you can't just throw money randomly at problems and expect results unless you know a line of attack that can get you all the way. OpenAI chose the line of attack only after it became known to them via rumors. They "front-ran" the researchers.
Yes. What the headlines hailed as an AGI discovery the facts show more to be someone spending years mining for gold, rumor gets to OpenAI that there might be gold in this specific place, they mine there and instantly discover gold, then tell the world they’ve developed the worlds best gold finding/mining machine.
Separate from all the allegations of more nefarious actions and ethical issues, that’s the most charitable version of what happened here.
they threw it on all the millenial math problems (I think there are 6 at this point unsolved, well, 5 now).
And according to them at some point they saw that one was close to being solved, so they pointed all the agents at it.
The same thing happens to humans - at this time there are no simple problems left, so solving the hard ones requires using prior knowledge and attempts at solving things.
Yeah but the one they decided the AI was close to solving may have been so because the researchers' progress on this problem became part of the training data for that AI...
Almost but not quite I think. You can throw money at parts of problems. I think it's helpful to think it kind of like supercomputer MD/MC or electronic structure calculations. A tool that can get you valuable answers but not necessarily aid understanding. Simulations can be used to aid understanding also, and are integral to theory development. In the same way the approach to this result is.
Plausible deniability - The line of attack is in their sessions/prompts data. Just make the prompt pointed enough that the search space is tractable and use your ginormous compute.
> "Of course we don’t know whether that is true"
Yep. Who is verifying these claims? We all know how trustworthy Altman & Co are.
“relying on” is misleading here relative to what the researchers have said.
If I write a book and pass it through a spelling and polish checker, I still wrote the book and its core IP. I didn’t “rely on” the tool to create the IP.
it’s much more like you come up with the premise and someone else writes the book. the released prompts for other foundational problems (like unit distance) prove that.
The tools the researchers used though was much more than an spellchecker, because spellcheckers don't come up with chains of reasoning for the arguments in the book. The LLMs did in the case of the Navier-Stokes problem.
that's not how AI was used in this case. It's more like a professor with assistants.
Professor says the assistants - why don't you dig in this direction, I have a hunch it might produce something valuable. And AI assistant does just that, proving or disproving a hunch. This would take the professor a lot of time if doing by themselves.
Yes, and keyboards also save a lot of time over handwriting. Numerical methods and proof engines save even more time. LLMs are just another tool in the kit.
Keyboards don't suggest chains of reasoning or words to type. When I press the K key, I know exactly what will happen. It's just a translation layer that gives an output known ahead of time and thus does not impinge upon the creativity of putting words together.
A better example would be playing chess against a player slightly stronger than me and using a chess computer to suggest some good moves. I could win, but it certianly wouldn't be just my brain that wins. It would be an amalgamation of my brain with a machine that suggests good moves.
This is straight up misleading. When you press your K key on a touchscreen, your keyboard program may decide you meant to press the neighboring L key (by dynamically inflating the collision geometry on it) because it was statistically far more likely that you meant to press the L key next.
This likely doesn't happen exactly on an analog keyboard, but then many text-processing environments that do the same thing in post. My keyboard just edited 'yuor' to 'your' even though I successfully input the prior string.
The researchers were driving prompts and trying to actually do math.
The OpenAI effort was a pure brute force attempt. I'm not even sure an LLM was actually involved. I think they just used their hardware to run the matrix multiplies required by the search for a counter example. Perhaps some clever approach guided the search but that seems to be about it.
You can’t do anything novel with these models from scratch and let it fly. I’ve observed something over the past few months
Work on something novel -> llm is kinda useless and low value-add -> Keep at it and in the process feed it more information -> keep doing this periodically -> a few months go by and you realise the model outputs are almost like-for-like regurgitations of what was inputted in some prior period.
Once it’s accumulated new info can it produce something automated that is somewhat useful? Sure.
But by itself - absolutely not.
I clearly see humans will be needed - the best ones that is. For ‘rote work’ and stuff that is not IP sensitive firms will be ok with employees putting that as inputs into models.
But I’d wary about trusting the labs. They will push the letter of the law to the max.
Personally I’ve stopped doing anything novel with these models. If I do use a model on something adjacent but not totally novel I have to craft the inputs in a strategic way not to give much away.
I’d wager firms will soon realise this and that growth rate of revenues of the frontier labs will become questionable. The economic cost that firms have brought out thus far is only financial. There’s a whole bunch of other costs people aren’t talking about.
Agree all. And as the revenues become questionable, the frontier labs practices will necessarily become (more) questionable. Vicious cycle.
To avert that dynamic, the frontier labs must deflect and otherwise act to prevent this controversy from breaking through. Both to the general public, but also more specifically to the firms' decision-makers. All of whom are generally aware of the IP issues, and some of whom are aware of what happened with Cursor and Figma, but with few exceptions have not yet themselves acted to protect their property.
This doesn't follow for me. There are what, Dozens or Erdos tier problems that got solved with no progress for decades? How does that factor in to your view?
IIUC the argument is that while unsolved there was much work done on them that shows up in the training data. The idea being that the LLM is limited to a small amount of inference over externally supplied data.
OAI doesn't need to mention Buckmaster's name directly in a prompt. They just need to select a basket of sessions that is guaranteed to contain Buckmaster's and then direct the LLM to attack only a specific method/angle. This is trivial to do while maintaining plausible deniability about not using his work.
OAI started working on this only after they found out it was close to being solved. They threw a team of researchers who spent sleepless nights + a ton of compute. This is not exactly healthy academic competition - it's like if you spend a year hunting for oil fields and finally find a very promising area to be explored, only to find that Exxon tapped their entire exploration unit to go all in and and find it overnight just to stake claim to the discovery. Tao said it right - math should not be treated as a non-renewable resource to be mined.
but a very large part of the whole model was trained on work in a manner the authors didn't consent to, the "for research purposes only" datasets of the entire Internet, etc
and you can argue this is "fair use" or whatever, not the point now, the point is that it definitely makes those accusations no longer "baseless".
in addition, it is not given freely to the world, it is the knowledge of the Internet/WWW being sold back to you as a subscription service. it's not free. and it's not even "given", because they can (technically) turn off the tap at any moment and you don't have it any more.
Humans do this all the time. You watch a YouTube video and subconsciously choose the same colour palette. They hear a rumour that it's a solved problem, i.e. they were pointed in the right direction that is all.
I mean some companies glean insight into new products merely by asking other people what they do for a living.
People need to get over this ownership thing, it's being taken too far. Humans benefit from the efforts of others simple as that.
it seems like OAI tried to share, but didn't want to share with an Ant employee. a bit childish, but understandable to want to avoid a headline "Anthropic researcher solves Millennium problem"
it seems like Buckmaster got one-upped and is upset. understandable, but I find their reaction childish as well
> OAI tried to share, but didn't want to share with an Ant employee
Why does OpenAI get to dictate who Buckmaster can claim co-authorship with?
> I find their reaction childish
OpenAI may have, with full plausible deniability, taken Buckmaster’s work and passed it off—in substantial part—as their own. (Fitting into a fact pattern of them having tried to do the same with Apple.)
There is a material takeaway for anyone who does creative or otherwise unique work from this. (Which is unfortunate. Whatever happened here, AI clearly accelerated the discovery process.) For anyone else, I agree it’s just drama.
More like OAI needed someone like Buckmaster's stamp of approval. They know they can't get Tao's. Now they won't be able to get Zoroa's. You have to hope the rest of the guys they didn't know to cite will readily give up honour and dignity
It doesn't seem like there's anyone at OAI who knows much about the problem they are solving. Seems like CS theorists or algebraists* trying to own the pros by driving a car that's beyond their skill level. For one, they didn't cite the guys that B&A based their work on.
*It would be most fair to say there are no analysts on board, nor are they likely hire any soon; those are the least impressionable people in math. Applied math PDE elves who hadn't already left on the world-model boats would have jumped off around the time that eg Ilya did because they wouldnt have been able to stand the three Bs pretending to be experts in fields they imagine to be "adjacent".. like Public Relations
> OAI didn't try to claim Buckmaster's work as their own
We can't say this until we have more information on what the OpenAI researchers prompted their model with and to what degree Buckmaster's work was fed into its training data, either by Buckmaster himself or by his co-author.
Apparently Tristan was working on this problem for years, with AI providing help. Vs OpenAI spending a week with their new model, potentially having access to Tristan’s work.
I'd think nothing is "safe". Anything you say can and will be used by the LLM if it has enough statistical similarity to the prompt. Call it "Ma Random Rights"
I don't think OAI should be given the benefit of doubt. They are doing the research equivalent of front-running. Knowing where to look is one of the main challenges in research. Tristan's argument from his essay was that it is hard to brute force with a vanilla prompt (even for seasoned mathematicians) unless you knew very specifically what to mention i.e the search space would have been intractable even for OAI's compute budget.
"deidentified data" isn't much to go by. Say I prompted the internal model this way - "Hey there's a solution to a unsolved problem X. The solution uses a less known Method Y so don't bother wasting time with the usual methods. Take papers A, B and C as references. Oh btw, here's the last year's worth of data of all prompt sessions that mention this problem. Pay special attention to the ones that mention Method Y and sub-keywords Z,W".
This is obviously all speculation but the timing is very suspect. If OAI actually did this (and I suspect whatever they did is pretty much close to this), I think it is highly unethical.
> Tristan's argument from his essay was that it is hard to brute force with a vanilla prompt (even for seasoned mathematicians) unless you knew very specifically what to mention i.e the search space would have been intractable even for OAI's compute budget.
This is a bad argument. This is clearly not how it works. And unless Tristan is some truly alien-like savant (and maybe he is), what's necessary to initiate the AI's work already exists in countless published research papers and not exclusively in his head or notes. AIs can survey the sum total of all prior work on a problem and discern reasonable paths for inquiry.
Tristan is acting as if he's working off of an outdated model of AI, similar to primitive chess-playing models that winnowed the search space much more deterministically. If someone this intelligent truly doesn't get that this is not at all what AI is anymore, then maybe there's no hope that we ever understand it.
But I think he does realize this and he's flailing about for counterarguments from a place of bitterness and dejection, accepting even those that are too weak to be defensible. And that is very human and even forgivable.
Are you saying there is no search space intractable to LLMs? That wouldn't be possible. AIs are statistical pattern-matchers on steroids. The prompt is key to getting anything useful out of them. They are incredibly useful and major game changers but ultimately that does not alter this fact. People (including OAI) have already tried to solve Millenium Problems with it. That OAI woke up last week and suddenly decided that throwing their researchers armed with millions of compute on one particular idea to a problem is highly suspicious in itself.
Even if OAI had zero data from Buckmaster's sessions, this is in very poor taste and highly unethical. You are front running a researcher just to be able to say you did it first? Tao is right - OAI is treating math results like oil. This is the like Exxon getting a whiff of a massive oil field and racing to the punch by deploying their full crew.
> Even if OAI had zero data from Buckmaster's sessions, this is in very poor taste and highly unethical. You are front running a researcher just to be able to say you did it first? Tao is right - OAI is treating math results like oil. This is the like Exxon getting a whiff of a massive oil field and racing to the punch by deploying their full crew.
I want to be clear that I agree with this view and with Tao more generally. But we're all just yelling at the wind now.
Try to increase the complexity of the work you're doing. It's not that AIs don't makes mistakes but they are able to pattern match to larger and larger pieces of the problem as more people use it and more iterations of the model are released. So your challenge is now in steering it in a way that these errors are minimised while reducing bloat.
Try to think of yourself as a professor who's trying to come up with a problem statement worth solving. The AI is your "lab" that will help you run experiments.
>> Test it yourself, GPT 120B OSS is cheap and available. BTW, this is why with this bug, the stronger the model you pick (but not enough to discover the true bug), the less likely it is it will claim there is a bug.
I guess this is the crux of the debate. All the claims are comparing models that are available freely with a model that is available only to limited customers (Mythos). The problem here is with the phrase "better model". Better how? Is it trained specifically on cybersecurity? Is it simply a large model with a higher token/thinking budget? Is it a better harness/scaffold? Is it simply a better prompt?
I don't doubt that some models are stronger that other models (a Gemini Pro or a Claude Opus has more parameters, higher context sizes and probably trained for longer and on more data than their smaller counterparts (Flash and Sonnet respectively).
Unless we know the exact experimental setup (which in this case is impossible because Mythos is completely closed off and not even accessible via API), all of this is hand wavy. Anthropic is definitely not going to reveal their setup because whether or not there is any secret sauce, there is more value to letting people's imaginations fly and the marketing machine work. Anthropic must be jumping with joy at all the free publicity they are getting.
In the Anthropic Mythos model cards they explicitly remarked that they didn't want Mythos to be specifically good at security. They trained it to be good at coding, and as a side effect the model is (obviously) good at security. This what happens with flesh hackers too, mostly. Hackers are very good programmers, as a side effect they understand systems well enough that their understanding has security implications.
You don't need to trust anyone. GPT 5.4 xhigh is available and you can test it for $20, to verify it is actually able to find complex bugs in old codebases. Do the work instead of denying AI can do certain things. It's a matter of an afternoon. Or, trust the people that did this work. See my YouTube video where I find tons of Redis bugs with GPT 5.4.
I did not claim or deny anything. You cited the model card, I just pointed out that this is no reliable source. If you have better sources, like your YT video, you should cite those instead.
You are claiming something: that the model card is not reliable, therefore it's as useful as nothing. Sowing doubt without a possible solution adds little value to the conversation. Moreover, your rebuttal is unsubstantiated.
Guys, think about all the security vulnerabilities you're aware of; now, think about how many of those you know how to technically reproduce. Now imagine that you actually don't know how to reproduce most things and you're never actually be able to judge the result.
Well, just cause these are all AI people doesn't mean they verified enough of the output of these models to actually provide the significant security implications they're advertising.
And overfitting benchmarks can easily be gamed. Yet here we are with the top HN comment on the HN Mythos thread outlining it's benchmarking performance gains.
The whole discussion started out as an attempt to disprove/verify anthropics (model card) claims.
He also transfers the logic of their claims to the actual real world. You can say that model cards are marketing garbage. You have to prove that experienced programmers are not significantly better at security.
> You have to prove that experienced programmers are not significantly better at security.
That has not been my experience. It's true that they are "better at security" in the sense that they know to avoid common security pitfalls like unparamaterized SQL, but essentially none of them have the ability to apply their knowledge to identify vulnerabilities in arbitrary systems.
An expert level human doesn't have to be expert at every programming category. A webdev wouldn't spot a use after free. A systems engineer wouldn't know about CSRF. That is if both don't research security beyond their field. Requiring a programmer to apply their knowledge to an arbitrary system is asking too much. On the other hand and LLM can be expert level in every programming field, able to spot and combine vulnerabilities creatively. That is all pretty hard and I don't think an security expert with vast knowledge would say "that's easy".
My point is that more experienced programmers are better at security on average, not that they are security experts.
I would think pwn2own competitions would signal the opposite. I'm consistently and often amazed at how a unique combination of exploits can bring a larger exploit and often in ways that most wouldn't even consider. I think it takes a level of knowledge, experience, creativity and paranoia to be really good with security issues all around as a person.
> essentially none of them have the ability to apply their knowledge to identify vulnerabilities in arbitrary systems.
I've found it to be the opposite. Many of them do have the ability to apply their knowledge in that fashion. They're just either not incentivised to do so, or incentivised to not do so.
The missing part of their intended meaning is "skilled hackers". Unskilled hackers are everywhere, and they're bad at programming, but so are unskilled programmers.
Yep. Some are while others are more or less forum leeching and exploiting known risks and use tools.
But the some that really find certain bugs are really exceptional. Almost all are very hardware prolific and do assembler stuff. This alone is an impressive feast, I still enjoy 6510 and M68000 assembler here and there as a former scener who mainly coded demos and here and there improved games (so called trainers) or cracked few.
To be honest, the assembler guys scare me always because with it you can poke a whole in almost anything. No one in his sane mind uses assembler on x86 for professional development besides few special cases. But Python etc serve many MB of executable code for the abstraction and 20 bytes just kills it…
Yes, that does track with my personal experience. More context, more params and no quantization is probably it. But my hunch is that all the training data they've been getting in the past year also plays a part here. More than any other lab, anthropic's focus on coding right from the beginning gives them access to the best training data (several githubs worth). Most of this code comes with human feedback and anthropic even has data on how many went to production, got reverted etc. No need to pay for human labeling when your customers are doing it for you. This is their secret sauce.
Mythos isn't restricted for marketing purposes - that would be incredibly dumb because Anthropic would be giving up first mover advantage for next gen models.
It's restricted because it's genuinely good at finding vulnerabilities, and employees felt that it's not a good idea to give this capability to everyone without letting defenders front-run.
That's it. That's all there is to it. It is not some grand marketing play.
>It's restricted because it's genuinely good at finding vulnerabilities, and employees felt that it's not a good idea to give this capability to everyone without letting defenders front-run.
It's a possibility, but it doesn't eliminate the possibility that it's hype. If these claims were indeed serious, they would submit it for independent analysis somewhere.
This isn't some crazy process. Defense contractors are required to submit their systems (secret sauce and all) for operational test and evaluation before they're fielded.
> If these claims were indeed serious, they would submit it for independent analysis somewhere.
They have. 40 different companies that have all committed resources to patching their systems based on vulnerabilities found by Mythos. One of them, Google, is a frontier AI lab that pointedly did not say that their own models have found similar vulnerabilities.
> Defense contractors are required to submit their systems (secret sauce and all) for operational test and evaluation before they're fielded.
Does this look something like having 40 separate companies look at the outputs of the system, deciding that it’s real and they should do something about it, and committing resources to it?
At some point, “cynicism” is another word for “lalala can’t hear you”.
Another cross-check I've run is, are the claims Anthropic is making for Mythos that out of line with the current status of AI coding assistents?
To which my answer is clearly, no, not even remotely. If Anthropic is outright lying about what Mythos can do, someone else will have it in a year.
In fact the security world would have to seriously consider the possibility that even if Mythos didn't exist that nation states have the equivalent in hand already. And of course, if Mythos does exist, nation states have it now. The odds that Antropic (and every other AI vendor) isn't penetrated enough by every major intelligence agency such that they have access to their choice of model approach zero.
I wonder about the overlap between people being skeptical of Mythos' capabilities, and those who are too skeptical of AI to have spent any time with it because they assume it can't be any good. If you are not aware of what frontier models routinely do, you may not realize that Mythos is just an evolution of existing capabilities, not a revolution. Even just taking a publicly-available frontier model, pointing it at a code base and telling it to "find the vulnerabilities and write exploits" produces disturbingly good results. I can see the weaknesses referenced by the Mythos numbers, especially around the actual writing of the exploits, but it's not like the current frontier models fall on their face and hallucinate wildly for this task. Most everything they produce when I try this is at least a "yeah, that's worth thinking about" rather than an instant dismissal.
Sure, I am not precluding the possibility that they've trained a genuinely great model. All I am saying is that the "this model better than that model" is moot when on one side you have model weights, and on the other side a whitepaper and some accompanying comments on the danger.
I'm not that old but have been here long enough that I remember when GPT-3 was considered too dangerous to release. Now you have models 10x as good, 1/10th the size and run on 8GB VRAM.
I don't think you can say this with confidence, outside-in. It's not just about safety. The additional unknown is cost - I don't just mean API cost, but fully loaded cost for a given task. Is the model cost effective for tasks such that it has product market fit?
We don't yet know if Mythos was a level shift in the capability/cost frontier, or a continued extension of the same logarithmic capability/cost curve.
I don't doubt it, I just mean the decision to release/not release generally may also be informed by the commercial/economic viability of the model for general usage patterns versus extremely high value patterns like vulnerability assessment
That safety stuff is almost always quacks whose job it is to exaggerate LLMs at their non profits or marketing hype that "our models are so powerful you should fear them". Then they release them and the world moves on and adapts.
Mythos will benefit security in the long run more than hackers, if it can do what they claim. And there's nothing that will stop an LLM like it from being released in the near term so it's very likely just resource constraints or marketing
it seems likely it's both a better model to some unknown extent and doing this "we have to give it to the defenders first" thing is super great marketing material. it seems an entirely natural marketing campaign "announce that we can't even give the model to everyone at first, it's so great!", plus there's some truth to it, even better.
unless you are an employee at anthropic and shouldn't be talking about any of this at all, there's no way to know what the model's capabilities are.
How do you know? If you have access you are not unbiased, otherwise you cannot know by definition.
AI companies routinely claim that something is too dangerous to release (I think GPT-2 was the first case) for marketing reasons. There are at least 10 documented high profile cases.
They keep it secret because they now sell to the MIC with China and North Korea bullshit stories as well as to companies who are invested in the AI hype themselves.
GPT-2 was obviously too dangerous to release at the time! It's OK-ish now, when the knowledge that AI can produce arbitrary text is widely shared. It would have been a disaster for scammers and phishers to get GPT-2 at a time when almost everyone still assumed that large volumes of detailed text proved there's a real human being on the other end of the conversation.
Maybe they did use small models but you couldn't make the front page of HN with something like this until Anthropic made a big fuss out of it. Or perhaps it is just a question of compute. Not everyone has 20k$ or the GPU arsenal to task models to find vulnerabilities which may/may not be correct?
Unless Anthropic makes it known exactly what model + harness/scaffolding + prompt + other engineering they did, these comparisons are pointless. Given the AI labs' general rate of doomsday predictions, who really knows?
papers are always coming out saying smaller models can do these amazing and terrifying things if you give them highly constrained problems and tailored instructions to bias them toward a known solution. most of these don't make the front page because people are rightfully unimpressed
The word "profound" is a bit overused when it comes to movies. I agree that The Battle of Algiers is an excellent film, one of the best ever made even. One Battle After Another is also excellent but it is not really political in the way the TBoA is. It uses a political setting very effectively in a chase thriller. A movie like The Parallax View is a better comparison. That movie used the post-60s paranoia very effectively in a great suspense thriller.
reply