Rhetorical question, not really self defeating.

Yes.

OpenAI could have done this same experiment with GPT-4, with possibly even worse results, depending on the quality of the sandbox. Even if the techniques used were not as sophisticated, the natural language output could still easily contain more unhinged sequences of words that lead to the techniques being used.

If the system generates strange conclusions as to when the task is done, or should be stopped, it wouldn't speak to the intelligence inherent to the system.

Not that the techniques used by the LLMs in the actual incident weren't unexpectedly sophisticated, but the outputs of each and every one of these processes could've been read at any time during the run. They just weren't.

Whether or not the AI has intelligence, the one thing that's clear is that it has terrible judgment. I would regard that as empirically proven.

The system that solved Navier-Stokes certainly was not about OpenAI engineers just typing and telling it what to do.

The recent advancement in math with the Riemann Hypothesis by Jared Sumner was basically him saying "you can do it! keep going!", however.

The whole latter part of the post is exactly about that.

This comment reads like Dario's a random tech blogger who's personally submitting these to HN

“Dario” is quite literally a random figure that just emerged out of nowhere one day. It’s not like it’s Eric Schmidt’s next company or Tim Ferris came out of the woodwork or even some Jack Dorsey backed underdog nobody’s heard of.

No, he’s more like an avatar with no history placed on the AI stage by the industry itself.

I’d respect a tech blogger’s opinion on the issue more, even if I reference them casually by first name (which you do with “Dario” btw)

5% is insanely generous for the YouTube example.

Quote from OpenAI in the NYT article: "In addition, since the completion of Navier-Stokes, we have made substantial progress on another Millennium Prize problem."

Which model were you using?

Chatgpt

It's been OpenAI both times though, going ham with poor sandboxing and lax supervision.

Should be treated like a digital cousin of gain-of-function research.

At least when it comes to chess, Magnus Carlsen has stated multiple times that he's been inspired by AlphaZero and adjusted his own style of play after studying its games.