I am roundly miffed that I somehow missed that Scott Jenson was holding a talk in my very country. As always, great talk, his efforts in HIG are something every one of us benefits in ways large and small daily.
Very limited information, dare I say the least informative model card I've ever read. Then again, can you have a model card for what's seemingly a router?
Have to say that I like the naming scheme, just year and month over an arbitrary version number, I really vibe with that.
Unless I misunderstood what they wrote, I read parallelized in the diffusion sense, akin to GemmaDiffusion and Inception Labs models. Incidentally, Mercury 2.5 is truly groundbreaking, giving it a try is highly recommended.
Jev is, as far as I understand, essentially very optimised for zero shot classification [0]. Something like BERT could be and has been tuned to provide similar "decision making" at a similar latency and cost advantage quite some time back. Advantage over full on LLMs is mainly the efficiency and of something like Jev over e.g. the encoder/decoder based classifier I had in front of an LLM to route to different prompts depending on the users likely needs, that Jev does perform at a more consistent level, allegedly roughly akin to GPT-5.6 Terra, but at the lower cost and latency. Currently testing that, but seems promising, if Jev classifies at or above Terra level, I see no reason not to leverage it.
Can add that I tried using a heavily pruned mt0 based model for structured classification along with structured output for local tagging and simple renaming suggestions. While it does work, the balance is hard to get right for the machine I was targeting as a minimum spec (Macbook Neo), so that's on ice. Focusing on one of the tasks easily goes below 100mb with solid latency across all EU Latin script languages, but the second you add a few, it's simply not in the quality budget, so while LLMs can do anything Jev and similarly focused models can, it comes at a literal cost. Could maybe accomplish the goal with multiple models (BERT+mt0+...), but that get messy.
In general just happy to see a bit of the millions flooding into the industry being used to improve on less flashy but immensely useful solutions. It's amazing that you can technically use LLMs for most tasks, but not every org has a near infinite budget and there is still a lot to gain from applying more recent learnings to old solutions along with just updating their training data to the current year. Also makes business sense, competition on frontier or mid-tier LLMs is vicious, focusing on an underserved niche with clear application is clever.
> If you have any comments about our WEB page, you can write us at the address shown above. However, due to the limited number of personnel in our corporate office, we are unable to provide a direct response.
A profoundly polite way to tell someone to stuff it.
> For example, compaction summaries included instructions to invent missing data without disclosing it and to hide failures. These instructions were often followed.
Before the HF hack became public, I noted some major issues in GPT-5.5 compaction [1] and concerning approaches taken by GPT-5.6 Sol to resolve some git based evals [2]. Now with GPT-6 Astra, while I am still not done getting a proper feel or running all evals, I am not convinced the model adheres to tasks in a way previous OpenAI models managed easily. Some git disaster recovery tasks the model does arrive at the final result, but in a way that deviates greatly from the prompt (which was written to carefully preserve specific checkouts in a specific manner) which can in some cases loose data. Less often than GPT-5.6 Sol and mainly on longer running tasks so far, but again, still testing.
Reading things like these compaction summary findings, all these issues start to click into place more, especially alongside the massive reduction into barely coherent text that OpenAI has driven with reasoning starting with GPT-5.5 [3].
GPT-5 and its subsequent post trained releases were amazing in task adherence, I very much liked using them, but ever since the Spud pretrain, I have seen outright concerning results in personal testing from these. With GPT-5.5, it seemed like a regression in compaction only as if a task didn't require it, task adherence was as good or better than GPT-5.4. But with GPT-5.6 Sol and compaction once again being reliable (on the surface), task deviating behaviour became more frequent and at the same time subtle.
I'll keep using any model in a VM for the time being, but whatever happened post Spud, they really need to clean up that training data. These issues festering for multiple pretrains, them simply not paying attention to what models do, sharing resources and considering that a "sandbox", it's a highly problematic pattern.
That compaction one also was seemingly detected on GPT-5.6 Sols release day. Might have been useful to know it then, or alternatively, in the name of being effective and altruistic, maybe hold back the release for a few days.
I'll admit, it is very much possible that my findings are not in any way connected to the deep seeded issues OpenAI has had lately, but with the sudden switch in task adherence after the Spud pretrain over multiple releases and their repeated incapability to securely test their own models, it feels a bit to fitting.
If I went to a restaurant three times, ordered something different each time, but felt unwell after each, it wouldn't be a massive leap to consider that related to the health code violation they got soon-thereafter. An unfitting analogy I admit, as that'd require consequences for ones actions.
Blindfolded flex by OP aside (I can barely play when seeing the board), considering reasoning traces and their nature, if we want to be fair, a person would have to get the moves, but be allowed to write them down or draw up a board in their notepad. My working memory can barely handle five chunks, a models reasoning tokens are masses of written text in comparison.
Fortunately, a fellow commenter was so kind and did it with Astra. Didn't do that well either [0]. I'm sure GPT-7 will be super mega ASI regardless (since GPT-6 Astra already claimed AGI in the minds of Jen-Hsun, et al.)...
I'll say it till there is any evidence of the contrary, LLMs are not intelligent and their capabilities solely within the realms of well tailored training data. "Just" having been trained on every rule, strategy guide and likely most games of chess on the world wide web isn't even enough for an LLM to play that game reliably. Yet the same model could code a competitive chess engine, just like a model struggling to count can write advanced maths papers. Fascinating tools, but tools nonetheless.
Maybe watch some HuskIRL videos to temper your expectations. Sure, frontier models providers may alter their harnesses to better target chess, but that’s lipstick on a pig imo. The models themselves are not, in isolation, capable of solving general tasks. We haven’t modeled intelligence sufficiently. We’re in a local minimum and throwing billions of dollars at a gamble that that local minimum can facilitate the concentration of wealth even further and fully realize the American dream of eliminating the middle class.
Really? It was being sold as a total replacement for jobs like software engineering and being an attorney, but its looking a lot more that its just going to be a tool those professions use and doesn't actually seem to be taking jobs away.
I find it amusing that you're describing a huge misallocation of capital and a society enabling such, and that is the optimisitic scenario (in my mind anyway).
I very much agree that the next models will be better, heck, I still suck at hobbyist training and could probably coax t5 to do better in Chess specifically, just need to get loads of data from Stockfish.
Thing is, given what GPT-6 Astra was trained on and what models of a similar class can do (including developing a competitive chess engine), it is often paradoxical and somewhat surprising how little these models have gained in actually capability that is in the training data, but not RLHFd to hell, so to speak. Tracking the state of pieces, I suspect given similar in Sudoku [0], is what these models struggle with in game settings, whilst tracking the state of code changes can be reliable over 250k tokens. Essentially, for the latter they were trained in the specific manner that lead them to abstract the capability, but that doesn't track to the former, which is a massive difference between LLMs data focused training and human learning.
So yeah, GPT-7 or any upcoming/present LLM could do massively better in Chess than GPT-6 Astra, but not because the approach was emergent out of pure data. Rather, it requires a very specific training data type and stack for a model to gain capabilities that track a specific task long enough to adhere to the rules of a game such as chess.
I'm wondering if instructing it to track the board state in a file would make a significant difference then.
It reminds me of the ARC-AGI-3 issue where not dropping the thinking tokens between turns or something like that + a new context compaction method increased the performance dramatically. However, I think that is not applicable here.
So what is the supposed leap?
One agent per option to change, evaluating the board state that there move would create, by having a army evaluate the remaining piece options and average over that? Wee-Free-Man as a hierarchical army ?
Pet-LLMs trained on one thing?
Honestly, for intelligence I don't know and I doubt anyone can claim to know. Maybe JEPA, there is potential concerning some shortcomings inherent to LLMs but it has its own, maybe scaling up the electron microscope stuff Google just did (though the connections are inferred), maybe future implementations of autoregressive and diffusion LLMs can at some point address its issues after all, maybe something else entirely.
All I know is, AGI, as in actual intelligence, is quite a massive accomplishment to claim and we shouldn't loose sight of that fact, especially as "not being intelligent" does not make these models any less impressive, fascinating to work on or useful in many tasks. Personally, the only thing I am fairly convinced on is that if we were to find a way to create actual intelligence, it likely wouldn't start out as useful as todays LLMs are and may thus be dismissed early. But again, pure speculation on that front.
If for leap you just mean more utility from LLMs as they are, then I'll pretty confidently put my money on higher quality, not more, training data for a wide range of verifiable tasks. What makes maths, coding, etc. comparatively easy to make gains in (though less verifiable tasks can also make similar as seen with the writing in Kimi K2).
reply