I’m on a strix halo @ GPU-5 with MTP and I get 600 prefill and 30 TG which pushes it into a very usable range. The odd thing is that Dflash2 is really slow for me, like sub 10 TG.
I've also got a strix halo box. 30tok/s would we usable, but I wonder how it compares to the Qwen3.8-Flash-next - I get about 40tok/s running that on Halogen and it feels like using Claude 4.6.
Vulkan, I have never used ROCm on it but have been debating since the latest big update. How is your prefill? Do you hit over 1K? If it’s 1000K prefill, and 40 TG, I might have to try this over the weekend. Also, can you fit 128K without offload the ngram onto SSD?
I have not measured pre-fill, but it's said to be around 1000.
It feels very snappy and unlike my experience with running 27B models the performance stays pretty flat even as the context increases. Unfortunately, we don't know how Halogen is doing this because it's closed source, but I think AMD should offer that guy some $$$ because he's done a lot of good work getting more performance out of Strix Halo.
I've found the opposite on my R9700 (n-max=7, no other speculative decoding like ngram-mod, which I found slows it down). I think it depends whether your workload and system are bandwidth limited or compute limited. I see draft acceptance around 0.55, so 0.55 * 7 = 3.8 tokens per pass, which on my bandwidth-limited card takes me from 30tps to a peak of 80tps on llama.cpp (MTP peaked at ~65tps). I'm also running a Qwen fine tune whose speculative execution is better than the base model.
Strix Halo has lower compute than the R9700 but the RAM is also slower, so not sure what would be the ultimate limiting factor.
I run llama.cpp with -ctv=8, -ctk=q4. Vulkan has better throughput if you're doing single-stream decode, but ROCm has better throughput if you have "--parallel 2" or higher. If supporting parallelism, unified kv cache should be off, especially with Vulkan.
Of course, some of these may be specific to my card so try variations for your hardware. Hermes can concoct a test suite and run some tests for different llama.cpp parameter permutations to find something optimal.
> audio-visual performance close to Gemini 3.8 Flash and overall audio performance that exceeds Gemini 3.8 Flash
Wow crazy if true. I think Gemini's audio capability and multi language was the "selling point" for a lot of people. Other capability also matches or exceeds 3.8 Flash.
They also made a new harness but github link seems to 404.
I tried adding my own endpoint, but the save button was still greyed out so it did not work. Maybe the feature is not fully implemented yet? In about:config, I could add the endpoint and api key, but not the model name.
Mark Gurman seems to get extraordinarily accurate and detailed leaks. I wonder how Apple got this leaky. Pretty much everything about the Duo was known months in advance.
Well, either corporate security got incompetent or they intentionally let select leaks exist to "test the market response". Established companies tend to get more risk averse overtime.
Has anyone have good success using AI generated CAD parts? I’ve been trying but it’s always 95% there, but with all hardware, you need 100% right. It’s often quicker and cheaper for me to do it by hand (but I was a mechanical design engineer for about a decade prior)
Just this morning I used OpenSCAD for the first time. I got Gemini (just the chatbot, no harness or MCP) to help me design a water-bowl holder for our senior dog (she kept knocking it over).
It was probably an easy model to get right, since the only "critical" dimension was the radius of the interior. But I was able to tweak the numbers on the support length & some other details relatively easily, and I had a working solution 2 hours of print time later.
(I had a CAD class in highschool, but haven't used it since).
I suspect this may be another case of "LLMs are mainly good at things you're bad at."
assemblies and tolerances are where AI generation quickly gets hard. but, for simpler things, it can be surprisingly good, especially for people who has no CAD knowledge. it certainly might feel a bit like brute-forcing through the LLM - but this also applies to vibe-coding..
reply