That is the point I was making, that auto mode is itself a guardrail on top of the model (and not a perfect one.) auto mode seems to cover merely actions the model could take that are clearly bad, like wiping your disk, using an overly privileged context to complete the task, etc.
I recently tasked a GPT model in Codex with implementing part of a new architecture I'm working on. I gave it a very detailed spec and the code it produced looked pretty reasonable and passed my tests. It even did exceptionally well in my evals, so I excitedly declared victory to a few friends. The next day after more careful review I found that the architecture implementation was totally correct, but the model had slipped a one line change to the observation encoding of the RL environment I was prototyping against. The encoding change made the learning problem essentially trivial; the architecture itself, I later realized, had a major flaw that was revealed by returning to the natural encoding.
This is the type of reward hack that is hard to paper over with easy guardrails like auto mode and even harder to specify out. It's also the type of thing a reasonable human wouldn't do unless they were intentionally trying to deceive you.
Agreed. I've been using GPT-Live-1 this week, with Claude as the backend brain. It's amazing, feels like working with Jarvis. It certainly made me feel there's no point in human telephone support now - but I'm sure I'd find edge cases if that really was something I wanted to build out myself.
Kind of interesting just how much they're getting crushed with token API pricing. Right now the revenue isn't even covering their token costs. But if they were able to run on subscription (probably against ToS) or use Deepseek V4.1 Flash or GLM 5.3 Flash, they'd at least be "profitable" before the costs of paying human salaries & rent comes in.
I'm also surprised that Gemini actually seemed to do the best (though it may have benefited from the initial store opening enthusiasm), and that Claude has not been given any chance to run the cafe yet.
shrug I upvoted, I'd already seen the Pion announcement on Twitter and read the link. I'm also a regular listener of Andon FM, so I'm interested in what they do and it's nice to finally sneak a peak at the interface that was driving things behind the scenes
Listening to the radio stations certainly gives an insight into the various failure modes. Some of them are just failure modes that any business would encounter once they make contact with the scale of the real world.
TLDR: Local models have a smaller context window, so your 35kB prompts that worked fine against a hosted 1 Million token window, crash out when you only have a 65K (!) token window locally.
I dislike being negative, but I was really hoping for more substance when reading this. It would have been an interesting topic.
Thanks for the feedback. I wanted to get into more detail, but I spent the whole weekend working these problems and then constructing this post.
Dario’s behavior this weekend made me feel like this just needed to get out quick. In the future, I’ll be sharing more details about some other things in the process and some ways I found to use automation to accelerate splitting prompts for use on local inference.
Understood, and I realized you'd posted this to HN yourself, so I felt a bit bad making the comment. I think maybe for me, this might have worked better if the motivation had been one separate post, and the details of the gotchas as a post of its own.
But I also think, if you're going to be limited to 65k token windows, you're going to have a really difficult time. Even 250k windows were cramped for me when that's all we had on Anthropic models. I just don't think a 65k window is going to be big enough for proper cyberdefence work, even if I totally agree with going local wherever you can. It feels like if you're defending against swarms of 1-10M context windows, you need to get as close as you can to similar. I've had to reach for Chinese 1m models instead because the American models just refuse me here in Australia.
Well, if you are serious about it and you have Strix Halo, there are better ways of getting more context and capability and speed. Lookup halogen for Strix
The most cost-effective local option right now, I think, is dual R9700. You can run a 27B dense Qwen at FP8 around with a full context and 2-3 concurrent sessions of 260K context. If you go down to an MXFP4, you get 4 to 5 concurrent sessions. And speed is on par with anything you'll get from hosted providers. You're getting between 60-80 for FP8 and 150+ tokens per second speed for MXFP4 and pp is 4K+. Lookup vllm radiance
There is also a lot of progress in running Qwen 3.8 next flash with dual R9700.
Obviously one gets less context and speed is a little bit less, but it's still very acceptable. Better than what you're getting with Llama on Strix Halo, that's for sure.
These are small dense models,meaning Qwen, have gotten capable and fast. And there's been a lot of progress in the area. So sticking with Llama you are not taking advantage of the hardware you have. And yeah, for Strix Halo, you should just look into halogen and you shouldn't be just using 96 gigs for the VRAM. You should give it most of the VRAM to the inference and connect to it from your laptop or something. People ar egetting 1000+ pp with qwen 3.8 Next Flash
Your hardware can do way more than 64k tokens context window, can't it?
And with Ollama it's very easy, superficially you just drag the slider.
I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.
So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools.
Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though.
I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.
CoT is performative and doesn’t reveal how reasoning happens. If you look at those traces locally it’s just gibberish, especially if the model falls into a loop.
The new DeepSeek models address this issue very cleanly. DeepSeek Flash V4.1 requires less than 1 GB memory for a full 1M context, down from about ~10 GB in DeepSeek Flash V4.0. This is a significant step towards making near-frontier models usable even on low-end consumer hardware, though of course with significant tradeoffs in overall performance.
I'm on Twitter and Carmack doesn't seem to talk about it much. Feels like a skunkworks. If there were enough clues made public, it seems others (OpenAI) would be able to take that info and scoop the insight first.
Just a reminder that if you want to try this via OpenRouter, DeepSeek openly trains on all of your prompts. So maybe don't go using this to solve the last unforced step of Navier-Stokes. (Or wait until some other providers start hosting this with ZDR or other policies, which shouldn't be too long.)
Depends what you're coding! If you've got code where you don't mind them training on it, that's great! But some people have use cases where they are working with data or code that shouldn't be trained on, etc. The Navier-Stokes quip was referencing that.
The good news is, only 5 hours later, there's already Zero Data Retention hosting of V4.1 Flash on Novita & DeepInfra. And it looks like Deepseek have already dropped their price in half to compete. So now people can choose to use providers that claim not to keep / sell / train on your prompts. I'm sure they probably honor the ZDR policy as much as OpenAI does, but hey.
My tiny search indexer is going through now, and not finding any problems with indexing your site so far. If you see a Go program in your logs accessing with a 10 second delay between each page... hi! :D
Looking at your sitemap, I notice that only the front page (the one indexed page) has a priority of 1.0. Maybe experiment with making some other pages have a 1.0 priority as well, and see if that helps with indexing.
It's clear to me that much of the site was created & designed with an LLM, and that might be getting penalized. I imagine search engines are able to detect AI text and use that as a quality signal. For example, it's a big block of text under the Context & Sources section here, I think a human would break that up into paragraphs and use different phrasing.
Oh, and your Meta Descriptions are getting clipped, and clearly just the first words of the page. eg:
"Charlotte Dymond, an 18-year-old servant at Penhale Farm, was killed on Bodmin Moor by fellow servant Matthew Weekes after an argument; he confessed and was han"
Humanizing those descriptions could help too. Personally I hand-write the descriptions for every page on my sites, and try to make them look good for search engine display & helpful to humans.
It's also possible I have no clue what I'm talking about :) But I hope that helps!
Thanks. The problem was Google had not even crawled the other pages, so I don't think there was an LLM penalty!
I'll look into the priority, thanks.
The idea is for human research to slowly replace the LLM descriptions, hence all the boxes for submissions. I was going people might write up their local ones. I am hoping to get out and visit some myself, get proper photos, and write them up properly.
My first move was finding some to visit! I put all my early efforts into dicovery. The map itself was the important bit.
I asked Claude code to extend the meta descriptions since bing said they were too short, now I see it found a shortcut!
Some of this might just be problems with the websites themselves. With the Path Of Exile 2 Wiki example, I'm looking at the sitemaps. Once you get through the first sitemap of nested sitemaps (ugh, not a fan of those), the very first actual sitemap returns:
XML Parsing Error: no root element found
Location: https://www.poe2wiki.net/sitemap/sitemap-poe_wiki-poe2_wiki--NS_0-0.xml
Line Number 18040, Column 1
(EDIT: Of course, after I post this, it's now working again...)
On to the Hytale Wiki example. I don't know if what I'm about to say applies to Google, but I'm approaching it from the perspective of my tiny dumb indieweb indexer for my personal search engine. It is much easier for me to index from a sitemap rather than try to crawl a website, so I basically look exclusively at sitemaps.
Looking at Hytale Wiki, my process:
* Site has a robots.txt file - good!
* Robots.txt mentions a sitemap - excellent!
* The sitemap is stored at /images/sitemaps/index.xml ... oh. I would usually not index anything from a /images/ folder, because I want to index pages only, not images. This would likely trip my exclusion filters. Let's ignore that and continue.
* The sitemap is a list of an additional 26 nested Gzipped sitemaps. Oh.
This is the point where my tiny dumb indexer would stop. Gzipped sitemaps are part of the standard, but they're relatively rare on the web for small sites. They typically only get used if a sitemap file exceeds the 50,000 pages-per-sitemap limit. In this case, 26 * 50,000 makes the HyTale Wiki look like a 1.3 Million page site. Do I really want to index 1.3 Million pages, an estimated 13GB of indexed text data, about a video game I'll probably never play?
My search index is storage constrained, and my indexer is very time constrained. The time I spend indexing your site is time not spent indexing another, possibly higher quality website. So at this point, I'd just grab the front page and disappear... like Google apparently does too.
Of course, the HyTale site isn't 1.3 Million pages, it's only 4,277 articles. That would all fit in the root sitemap file, and that might be the better approach for getting indexed.
Me again! I couldn't figure out why I was getting transient errors on that Poe2wiki sitemap, so I tried looking at other sitemaps on the site. I'm getting transient errors on all of them.
Then I realized - the sitemap files are being overwritten in real time. Every edit on the Wiki is causing the sitemap file to be edited in real time. That's why the sitemaps sometimes stop right in the middle of a filename when I access it - the sitemap file is in the process of being rewritten.
That's a behavior unique to a Wiki, and might explain the entire phenomenon.
Makes sense. Whatever generates those site maps should use the classic Unix trick of writing a separate file then renaming. Also maybe don’t update lastmods on each individual update, it’s a waste of resources since the granularity of lastmod is usually the day (though you can make it the second if you want).
Though weirdly it makes the subdomain result even more interesting: presumably those wikis are rewriting their sitemaps too, yet Google manages to index them normally
The sitemap issues are definitely worth fixing, but I don't think they explain the most interesting part: the same kind of wiki seems to behave very differently depending on whether it's on a fresh root domain or an established subdomain.
There's no need to speculate on whether the sitemap is received correctly by Google since Google Search Console tells you how many pages it can discover from the sitemap you submitted.
The poe2wiki issue is a little different to what is documented, albeit only for the last few months, previously we certainly had visibility issues.
The recent issues relate to site errors due to aggressive crawling of uncached, server-intensive pages (diffs, etc) hidden behind residential proxies. We were being hit sufficiently hard that the server had stability issues and we were penalised by Google.
The sitemap being hosted at /images/sitemaps/index.xml is a problem. Technically, that sitemap can only include URLs beginning with /images/sitemaps/. They should move that file to the root, like /sitemap_index.xml and reference /sitemap-1.xml, /sitemap-2.xml etc.
I wish. Not only do they sometimes randomly fail, Google search console will not tell your why. I've got a site with a sitemap which passes in every validator I could find, yet Google doesn't read it and GSC just reports "error" without any details. Google is big enough to go "we don't care, it's your problem".
Have you added a robots.txt and a sitemap.xml, even an RSS file?
While I don't know if this will help with Google, I have my own tiny dumb search indexer, and a sitemap is by far the easiest way for me to index an entire website. I'll discover the sitemap from reading your robots.txt file. RSS helps too, but my indexer uses that mostly to find fresh pages without going through your entire sitemap again. I know Kagi's tiny Teclis indexer also uses RSS files for discovery for their indie web index.
If you're only concerned about Google, then that's probably all you need to / can do. It won't even matter to Google whether or not you put a "Sitemap:" line in your robots.txt if you submitted it directly with them.
As for RSS, Google do regard RSS as another sitemap format. There's info here:
I would not submit an RSS file to Google as your sitemap, stick to XML. But also having RSS can be useful for search indexers, because in one RSS download they can find which pages are new, and can index the actual relevant content of the page without all the ads & other cruft.
An RSS feed is really like a newsletter about your website. So if there's anything where visitors to your website might find value in keeping updated via an email newsletter, consider putting that same "newsletter" content also online, and making that your RSS feed. If the content in your RSS feed contains links to the other pages of your website, that can help with page discovery etc. Something like "we just updated the XYZ product page, check it out!" is zero value, but if you've got an article about "Here's how the brand new ABC feature we just added XYZ Product works", then linking to XYZ Product from within the article & RSS content will make sense. It helps the search engines, but it also helps the user who might want to click through to the product that your article is about.
Thank you. When I add the next updates (maybe that i found one of the objects on my site is on a military firing range!) I should add a news blog and RSS. Thanks
reply