Dagens Vibes — 28. august 2026

Morgan@morganlinton · 28. august · 04:52

I am working on Eval Suite 4 for @VulcanBench, and it has by far been the most deep complex eval work I have done so far. Building it has been also incredibly interesting, and challenging in a way I wasn't expecting. What I have been learning is that, yes, I can create an eval suite that's hard, and truly challenges current frontier models on coding tasks, and gets scores in the 40 - 60 range. But, in most cases, making it that difficulty causes the eval suite to have tasks that I don't think represent real, every day engineering work. Through the process I have actually become even more excited about the current state of frontier models, and, at the same time, more convinced than ever that benchmarking models for coding should be done in a harness, and compared across harnesses. Because holy shit do harnesses matter, like they really really matter. In some of the testing I've been doing, building my eval suite, I've found cases where harnesses like Pi can take what I think is a challenging eval suite for a model, and suddenly give it super powers that allows it to ace the suite. More to come but I can tell you, if you're just using models in one harness, you might be surprised how much of a difference, both in accuracy and token use, you'll find with different harnesses. But here's the thing, there are a lot of different variables to test, and a model x harness combo that does well for certain Rust tasks, might not be optimal for Typescript. This is where I am now more convinced than ever that we can do a better job with benchmarks. Not by trying to trick models with harder and harder tasks, but with creating eval suites that actually represent more specifically the kind of engineering tasks that teams will use with these models daily. I am also starting to build out custom eval suites, and seeing way more interest in this than I was expecting. You can read a bit more about these here: https://t.co/FkFr8ZrHTT Okay, now back to my evals 🖖

Howaboua@Howaboua · 27. august · 21:45

I ported basically all my personal agents into GrokBot in about five hours. And then, as your Clankmaster, I decided I shall not rest until what I learned while making Pi-Codex-Conversion lands in GrokBot :D https://t.co/3saZRndqaA As opposed to some solutions, I am doing proper Responses implementation, proper caching, delivering UI and all that jazz you expect from yours truly. Transport is fully ported. The only thing I am not doing is OAI compaction, because that would dig too deep into the guts of Anysphere's agent loop. One might say I am late to GrokBot. Nah. I was patiently waiting until the pricing became realistic. My X Premium+ quota lasted around five hours. Apparently that was enough. Mad props to everyone at xAI/Cursor. This is what normies will consider AGI. Disclaimers: a. I take zero responsibility for shenanigans on your accounts and VMs. b. It works today. It might not work tomorrow. c. I am basically unemployed. Please don't sue me xD Funnily enough, literally anyone can do this right now if they have enough experience with how harnesses work and can prompt a Clanker hard enough. So don't expect it to last. Features: - Keeps Grok Bot's native UI, tools, permissions, delivery tools, and agent loop - Uses an existing Pi or Codex CLI OAuth login, with no extra login flow - Supports GPT-5.6 Sol, Luna and Terra - Selects model and reasoning effort per Grok Bot agent - Separately routes summarisation, subagents, browser, computer, automation, and group turns - Reuses live Websocket connections without arbitrary expiry timers - Sends validated continuation deltas instead of rebuilding every cached turn - Preserves Codex turn state across tool loops and retries - Supplies stable prompt-cache identity and reports provider-confirmed cache hits - Falls back to SSE when WebSockets genuinely fail - Preserves native tool-call identity, results, images, and delivery schemas - Tracks tokens, cache reads, inference time, failures, and activity per agent - Includes a local control UI for routing, settings, stats, logs, and recovery (http://localhost:21371/) - Automatically reapplies the host patch after compatible Grok Bot updates - Fails closed on unfamiliar host bundles instead of blindly patching them - Keeps prompts, tool arguments, credentials, and account identifiers out of telemetry - One-shot Bun installation with automatic VM checks - Bundled skill for agents - run Pi/Codex CLI inside Grok Bot VM and it will install, troubleshoot etc Happy Clanking?

kwindla@kwindla · 27. august · 18:36

Introducing PhoneLLM, an open model for voice agents. GPT 5.6 Terra performance on typical voice agent tasks at 1/3 the latency and 1/18 the cost. For voice agents, we need models that are both very low latency and very good at tool calling and instruction following. There's a trade-off here, and we often have to compromise on either latency or capability when building voice agents. With PhoneLLM (and the training and data stack that made this model possible) we're fixing this problem. For the last couple of years, most of the effort in frontier model development has gone towards leveraging test-time compute. Which is awesome! Models of all shapes and sizes are available that perform really, really well ... if you have "thinking" turned on for your model. But if you need your agent to respond at voice conversation speed, you can't use thinking models. PhoneLLM is a full-weights fine-tune of NVIDIA Nemotron Nano 30B. We trained on a wide range of real-world telephone and customer support use cases. The training focused on taking the excellent Nano 30B base capabilities and teaching the model to do typical voice agent tasks with thinking disabled. The results are really good: accurate tool calling and concise, on-topic responses in long conversations. And fast: TTFAT measured server-side is <100ms if you run PhoneLLM on a lightly loaded B200. :-) But seriously, when we characterize model latency, we do it with full, end-to-end, batched request simulations using real Pipecat voice agent pipelines. You can serve more than 80 concurrent agents on a single B200 with P95 end-to-end TTFAT <600ms. Including network overhead. That's an LLM cost-per-minute around $0.0025. (1/4 of a cent.) At a latency lower than any third-party API offers today. More details about this model, including weights on @huggingface, how to spin it up with one click on Modal, and a starter project repo you can clone, are in the thread ...