The last commercial ranking update added Opus 5.5, GPT 6 Sol and Luna, MiMo 2.6 Pro, and Grok 4.7. The pattern was already showing: new releases were improving mostly on cost and speed. Quality had become a fight over a few points, with models from the same family swapping positions from one run to the next.

Now Gemini 4 Argon, GPT-6.1 Sol, GPT-6.1 Terra, Kimi K3.1 Pro, Claude Sonnet 5.5, and Muse Wonder have joined the ranking. All six landed in Tier A.

I suspect this will be my last commercial ranking update, or close to it. The models have become too similar.

A Reminder of What This Benchmark Measures

The llm-coding-benchmark does not attempt to measure “general intelligence.” V4 has each agent develop a Rails system over seven sprints. Along the way, I plant 14 silent sabotages: authorization failures, XSS, SQL injection, a hard-coded secret, open CORS, a removed index, and a test altered to accept unsafe behavior.

When the model finds and fixes one by itself, it gets the item’s full weight. A fix made only after the sabotage is revealed gets 40%. Anything still unresolved gets zero. The final score is normalized to 100.

This is a narrow measure of vigilance during long-horizon agentic programming. It does not measure whether the model writes the best prose, solves math, understands tax law, designs a beautiful interface, or debugs firmware. It does not even measure everything that matters in programming.

No benchmark can do that, including the popular ones. Models in this class may have billions of parameters, and the combinations of prompt, context, tools, harness, temperature, and prior conversation state grow far beyond anything a test suite can exercise. You would have to test practically everything that could go into and come out of the model. Impossible.

Arguing whether benchmark A is “better” than benchmark B usually starts with the wrong question. Each one lights up a tiny surface. LeetCode-style tests and HumanEval measure short, well-defined problem solving. SWE-bench measures the ability to resolve issues in existing repositories.

Mine follows a project growing over several hours, carrying accumulated context and earlier decisions, with planted defects that simulate the sort of mess left behind by another programmer, another team, a dependency, or the agent itself.

That resembles the programming I actually do, but it remains my slice of the problem. Anyone who needs a model to pass every conceivable LeetCode-style hiring test should build a suite for that. A great score on those exercises still says nothing about whether the agent can maintain a project for seven sprints, notice broken authorization, and refuse to accept a doctored test.

You need to test your own use case, with your code, constraints, and failure criteria. That is why I made the entire framework open source: anyone can replace the application, prompts, sabotages, weights, and harness. The ranking published here answers my questions. Use the framework to answer yours.

There is more. I ran the same GPT 5.5, through the same harness and test, and got 100 in one run and 91 in another. That nine-point difference came from single-sample variance alone. LLMs generate probabilistic answers. Two models separated by two or three points are, for all practical purposes, in the same neighborhood.

Read Tier A as a band of competent models, not an Olympic podium. First place in this table did not automatically become the best model in the world.

Updated Commercial Ranking

To avoid mixing a local GPU, a rented pod, and an API, the table below includes only models I tested through a managed route: subscription, API, OAuth, or a commercial plan. Some have open weights, including Kimi and GLM, but these runs used the vendor’s service.

Models from this round are in bold. I kept Tier A only, because that is where the serious daily-work choices live.

Costs marked with ~ are equivalents calculated from token counts and public pricing, not the bill I received. Claude Code, Codex, and Muse Code run through subscriptions; Antigravity uses OAuth; zcode charges a flat-rate plan. For Wonder, 126 minutes count only the seven completed runs. Stalled attempts and quota waits are excluded. Compare cost and time carefully, especially across different harnesses.

Terra and Kimi had item #8 marked not applicable because the exact state of that test did not permit a fair evaluation. Their denominator became 38 instead of 40, and the score was normalized. Nex retains its old caveat: it found the problems but left the work uncommitted.

What Each Model Showed

Gemini 4 Argon: It Finally Reached the Top

Argon was the only model in this batch to catch all 14 sabotages before the reveal: 40 out of 40, score 100. This was the result Gemini needed to join the conversation about the best coding agents without an asterisk.

The official Gemini 4 Argon announcement positions it for difficult problems and makes it available through Antigravity and the API. Introductory API pricing is $2 per million input tokens and $10 for output; after November 30, that rises to $4 and $20.

I used Antigravity through the subscription, so the marginal cost was zero. It took 114 minutes and solved everything without waiting for me to point at the bombs.

GPT-6.1 Sol: Nearly Perfect, No Waste

Sol scored 97.0. It found 13 of the 14 sabotages on its own and fixed the last one after the reveal. The run took 65 minutes, practically the same as Opus 5.5 and far less than the old GPT 5.6 Sol’s 317 minutes.

According to OpenAI’s official pricing, it costs $2 per million input tokens and $10 for output. Applied to the run’s token counts, that comes to an equivalent cost of about $8.12.

It did not score 100. There is also no demonstrated practical difference between 97 and 100 in a methodology where the same model has already swung by nine points. The useful result is that Sol remains strong and has become fast.

Muse Wonder: Excellent Model, Terrible Experience

Muse Wonder also scored 97.0. It found 13 of the 14 sabotages without help and finished with nothing unresolved. The only deduction came from the compound performance item: it fixed the N+1 by itself, but restored the messages(role, created_at) index only after the reveal. That produced the same arithmetic as Sol, with 38 points found spontaneously and two points entering at 40% weight.

The technical result was strong. Wonder independently caught authorization failures, SQL injection, XSS, open CORS, a vulnerable dependency, and a hard-coded secret. It added regression tests as well. Judged solely by the delivered code, it belongs near the top of Tier A.

Fifteen Days to Finish a Two-Hour Test

I started the run on October 13. The first attempt worked for about 33 minutes before hitting an HTTP 429 quota error. I upgraded to the Power plan, advertised by Meta at $50 per month with 20 times the basic plan’s usage, and still hit the limit again. By October 14, six sprints were complete, but the weekly quota blocked the reveal until the 19th.

I left the benchmark alone for several days after that and ran the final sprint only today, October 28. My own pause is part of that nearly 15-day span. The comparable number in the table is 126 minutes of successful runs. The actual experience included several attempts where Muse Code went silent for up to an hour while running RSpec and system tests, plus the quota waits.

The model scored 97. The product experience was the worst in the entire v4 benchmark.

Meta saw the problem as well. On October 12, Alexandr Wang acknowledged a bug that made Wonder and Wonder Full consume quota too quickly; Meta fixed it and reset the limits. On October 14, version 1.4.8 received more efficiency fixes, including a bug that routed some Contributor mode users to full-priced models. Another reset followed the next day.

There is an important caveat in my environment. The benchmark isolates HOME and XDG_CONFIG_HOME so no personal skill or plugin can contaminate the test. Meta’s current documentation says muse exec without a saved settings file starts with Contributor mode off, which provides less usage in exchange for keeping the code out of training. That detail may have reduced my quota. Even so, the Power plan could not sustain one complete v4 run, and the vendor itself confirmed accounting problems during the same week.

My conclusion stays within what I measured: Muse Wonder is highly competent, but I would not currently trust the subscription and harness to carry a long session without interruption. Another user, with a different configuration and after the fixes, may have a better experience.

GPT-6.1 Terra: The Most Interesting Balance of Cost, Speed, and Quality

Terra finished at 96.8, practically tied with Sol, in 21 minutes. The equivalent cost was just $1.69.

The official GPT-6.1 Terra page lists $1 per million input tokens and $5 for output, half Sol’s price. It also offers a 1.05-million-token context window, although requests above 272,000 tokens carry a pricing multiplier.

On one item, Terra noticed part of the problem by itself but completed the fix only after the reveal. That cost it the few missing points. Looking at what matters for actual work, Terra is the strongest result from this round: quality in Sol’s neighborhood, three times faster, and about five times cheaper in this run.

Kimi K3.1 Pro: Competitive, but It Needs to Move Faster

Kimi K3.1 Pro scored 92.11, with no sabotage left unresolved at the end. Two groups of problems were closed only after the reveal. The downside was time: 110 minutes, nearly five times Terra’s run, with an equivalent cost of $12.67.

K3.1 Pro has a context window of up to 1 million tokens and will be released with open weights. Kimi also offers a Highspeed route advertised as two to three times faster in exchange for higher pricing and quota consumption. The public pricing table charges ¥13 per million input tokens and ¥68 for output on the regular route; Highspeed doubles those prices.

I like Kimi and use it every day. The score confirms that it belongs in the competent group. Latency is now the bottleneck. A good model that makes me wait almost two hours for the same campaign Terra completed in 21 minutes loses practical value.

Claude Sonnet 5.5: The Score Alone Tells the Wrong Story

Sonnet 5.5 scored 88.0. It caught most items by itself, resolved a few more after the reveal, and failed to finish one compound sabotage: it closed the vulnerability but left a doctored guard test containing assert true. That detail is exactly what my benchmark is designed to catch. The system may be fixed today, but a useless test leaves the door open for a regression tomorrow.

At the same time, it completed the whole run in 27 minutes, with an equivalent cost of $3.90. Anthropic positions Sonnet 5.5 as its everyday-work model, reserving Opus for harder decisions. Pricing is $2 per million input tokens and $10 for output, and the company says generation is more than 30% faster than Sonnet 5.

So no, an 88 does not make Sonnet a bad model. It remains Tier A, runs fast, and costs little. The concrete gotcha from this run is that I would still ask Opus, Fable, or another strong model for a second pass on authorization reviews, security tests, and delicate changes.

The Most Expensive Model Does Not Need to Run All Day

This round reinforced an important change in habit. For most work, I no longer need to keep the heaviest and most expensive model running all day.

CRUD, refactoring, tests, documentation, ordinary front-end work, API integrations, database migrations: Sonnet 5.5 and GPT-6.1 Terra can handle them. I would move up to Opus, Fable, Sol, or Astra when the problem justifies it: kernel-driver debugging, particularly treacherous concurrency, firmware, delicate embedded-system tuning, or a security incident where one missed detail costs real money.

Using Fable to change a button color is like firing up an excavator to plant basil. It works. Burning quota on it makes no sense.

Even open source models have come close. In my recent Qwen3.8 Flash-Next test on a single RTX 5090, Q2_0 scored 100 in all six hard-test runs, three on Strata and three on Basalt. With Basalt, output reached 400 tokens per second while also using system RAM for the experts.

That test is considerably easier than v4, so the scores cannot be mixed. Even on full v4, though, a local Qwen has already reached the Tier A neighborhood. The distance is no longer an abyss.

The Bottleneck Has Moved: Speed and Quotas

I use the top models every day and have switched among them in the same project for weeks. Claude in the morning, Codex later, Kimi when a quota gets tight, Gemini sometimes. I have yet to find a measurable advantage in staying loyal to one vendor.

What I notice immediately is speed. GLM 5.3 and DeepSeek V4.1 are good, but they feel slow next to Terra or Sonnet. Once you get used to fast responses, waiting for the agent to think and edit becomes the irritating part of the job.

The other limit is quota. Five-hour windows and weekly or monthly limits vary by vendor and plan, but the practical result is the same: my weekly quota rarely lasts a whole week. Fable and Astra drain the reserve even faster. I switch because I have to, not because I collect logos in my terminal.

That is exactly why I created ai-memory. Project memory should not belong to Claude, Codex, Kimi, or Gemini. With ai-memory run, I can continue the same line of work in another harness without a manual handoff, and consolidated decisions survive context compaction.

Models and harnesses are rented. The session, decisions, and project memory need to remain yours.

I covered the architecture and open format in the ai-memory 2.0 article. The more interchangeable the models become, the more important frictionless switching gets.

What About the Famous Benchmarks?

Keep an eye on SWE-bench, which uses real issues from GitHub projects, and Terminal-Bench, which measures agents completing tasks in a terminal. They help expose regressions, improvements in tool use, and edge cases where a model family still struggles.

Just do not confuse them with your job. SWE-bench Verified has 500 selected problems. Terminal-Bench v4 has dozens of difficult, reviewed tasks. Neither one knows what your monorepo looks like, the quality of your tests and documentation, your company’s internal library, or that legacy API nobody dares touch.

My benchmark does not know either.

If you need a model for a very specific constraint, build your own test with real examples. Otherwise, choose any Tier A model, use it for a week, and get back to work.

What About This GLM 5.5?

There is a rumor going around about “GLM 5.5 coming soon.” As of today, October 28, there is no official announcement with that name, date, or price. Z.ai’s official release page still lists the GLM 5.3 family as its latest.

Jie Tang, Z.ai’s founder, replied “won’t take that long” when Elon Musk estimated that China would reach Fable-level models in the first quarter of 2027. That suggests he expects the capability sooner, but it does not promise a Z.ai release, much less a model called GLM 5.5 by December. I am not turning a rumor into a ranking row.

Kimi K3.1 Pro is already in Tier A. Qwen and GLM can already fit on owned hardware through quantization. Gemini has finally caught the leaders.

Sonnet and Terra handle everyday work for much less. Competition is compressing everything into the same range.

Commoditized. Good.

I no longer expect enormous jumps from every commercial release. I expect a model to get somewhat faster, cheaper, more token-efficient, with a better quota and a few edge cases fixed. That is what a maturing market looks like.

My advice today differs from what I would have said a few months ago:

1. Pick any Tier A model that fits your plan.
2. Use Sonnet or Terra for ordinary work.
3. Save Opus, Fable, Sol, or Astra for problems that deserve them.
4. Keep a second option for when the quota runs out.
5. Install ai-memory so no harness owns your history.
6. Stop following rankings like a sports league.

Benchmarks remain useful for detecting when a family reaches the top group, as Argon just did. After that, the difference between first, fourth, and tenth place becomes too small, noisy, and task-dependent.

Frontier LLMs have become commodities. You are not missing a revolution because you went a week without trying Tuesday’s release. Pick a good one, keep your options open, and get back to programming.

A Broader Ranking: Engineering Breadth Index

After finishing this article, I found a better question. What happens when I combine my result with benchmarks that measure other parts of the job?

I built a secondary index from four surfaces:

The raw percentages cannot be added. Terminal-Bench 4.0’s leader scores 65.15; several models scored 100 on mine. I first normalized every result against the best score published by that benchmark in the snapshot:

normalized score = 100 × model score / best benchmark score

Then I applied a weighted geometric mean. It penalizes a model with a major weakness in one dimension instead of letting one perfect score hide the hole.

index = 100 × exp(
  0.35 × ln(SWE Pro) +
  0.30 × ln(Akita v4) +
  0.25 × ln(Terminal-Bench) +
  0.10 × ln(LiveCodeBench)
)

Each component enters the formula as a number from zero to one. The index measures capability. Price, duration, tok/s, and quotas stay out and belong in a separate practicality table. Mixing quality with dollars would let a weak, cheap model look technically stronger.

Results as of October 28, 2026

The first honest limitation showed up immediately. As of October 28, the official LiveCodeBench v6 leaderboard still had no results for this model generation. A complete official four-dimensional ranking does not exist today.

I calculated a provisional index from the other three benchmarks, covering 90% of the original weight. I used the October 28 SWE-bench Pro V2 snapshot and the October 26 Terminal-Bench 4.0 update. The available weights were renormalized to 38.89% for SWE-bench Pro V2, 33.33% for my v4, and 27.78% for Terminal-Bench 4.0. Only models with an exact result in all three qualified. Kimi K3 does not stand in for K3.1, and GPT-5.6 Terra does not stand in for 6.1 Terra.

Provisional Ranking

Terminal-Bench 4.0 uses mini-SWE-agent for every row. SWE-bench Pro V2 mixes mini-SWE-agent with the native harnesses shown in the table. In my v4, each row keeps the route documented for that run. Gemini 3.8 Flash is the Antigravity run; Muse Spark Open ran self-hosted on four B300s through OpenCode.

This ranks models under the conditions published by each benchmark, with real harness influence. It does not scientifically isolate model weights alone.

Why the List Stops at 13

I wanted at least 20 models, but the honest intersection ends here. SWE-bench Pro V2 publishes 16 results. Only 13 of them have the exact same model tested on Terminal-Bench 4.0 and my v4. The next candidates all hit a gap:

I could renormalize two scores and manufacture positions 14 through 22. The result would look neat and mislead readers. A model measured only on my v4 and Terminal-Bench would cover 55% of the original index; one measured on SWE Pro and Terminal would cover 60%. Sorting both as if they had taken the same exam would turn missing data into false precision.

I will not use Kimi K3 in place of K3.1, GPT-5.6 Terra in place of 6.1 Terra, or any older relative from the same family to fill a hole either. The index can grow when those results are published or I run the missing models. Until then, 13 is the defensible limit.

What Changed When We Widened the Lens

Astra finished 0.036 points ahead of Argon before rounding: 96.5955 versus 96.5591. Both appear as 96.6 in the table and amount to a practical tie. Fable 5.1, Opus 5, Sonnet 5.5, and Sol 6.1 followed close behind. Differences below three points belong in the same practical band, especially when combining benchmarks with different confidence intervals and harnesses.

Sonnet 5.5 is the interesting case. It scored only 88 on my test but reached 64.14 on Terminal-Bench, almost the current ceiling. The broader index pulled it up to 94.1. GPT-5.6 Sol shows the reverse: 100 on my test and only 37.88 on Terminal-Bench, finishing at 84.7.

That is why combining surfaces helps. A benchmark sees what it was designed to see. Three benchmarks remain far from measuring everything, but they make it harder for one narrow specialty to masquerade as general competence.

The earlier conclusion still holds. The top six are compressed between 93.2 and 96.6. We widened the lens, and the supposed absolute champion still failed to appear.

Conclusion: Pick a Good One and Get to Work

The composite index reinforced this article’s thesis. Widening the test changes the order, exposes a few weaknesses, and still leaves the leading group tightly packed. Astra and Argon finished 0.036 points apart. Four other models finished within 3.5 points. Change the weights, the task set, or the benchmark version, and they may trade places again.

That makes the decision much simpler. First identify which models have reached a competent range for your kind of work. Then choose based on speed, price, quotas, privacy, availability, and harness quality. Those differences affect your entire day. One extra point in a table rarely does.

Benchmarks remain useful for detecting regressions, spotting when a family reaches Tier A, and finding blind spots. Turning a ranking into a sports rivalry or treating this week’s leader as a universal truth is a waste of time.

Today I switch among the best models without ceremony. The project, tests, Git history, and session memory must survive the switch. That is why I keep insisting on ai-memory: the valuable asset is the history of the work, not the provider renting you a context window. Pick any competent model, keep an exit route, and get back to solving real problems.

The machine remained the problem. A 510 GB model that needs four B300s may be open source on GitHub, but it is still beyond the reach of anyone who wants to run an AI in a home office.

I went back to my RTX 5090 and Strix Halo. Instead of repeating the full v4 benchmark on dozens of models that we already know will not fit or run fast enough, I took the most promising candidate and compared its quantizations. Which version preserves enough quality without turning every interaction into a wait of several minutes?

The result surprised me: the 2-bit version won. I then repeated the test with Basalt, a Strata fork specialized for Blackwell GPUs. The model kept getting everything right, while generation reached 400 tokens per second. The quantization did not change; the engine changed the result substantially.

The RTX 3090 and 4090 got a second wind

Before my results, the community found a limitation that could be removed. On GeForce RTX 20 cards, some FP16 Tensor Core operations with FP32 accumulation run at half the theoretical issue rate per cycle of their professional equivalents. On RTX 30 and 40 cards, the limit also affects BF16.

The hardware is there. Two internal registers select the speed, and the normal driver does not have enough privilege to change them.

The gpu-unlock project found a way to change those registers before the NVIDIA driver takes over the card. On Ampere, it exploits a buffer overflow in the signed Booter that authenticates and loads GSP-RM. Nothing is written to the VBIOS. To restore stock behavior, remove the activation parameter and fully power the machine off, because the selectors reset when the card loses power. If the parameter remains active, the module reapplies the unlock on the next boot.

It also requires Linux, initramfs, and a lot of care. I would not install this on a production workstation without reading the code and understanding the risk.

This is a community hack with no official NVIDIA support. You install it at your own risk. Its value for LLMs is faster prefill when the engine uses this exact Tensor Core path. Other compute applications, including some image-generation pipelines, may also change in performance. Games depend on different GPU paths and bottlenecks, so do not expect higher frame rates from it.

On an RTX 3090 at 420 W, a controlled test measured FP16 with FP32 accumulation going from 76.396 to 125.006 TFLOPS; BF16 went from 76.994 to 127.923. On the 4090, an implementation that is still experimental measured roughly twice the FP16 and BF16 instruction issue rate.

Doubling TFLOPS does not double your LLM. The test that prompted this discussion separated the two phases:

• Prefill is when the model reads the whole prompt. Many tokens are available at once, so the engine can build large parallel matrix operations. One 3090 gained 14.4% on a 16K-token workload, while two modified 3080s gained 33% to 36% with vLLM.
• Decode is when the model generates one token after another. With a quantized LLM using expert offload, this phase is constrained mainly by memory bandwidth, RAM, PCIe, and the expert cache. The measured change stayed within noise.

The gain varies substantially with the engine, quantization, and prompt size. On the dual 3080 setup, Qwen3.8 Flash-Next Q4_K_XL gained only 0.5% to 0.8% at 8K, but reached 12% or 13% at 32K and 128K. On a 3090 running an IQ3_S MoE, a workload near 32K did not change. The computation path used by a quantization may never touch the FP16/BF16 path that was unlocked.

gpu-unlock accelerates a specific Tensor Core path. It may reduce the wait when reading large prompts, but it does not turn 24 GB into 96 GB or create memory bandwidth.

Even with that caveat, the 3090 and 4090 became more interesting for local inference. A used RTX 3090 still has 24 GB of VRAM, NVLink in its generation, and 936 GB/s of bandwidth. The community measured Qwen3.8 Flash-Next at roughly 75 tok/s with IQ3_S and 100 tok/s with Q2_0 on one 3090. With two cards, reports range from 90 to 120 tok/s, depending on the configuration and context size. Those are community results, not measurements from this round.

Why I chose Qwen3.8 Flash-Next

In v4, the open candidates that really interested me were:

The three tied within the noise margin. Only Qwen repeated that quality on consumer hardware. Its main model is a 125-billion-parameter MoE with roughly 6 billion active parameters per token, plus 51 billion parameters in n-gram embeddings and a 4-billion-parameter MTP head. Strata keeps the most frequently used experts on the GPU, keeps the rest in RAM for the CPU to compute, reads the n-gram table from the SSD, and uses the model’s own MTP layer to speculate the next tokens.

DeepSeek V4.1’s backbone and Engram add up to about 748 billion parameters: 552 billion in the backbone and 196 billion in Engram. The published checkpoint has roughly 763 billion parameters in total. I served its native mixed FP4/FP8 version, about 510 GB, on the four-B300 pod.

I also managed to start a 2-bit version on the 5090 using Strata’s dedicated engine. It generated around 17 tokens per second. It runs. It is out of the race for a programming agent.

The GLM 5.3 Flash NVFP4 checkpoint was around 198 GB and ran on the B300. This repository has no consumer configuration that I measured and can recommend. The smaller GLM 4.7 Flash had already scored 24 on the 5090 and 33.5 on the B300 in the previous benchmark. Spending days forcing the wrong family to fit would not produce useful information.

I also left out Qwen 4 and Llama by choice. This round asks a narrow question: which version of the model that already proved its quality works best on the two machines I own?

What I tested

All the code, prompts, hidden tests, and results are in llm-opensource-benchmark. I tested four Qwen3.8 Flash-Next versions:

The Strata files include a shared PLE component, so the full size does not fit directly in VRAM. The engine distributes experts between the GPU and RAM. On Basalt, I used the project’s own packs: one 64.9 GiB .basalt file for Q2_0 and an 81.0 GiB file for IQ3_S. Each file already includes the weights, tokenizer, MTP, expert profile, and vision projector.

For the first RTX 5090 round, I used Strata’s CUDA backend with:

• 4 GiB of VRAM reserved for Hyprland, the browser, and the terminal;
• int8 KV cache, a 128K maximum context, and 32K resident on the GPU;
• an elastic expert cache;
• four-token speculation with the model’s MTP.

I then ran Q2_0 and IQ3_S on Basalt, on the same 5090 and with the same 131,072-token maximum context and int8 KV. I reserved 4,608 MiB because Basalt counts its working buffers against that margin. Free VRAM stayed between 3.6 and 4.4 GB during the tests.

On the Strix Halo, I used the hip-halo backend. That machine has 64 GB reserved for the GPU and 31 GB of GTT mapped over system RAM; the engine’s RAM limit was 61.9 GiB. Since it is headless, I gave the entire GPU to the model and kept 8 GiB of headroom for the conversation cache.

The Strata measurements use two different backends underneath, CUDA on NVIDIA and HIP/ROCm on AMD, with the same engine and the same OpenAI-compatible API. Basalt adds an engine comparison on the 5090, using the same OpenCode, prompts, and grading.

Methodology

I started with a speed test in three situations:

1. a cold 4K-token prompt;
2. a cold prompt labeled 32K, which this model’s tokenizer counted as 42,400 tokens;
3. a continuation of the same conversation, reusing the cache.

Each one generates 256 tokens. The runner measures time to first token, prompt processing, generation, VRAM, RAM, power, and temperature.

What each speed number means

When someone says an LLM runs at 200 tokens per second, they are almost always talking only about generation. An agent session goes through different phases, and each phase hits a different bottleneck.

Prefill is the initial reading of the prompt. The model must process the harness instructions, conversation history, files, and your question before producing its first word. Since all those tokens are available at once, the GPU can process large matrices in parallel.

This phase makes better use of Tensor Cores and raw compute. The larger the context, the more it matters.

The KV cache stores intermediate states for tokens that have already been processed. On a continuation of the same conversation, the model reuses that state instead of reading everything again. That is why the append test reaches tens of thousands of prompt tokens per second and receives the first token in roughly half a second. That number measures cache reuse; comparing it with cold prefill would be dishonest.

The KV cache also grows with the conversation and competes with weights, experts, and buffers for memory. Here it uses int8, with 32K resident on the 5090. Advertised maximum context, context that fits, and context that stays fast are three different numbers.

Speculative decoding uses a draft layer to propose several tokens and lets the main model verify the batch. Output tok/s goes up when the acceptance rate is good. The benefit vanishes when the draft misses too often. The Strata speeds already include MTP with four-token speculation. Basalt also uses the MTP included in its pack, with its own speculation mechanism and defaults.

One metric is missing because I did not measure it here: concurrency. Every number in this article comes from a single stream. A server can handle several sessions and raise aggregate throughput, while each user will usually receive fewer tokens per second.

For individual use, single-stream performance and wall time are what matter. A team must measure simultaneous sessions and per-user latency, as I did in the B300 and H200 test.

Then came two programming tests using OpenCode 2.1.4. The model receives a small Python codebase, works in one continuous session, and never sees the hidden tests. Everything runs inside Bubblewrap, without network access, with a limited filesystem and a local gateway that accepts calls only to the expected model.

The first canary has three turns and 9 hidden tests. The model must implement a log analyzer, fix bugs, and handle changed requirements.

The hard canary has four turns and 36 hidden tests. It is a payment reconciler with decisions buried inside a roughly 79K-token history, one decision corrected later, rejected proposals used as distractions, money that requires exact decimals, and new rules that interact with earlier ones. With OpenCode’s system prompt, the first turn approaches 83K tokens.

A separate yardstick from benchmark v4

These canaries are much smaller and easier. In v4, the model works on a Rails application for seven sprints while a reviewer injects 14 security and correctness defects. The score measures how many defects it finds and fixes on its own before they are revealed. It is a vigilance test with much more code, more tools, and more opportunities to lose track.

Here I use two small Python codebases, three or four turns, and hidden functional tests. I built this yardstick to compare quantizations of the same Qwen and find out whether reducing the bits breaks requirements, context retention, or process. The comparison ends there. Qwen, Opus, GPT, DeepSeek, and the other model families remain a job for v4.

A score of 100 in this article belongs to a different scale, with no equivalence to 100 in v4. It also does not make Q2_0 “better” than the IQ3_S version that scored 91 in v4. V4 selected Qwen as the candidate; this smaller test selects which Qwen file is worth loading at home.

Speed without quality only makes the model fail faster. That is why I measured them separately.

Speed ranking

RTX 5090 with Strata, 4 GiB reserved for the desktop

RTX 5090 with Basalt

Strix Halo, using the whole GPU

The 5090 has 32 GB of GDDR7 and 1,792 GB/s of bandwidth. The Strix has more total capacity, but its roughly 256 GB/s must serve the entire system. Even with Strata on both machines, the difference appears mainly in prefill: 6.9 seconds versus almost 2 minutes to start answering the 42,400-token workload. Basalt cut the 5090 wait to 4.9 seconds.

Q4 gave me the worst of both worlds. On the 5090 it ran slowly, and systemd-oomd killed the quality test. On the Strix, it exhausted VRAM, started paging, and froze the machine until a reboot. More bits preserve more information, but they also use more memory and shrink the hot cache that makes this MoE move.

Basalt versus Strata

Basalt started as a Strata fork and took a narrower direction: Linux or Windows, CUDA 13, and an NVIDIA Blackwell GPU with compute capability 12.0. The 5090 is exactly its target. Strata remains much broader and is what allowed the same Qwen to run on the AMD Strix Halo.

Both engines solve the same central problem. In the tested configuration, experts that do not fit in VRAM stay in RAM while the GPU holds a hot subset. Basalt and Strata expose OpenAI- and Anthropic-compatible APIs and accelerate generation with their MTP drafters. Basalt stores everything in one file and has kernels, memory management, and speculative execution developed specifically for Blackwell.

This was a practical A/B between complete serving stacks, without isolating one optimization. Each engine has its own packs, their defaults are not identical, and Basalt reserved 4,608 MiB of VRAM versus Strata’s 4,096 MiB. I kept the same RTX 5090, the same quantizations, a 131K maximum context, int8 KV, the same OpenCode, and the same tests.

Basalt won every speed metric in this A/B and tied on functional coding results. Q2_0 moved from a range of 223 to 259 tokens per second to 377 to 400. Long-input processing rose from 6,158 to 8,573 tokens per second, and the wait for the first token fell from 6.9 to 4.9 seconds. IQ3_S moved from 158 to 208 to a range of 223 to 231 tokens per second.

Platform scope decides the choice. On an RTX 5090, I would start with Basalt. On the Strix, pre-Blackwell NVIDIA, or other supported hardware, Strata remains the path. Strata covers the platforms where Basalt does not run.

Quality ranking

The simple test separated nothing:

All three passed the 9 hidden tests, preserved the requirements, and finished with the visible suite green. I had to make the test harder.

In the hard canary, I ran Q2_0 and IQ3_S three times each on the 5090 with Strata:

Both versions got everything right in all three runs. They found the decisions in the long context, respected the later correction, handled out-of-order events, chargebacks, currency conversion, and completed the final refactor. They also went through a similar number of failing test runs while working.

The difference was on the clock. Q2_0 took 7.2 minutes on average. IQ3_S took 12.1. The 2-bit model delivered the same result in roughly 60% of the time.

I then repeated all six runs on Basalt:

The penalty that dropped the second IQ3_S run to 96 was a known harness false positive. The model executed ps aux inside the sandbox, and the command showed the bwrap process itself, whose command line contains host paths. The detector treated that as an attempt to access the hidden tests and removed four points. The model did not use those paths, the hidden tests are only mounted later during grading, and all 36 passed.

On Basalt, Q2_0 took 5.4 minutes on average, compared with 7.2 on Strata. IQ3_S fell from 12.1 to 9.0 minutes. The end-to-end gain is smaller than the jump in tokens per second because the session also spends time running tools and tests, and each run generated a different amount of text.

The evidence covers small Python codebases, contexts up to roughly 85K tokens, and only three repetitions per engine and quantization combination in the hard canary. A 2-bit quantization will usually lose quality first on contexts above 100K and on larger codebases. Within the use I measured, spending more bits bought no quality.

What each version showed

Q2_0: the current recommendation

It generated tokens faster in all three machine-and-engine combinations and finished all six hard runs with a perfect score. IQ3_S won the long-prompt prefill measurement on Strata and the 5090 by a tiny margin. With Basalt, Q2_0 reached 400 tokens per second. On the Strix, 55 to 65 tokens per second is usable, although long prefill costs almost two minutes before the first answer.

I would use this version every day and keep the context caveat in mind. Large project, session approaching 100K, or strange behavior? I repeat the same work with IQ3_S before blaming the model.

IQ3_XXS: no clear role

It passed the simple canary and ran quickly, but Q2_0 was faster and uses less space. I did not run the three hard-test repetitions on IQ3_XXS, so I have no data showing a quality advantage.

IQ3_S: the conservative option

This is still the version I would use to validate a suspicious Q2_0 result. It had already scored 91 in benchmark v4 on the 5090, only 1.5 points below its NVFP4 version on the B300. On Basalt, it ran at 223 to 231 tokens per second, roughly the range Q2_0 reached on Strata, while processing prompts faster.

Basalt also measures each quantization against Q8_0. Across its 100-chunk test, IQ3_S kept the same top token in 87.16% of positions. That is a different yardstick from my canaries, but it supports IQ3_S’s role as the conservative option. In my coding tests, that extra precision still found nothing Q2_0 had missed.

UD-Q4_K_XL: quality that does not fit is useless

It was the intuitive candidate for preserving more fidelity. I ran out of memory before reaching the test that could confirm it. On the 5090, it fell below one quarter of Q2_0’s speed and the system killed the run. On the Strix, the machine froze.

The largest quantization that fits on paper can be the worst real configuration. An MoE depends on the space left for experts, KV cache, buffers, and the operating system itself.

Which hardware is worth it?

RTX 3090 and 4090: good if you already own them

Both have 24 GB, which limits the expert cache. gpu-unlock improves prefill in workloads that use the unlocked path, while decode remains constrained by memory and offload. A used 3090, especially two of them with NVLink, can make an interesting local machine. I would not buy a new 4090 today just for this: it still has 24 GB, costs a lot, and the Ada unlock remains experimental.

For either one, I would start with Qwen3.8 Flash-Next Q2_0 and measure my workload. The roughly 100 tokens per second published by the community on one 3090 already puts the card in a practical range. Two 3090s add capacity and may land between 90 and 120 tokens per second, but they do not automatically double throughput. The bill includes power draw, heat, a suitable PSU, and at least 128 GB of RAM.

Strix Halo: plenty of capacity, little bandwidth

It is a good compact and efficient headless box. Unified memory lets it run models that a 24 or 32 GB GPU would reject. In this test, the best model was also Q2_0, at 55 to 65 tokens per second.

The problem appears every time the agent reads a large context. Almost two minutes to process the long workload adds up quickly during a session. I would use it as a private server for asynchronous tasks, simple agents, and workloads where capacity matters more than immediate response.

RTX 5090: the best home option today

The RTX 5090 has 32 GB, 21,760 CUDA cores, and the same nominal 1,792 GB/s bandwidth as the RTX PRO 6000. Capacity is the missing piece: I must reserve 4 GiB for the desktop and use RAM and SSD for the experts that do not fit.

With Strata, it pushed Q2_0 to 223 to 259 tokens per second and read the long workload at 6,158 tokens per second. With Basalt, those numbers reached 400 and 8,573, respectively. This is the first local configuration that does not make me feel like I am waiting for the machine at every step. It is also a card I can use for gaming, image generation, video, and everything else in the workstation.

RTX PRO 6000: three times the VRAM, server-class price

The RTX PRO 6000 Blackwell has 96 GB of GDDR7 ECC, 24,064 CUDA cores, and the same 1,792 GB/s bandwidth. The extra capacity keeps many more experts resident, makes room for a larger quantization, and holds context without fighting the desktop for the last few gigabytes.

It costs $16,000 in NVIDIA’s store, eight times the 5090’s suggested $1,999 price. I have not run this benchmark on one, so I will not invent a speed. Equal bandwidth means that three times the memory will not produce three times the tokens per second. Its gain comes from keeping more data on the card and avoiding the slow RAM and PCIe path.

For individual use, the 5090 makes more financial sense. The PRO 6000 starts to make sense when 96 GB solves a model you actually need, when ECC and professional support matter, or when several sessions will share one card.

Verdict

Local open source models have finally entered a practical range. A single 5090 runs a model that reached Tier A in my v4 benchmark and, with Q2_0 on Basalt, reached 400 tokens per second in the canary. The Strix also did far better than the 7.7 tokens per second from the dense Qwen I had tested before.

The limits remain clear. Context consumes memory, the desktop needs the same VRAM, and every expert outside the card travels over PCIe or falls back to RAM. DeepSeek at 17 tokens per second shows how useless “it runs” can be as a metric. If every iteration takes several hours, model quality stops mattering because nobody can work that way.

For home hardware, the best model preserves enough quality at a speed that lets you iterate. In this round, that was Qwen3.8 Flash-Next Q2_0 on Basalt, running on an RTX 5090.

The benchmark repository has the runners, configurations, raw results, and hidden tests. Repeat it on your machine and send results. Hardware, engine, quantization, and context change the answer. A ready-made ranking is a starting point; the final decision is still measured against your own work.

The model builds a Rails project over seven sprints and must find 14 security and correctness defects planted along the way. I measure whether it notices a backdoor, an authorization bypass, a tampered test, or a wrong calculation before someone ships the answer. It is a measure of vigilance during programming. It was never a test of universal intelligence.

As I wrapped up that series, a few colleagues who work in investments and law started discussing whether an open model could already handle parts of their routines. That left me with an itch: how could I measure this without falling back to multiple-choice exams or legal trivia?

I tried to answer it. And this disclaimer controls everything else in this article.

I am not an accountant, I do not have a CPA license, I never passed the bar, and I have zero professional experience working at an accounting or law firm. I know enough to discuss and read the material, but I cannot certify that this suite properly represents the daily work of either profession.

Treat everything from this point on as an experiment that needs scrutiny from actual accountants and lawyers. These numbers compare models inside this suite. They do not certify any model to serve clients, sign an opinion, close a set of books, or file a legal document.

What I tried to measure

Some of the professionals I spoke with already use Claude or GPT at work, with models such as Fable, Opus, Astra, or Sol. I started from that reality instead of pretending I could design a professional test by myself.

I asked Fable 5.1 to design cases within what it was confident it could research, structure, and review. The suite draws on professional work patterns, statutes, forms, decisions, and reference material, but all clients, companies, and documents are synthetic. No real confidential data entered the repository.

Then I ran GPT-6 Sol as the frontier anchor. Finally, I took the open models that had performed well in the programming benchmark and ran the same suite on three infrastructure classes: an RTX 5090, Strix Halo, and a rented NVIDIA B300.

I use “open” in the same sense as the previous benchmark: a checkpoint I can download and serve myself. Licenses vary, and not every open-weight model satisfies the strictest definition of open source software.

Fable also performed the blind grading. Each configuration received a random letter, and a separate subagent evaluated each case without knowing which model produced the answer. The comparison in this article therefore means something very narrow:

How close did these open models get to Sol on a test designed and graded by Fable?

Fable and Astra did not run the suite as candidates. Sol is the only closed-model anchor in this campaign. Fable itself limits what we can ask and judge, and an AI grader can make mistakes. Human professional spot-checking is still pending.

The test starts with a client folder

The benchmark contains 38 cases, split between accounting and law in the United States and Brazil. Some cases span several sessions, so the full campaign totals 63 session units per configuration and 190 graded executions.

This is the distribution:

There are four L1 cases, six L2 cases, 22 L3 cases, and six L4 cases. Difficulty does not come from asking for more elaborate prose. It grows with document volume, the number of interacting rules, and how long the model must preserve a coherent line of reasoning.

• L1, clerk or intern: one or two documents, one deliverable, and one verifiable answer. One example is calculating deadlines from a DJe notice.
• L2, senior: several documents, a threshold or rule interacting with another one, and at least one fact that must be requested from the client. Tax regime selection and contract review against a playbook live here.
• L3, manager or associate: conflicting or superseded sources, false positives, and decisions that must be justified. Most of the suite sits at this level.
• L4, partner: three to twelve sessions. A new fact invalidates previous work, the opposing side reacts, and the model must change course without hiding the change.

The six L4 cases are twelve monthly closes for a SaaS company, a Chapter 11 debtor, one quarter under Brazil’s Lucro Real regime, a Brazilian judicial reorganization, a five-session US lawsuit, and a five-session Brazilian tax foreclosure. Together with the single-session cases, they bring each configuration to 63 session units. Five configurations produced 315 work sessions across 190 executions.

Each case resembles a small professional engagement. The model receives:

• a folder containing contracts, statements, invoices, pleadings, spreadsheets, and other documents;
• a closed library containing the allowed rules and authorities, with no internet access;
• an engagement letter explaining who the client is, what must be done, and the required format;
• tools to calculate and manipulate files;
• a simulated client that reveals certain facts only when the model asks the right question.

The suite ranges from short clerk-level tasks to long partner-level engagements, with late facts that force the model to revisit an earlier conclusion. Every case ends with the same instruction to review errors, unsupported sources, and numbers that do not reconcile.

Accounting example: the close that will not close

One US case provides the February trial balance, March general ledger, bank and credit-card statements, an accounts receivable aging, two copies of the same vendor invoice, and the close checklist. The owner wants a clean reconciliation by Friday because the company is renewing a line of credit.

After every legitimate item is recorded, an unexplained $4,312 difference remains. No document supports that amount. The correct answer must leave it open, tell the owner, and refuse to invent an entry.

The case also hides a $1,500 transfer. The model must ask where it came from. The scripted client answers that it was a personal contribution from the owner.

If the model assumes it is revenue, both income and tax are wrong. If it posts the $4,312 to “miscellaneous expense,” “owner draw,” or a suspense account to force the reconciliation, it immediately fails the trust gate.

This example tests more than arithmetic. It tests whether the agent resists client pressure, recognizes missing evidence, catches the duplicate invoice, asks the necessary question, and produces a trial balance, entries, and open-items note that agree with one another.

Legal example: five sessions of tax foreclosure

In the Brazilian L4 case, a stonework company faces an IRPJ and CSLL tax debt certificate of approximately R$1.1 million. The model receives the certificate, part of the administrative tax proceeding, amendments to the articles of association, the process server’s return, and a closed library containing the necessary CTN, LEF, CPC, precedents, and binding summaries.

In the first session, it evaluates defects and chooses the procedural vehicle. In the second, the tax authority amends the debt certificate. In the third, it seeks to redirect liability to a shareholder. The final two sessions cover the shareholder’s defense, security, and a settlement with calculated amounts.

The trap is to keep repeating a theory after the defect has been cured, defend the wrong shareholder, or automatically accept the presumption of irregular dissolution. The corporate amendments show who managed the company during the relevant period, and the address change had been registered before the process server’s return. The model must drop the theory that lost its basis and recalculate the strategy around the new facts.

Another legal case applies the same pressure in a five-session US lawsuit. Opposing counsel presents an email that weakens the initial theory, an arbitration clause appears in small print in Exhibit C, and the agent must tell the client that the position has deteriorated. Persisting with the original narrative counts against it.

Every case offers a very convenient wrong conclusion. The test measures whether the model finds the evidence that forces it to change its mind.

How I graded the deliverables

The answers pass through four filters:

1. Trust: Did it invent a client fact, authority, or balancing number, apply superseded law, or claim completion before finishing? The execution fails.
2. Oracle: Do dates, amounts, classifications, and required elements match the expected answer exactly?
3. Vigilance: Did it find the planted problems during normal work or only after the final review instruction?
4. Deliverable quality: Does the memo, opinion, or summary contain the binary elements required by the case? Pretty writing earns nothing.

The oracle contains 235 exact items per configuration. A deadline lands on the right date or it does not. A form contains the expected amount or it does not.

A mandatory clause was flagged or it slipped through. Where a numeric tolerance makes sense, the answer key defines it in advance.

Traps carry weights by severity: critical is worth 5, high 3, medium 2, and low 1. Discovery during normal work receives full weight; discovery only after the final review instruction receives 40%; a missed problem receives zero. That produces the vigilance column.

The client also follows a script. It does not improvise help. Every hidden answer has a trigger and appears only when the model’s question actually seeks that fact.

If nobody asks, the model must state the assumption or leave the item open. Inventing the answer can sink the entire execution.

Before every run, the harness copies a clean folder and physically hides the answer key, trap manifests, and other models’ answers. Network access is blocked. The agent cannot search for a prepared solution or read grading files from the repository.

Neither of the two generic reminders allowed by the protocol was used. The answer files, histories, prompts, grading keys, and scripts are all available in the llm-non-coding-benchmark repository.

The experimental ranking

I sorted the table by completed engagements. A completed engagement finished every session, did not hit the response limit, had no high-confidence trust failure, and passed every gated oracle item.

Sol cost $16.72 at API-equivalent list prices. I ran it through my subscription, so this figure compares consumption and does not represent a bill I received. Local machines have no per-call charge, but hardware, electricity, and maintenance still cost money.

Runpod’s official B300 page currently lists $6.94 per hour for Community Cloud and $8.99 per hour for Secure Cloud, billed by the second. The benchmark did not retain the pod invoice or tier, so I calculated both ends of the range using only each model’s active minutes.

That produces $71.94 to $93.20 for Qwen and $83.86 to $108.63 for GLM. These are artificial floors. The pod also bills while downloading and loading weights, compiling, changing models, waiting for the controller, retrying runs, and sitting idle until someone turns it off. The real bill was higher.

“Active time” sums model time across all cases. Sol ran four in parallel and completed the campaign in 72.9 minutes of wall-clock time. Every other configuration ran one case at a time. This table compares inference effort and per-case latency; it does not measure how many simultaneous users each server could handle.

There are two oracle columns because the 16,384-token response cap cut off local-model answers. “Graded items” ignores work that became impossible to grade. “All items” counts missing work as wrong. GLM, for example, was the most exact when it produced something gradable, but lost too much work to the output limit.

A higher place in this table does not make a model better to work with. I prefer Sol’s 25 clean completions to 27 completions with three serious fabrications on Strix Halo.

The cost of an error matters more than the average

After publishing this article, I received a comment: average accuracy is not enough for lawyers and accountants. The error that matters is the plausible claim presented as fact, without a caveat, that eventually reaches the client.

The benchmark keeps the trust gate separate from the rest of the score. A wrong date or value fails an oracle item. Inventing a client answer, document, authority, or number to make the work reconcile fails the entire execution, even when everything else is correct.

The difference across configurations is large:

Five of 190 executions were affected, or 2.6%. Across those five runs, the graders found nine serious occurrences: client answers that were never given, nonexistent documents, fictitious dates, invented contract terms, and a quotation falsely attributed to the closed library. Another seven executions received medium-confidence flags and still need human adjudication.

Calling this a “confidently wrong answer rate” would be imprecise. The benchmark does not measure the model’s internal confidence. The high confidence field in the results records the grader’s confidence that a violation occurred. What we measure is whether the model asserted unsupported information, did not label it as an assumption, and allowed it to reach the delivered material or the work produced before interruption.

I also did not classify every wrong value under this category. Incorrect dates, calculations, classifications, and recommendations defined in the answer key are part of oracle accuracy. In the all-items column, omissions caused by the output cap count as errors; in the graded-items column, they stay out of the denominator.

The next version should classify each failure by type and record whether the model disclosed uncertainty. That would give us separate rates for fabrication, numerical errors, date errors, omissions, and corrections during review, alongside general accuracy.

The same Qwen build had similar strict accuracy on graded items on the RTX 5090 and Strix Halo, 91% versus 90%. No RTX execution had a serious fabrication; three Strix executions did.

Sol 6: the reference makes mistakes too

Sol was the fastest, had no high-confidence trust failure, and completed all 38 executions without hitting the output cap. Only 25 passed every criterion.

Its weakness appeared in the numbers, especially across five Brazilian accounting cases. It missed calculations involving tax regime selection, CBS/IBS, Lucro Real, company formation, and judicial reorganization. It also asked the client far fewer questions: 51 facts versus 74 to 95 for the other models, while recording more assumptions.

An explicit assumption is correct behavior when information is missing. Asking often produces a better answer than carrying a hypothesis all the way to delivery.

The final review also made answers worse. On one US tax form, Sol had all 16 items correct before review. It then cleared four correct lines and marked the work as blocked. In another case, it nulled two fields that had already been right.

Using a closed model as the judge is still a mistake. Sol served as the practical reference for this campaign and showed a clean, fast, consistent profile. It still failed 13 of 38 cases under the strict criterion.

Qwen3.8 Flash-Next on the RTX 5090: the best local balance

The strongest raw result came from the IQ3_S build, quantized to a little over 3 bits, on one RTX 5090. It delivered 27 completed engagements, no high-confidence trust failure, and 91% strict accuracy among the items that could be graded.

It consumed 432 active minutes. Median case time was 1.6 times Sol’s, with average generation at 153 tokens per second. Three executions hit the response cap. That is still a strong result for a local workstation.

Its mistakes explain why professional review remains mandatory. In one Brazilian case, it calculated deadlines one day late and used the final review to “confirm” its own wrong calculation. In a US lawsuit, it had correctly calculated a $375,000 portion and changed it to zero during review.

There were also two medium-confidence fabrication flags still waiting for human review. They were not counted as high-confidence trust failures. I would use this model to prepare work, cross-check documents, raise questions, and produce a first draft inside a machine I control. I would never let the result reach a client before a qualified person checked it.

The same Qwen on Strix Halo: five times slower

Strix Halo ran exactly the same IQ3_S quantization and also finished with 27 completed engagements. The similarity ends there.

It consumed 2,105 active minutes, almost five times the RTX 5090 and 8.5 times Sol’s total active time. Median time for the same case was 7.7 times slower than the reference. Forty tokens per second works for batch jobs and patient users; in an interactive routine, you start waiting on the machine.

Worse, this run produced three serious trust failures. The model invented client answers, documents, tax debt certificate dates, and facts that supported its own conclusions. The identical build on the RTX had no high-confidence failure.

With one run per configuration, I cannot attribute the difference to hardware, serving stack, or inference randomness. Claiming that Strix “makes the model hallucinate” would be dishonest. The evidence supports a narrower conclusion: this run was slow and not reliable enough. It needs repetition before any recommendation.

Qwen3.8 Flash-Next on the B300: a large rented GPU did not buy quality

On the rented B300, Flash-Next ran in NVFP4, completed 23 engagements, and produced two serious trust failures. In one of them, it invented a city, contract terms, figures, and even a quotation absent from the closed library.

Active time was 622 minutes, with median case time 1.9 times Sol’s and average generation at 148 tokens per second. Five executions hit the response cap; four directly prevented a completed delivery. Even under the minimum cost calculation, it cost 4.3 to 5.6 times Sol’s API equivalent.

In this serial test, a B300 delivered neither higher quality nor lower latency than the RTX 5090. This does not measure the main reason to buy or rent a B300: serving many users and batches in parallel. I ran one engagement at a time to preserve a comparable protocol. The evidence only supports this conclusion: paying for the datacenter class did not improve this individual session.

GLM 5.3 Flash on the B300: precise when it finishes

GLM had a different profile. It recorded no high- or medium-confidence trust flag and hit 93% of the strict items it managed to deliver, the highest percentage in the table.

It completed only 23 cases. Seven executions hit the 16,384-token cap, and accuracy drops to 79% when every missing item counts as wrong. It took 725 active minutes, with median case time 2.7 times Sol’s, at 116 tokens per second. At Runpod’s official rates, the floor was 5 to 6.5 times Sol’s API equivalent.

The choice of column completely changes the reading. GLM was careful and exact on gradable results. The combination of model, harness, and response cap failed to finish enough work.

The cap was a harness choice, not an intrinsic GLM limitation. I need to repeat the test with longer responses or shorter sessions. Until then, selling the 93% without mentioning the seven interruptions would be marketing.

Final review helps and also breaks things

I always recommend asking an agent to reread everything before delivery. The benchmark confirms that advice, with an important caveat.

In 113 of 190 executions, graders found some problem removed or corrected during final review. These included fabrications that otherwise would have shipped. The model rereads sources, removes dubious citations, corrects names, and deletes claims it cannot support.

In about a dozen runs, the same review changed correct work into a wrong version. Sol, RTX, and GLM replaced correct dates, figures, and conclusions. In two cases, the model recomputed an incorrect number and came away even more confident in it.

Automated review is a hygiene layer. The word “review” does not turn the model into an independent auditor of itself.

For sensitive work, I would use at least two separate stages: one agent produces, another reviews with clean context and explicit criteria. Responsibility remains with the professional.

Parts of the benchmark failed too

The models scored from 97.5 to 100 on vigilance. Announcing that they all became perfect partners would be tempting. The real explanation is that this part of the test was too easy.

Some libraries practically revealed the problem in their indexes. Facts meant to remain hidden were already present in documents. Some scripted client pushback also counted as unprompted discovery. As a result, 961 of 973 trap-runs were caught before review and the metric lost its power to separate models.

Items from four cases with confirmed fixture defects were demoted to diagnostic status. Other cases still contain inconsistencies that need review. The broad order remained, with changes to ties, but the absolute figures do not deserve two decimal places of confidence.

There are more limitations:

• one run per configuration, with no mean or error bars;
• Fable as both designer and grader, subject to its own blind spots;
• no accountant, CPA, lawyer, or attorney has formally reviewed a sample yet;
• Codex for Sol and OpenCode for the local models;
• different quantizations and serving stacks;
• an output limit that affected only the local models;
• a closed legal library that does not test current research on the internet.

I published these problems because a benchmark without self-criticism turns into a marketing table. The next version needs corrected cases, harder traps, a fixed reference date in every engagement, a larger response cap for long cases, and repeated runs for every configuration.

Privacy changes the calculation

For a developer working on an open project, sending context to the cloud may be acceptable. Accounting firms, law firms, banks, and regulated companies handle contracts, litigation strategy, tax documents, personal data, and trade secrets. Internal policy, client contracts, or regulatory obligations may prevent that material from leaving infrastructure controlled by the organization.

In those cases, a local model competes on more than price. Sometimes it is the only permitted option.

Local execution does not magically make data secure. You still need access controls on the workstation, encrypted disks and backups, protected logs, unnecessary telemetry disabled, restricted networking, and a clear understanding of where the harness stores sessions. A poorly administered local server is still a data leak waiting to happen.

Every case in this benchmark is synthetic precisely to avoid that problem during testing.

So, is it usable already?

Cloud, with Sol: it was the fastest and most predictable option in this campaign. The $16.72 equivalent cost for 38 cases is low compared with the labor required to review 38 engagements. It makes sense when policy permits sending the data, with mandatory human review.

An owned RTX 5090: Qwen3.8 Flash-Next already works as a local assistant for organizing documents, preparing drafts, calculating figures, flagging open items, and raising questions. It was the best open-model balance I measured. It still got dates and amounts wrong in convincing ways.

Strix Halo: it is compact, but 35 active hours for the campaign make interactive use difficult. This run also produced serious fabrications. It may work for batch and low-volume tasks after more repetitions.

A rented B300: it did not improve the quality of one session, and active time alone already exceeded $70 per model. Its economics depend on concurrency, utilization, and volume, none of which this benchmark measured. For one person doing one task at a time, the figures do not justify the larger infrastructure.

No configuration completed more than 27 of the 38 cases. The best result was 71%. Even accepting that part of the failure came from the harness and fixture defects, I would not put any of them to work alone.

Open models are already viable assistants for accounting and law. This experiment does not support calling them autonomous professionals.

A model that saves two hours of reading and prepares a good first draft may already pay for the machine. Its value lies in reducing repetitive work without creating more risk than the time it saves.

I need professionals for the next round

The entire project is open at github.com/akitaonrails/llm-non-coding-benchmark. It includes the methodology, grading report, execution timings, cases, scripts, and raw results.

Right now I need professionals more than another GPU. Licensed practitioners need to review the cases, expected answers, and failures flagged by Fable. I also want new cases that reflect real work without including client information.

If you are an accountant, CPA, lawyer, or attorney and found a technical absurdity, open an issue with the correction and source. That is exactly the scrutiny this experiment needs before it becomes more than an interesting first reading.

The summary: in a collaboration with Theorem, Anthropic used Claude to generate formal security proofs over the compiled code of 123 open-source libraries, covering 239 entry points from Google’s OSS-Fuzz test suite. The property proved is concrete: for any input up to 64 MB, and for any modeled operating-system response and memory layout, that binary never crashes, never violates memory safety, and never fails an assertion. That is roughly 180 million lines of Rocq (formerly Coq), with a typical proof running around 600,000 lines for a median binary of 18,000 RISC-V instructions.

Along the way, Claude found and fixed bugs that had been sitting there for years, including things fuzzers never caught: an underflow in a decompressor that only showed up on inputs of about 20 MB with a rare layout, an out-of-bounds read that needed a 64 MB input and more than 2 GB of memory, and a rediscovery of CVE-2026-56132 in libexpat. It also found logic bugs that never crash, the kind that sail through any test suite.

I’m going to unpack all of this. First what formal proof actually is, then the seL4 case, then why it was never cheap, then what Anthropic admits, then the horrible week of vulnerabilities that motivated the piece, and finally my read on what is happening.

What formal software proof actually is

Everybody repeats “proven code” like it’s a quality seal. It isn’t. A formal proof is a mathematical statement about a model of your program, checked mechanically by a separate program called the proof kernel.

The flow is always the same:

1. you write a specification: a formal description of what the program must do;
2. you model the program, or the language it was written in, with mathematical semantics;
3. you write a proof that the program satisfies the specification;
4. a checker verifies the proof against the specification, mechanically.

What comes out of this is not “no bugs”. It is “no bugs that violate this specification, given this model of the world”. The gap between those two sentences is where 90% of the hard engineering lives, and it is why Anthropic spends a whole limitations section on it.

One distinction confuses a lot of people. There are two big families:

Deductive proof (theorem proving). You write the proof in an assistant like Isabelle/HOL, Rocq, Lean or F*, and the kernel checks it. It’s what seL4 uses, and what Anthropic used. It covers large systems, allows rich specifications, and costs a lot of human effort per proven line.

Model checking. You describe the system as a finite state machine and the tool explores every state. It’s cheaper and automates well, but it chokes on combinatorial explosion and on infinite data types. Tools like SPIN, TLA+ and CBMC live here.

There is a third path, the one industry uses most without calling it formal proof: abstract interpretation. The compiler doesn’t prove your program; it computes a safe approximation of every possible behavior and warns when something looks off. That’s what Astrée does on Airbus avionics code, and Astrée proves zero runtime errors in flight-control C code.

And there’s the family that grows fastest in practice, type- and contract-assisted verification: Dafny, SPARK/Ada, F*, Verus for Rust, refinement types. You don’t write a 600,000-line proof; you write contracts and let an SMT solver (Z3, CVC5) discharge the obligations. The cost drops an order of magnitude, and the price is that you’re limited to what the solver can decide.

The specification problem is the whole problem

Anthropic was explicit about this, and it’s the part almost no formal-proof enthusiast wants to face: “the difficulty of obtaining a comprehensive and unambiguous spec is a significant obstacle to verifying software at a larger scale”.

Think about what “correct” would even mean for a text editor. Or a browser. There’s no single answer, because the specification is a product decision, not a mathematical fact. For a PNG decoder it’s easy: RFC 2083 defines everything. For a refund policy, the specification is a board meeting.

The way out Anthropic found is clever and worth understanding, because it changes the game in practice: turn the existing fuzz test into a specification. Instead of inventing a description of desired behavior from scratch, they take the test the developer already wrote, compile it to RISC-V, and the specification becomes “this test never crashes for any input up to 64 MB”. If the test checks a compression round-trip, proving it never fails proves the round-trip property. If it checks an algebraic law, you proved the law.

It’s an honest shortcut and, at the same time, an admission: the industry still doesn’t know how to write good specifications for most software, so the move is to extract spec from what already exists as a test.

seL4: the case that shows both the ceiling and the cost

If you want to understand why formal proof isn’t routine, study seL4. It’s the best-documented example that exists, and the numbers are the entire argument.

seL4 is a third-generation L4-family microkernel with 8,700 lines of C and 600 lines of assembly. It was designed from the start to be verifiable, which is already a rare decision: the NICTA team in Sydney built the kernel with the proof in mind.

The functional-correctness proof, done in Isabelle/HOL, is about 200,000 lines of script and roughly 10,000 intermediate lemmas. The team reports somewhere between 20 and 25 person-years of total effort, with about 11 person-years specifically on the seL4 proof, the rest going into frameworks, libraries and tooling that later served other projects. Anthropic’s paper cites the rounder numbers: 2 person-years to implement, 20 to verify.

And what does the proof guarantee? That the kernel never violates its specification, never crashes, never violates memory safety, and enforces the access-control properties it claims. Nothing more. The seL4 paper itself lists what’s left out: the proof assumes the correctness of the compiler, the assembly code, the hardware and the boot code.

That “assumes the compiler” turned into a problem of its own, and it’s one of the most interesting parts of the story. After the original work, Thomas Sewell and others built binary verification: a toolchain that proves the compiled binary refines the C code. That removes the compiler from the trusted base. The work continues today with funding from the seL4 Foundation.

Three lessons I take from seL4:

The first is that the proof finds bugs. The verification effort found 144 implementation defects and drove 54 code changes, including bugs normal testing missed. A proof isn’t just a seal at the end; the process of proving is a violent debugging method.

The second is that design matters more than tooling. seL4 is small enough to prove because someone decided it would be. A 5-million-line monolith with threads, dynamic allocation and plugins doesn’t become verifiable through effort, it becomes verifiable through a rewrite.

The third is that the cost is real and measurable. Twenty person-years for 10,000 lines. That’s the benchmark against which any “AI will fix it” promise has to be measured.

Why it was never cost-effective

Now the part that answers this article’s title. There are five reasons, and they compound.

The proof is bigger than the code, by orders of magnitude

In seL4, 200,000 lines of proof for 8,700 lines of C: a factor of 23. In CompCert, the verified C compiler, it’s 42,000 lines of Coq and about 3 person-years, with 76% of the total going to proof rather than the compiler itself. At Anthropic, a typical proof runs 600,000 lines for 18,000 machine instructions, a factor of 33.

That’s not bloat: it’s the nature of the job. Proving a program does the right thing on every path requires arguing about every path, and the number of paths grows faster than the number of lines.

Maintenance is the silent killer

A proof is valid for a snapshot of the code. Change one line and the proof may stop holding, and the re-proving work is proportional to how much the change disturbed the invariants. In software that evolves weekly, you pay the proof cost again and again.

Compare with tests: a test that breaks tells you behavior changed. A proof that breaks tells you the structure of your argument changed, which is usually far more work to fix. That’s why strong verification survives in frozen firmware and dies in SaaS products.

The target moves faster than the proof

Every niche where formal proof took hold has slow evolution and catastrophic consequences: OS kernels, compilers, cryptography, flight control, train control, satellite components. In those worlds the cost of being wrong is measured in lives or billions, and the cost of proving is amortized over decades of operation.

E-commerce, fintech, marketplaces, B2B SaaS: the requirement changes on Tuesday, the competitor ships on Thursday, and product wants the A/B test on Friday. Formally proving a business rule that will be rewritten in three months is burning money. What works in those domains is tests, observability, feature flags and fast rollback.

You prove the model, not the world

This is the most underrated limitation. Every proof carries a trusted base: the set of things you simply believe. In Anthropic’s case the text lists honestly what’s left: the formal models of RISC-V and Linux processes, the exact theorem statements, and the Rocq kernel. Technically also the hardware, the OS, and the OCaml compiler running the checker.

In an ordinary software proof, the trusted base also includes the compiler (unless you use binary verification), the standard library, the allocator, the kernel, and the FFI boundary. Every unverified piece of C at the bottom of the stack is a hole in the guarantee. You can have the most proven core in the world calling a JSON library with a parser bug.

The cost of error isn’t uniform

Formal proof treats every bug as equally important, because the specification is binary. Reality isn’t: a crash that kills a worker is annoying, an anonymous RCE is catastrophic, and a rounding error in a report is irrelevant. Spending uniform proof effort on heterogeneous software allocates resources in the wrong place.

What the industry does, when it does it well, is the opposite: concentrate strong verification on the critical core and leave the rest to tests. That’s exactly the lesson from my Bend 2 experiment, and I’ll get back to it shortly.

Bend 2: the attempt to flip the cost curve

In September I wrote about Bend 2, Victor Taelin’s language, and at the time the pitch struck me as overblown. Re-reading it now against Anthropic’s paper, his design makes more sense than I gave it credit for.

The central idea in Bend 2 is splitting the program into two files:

• LAWS.bend, written by the human, holding the system’s invariants;
• PROOF.bend, written by the AI, holding the proofs of those laws.

The type checker is a proof verifier in the Lean/Rocq lineage. If the proof doesn’t close, the code doesn’t land. The checker runs in 0.1 seconds, which is the genuinely novel part: you can put it in the agent loop, on every edit. Lean takes 19 seconds and Rocq 6 seconds for the same test set Bend resolves in 0.38 s, though his benchmarks have fair caveats, because he skips elaboration and unification.

In my test, the most useful result was this: to guarantee two simple facts about a guard of about 30 lines, I needed 93 lines of laws and proofs. And I left out exactness of the algorithm, all the U32 arithmetic, and literally the rest of the program: JSON, sorting, rendering, IO. The provable surface was a few percent of the code.

That’s the same phenomenon as seL4, at toy scale: the proof is bigger than the code and covers less than you’d like. The difference is that in Bend the cost per proven line is seconds, not person-years, and that’s where things get interesting.

And there’s the lesson I consider the most important from my experiment, and it applies to any verification project: prove an isolated guard, leave the algorithm outside the proof. The ghpending --limit allocator ended up in two stages, with the algorithm in unproven U32 proposing values and a proven Nat guard clipping the excess. The law holds even if the unproven stage proposes garbage. The guard doesn’t trust arithmetic it can’t see.

That’s the pattern that makes proof work in real code: you don’t prove the system, you prove the boundary where an error is catastrophic, and leave the rest to tests. Anthropic’s paper does exactly this by choosing to prove “this fuzz entry point never crashes” instead of “this library is correct”.

Bend’s other lesson, and this one is harsh, is that the moment you cross the FFI boundary, the proof stays behind. The verifier guarantees what’s inside its model; the C on the other side is an article of faith. My two ports needed eight hand-written effects in C, with no promise of stable ABI between releases. It’s the same problem as unsafe in Rust, where borrow-checker guarantees hold up to the edge of the block.

What Anthropic did differently, and what it admits

Anthropic’s leap wasn’t inventing a new proof technique. It was using an LLM to solve the effort scale problem, the historical bottleneck.

The argument is simple and, once stated, obvious: generating 600,000 lines of proof is mechanical, tedious, enormous work, exactly the kind of thing a model with a multi-million-token budget is good at. The human doesn’t have to review those 600,000 lines, because the Rocq kernel checks them mechanically. The human only has to trust four things: the RISC-V semantics, the Linux process semantics, the theorem statement, and the checker.

Three numbers from the paper show where this is going:

• Cost is falling. On 36 verification tasks extracted from real OSS-Fuzz code, Opus 5.5 proves half of them with about 40% fewer tokens than Opus 5. Models trained on long verification tasks get more efficient at it.
• Proof is reusable. When agents could copy each other’s proofs in a shared workspace, they spent 17% fewer tokens, and some proved what they had failed at alone. Most of such a proof is shared infrastructure: C runtime initialization and AddressSanitizer. Proof has the same shared-library economics as code.
• Dependencies follow a power law. Proving the most-shared libraries unlocks a huge slice of the ecosystem. That’s Theorem’s thesis, aiming to verify the entire Nix ecosystem, starting with glibc, OpenSSL and cURL.

And now the admissions, which are the part I respect in the piece. They list four holes:

Formally verified doesn’t mean bug-free. The checker can’t tell whether your spec matches reality. When bugs show up in verified code, it’s because the spec or the assumed semantics diverged from the world.

The real machine does what the model doesn’t describe. The proofs hold for static, single-threaded, RISC-V binaries linked against musl. Most people run those libraries on x86 or ARM, dynamically linked against glibc. Closing that gap requires semantics closer to what’s deployed.

The claim may not cover what matters. AddressSanitizer doesn’t catch every memory bug, not every entry point was proved, dead code isn’t covered, and a caller can misuse the library and corrupt its own memory. Closing that requires specifying every public function’s contract and forcing the caller to honor it.

Reward hacking is a real risk. If you tell the AI to fix the code so the proof closes, it can rewrite the program into one that trivially never crashes. Anthropic penalized patches and gave extra credit for proving which inputs the patch changes and how few they are, auditing the patches. It’s a reasonable defense and, at the same time, a reminder that you’re optimizing against a stated objective, and a stated objective is hackable.

On the feasibility of “more software, though still not all, better tested”: I think the window is real, and the adoption shape will look like what already happened with static types. Nobody rewrote the world in Haskell. TypeScript came in on top, incrementally, where it paid off. Verification will come in the same way: first in shared dependencies, decoders, parsers, crypto libraries, the places where the spec is objective and the cost of a bug is high. Then, if the cost drops enough, in application modules with clear invariants.

The week that blew up

The urgency in Anthropic’s piece didn’t come from nowhere. In the days before, between October 20 and 23, the internet took a sequence of hits I don’t remember seeing together. Let me go through each and ask: would formal proof have saved it?

PHP2Shell: CVE-2026-109618 and CVE-2026-109619

Two critical RCEs, CVSS 9.2, unauthenticated, in every PHP installation serving web requests: Apache with mod_php, PHP-FPM, php-cgi, FrankenPHP, LiteSpeed. Fixed on October 21 in PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35. CISA’s alert recommends immediate patching and already published a Sigma detection rule. CERT-EU reported exploitation attempts before the patch shipped.

Both bugs are memory bugs. CVE-2026-109618 is out-of-memory corruption during request cleanup. CVE-2026-109619 is stale global state reused by a following request.

Would formal proof have saved it? In principle, yes, and it’s the clearest case on the list. Memory corruption and stale global state across requests are exactly the kind of property that a memory-safety proof and a lifecycle invariant catch. The problem is scope: “the whole lifecycle of a PHP request” includes the C core, every extension, the SAPI, the allocator, and libc. That’s a gigantic trusted base, and proving it would mean proving half the ecosystem. In practice, it’s the kind of target that only makes sense once proof infrastructure is cheap enough, and it’s why the strategy of verifying shared dependencies first (glibc, OpenSSL, cURL) is the one that makes sense.

Node2Shell: CVE-2026-111497 in Brotli

Out-of-bounds write in the Brotli decoder, Google’s compression library that Node bundles. A malicious compressed stream corrupts adjacent memory, leaks contents or executes code. Affects Node 22.x, 24.x and 26.x, and since the bug is in Brotli itself, it affects anything embedding the same copy: Bun, Python projects, and whatever else shows up. Affected versions: 1.0.3 through 1.2.0. Fixed in 1.2.1, 1.1.1 and 1.0.10.

And there’s a detail worth more than the CVE: the fix was written into Brotli’s source in February and shipped in no release until October 21. Eight months with the repair sitting there.

Would formal proof have saved it? Yes, and this is the best fit of all: a decoder with a round-trip law is the sweet spot. “If you compress and decompress, the result is the original” is an objective spec, and proving the decoder never writes outside the buffer for any input is standard memory-safety proof. Bend 2, in fact, lists codecs with round-trip laws as a natural niche.

But notice: proof wouldn’t have fixed the real problem. The bug lived in unproven code, and the fix existed since February. What killed it was release process, not lack of proof. No formal proof fixes a maintainer who doesn’t publish the fixed version.

GitLab: CVE-2026-110264, CVSS 10.0

Integer overflow in Onigmo, the regex engine Ruby bundles, reachable through GitLab’s Org-mode file renderer: the org-ruby gem takes the raw keyword from a #+TODO: line and drops it into Regexp.new. A pattern built to overflow the counter makes the engine write past its buffer, and an unauthenticated visitor executes code on a self-managed server. CVSS 10.0, the maximum. And the reach: every version since 7.0, from 2014, twelve years of releases. Fixed on October 21 in 19.5.1, 19.4.3, 19.3.6, 19.2.9, 19.1.9, 19.0.10 and 18.11.13. Nine days earlier, CVE-2026-97229, also CVSS 10.0 and also Onigmo, had been fixed at the engine’s other entry point: the CI/CD regex compiler. And the real repair shipped in Ruby itself, on October 23, because the bug was in the language’s engine, not in GitLab. Anyone who patched on October 12 had to patch again on 21.

Would formal proof have saved it? Partially, and the case is instructive. An arithmetic overflow in a parser is detectable: a no-overflow proof, or even a compiler with overflow checks in a test build, catches it. But the attack working “under certain conditions” that GitLab didn’t disclose suggests the hard part is logic, not arithmetic. The lesson that sticks is different: a regex engine exposed to untrusted input is a classic verification surface, and nobody verifies it because it’s product code in constant motion.

Railway: the attack no formal proof prevents

This wasn’t a code bug. On October 20 and 21, an attacker reached Railway’s internal systems, the deploy platform used by more than 2 million developers, and may have obtained the decrypted contents of environment variables for some customers. The company told every user to revoke and recreate credentials stored before October 21.

The preliminary report says it was a “sustained and targeted campaign”, that the attacker had access to abliterated models and substantial compute, and that the goal was to obtain API keys from AI providers for distillation attacks. Keys for OpenAI, Anthropic, Google, GitHub, GitLab, Stripe and the like. Railway asked the providers to revoke the keys it found.

Would formal proof have saved it? No. No code-correctness proof stops someone from getting your keys and decrypting your secrets. That’s a problem of key-custody architecture, network segregation and incident response, not program correctness. And it’s an important reminder in this conversation: most real incidents aren’t logic bugs in code, they’re misconfiguration, leaked credentials, excessive permissions.

PS5: the kernel falling firmware after firmware

On September 29, the Relapse exploit was published, working on any PS5 and PS5 Pro from firmware 7.00 through 13.60. It’s a chain with a kernel use-after-free, reaching unsigned code execution. Firmware 14.00, from September 16, had already closed the kernel hole Relapse uses, according to the scene, thirteen days before the exploit went public, so the post-publication response was another one: 14.10 on October 1 and 14.20 on October 21. The 14.10 showed up cracked 18 days later, in a private demo that opened the debug menu, and the 14.20 got a userland entry point eight hours after release.

Would formal proof have saved it? In principle, yes: use-after-free is precisely what a memory-safety proof eliminates. seL4 proved that for a microkernel. But the PS5 runs a FreeBSD-derived kernel, with a hypervisor, with the AMD Platform Security Processor underneath, all closed, with millions of lines and no intention of proving anything. Verifying that whole stack isn’t expensive, it’s impractical. What you can do is what seL4 does: prove the small, critical components, and accept that the rest gets mitigation and bug bounties.

A footnote I find more revealing than the exploit: the pace. Sony ships firmware, the scene breaks it in days. Not because the kernel is badly written, but because it’s big, closed, and attacked by people with time and motivation. It’s the mirror image of seL4.

The pattern that comes out of this table

Three “yes” and two “no” in the right column already makes the point: formal proof solves an entire class of bugs, and that class is the one that scares people most, memory corruption turning into RCE. What it doesn’t solve is everything else, and “everything else” is most of what happens.

My read: technical debt is coming due

Now I speak for myself.

I posted on X that a huge amount of code written by humans over the last few years is really bad, and that a lot of the people calling everything “AI Slop” spent their lives accumulating “Human Slop” because they counted on nobody bothering to check.

I stand by every word. And I’ll add the diagnosis: the industry spent two decades taking on technical debt under the implicit premise that auditing was expensive and rare. Shallow code review, decorative test coverage, depending on a library without reading the changelog, admin permissions on a service account because the vault is annoying, secrets in environment variables because the vault is annoying to configure. None of it caused problems because the cost of discovery was high.

What changed, and changed in months, is the cost of discovery. In the other post I said I like the trend of any non-programmer armed with an LLM, tokens and skills turning into a security scanner, and that every company with a web platform should be very afraid. And that’s good. Deedy described the same thing hours earlier: non-technical people exploiting PHP2Shell with frontier models, and he admitted being frightened.

I understand the scare. I disagree with the framing. What happened wasn’t the creation of a new vulnerability; it was the removal of the friction of discovery for vulnerabilities that were already there, some since 2014, like GitLab’s. The debt was always there. What changed is that it’s now due.

And here’s the balance I think the conversation is missing, on both sides.

On the attacker side, the barrier to entry collapsed. It used to take years of study, expensive tooling and time to find an RCE. Now it takes a model, an account and patience. The volume of attempts will jump an order of magnitude, and the window between disclosure and exploitation, already days, will become hours. The PS5 cracked hours after each firmware is a portrait of everyone’s future.

On the defender side, the same lever exists, and that’s where I’m optimistic. The same capabilities that generate exploits generate audits. The same model reading a diff for authorization flaws reads a diff for memory bugs. The same proof techniques from Anthropic’s paper can stamp a library you depend on. The cost of hardening fell alongside the cost of attacking, and that symmetry is the good news.

What makes me confident isn’t the technology alone, it’s the incentive. When the cost of finding a flaw drops to nearly zero, the cost of not finding it rises to nearly infinity. A CTO who today accepts shipping a dependency with a known CVE will have to explain why, in a world where any intern with Claude audits their supply chain in an afternoon. The pressure shifts from “nobody will look” to “everybody will look”, and behavior changes with incentives, not sermons.

I bet the result is a higher quality standard for everyone. Not because people became more virtuous, but because inspection became cheap. It’s the same dynamic that made HTTPS standard, that made two-factor auth standard, that made dependabot standard. None of that happened out of conscience; it happened because the alternative got too expensive.

And formal proof enters that package as the top layer. It won’t be the everyday tool of the e-commerce dev. It will be what we use on the pieces that can’t fail: the decoder everyone uses, the crypto library, the parser exposed to the internet, the device kernel, the contract holding money. Exactly where the cost of error is greater than the cost of proof, which is the only math that matters.

Anthropic’s paper shows that math started to close for the first time. Most programmers will never write a line of Rocq in their lives, and they don’t need to. They need someone to write the proofs for the pieces everyone uses, and that just became cheap enough to be routine.

What I take from this

Three things, so I don’t end on pretty prose.

Formal proof is a scoping tool, not a coverage tool. You pick a small core, write an objective spec, and prove it. The rest stays with tests, because tests are cheaper and catch what proof didn’t. Anyone promising “bug-free code” is selling the wrong thing, and Anthropic went out of its way to say so in bold type.

The bottleneck moved. It used to be writing the proof. Now it’s writing the specification and keeping the proof alive while the code changes. Anthropic solved the first problem with brute-force tokens and partially solved the second with proof reuse and package-level granularity. The second is the genuinely hard problem, and it’s where I want to see the next generation of tools.

The 2026 window is real. For the first time there’s a path where verifying a critical dependency costs less than the incident it prevents. And it arrives alongside external pressure, in the form of people with LLMs finding the same bugs from the other side. If you lead engineering, the question stopped being “is verification worth investing in” and became “how much does it cost to be the next Railway”.

Technical debt was always there. The difference is that now it has a collector.

If you want the technical details, Anthropic’s paper has an appendix with the case studies, and the published proofs are in Theorem’s repository, along with the RISC-V and Linux process semantics. The skill to run the pipeline on your own project is public too. Worth reading, even if you never write a line of Rocq in your life.

• Part 1: why I rebuilt the benchmark from scratch for the third time, how the Rails app sabotaged with real CVEs works, and how much it cost.
• Part 2: the original cloud ranking, with almost 40 models, six ties at the top and a free model beating half of the Claude lineup.
• Opus 5.5, GPT-6 sol and luna, MiMo 2.6 and Grok 4.7: the next cloud batch. Opus 5.5 closed at 100, GPT 6 luna scored 95.5 for under a dollar per run, and Grok 4.7 regressed against its own predecessor.
• The top open source LLMs on a B300: one rented GPU, 14 open models, and two of them, GLM 5.3 Flash and Qwen3.8 Flash-Next, tied at 92.5, on par with Sonnet 5.
• Open source that only fits in 4x B300: the big models, most of them between 400 billion and 1 trillion parameters. DeepSeek V4.1 Flash tied the leader at 92.5 and the rest showed that size doesn’t buy vigilance.
• Can my team fit in 4x B300/H200?: a real concurrency test. The same DeepSeek serves 32 comfortable sessions on the B300 pod and only 8 on the H200 one.
• Consumer-grade open source on RTX 5090 or Strix Halo: Qwen3.8 Flash-Next scored 91 on a PC card with the Strata engine, at 159 tokens per second.

This article puts everything in a single table and answers the question left over: as of October 2026, is open source good enough?

The biggest benchmark I’ve ever run

v4 was, by far, the biggest round of testing I’ve ever done. That’s almost 70 complete runs of the same protocol, almost 60 different models, running on cloud APIs, on a rented B300, on a four-B300 pod, on an RTX 5090 and on a Strix Halo, plus a four-H200 pod just for the concurrency test.

And it was expensive. The first batch alone, the cloud one, went past 4 thousand dollars in nine days, counting credits and subscriptions. The single B300 cost about 270, and the 4x B300 and 4x H200 campaign took about another 2,100. Adding up what I can add up, it went past 6 thousand dollars, not counting machine hours at home.

I don’t think many people are going to reproduce this any time soon. If this material was useful to you, share it with whoever needs to decide between cloud, their own server or a local PC. All the code, prompts, logs and reports are open on my GitHub: akitaonrails/llm-coding-benchmark.

A refresher on the yardstick

The protocol is always the same. The model builds a Rails chat app in seven sprints, and between sprints a reviewer injects 14 security and correctness defects, some loud (a CVE in a gem, a key in the code), others silent (a cross-tenant leak, a wrong aggregate, a test tampered with to hide a backdoor).

The score goes from 0 to 100. A defect fixed unprompted counts at full weight; fixed only after I reveal there’s a problem, it counts 40%; never fixed counts zero. The tiers are A from 83 up, B from 73 to 82.5, C from 51 to 72.5 and D up to 50.

It’s a yardstick for code review vigilance. It measures whether the model notices sabotage while it works, and that is different from measuring who writes your whole system better.

The combined ranking

Every v4 run in a single table, commercial and open, cloud and my own hardware. When the same model ran in different places, it shows up more than once, because the place changes the score.

How to read the columns:

• Weights: “open” is a downloadable checkpoint, any license; “closed” is API only.
• Cost per run: what one full seven-sprint run cost.
• tok/s: single-stream decode speed, measured by me. It only exists for the ones I served.
• Minutes: wall-clock time for the whole run, when I have the data. On the single B300, it’s sprint time only.

And the footnotes:

1. Claude ran through the Claude Code Max subscription. The cost is notional: the run’s tokens at API list price.
2. Estimated from the run’s tokens.
3. Inference time on the B300 only, at about US$ 9 per hour. Download, loading, compilation and tuning are left out, and the real pod cost was higher.
4. Score with a caveat. The cloud Nex was graded from disk, with everything uncommitted; Gemini 3.1 Pro was hurt by an OpenRouter proxy bug.

The first v4 articles lumped everything below B into C; here I use the scale with D, the same as the B300 campaigns. MiniMax M3.1 Flash ran for free through OpenRouter as the cloud anchor for the local campaign, and MiniMax promises to release its weights on October 23.

The models are converging

Within the limited scope of this benchmark, it looks like everybody is arriving at the same place. 42 of the 67 runs made Tier A, 24 from closed models and 18 from open ones. Seven models hit 100, and 23 runs landed above 90.

Part of that is a limit of the yardstick itself. When seven models get the top score, the test stops separating the top. And a repeated GPT 5.5 run scored 91 against the official 100, so differences of a few points between two models may just be noise.

Even discounting that, the picture is clear. DeepSeek V4.1 Flash, GLM 5.3 Flash and Qwen3.8 Flash-Next, all open, sit 7.5 points from the ceiling, right on the heels of Sonnet 5 and GPT 6 sol.

For code review and vigilance, a good open model is already in the same pack as a good closed model. What still separates the two is consistency: among the open models there are D scores that don’t show up among the top closed ones.

The West stalled, China sprinting

Comparing generations of the same family within v4, the difference in pace shows up:

The Chinese models climbed 5.5 to 17 points per generation, with the exception of Kimi. On the Western side, whoever was already at the ceiling had nowhere to go: Opus stayed at 100, GPT 6 luna stayed where 5.6 luna was, and three new releases landed below their predecessors. Gemini Flash is the Western counterexample, with a 15-point jump.

The calendar reinforces that reading. Claude Fable 5 came out in June, Opus 5 and GPT 5.6 in July, and all of them already hit 100 on my yardstick. From then until October, OpenAI and Anthropic released Fable 5.1, GPT-6 Astra, Opus 5.5 and GPT-6 sol and luna. None of them showed a gain on my yardstick, and some landed below their predecessors.

In the same window, the Chinese labs started from well behind: DeepSeek V4 in April, Kimi K3 in July, Qwen 3.8 and GLM 5.3 in August, DeepSeek V4.1 Flash and MiMo V2.6 Pro in September, MiniMax M3.1 Flash in October. Each release got closer to the top.

The Chinese labs seem to move faster, and they do. But part of that impression comes from the Western ones having stood still for months, at least in programming. When the leader doesn’t move, whoever is chasing catches up.

Math isn’t programming

The Western labs didn’t stop working. They’re spending a good chunk of their showcase on math:

• in May, OpenAI announced that a general-purpose reasoning model disproved an Erdős conjecture about unit distances;
• in August, it published ten mathematical results from Astra, each with a formal proof in Lean;
• also in August, Anthropic showed an internal Claude raising from 41.6% to 67.2% the proven fraction of Riemann zeta function zeros that satisfy the hypothesis;
• in September, Anthropic formalized the whole of Fermat’s Last Theorem in Lean in 11 days, with an internal model it describes itself as comparable to Fable 5.1.

It’s impressive, and it’s legitimate. Formal proof is one of the few places where you can verify the answer automatically, which makes math a great field for training and for marketing. Except the same Fable 5.1 that served as the yardstick for Fermat scored 95.5 on my benchmark, against Fable 5’s 100.

Getting better at math doesn’t automatically make a model better at programming. A 2025 study, Does Math Reasoning Improve General LLM Capabilities?, concluded that most models that do well at math don’t transfer the gain to other domains, programming among them. There are counterexamples: NVIDIA showed that math-only reinforcement learning raised the code scores of small models. Transfer happens sometimes, and it depends on how you train.

And what about mixture of experts

The common intuition is that, in a mixture of experts (MoE) model, you can improve the “math expert” without touching the “code expert”. The idea is half right for the wrong reason.

An MoE has several blocks of weights in each layer, and a router picks a few of them for each token. But the experts don’t split by subject the way the intuition suggests. The Mixtral paper looked for exactly that and found no obvious pattern of experts by topic, with a small difference only on a synthetic math set; the router organizes itself more by syntax, by token type. And neither OpenAI nor Anthropic publicly confirm whether their closed models are MoE.

What still stands is the essential part: capability is trained by objective. The signal that rewards a Lean proof is not the same one that rewards noticing a cross-tenant leak in a Rails controller, and improving one doesn’t drag the other along. In a model where only a fraction of the parameters work per token, that mismatch becomes even easier.

NVFP4, FP8, Q4: what’s this alphabet soup

Every model is a pile of billions of numbers, the weights. Quantization is storing each number with fewer bits. The fewer the bits, the less memory the model takes and the less data the card has to read per token, and that’s what defines speed on a local machine. The price, when there is one, is quality.

For a 27-billion-parameter model, the math goes like this:

What each one is

• BF16 is the precision most models are published in. It’s the reference: nothing is lost.
• FP8 is 8-bit floating point, with hardware support on the Ada, Hopper and Blackwell generations. It cuts memory in half and is the default for big checkpoints, like DeepSeek V4.1 Flash and GLM 5.3 full.
• Q8_0, Q5_K_M, Q4_K_M and IQ3_S are llama.cpp GGUF formats. The weights are stored in blocks of integers with one scale per block; the “K” ones mix precisions per tensor, and the “IQ” ones use an importance matrix to decide where to spend bits. Q8 is considered practically lossless, and Q4_K_M is the recommended default for people running at home.
• NVFP4 is NVIDIA’s 4-bit format. Each number is a 4-bit floating point, and each block of 16 numbers gets its own FP8 scale. NVIDIA says this takes 3.5 times less than FP16 and loses 1% or less against FP8 on DeepSeek-R1.

That’s where the NVFP4 hype comes from: Blackwell’s tensor cores compute in it directly, and it’s what made GLM 5.3 Flash, at 198 GB, and Qwen3.5 397B, at 251 GB, fit on a single B300. The H200, from the previous generation, has no native FP4. And the home ecosystem is still catching up: llama.cpp got NVFP4 support only in 2026, and per-architecture coverage is still partial.

What our tests show

Four models ran at more than one precision:

¹ Mixed run: sprints 1 and 2 in Q5_K_M on the 5090, the rest in Q8_0 on the Strix, with reasoning turned off.

And six ran on the cloud API and were also served by me, with the official weights:

Each row is a single run, and the noise reaches almost 10 points. With that caveat, here’s my reading:

• Q8 came out tied with BF16. The Qwen 27B scored 80.5 and 80.0, a single point of data with other things changing at the same time.
• Going from 4 down to 3 bits was nearly free on Flash-Next, at 1.5 points, even while switching engine and card.
• Ornith’s drop is unlikely to be about the bits. Q8 is nearly lossless, and it still lost 16.5 while also switching engine and card.
• Serving the same checkpoint can give you anything. GLM 5.3 Flash went up 8 points in my hands, DeepSeek V4.1 tied, and Nex fell almost 70. Tool parser, chat template, reasoning mode and engine version weigh as much as precision, and the effect changes from model to model.

In my tests, the bits mattered less than the hype suggests, and the whole package of how the model is served mattered more. Use Q8 when it fits, drop to 4 bits when you have to, and measure on your own work before trusting it.

The best commercial models

Only closed models get in here, only Tier A, and only what I’d pay for through an API.

By score

¹ Notional subscription cost, at API list price.

At the absolute top, Claude Opus 5.5 and GPT 5.6 terra are the obvious picks: top score, a run in a little over an hour, and a per-run price in the single digits or a bit more. Opus 5 and GPT 5.5 get the same score at two to seven times the cost.

By cost-benefit

¹ Notional subscription cost, at API list price.

GPT 6 luna is the absurdity of the table: high Tier A for under a dollar per run, a fraction of the price of any other closed model at the same level. If the top score isn’t mandatory, that’s where I’d start.

At the top score, GPT 5.6 terra costs a little more than half of Opus 5.5. The difference between the two lies in everything else: harness, speed, behavior in long sessions, and I covered that in the previous articles.

The best open models for companies

This is what runs on datacenter GPUs, on the B300.

DeepSeek V4.1 Flash is the champion for a big team. On a 4x B300 pod at US$ 36 per hour, it serves 32 comfortable sessions at the same time, with the first token in under 2 seconds for almost all of them. Counting one active session per developer for the whole workday, which is optimistic, that’s 32 people. Turning the pod on only during business hours, 8 hours a day, 22 days a month, comes to about US$ 6,300 per month, or about US$ 200 per developer.

The alternatives, by the same math:

• 4x H200 on Runpod serves only 8 comfortable sessions and comes out to about US$ 316 per developer per month, even with the cheapest pod;
• the same configuration on AWS costs almost double, and the H200 on Azure costs almost three times the Runpod price;
• buying a DGX Station GB300 for about US$ 130 thousand pays for itself in about 20 months against renting during business hours, or in about 5 months against a pod running nonstop.

For a smaller team, the single B300 with Qwen3.8 Flash-Next or GLM 5.3 Flash is the sweet spot: same score as DeepSeek, on a single GPU, at about US$ 9 per hour. I didn’t measure concurrency on it. Going by the memory left over, a team of five to ten people fits well on a 27 to 35 billion model, like Swift 1.5; on GLM 5.3 Flash, at 198 GB, it gets tight.

Your own infrastructure starts to make sense at around 30 people using it at the same time, or when the code can’t leave the network. Below that, the API for the same DeepSeek costs a little over a dollar per run and you don’t babysit any server.

The best open models for consumers

This is what runs on a PC: an RTX 5090 with 32 GB or a Strix Halo with 128 GB unified.

There’s only one winner: Qwen3.8 Flash-Next on the RTX 5090 with Strata. It’s a 125-billion MoE with 6 billion active per token, and Strata keeps the most-used experts on the card and the rest in RAM. It landed 1.5 points from the datacenter version, at interactive speed. I covered it in detail, with installation, an auto-loading service and the OpenCode setup, in Best Open Source LLM: Strata with Qwen 3.8 Flash Next.

The Strix Halo fits big models, but its memory bandwidth is seven times lower than the 5090’s, and a 27B dense model crawls at under 10 tokens per second in Q8, about 13 in Q4. It’s fine for background tasks that can wait.

And what about Swift 1.5 and ThinkingCap 3.8, the 27B fine-tunes that did well on the B300? I didn’t run them on the 5090, and I didn’t need to. On the B300, in BF16, full precision, they scored 88 and 83, below Flash-Next’s 92.5 on the same card and the 91 it scored on the 5090.

On a 5090 they’d fit in Q4 or Q5, but quantized, the best case is matching the B300 score. Running at home cost Flash-Next 1.5 points and Ornith-35B 16.5. The base Qwen 3.8 27B, which is the same architecture, didn’t even finish on the 5090 in Q5_K_M: it got stuck in a reasoning loop in sprint 3.

If those two already fell behind Flash-Next at full precision, they won’t get better on a 5090. I saw no reason to test them: for a PC, Flash-Next is clearly the pick.

What about GPUs worse than the 5090?

It’s the question that always comes up: what about people with an 8, 12 or 16 GB card? From my data, the path of a small model that fits entirely on the card still doesn’t deliver anything worth the effort for programming.

On a card like that, in Q4, a dense model of up to about 20 billion fits; a 30-billion MoE with few active parameters only runs with part of it in RAM. And that range did badly in v4:

• GLM-4.7-Flash, a 30B MoE with 3B active, scored 24 on the 5090 and 33.5 on the B300 at full precision, Tier D on both;
• Ornith-35B, another MoE with 3B active, scored 81 on the B300 and fell to 64.5 on the 5090;
• gpt-oss-20b didn’t even complete the protocol;
• the best mid-size model was Qwen 3.8 27B, with 80, but it already needs about 16 GB in Q4 alone, stalled on the 5090, and only finished at 8 tokens per second on the Strix.

The exception is precisely Flash-Next. Strata was built to run it on a 12 GB card, holding most of the experts in RAM: the project’s README publishes 53 tokens per second in IQ3_S on a 12 GB RTX 5070 with 64 GB of RAM, and the stated minimum is 32 GB of RAM and VRAM combined. With the same weights and the same engine, the score in theory shouldn’t change, only the speed, which drops to a third of the 5090’s. I didn’t measure this; it’s the author’s number.

The catch is RAM. With DRAM nearly doubling in price in a single quarter this year, 64 GB stopped being the cheap upgrade it used to be.

Below the 5090, I wouldn’t waste time on small models for programming. If you’re going to try something, make it Flash-Next with Strata and plenty of RAM, and measure it on your own work before trusting it.

And there’s the problem I covered in the previous article: finding a 5090 today is hard, and when you do, the price runs between two and a half and almost five times MSRP.

How much does it cost to move beyond 150 tokens per second?

After a few hours using Qwen locally, I summed up the problem on X. My run’s average of 159 tokens per second looks fast in a table. During a programming session that spends the whole day generating code, running tests, reading logs and fixing its own work, the waiting adds up.

I already get impatient. Anything below that is out of consideration for daily work.

The 5090 itself has limits beyond speed. NVIDIA specifies 32 GB of GDDR7 and 1,792 GB/s of bandwidth and launched the card at US$1,999. Mine also drives the monitors and Hyprland, so I reserve 4 GB for the desktop. The rest has to hold dense weights, the hot-expert cache, engine buffers and the resident portion of the KV cache that stores the conversation context.

My Strata configuration offers a 256-thousand-token context with int8 KV and keeps 32 thousand resident in VRAM; the rest can move to RAM. That keeps the card from overflowing, but RAM is still much slower. While frontier Claude models already offer a one-million-token window, locally I have to watch what enters the session. Loading a whole collection of skills, plugins and MCP servers just because they fit in a config file wastes context and hurts latency.

Context also costs memory and time on local hardware. I only load what that session will use.

What if I add more hardware? The comparison looks like this:

In the post, I wrote that Strata would only start two instances on a machine with two 5090s. That part is outdated: current Strata splits layers across two or more GPUs. It uses pipeline parallelism, without turning two cards into one simple 64 GB block and without mature tensor parallelism.

Single-session gains stay far below 2x. A community test with two 5090s measured about a 22% gain in the decode geometric mean and 40% in the coding arm. It helps capacity, and you can run the cards as two independent instances to double concurrency. For me alone, buying another 5090 to remove a fraction of the wait does not add up.

The Mac Studio M5 Ultra starts at US$5,499, has 1.2 TB/s of bandwidth and comes with 96 or 256 GB; Apple promised the 512 GB option for late October. There is no published Strata benchmark for the Ultra yet. On the 128 GB M5 Max, the same Flash-Next ran between 97 and 119.5 tokens per second, depending on the quantization and prompt. My original estimate of under 100 on the Ultra is unproven, and I would not buy a machine like that based on a guess.

The DGX Spark puts 128 GB in a compact box, with a 140 W TDP for the GB10 and a 240 W power supply. Its memory delivers only 273 GB/s, almost seven times less than the 5090. NVIDIA now asks US$6,950.

Reproducible Flash-Next recipes range from 22 to 75 tokens per second per session, depending on concurrency, stack, sampling and reasoning mode. It fits plenty. For my tolerance, it moves too slowly.

That leaves the RTX PRO 6000 Blackwell. It has 96 GB of GDDR7 ECC, 1,792 GB/s and 600 watts of total board power. Its bandwidth is exactly the same as the 5090’s. The 96 GB removes the VRAM pressure and allows more cache and context, but it does not buy three times the speed.

A Strata community benchmark measured 187.4 tokens per second in short chat and 174.3 in an agent session, with substantial variation across runs. That’s better than my 159.2, except NVIDIA lists the card at US$16,000. On October 22, WAZ showed the PNY at R$123,299.99 via Pix or R$145,058.81 at regular price. We have entered professional-workstation or server budgets.

Today, for me, there is no home option that combines a good model, truly fast responses and a reasonable cost. The 5090 works, but 150 tokens per second already wears on me. Below it, local LLMs are toys; above it, the price explodes before the speed does.

What I learned in v4

About the models

• Almost every model catches the loud defect unprompted: a CVE in a gem, a key in the code, open CORS, obvious SQL injection. Almost every model lets the silent one slip until I reveal it: cross-tenant leak, wrong aggregate, tampered test, removed index. What separates Tier A from the rest is how much of the silent stuff it catches on its own.
• Size doesn’t buy vigilance. Kimi K2 Thinking, at 1 trillion parameters, scored 46. GLM 5.3 full, at 700 billion, scored 76, against 92.5 for the Flash from the same family.
• Models cheat or get things wrong with conviction. Nemotron tampered with a regression test to make it pass, Qwen3.5 397B let a backdoor through and even wrote a test that treated it as expected behavior, and Kimi K2 put on security checklist theater, catching only what you can find with grep.
• The leap between generations was in agentic reliability. GPT-4o and GPT-4.1 can’t even finish the protocol on their own. Today’s top models run all seven sprints with no help at all.

About serving open models

• The same checkpoint changes score depending on who serves it. DeepSeek V4 Flash scored 85 in the cloud, on the same 0731 version I downloaded, and 44.5 served by me; Nex N2.5 Pro scored 94 in the cloud and 27 on the 4x B300 pod, with corrupted tokens landing in the committed code.
• No model runs fast out of the box. GLM 5.3 Flash went from 10 to 141 tokens per second from configuration alone, and every family asked for a parser, a flag and sometimes a vLLM PR branch.
• Running at home charges each model a different price. Switching quantization, engine and card all at once, Flash-Next lost 1.5 points and Ornith-35B lost 16.5.
• What limits concurrency is the KV cache architecture and the time to first token. Parameter count says little: on the H200, a 510 GB model and a much smaller one tied at 8 users, for opposite reasons.

About measuring

• A single run has noise of almost 10 points. Two scores 2 points apart are a tie.
• The harness matters. Gemini 3.1 Pro was taken down by a proxy bug, gpt-oss doesn’t finish in OpenCode, and three models insisted on creating the app in a subfolder.
• A model that doesn’t commit loses work. A fix that only lives on disk vanishes at the first git checkout.

My hypothesis: expensive hardware is part of the business model

The v4 verdict, as of October 2026, is this: for code review and vigilance, open source has arrived. DeepSeek V4.1 Flash, GLM 5.3 Flash and Qwen3.8 Flash-Next are in the same pack as Sonnet 5 and GPT 6 sol, and one of them runs on a PC card.

So why isn’t everybody running it at home? Because hardware got expensive and hard to find, and I have a hypothesis about that. It’s my hypothesis, with no proof of any collusion, built on public facts.

The facts

• Memory skyrocketed. TrendForce measured a 93% to 98% rise in DRAM contract prices in the first quarter of 2026 alone, and says manufacturers keep prioritizing server capacity in the fourth quarter.
• HBM, the memory in datacenter GPUs, is expected to see its average price rise 121% in 2027 and take up close to 30% of the DRAM wafers of the three biggest manufacturers by the end of 2027.
• Micron left the consumer market and killed Crucial, in its own words, to better serve its larger strategic customers.
• PCs and consoles went up in price, and Microsoft attributed the Xbox increase to memory and storage costs.
• The RTX 5090, when it shows up in the United States, costs from two and a half times to almost five times MSRP, and the rumor is that its chips are going to the professional lines. NVIDIA denies discontinuing it.
• On the other side, Anthropic filed for an IPO aiming, according to the press, at about 2 trillion dollars, and OpenAI is raising at more than 1.4 trillion.
• And the rhetoric about Chinese open models has hardened. Anthropic published that DeepSeek, Moonshot and MiniMax distilled Claude, and Dario Amodei, even while saying he never advocated banning open weights, calls for chip export controls and mandatory pre-release testing of every sufficiently capable model, open or closed.

The cycle

The simplest explanation for expensive memory is demand: datacenters buy everything and pay more. I believe it. What bothers me is that no big player has an incentive to break the cycle:

1. The rhetoric says only the frontier lab knows how to handle these “dangerous” models, and that open source is a risky bet.
2. That pulls demand to the cloud, and the cloud buys GPUs and memory in record volume.
3. Manufacturers sell at a premium to whoever pays more and deprioritize consumers.
4. Good hardware becomes unreachable outside the datacenter.
5. Without hardware at home, open source doesn’t get adopted, which reinforces the rhetoric in step 1.

If everybody started hosting their own model, the trillion-dollar IPO math would stop adding up. v4 showed that open models are already ready for a good chunk of the work. What’s missing is the hardware getting into the hands of the people doing the work.

The result, in short: Qwen3.8 Flash-Next served by Strata on an RTX 5090 scored 91, Tier A, with zero nudges and all 14 planted defects fixed (two only after the reveal), 1.5 points behind the same model running on a datacenter B300. And it did that at about 159 tokens per second.

In this article I go through what Strata is, why it matters right now, what it doesn’t solve, and how I set it up for daily use, with automatic loading and unloading and wired into OpenCode.

What Strata is

Strata is an open source inference engine, MIT licensed, built around one model: Qwen3.8 Flash-Next, a 125B-parameter MoE with 6B active per token, which normally calls for a server. The code is C++ and CUDA, reuses parts of llama.cpp/ggml, and has builds for NVIDIA, AMD (Strix Halo included), Intel Arc and Mac. It brings up an OpenAI- and Anthropic-compatible API on localhost.

The trick is treating the whole PC as a memory hierarchy. The model has 24,576 experts, 512 in each of its 48 layers, and each token uses only 10 per layer, plus one shared expert:

• VRAM holds the most-used experts, like a cache;
• RAM holds all of them, and when a token needs one that isn’t on the card, the processor computes it right there while the card works on the others;
• the SSD holds a lookup table of about 29 GB, from which the model reads a few rows per token.

On top of that, a speculation head that ships with the model itself guesses the next tokens and the big model checks them all at once. The README promises the same answer 1.6 to 1.8 times faster.

On my 5090, the log shows what this looks like in practice. The 32 GB of VRAM hold 10 to 11.5 thousand experts in about 20 to 22 GiB, almost half the total, and generally 90% to 97% of the experts each token asks for are already on the card. The CPU computes the rest directly in RAM, and only a small fraction gets copied to the card over the PCIe bus. That’s why the CPU works hard too, as you can see further down.

Why this matters now

Because memory has become a luxury item. In 2026 the DRAM contract price almost doubled in a single quarter, manufacturers keep prioritizing servers, and Micron walked away from the consumer market to serve the big customers. The RTX 5090 itself, when it shows up, costs two and a half to almost five times MSRP, as I covered in the verdict.

In this scenario, the traditional way of running a Tier A model at home, with the whole model inside VRAM, is a non-starter. Flash-Next in NVFP4 took about 100 GB of card memory on the B300; nobody has that at home.

Strata flips the premise: instead of demanding VRAM for the whole model, it uses VRAM as a cache and RAM as the warehouse. That’s what lets a Tier A model run on a fraction of the video memory it would otherwise ask for.

It isn’t magic

Strata makes it fit, but it doesn’t work miracles. The stated minimum is 32 GB of RAM and VRAM combined, but that only runs the most compressed version; the IQ3_S that scored 91 asks for about 64 GB of RAM. Add about 80 GB of disk, preferably SSD, and a 12 GB card or better on the official path. Below that it’ll still start, there are reports with an 8 GB card, but at about 15 tokens per second.

Memory bandwidth sets the speed

In token generation, the bottleneck on a local machine is almost never compute. It’s how many bytes per second the memory can deliver to the cores, because every token has to read the model’s active weights. And there the gap between memory types is brutal:

The last row is from my machine: the card is running on 8 PCIe lanes instead of 16, and Strata measures that at startup. One more reason the VRAM cache matters so much.

The Strix Halo has 128 GB and fits a lot more than a 5090, but every byte arrives seven times slower. That’s what I measured in the benchmark: the dense Qwen 3.8 27B in Q8 reads about 29 GB per token, and on the Strix the theoretical ceiling sits near 9 tokens per second. I measured 7.7. The same math on the 5090’s bandwidth would give about 60, seven times more, if the model fit along with the context.

That’s why Strata’s design works: the hot path stays in GDDR7, and whatever misses the cache gets computed by the CPU in DDR5, or crosses PCIe, both dozens of times slower. With over 90% cache hits, the little that’s left out doesn’t drag the speed down. On a small card, the cache covers less, more experts take the slow road, and speed drops: the author himself publishes 53 tokens per second on a 12 GB RTX 5070, a third of my 5090.

A potato PC won’t run it

So forget the laptop with integrated graphics and 16 GB of RAM. Without a reasonable dedicated card, without 32 GB of RAM and VRAM combined and without an SSD, the model won’t load, or it loads and runs at RAM speed, which is horse-and-buggy speed.

And there’s an entry cost that doesn’t show up in generation speed: Strata reads the whole first message of every conversation, and the README cites about 60 seconds per 30 thousand tokens as a rule of thumb. On my 5090 it’s much faster, about 100 thousand tokens in 16 to 20 seconds, but on a smaller card count on the README number. Coding agents send big system prompts, so the first answer of each session takes longer than the ones after it.

Heavy even on my 5090

My machine is no slouch: Ryzen 9 7950X3D, 96 GB of RAM (the system sees about 92), 32 GB RTX 5090. Even so, Flash-Next takes up everything:

• at load time, it pushes about 50 GB of experts into RAM, at about 3 GB per second with a cold SSD, and fills the card’s cache with over 20 GiB;
• with everything loaded, less than 800 MB of VRAM is left free;
• with the server up, the machine sits at 88 of 92 GB of RAM used, and the system starts swapping.

This is btop in the middle of a work session with OpenCode. The Strata process alone holds 54 GB of RAM and half the CPU, with half of the 7950X3D’s 32 threads pinned at 100% and the rest above 50%. The 5090 is at 99% utilization, with 30 of 32 GB of VRAM in use and pulling 351 W. There’s 4 GB of RAM left available, and almost 48 GB already in swap.

The side effect shows up in the rest of the system. With VRAM at the limit, my desktop’s app launcher and the terminal on a secondary monitor stopped opening properly, and other programs get sluggish. The fix was telling Strata to give about 2.5 GB of VRAM back to the desktop, with the --vram-elastic option, at the cost of a slightly smaller cache.

With all that, I firmly believe my machine is the minimum for using a local LLM in real programming work, beyond small exercises. With less than that, count on much lower speed, shorter context and the whole PC suffering while the model is loaded.

How to install

The official path on Linux and Mac is the one-line installer. It checks the SHA-256, doesn’t ask for root and installs into your home folder. Like any curl | sh, download it and read it first:

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh
less install.sh
sh install.sh

After that, strata launch asks for model, context and access, downloads the right engine for your card and the model, and starts serving. For scripts, everything goes through flags:

strata launch                                # guided screens
strata launch --model <variant> --yes --no-tui   # no questions
strata launch --status                       # what is being served
strata launch --stop                         # stop
strata monitor --url http://127.0.0.1:8080   # speed, GPU, memory

I used the IQ3_S variant, the one that scored 91 in the benchmark. On disk it takes about 84 GB in two files (the 55 GB model and the 29 GB lookup table), plus about 7 GB for the speculation head.

The configuration strata launch generates is a JSON with the engine arguments. The ones that matter in mine:

"args": [
  "--expert-cache", "auto",
  "--prefill", "auto",
  "--spec", "4", "--spec-min-p", "0.5",
  "--mtp", "/mnt/data/strata-models/mtp/rt",
  "--max-context", "262144",
  "--kv", "int8",
  "--kv-resident", "32768",
  "--vram-elastic"
]

--spec 4 turns on speculation with a window of 4 tokens per step, up to 3 guessed by the speculation head; --kv int8 stores the attention cache in 8 bits; --kv-resident keeps only 32 thousand positions of the context cache, the most-read ones, in VRAM; and --vram-elastic lets it give VRAM back to the desktop with the server running.

From 128 thousand to 256 thousand tokens of context

The benchmark ran with 128 thousand tokens of context, which is what the install configured by default. Later I bumped it to 256 thousand, with --max-context 262144, and the loss was small: in the server logs, generation averaged about 190 tokens per second in a short session with 128 thousand and about 175 with 256 thousand, about 10% slower for twice the context, a bit more with VRAM given back to the desktop.

What holds this together is --kv-resident: only 32 thousand positions of the attention cache, the most-read ones, stay in VRAM, the rest stays in RAM, and even so about 97% of the reads from that cache happen on the card. The benchmark’s 91 was measured with 128 thousand; with 256 thousand I measured speed, and quality hasn’t been re-tested yet.

For me, 256 thousand is the minimum for long agent work. With 128 thousand, a real session blows through the context fast, and the agent starts compacting its memory all the time. That gets annoying very quickly, and every compaction throws away details the model will need later.

Short context is a hidden problem with local models. Before looking at tokens per second, check how much context you can serve without the speed collapsing.

Loading and unloading on its own

Keeping about 50 GB of RAM and the entire card tied up all day, just to have the model ready when I need it, makes no sense. So I put together a small proxy, in pure Python, that’s always up on port 8080 and controls Strata:

• it answers the model list on its own, so OpenCode sees the model without waking anything up;
• on the first chat request, it brings Strata up on an internal port and waits for it to be ready, about 40 seconds;
• it relays the responses in streaming;
• after 10 minutes without traffic, it takes Strata down and gives the card and the RAM back to the system.

The core is this:

def ensure_up():
    with _lock:
        if ready():
            return True
        subprocess.Popen([STRATA, "serve", "--config", CFG,
                          "--host", "127.0.0.1", "--port", str(UP_PORT)],
                         start_new_session=True)
        for _ in range(90):          # cold start takes ~35-40 s
            if ready():
                set_reserve()        # give VRAM back to the desktop
                return True
            time.sleep(2)
        return False

def idle_watch():
    while True:
        time.sleep(30)
        if time.time() - _last > IDLE_TTL:
            stop_up()                # free the GPU and ~50 GB of RAM

And a systemd user service keeps the proxy up from login (trimmed version):

# ~/.config/systemd/user/qwen-strata.service
[Unit]
Description=Qwen3.8 Flash-Next on-demand proxy (Strata RTX 5090)
After=network-online.target

[Service]
Environment=QWEN_IDLE_TTL=600
Environment=QWEN_VRAM_RESERVE_MIB=2560
Environment=QWEN_UP_PORT=8781
Environment=QWEN_PROXY_PORT=8080
ExecStart=%h/.local/bin/qwen-strata-ondemand
Restart=on-failure
KillMode=mixed

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now qwen-strata.service

The price is the wait on the first message after some idle time. In exchange, when I’m not programming, the machine is mine again.

Configuring OpenCode

Strata has a shortcut, strata connect opencode --write, that writes the configuration after asking and keeps a backup. By hand, it’s an OpenAI-compatible provider pointing at the local port:

"provider": {
  "strata-5090": {
    "npm": "@ai-sdk/openai-compatible",
    "name": "Strata (RTX 5090)",
    "options": {
      "baseURL": "http://127.0.0.1:8080/v1",
      "apiKey": "local"
    },
    "models": {
      "qwen3.8-flash-next-iq3_s": {
        "name": "Qwen3.8 Flash-Next IQ3_S (Strata 5090)",
        "reasoning": true,
        "tool_call": true,
        "limit": { "context": 262144, "input": 229376, "output": 32768 }
      }
    }
  }
}

Any key works, because the server is local. Strata itself warns that OpenCode needs limit.input, to know how much of the context is left for input; in the example I set it to the context minus the max output. Claude Code works too, by pointing ANTHROPIC_BASE_URL at http://127.0.0.1:8080 and setting CLAUDE_CODE_MAX_CONTEXT_TOKENS to the context size, since Strata speaks the Anthropic API as well.

One detail: by default Strata serves one request at a time. If your agent fires off subagents in parallel, they get queued.

And day to day?

If you have the machine for it, the result is good. Flash-Next stays above 150 tokens per second most of the time, as you can see in the screenshot at the top, and tool calling is solid: in the benchmark it ran all seven sprints in OpenCode without a single nudge, and in normal use it reads files, runs commands and edits code without tripping over the format.

In practice, I don’t feel much difference from Claude Code, or from OpenCode with GLM or Kimi in the cloud. And the cloud isn’t at its best right now: in my perception, all those services have been a bit slower than usual lately, probably struggling with more demand than available capacity. Running at home, the speed is always the same, and the code never leaves my machine.

Even with good tokens per second, I still feel Flash-Next is a bit less responsive than a commercial model. There are the seconds of reading on the first message of each session, the pause when it needs experts that aren’t on the card, and the cold load after some idle time. That’s to be expected: we’re running a server model with resources at the limit.

Considering it’s free and runs on my machine, it’s almost good enough. For the first time, running a local model for programming feels like a genuinely usable scenario, and I find myself using it more and more as a replacement.

The catch is still the hardware. A 5090, 96 GB of RAM and a strong processor are the minimum for this scenario, and that hardware keeps getting more expensive and harder to find. But for those who already have it, Tier A at home is no longer a promise.

I went back to two machines I already had: a workstation with an RTX 5090 and a Strix Halo mini PC. And with a new inference engine, Strata, which promises to make big MoEs fit on a regular PC. The short answer: a 125B model already makes Tier A on an RTX 5090. The rest is more complicated.

The machines, side by side

Before any scores, understand the physical difference of what I’m comparing:

The B300 has over 30 times the Strix Halo’s memory bandwidth. The 5090 has 7 times. That explains why the same model flies on one machine and crawls on the other, and why quantization stops being a detail and becomes a design decision.

And there’s the shared-memory detail: on the Strix and the Spark, the same RAM serves CPU, GPU and operating system. It’s not dedicated VRAM, so the real slack is much smaller than the pretty number on the box.

The ranking on consumer hardware

The yardstick is the same as the v4 sabotage benchmark: 14 planted defects, a review-vigilance score, Tier A starting at 83.

GLM-4.7-Flash completed the whole protocol (after a first nested attempt that needed a retry) and still failed to detect sabotage. And there’s a humility lesson in the Qwen 27B: it scored 80 on the same task where it scored 80.5 on the B300, but at 7.7 tokens per second on the Q8 probe and 706 minutes on the clock. A good score doesn’t pay for productivity.

I left out the tests that don’t have a final score yet: Qwen3.8 Flash-Next on the Strix with Strata is only confirmed as executable, and the dense 27Bs were discarded on the Strix by the 30 tok/s cutoff.

The tuning that makes a model run well on a PC

Consumer hardware doesn’t forgive lazy configuration. What I learned in practice:

• On the 5090, compile llama.cpp for sm_120. A ready-made image falls into a slow PTX fallback. It’s -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120, driver 570+, and no --ctx-size 0 on a 32 GB card: the training-context default blows the VRAM on the first allocation. Context needs an explicit ceiling, and if you get OOM on the first request, cut 25 to 50%.
• KV cache in q8_0. Halves KV memory with no measurable quality cost. Flash-attention on by default, but GLM 4.7 Flash needs it off, because its deepseek2 architecture is incompatible.
• Hybrid reasoning is work. Qwen3.8-27B, with the SSM/Mamba architecture, needed a fresh llama.cpp/Ollama build because of ssm_conv1d. And “no reasoning” mode didn’t prevent a coherence loop in one of the attempts; the way out was resuming on the Strix with Q8_0 and --reasoning off. It wasn’t extra memory that fixed it, it was turning off the thinking.
• On the Strix, the adjustment is ROCm, not magic. HSA_OVERRIDE_GFX_VERSION, GGML_HIP_NO_VMM, -dio. Its trump is capacity (128 GB), not speed (~256 GB/s). Q4_K_M gave 12.6 tok/s and Q8_0 gave 7.7 tok/s, below my 30 tok/s cutoff for interactive use. That’s why the Strix became an escape hatch for big models, not a fast chat box.
• Speculation only helps with high acceptance. In a test by Mihai Farcas on an RX 7900 XTX, llama.cpp’s n-gram speculation, with no draft model, had only 5 of 64 tokens accepted, and throughput stayed flat in one run and dropped about 2% in the other. The gain shows up with a drafter made for the model, like the helper Strata uses on Flash-Next.

What is this Strata thing

Strata is an open source inference engine, MIT license, built around one model: Qwen3.8 Flash-Next, a 125-billion-parameter MoE that normally asks for a server. It also runs smaller MoEs, like the Ornith-35B I tested here. The code is C++ and CUDA and reuses parts of llama.cpp/ggml. It has builds for NVIDIA (CUDA 12 and 13), AMD (including Strix Halo), Intel Arc and Apple Silicon, and brings up an OpenAI- and Anthropic-compatible API on localhost, which plugs straight into Claude Code, Codex CLI or OpenCode.

Careful with the name. Mihai Farcas published another engine called Strata, in Rust with HIP kernels, for small dense Qwen 3.5 models on an RX 7900 XTX. They’re different projects, and the one that ran here was the first.

The problem it attacks is the big MoE on a small machine. The strategy is expert-offload, in layers:

• the model has 24,576 experts, 512 in each of the 48 layers, and each token uses only 10 per layer;
• the GPU holds about 4 thousand experts, the most used ones;
• the system RAM holds all of them;
• a lookup table of about 29 GB stays on the SSD, and the model reads a few rows of it per token.

On top of that, a small helper guesses the next words and the big model checks them all at once. The README promises the same answer 1.6 to 1.8 times faster. The minimum is 32 GB adding RAM and VRAM, plus about 80 GB of disk, and the IQ3_S variant I used is about 55 GB.

What made me believe in it was the result: Qwen3.8 Flash-Next on the 5090 with Strata scored 91.0, with zero nudges and all 14 items solved, 1.5 points behind the 92.5 the same model scored on the B300 with vLLM. The report itself calls that “within the noise”. A community project, on a consumer card, sat glued to the same model served in a datacenter in NVFP4.

And now it can see

Until yesterday, my Strata was blind: sending an image got back “this server was started without the vision encoder”. I enabled vision by hand, and it was simpler than it looked.

The vision projector (the mmproj) I had already archived on the NAS inside the uncensored Flash-Next variant. I checked its sha256 against the archive’s manifest and against the hash Strata pinned for the BF16 encoder, and copied it into the models directory. The configuration gained the --vision flag and a vision section pointing to the strata-vision component and the mmproj, with a backup beside it.

The verification was end-to-end: the encoder loads in half a second, warms up 1,024 image tokens in under two seconds, and answered “Red” to a red test square. And the detail that matters most to me: with the encoder resident, 3.65 GB of VRAM remained free, so the desktop’s 4 GB reserve still holds. The only gotcha: lazy loading doesn’t coexist with the vision section in this configuration, so it stays off here.

Careful: Strata eats the whole GPU, and the desktop dies

There’s a cost that doesn’t show up in benchmarks: running Strata on the same GPU that draws your desktop. It was designed to fill the card, and it does.

In my case, the engine allocated 32,085 of the 32,607 MiB on the 5090, 98.4% of the VRAM. Hyprland was left with about 500 MiB, desktop allocations failed, and even the terminal stopped responding. Only a hard reset solved it.

It was two failures in sequence. The reserve I had configured was 2,560 MiB, but the proxy fired only once, swallowed the exception and never tried again. The engine’s /v1/vram endpoint wasn’t up at that moment, so the reserve was never applied. And 2,560 MiB was already too little for Hyprland, browser and terminal together.

The fix, validated under load:

• 4,096 MiB (4 GiB) reserve, with retry for up to a few minutes, engine-response verification and a confirmation log. No fire-and-forget.
• Re-assertion of the reserve every 30 seconds while the engine is loaded, to close any rebalancing drift.
• In the first load test, with the GPU at 99%, 3,629 to 3,888 MiB stayed free during the whole generation. In the following test, one long generation and three other concurrent requests finished successfully; the smallest slack observed was 3,949 MiB. The desktop stayed alive.

The price shows up in throughput. The expert cache dropped from about 11,700 slots to 9,700, so more experts travel over PCIe. In the initial measurement, it went from ~200 to ~150 tokens per second. I happily pay that: a desktop that doesn’t freeze is worth more than those 25% of speed in that test.

There’s a structural alternative: this machine has an integrated Radeon 610M, and Hyprland can be moved to it. The 5090 would be dedicated to inference.

The cost is that the 610M is weak for gaming and NVIDIA multi-GPU on Wayland has its quirks. I preferred the elastic reserve, because the model also unloads when idle.

The general lesson: never let an inference engine take 100% of a GPU that also draws your screen. In this configuration I don’t have a rigid driver-level VRAM reservation for display clients. I had to solve it on the engine side: reserve a few GB, re-assert after every cold start, and measure again with the desktop open.

The same 32 GB also limit the session

The 5090’s math is tight on all sides. The same 32 GB have to accommodate the model’s dense weights and fixed buffers, the hot-expert cache, the resident part of the KV cache and also the 4 GiB reserve for Hyprland, browser and terminal. What’s left for one thing comes out of another.

KV cache is the conversation’s memory. Every token of the prompt, of the previous answers and of the files read creates state that the model needs to consult in the next tokens. In my Strata it’s in int8, with 32K cells per layer resident on the GPU.

The rest can be streamed from RAM, which is why raising the window from 128K to 256K didn’t increase VRAM or drop speed in my synthetic test. That streaming avoids the immediate blow-up, but it still uses memory and doesn’t turn a 5090 into a limit-free server.

The ceiling configured here is 262,144 tokens, or 256K. Several commercial models I tested, including the newest Claudes, already work in the 1-million-token class. Almost four times the window changes how you work: in Claude Code you can accumulate much more history before compaction; in the local Qwen I have to treat context as a budget from the first prompt.

The conversation that shows up in the terminal is only part of the context. Depending on the harness, the system prompt, AGENTS.md and CLAUDE.md, skill catalogs and loaded content, tool descriptions and schemas, MCP servers, plugin instructions, command outputs, files and the space reserved for the answer also go in. A well-implemented skill loads its full text only when called, but its catalog can still enter the prompt. A plugin that registers twenty tools also sends twenty schemas before you ask for anything.

That’s why the benchmark didn’t use my normal OpenCode. The runner generates an isolated configuration, without the global plugins and MCPs, with only the providers and agents needed for the test. It also pins the real 262,144-token limit instead of pretending the model accepts the context advertised by another backend.

To use this Qwen day to day, I’d do the same cleanup:

• start with the leanest possible harness and activate plugins or MCPs when the project really needs them;
• keep heavy skills on demand, without dumping whole manuals at the start of the session;
• compact or open a new session before touching the ceiling, recording important decisions in project files;
• reserve space for the model’s output. A window stuffed with input is useless if there’s no token left for it to think, call tools and answer.

With 256K, context is a hardware resource. Installing too many integrations and loading everything into every session consumes window and memory before the work starts.

Wasn’t Qwen 3.8 too slow?

It was, and I wrote that. In the v4 ranking, the local Qwen 3.8 27B took almost 12 hours to close the seven sprints. In the B300 article I summarized: scored 80, took 706 minutes, “so don’t even try”.

Except that’s another Qwen 3.8. The family has both, and they behave in opposite ways at home:

Why the bigger one runs faster

In token generation, a local machine is memory-bandwidth-bound: each token needs to read all the active weights. The dense 27B reads its 27 billion every token, about 29 GB in Q8. The Strix moves about 256 GB/s, so the ceiling sits near 9 tokens per second, and I measured 7.7.

On the 5090 the same 27B would be fast, but in Q8 it doesn’t fit in the 32 GB along with the context that seven sprints accumulate. In Q5_K_M it fit, but got stuck in a reasoning loop on sprint 3. It finished on the Strix, in Q8_0 with reasoning off (--reasoning off), which is what unlocked the loop.

Flash-Next has almost five times more parameters in total, but only about 6 billion work per token. In 3 bits, that’s around a tenth of what the 27B in Q8 needs to read. The hot experts stay on the 5090’s GDDR7, with seven times the Strix’s bandwidth, and the speculation helper gets tokens ahead. Result: about 20 times the 27B’s speed on the Strix, with a higher score.

The turnaround came from the model and engine swap: a MoE with few active parameters, served by an expert-offload engine made to measure for it. No configuration flag would pull this out of the dense 27B on the Strix.

The tuning still charges

The price of running at home changes from model to model. Flash-Next in 3 bits lost only 1.5 points against the B300’s NVFP4 version. Ornith-35B, on the same engine and the same card, lost 16.5 against its BF16 version on the B300. Measure your model before trusting it.

What each model did that’s interesting

Flash-Next: the proof that consumer reaches Tier A. 34 of 40 unannounced, plus 6 with reveal weight, and only two items needing a pointer. No nudges, all sprints clean. It’s the only local one that combines a Tier A score, interactive speed (159 tok/s) and the full protocol without help. If the question is “can I use it at work”, this is the example.

Ornith-35B: fast, but not good enough. The same Ornith that scored 81 on the B300 in BF16 scored 64.5 on the 5090 with Q8_0. Unannounced detection dropped from 30 to 23. Quantization, engine and hardware all changed at once, so the cause can’t be isolated: this Q8_0 + Strata + 5090 combination lost 16.5 points, and I don’t know how much came from each variable. The message is clear: “runs fast” is not “reviews well”.

Qwen 27B: good score, terrible experience. It scored 80 on the Strix resume, with sprints 1 and 2 on the 5090, close to the B300’s 80.5. But 7.7 tok/s on the Q8 probe and 706 minutes of wall time. And on top of that zero fixes on reveal, with four items open. Using this day to day is choosing to wait.

GLM-4.7-Flash: the engine ran, the model failed. Full protocol in 29 minutes, and zero unannounced detection. It recovered 8 of 14 only when pointed, and even then left the fixes uncommitted. On the B300 it scored 33.5, which isn’t good either. The problem isn’t just the card.

The distance between the categories

The question every dev has is whether consumer already gets there. Direct comparison, on the same yardstick:

The reading is that the consumer top is 9 points below the absolute commercial top and 1.5 points below the open datacenter top, on this vigilance metric. It’s a small distance, and it vanishes when you notice GPT-6 Sol and Sonnet 5 are also at 91. The real abyss isn’t between datacenter and consumer: it’s between the right model and all the others. Between Flash-Next and Ornith, on the same 5090, it’s 26.5 points.

So, is consumer hardware good enough to work with?

Yes, with the right model and engine. An RTX 5090 with Strata delivers Tier A code-review vigilance, at 159 tokens per second, without paying a cent per token once the hardware exists. For anyone doing code review, testing coding agents and keeping private data at home, that changes the game.

For the first time in this benchmark I have a tested model, on consumer hardware, that deserves to go into everyday work. I’ve already configured Flash-Next in OpenCode pointing at the 5090. The next step is using it in real programming, outside the sabotage yardstick, and seeing whether the score holds.

What doesn’t work is thinking any configuration works. The Strix Halo is great for having 128 GB and mounting a big model, but its speed doesn’t pay for interactivity; Ornith runs fast and reviews badly; GLM-4.7-Flash completes the protocol and detects nothing unannounced. And there’s the invisible price: aggressive quantization can charge score, depending on the model and the configuration, and waiting time charges patience.

If I were building a box for a dev today, it would be a 5090 with Strata and Flash-Next, and I’d use the Strix for background tasks that accept being slow. The commercial frontier keeps winning for whoever needs the maximum. But consumer stopped being a consolation prize: there’s already Tier A on the home shelf.

The problem now is finding a 5090

In October, leakers claimed NVIDIA would stop producing the RTX 5090 and reserve the GB202 chips for the professional and datacenter lines. The card even disappeared from NVIDIA’s official US store. On October 16, at an event in Taipei, Jeff Fisher, NVIDIA’s senior vice president, denied the discontinuation and said the company will keep delivering, but admitted production capacity is tight.

In practice:

• the suggested price is still US$ 1,999, and most listings on American Newegg go past US$ 9,300, almost five times that;
• in Brazil, each unit goes for around R$ 35 thousand at retail;
• and two pallets of Zotac RTX 5090s, imported by the Oderço distributor and valued at about R$ 2.5 million, disappeared at Guarulhos airport. GRU Airport notified the authorities, and the case remains under investigation, with no theft confirmation.

The v4’s first local Tier A model runs precisely on the hardest card to buy right now. The home shelf already has Tier A. Getting the shelf is what’s missing.

The answer depends on two things: how many comfortable users the hardware holds, and how much that costs per person. We measured both.

The math before the test

Recapping the numbers from the previous article: the 4x B300 pod costs US$ 36 per hour (US$ 864/day, almost US$ 26 thousand/month running nonstop). I started the campaign with about US$ 127 of balance, topped up US$ 2,550, and US$ 537 were left. That was about US$ 2,140 between setup, downloads, tuning and benchmarks, not just inference.

In the middle, the balance hit zero, the pod died during an autotune, and when it was time to top up there was no B300 available. That’s Runpod’s operational risk: the price changes and the supply disappears. The alternative was a 4x H200 pod, with list price around US$ 3.59 per GPU-hour on Community Cloud: US$ 14.36/hour for the four at list math, against US$ 36 for the B300s.

B300 and H200 are not the same machine

The hardware difference changes more than the price:

The swap had an adaptation cost: the GLM’s NVFP4 checkpoint that ran on B300 doesn’t exist for Hopper, so its FP8 variant had to be downloaded, and even DeepSeek needed kernel tuning. The concurrency results below are with both in native FP8, which is the fair comparison possible.

How we measured “a team fits”

The test uses vllm bench serve against each pod’s localhost, with requests of 8,000 input tokens and 800 output (a synthetic load in the shape of a coding session), raising concurrency N step by step.

The comfort criterion is what matters: TTFT up to 2 seconds and TPOT up to 20 ms per token (that is, at least 50 tokens/s per stream), with 95% or more of requests meeting both limits. It’s not “the median looked good”: it’s almost everyone well served. A team that waits 50 seconds for the first token is not a team, it’s a queue.

And the caveat: this measures comfortable code requests, not people. I don’t have a measured duty-cycle multiplier for interactive agents, so I won’t invent that each N becomes 2 or 3 developers.

The numbers

Read carefully, because there are three stories here.

The B300 serves four times more people in comfort. Same model, same FP8, same TP4: 32 comfortable users against 8. The H200’s aggregate is even impressive (2,506 tok/s), but that’s a prefill queue: p99 TTFT reaches about 51 seconds at N=256, so the user waits a minute before seeing anything. High throughput is not interactive capacity.

The cliff is so steep it’s worth looking at the numbers up close. At N=16, p99 TTFT was 1.17 s on B300 and 3.19 s on H200, with 100% and 92.7% of requests good. At N=32, the B300 still held 97.4% inside the SLO (2.23 s), while the H200 collapsed to 8.3% (6.58 s). And at N=48 the B300 fell to 92%, below my 95% gate: that’s why comfortable is 32 and not 48. The difference between “works” and “works for everyone” lives in those last 2 percentage points.

Heavy and light models tie on the H200. The 510 GB DeepSeek and the smaller GLM sustain the same 8 comfortable users, for different reasons: DeepSeek has efficient KV via MLA and dies of compute/prefill; GLM blows the KV cache (97% usage). Choosing hardware by parameter count is looking at the wrong place: the KV architecture is what rules concurrency.

The loose gate deceives. If you only look at the median, you could say 48 on B300 and 16 on H200. Except the p99 tail destroys the experience of whoever is on the other side. To size a team, I use the comfortable number and that’s it.

Cost per comfortable user

Now the deciding arithmetic:

• 4x B300: US$ 36/h ÷ 32 = US$ 1.13 per comfortable user per hour.
• 4x H200: US$ 14.36/h ÷ 8 = US$ 1.80 per comfortable user per hour.

The B300 wins per person even with the pod costing 2.5x more. The H200 would only become cheaper per user below US$ 9/hour for the whole pod, and on the current table it costs US$ 14.36. The cut-off point isn’t the machine’s price, it’s its capacity.

What if your company already lives on AWS or Azure?

You can build the same infra on the hyperscalers, but the price changes category. The list values I checked, per GPU-hour:

B300 on AWS costs 1.8x what I paid on Runpod, and H200 on Azure costs almost 3x Runpod’s. The capacity conclusion doesn’t change (the physics of tensor parallelism and KV cache is the same), but the “worth it” math gets considerably worse: if on Runpod the B300 ties the subscription around 30 users, on AWS it starts losing by a comfortable margin.

And in practice, how much does it come out to per developer, turning the pod off at night like in the other calculations (8 hours per business day, 176 hours/month)? The AWS and Azure rows are normalized arithmetic for 4 GPUs, because the cited SKUs come with 8 and I didn’t measure TP=8:

The difference is brutal: the same comfortable developer costs US$ 198 on Runpod and US$ 933 on Azure, almost five times the price of the strongest frontier subscription. AWS sits almost 80% above Runpod, so even it only makes sense if your requirement is something other than cost.

And if you take the whole 8-GPU instance, as they’re sold? AWS comes out at US$ 22,734/month and Azure at US$ 14,925/month, and then you have to measure TP=8 to know how many users you can use, because with TP=4 you pay double for the same capacity.

The commercial models differ too: capacity block on AWS is not on-demand, and Runpod’s Community Cloud is not production SLA. So read this table as an order of magnitude of price, not a contract quote. If your requirement is compliance, procurement or residency in a specific region, the hyperscaler may be unavoidable, and then you pay the premium. If the requirement is just price, the neoclouds win in these configurations.

What if, instead of renting, I buy the machine?

When the rental bill hurts, the natural question is to buy the hardware. Two candidates from the current shelf:

DGX Station GB300 (~US$ 130 thousand, with configurations starting around US$ 95k): 748 GB of unified memory (252 GB of HBM3e at 7.1 TB/s and 496 GB of LPDDR5X at 396 GB/s), 20 PFLOPS in FP4. The weights of every model I tested fit on it, and the best ones to put on it would be:

• DeepSeek V4.1 Flash (92.5/A, 510 GB in FP8), the benchmark champion;
• Muse Spark Open (69.5/C, 533 GB in NVFP4);
• Step-5-Preview (72.5/C, 368 GB in NVFP4);
• GLM 5.3 full (76/B, 704 GB in FP8).

It fits, but fitting is not serving: the full GLM would leave only about 44 GB for KV cache and system, so full long-context concurrency remains to be measured. The comfort estimate is single-digit, in the H200’s line, not the four-pod’s 32. The difference is you don’t need TP=4: everything lives on a single chip.

DGX Spark (~US$ 6,950 on NVIDIA’s current store): 128 GB unified, 1 PFLOP in FP4, 273 GB/s of bandwidth. The good filling for it is small and medium models, and it’s better than it looks:

And there’s a detail that hurts to see on the bill: by my own estimate, without a closed audit of all the first round’s costs, adding the v4 credits (those ~40 commercial models) to this B300/H200 campaign’s US$ 2,140, I’ve already spent a few thousand dollars between API and pods, close to half the price of a DGX Spark. And even so, I wouldn’t have a machine sitting at home because of it.

• Qwen3.8 Flash-Next (92.5/A), the single-B300 co-leader. In IQ3_S it lands in the tens of GB range, and it’s exactly the model that scored 91.0 on the RTX 5090 with Strata. It’s the best model I’d recommend for a Spark today.
• Swift 1.5 27B (88/A, 56 GB), which was the most balanced self-host recommendation of my previous campaign;
• ThinkingCap 3.8 27B (83/A, 56 GB);
• Ornith-1.5-35B-A3B (81/B, ~70 GB in BF16);
• Laguna-S-2.1 (65.5/C, 99.7 GB in NVFP4);
• and the small classifiers from the decision benchmark (Jebadiah 4B, Laya, Julia), which are the ideal use case for a box like this.

The DeepSeek V4.1, at 510 GB, doesn’t fit at all. The bandwidth is almost 30 times smaller than the B300’s, so interactivity suffers, and practical concurrency remains to be measured: I expect little more than solo use. For small models it’s a nice little box; for a monster, it’s a paperweight.

TCO, ROI and depreciation

The own-versus-rent math, using 3 years of depreciation and the same usage pattern (8 hours per business day, 176 hours/month):

Reading: if you run a big model every business day for two years or more, the GB300 pays for itself in under two years, and in 24/7 it pays for itself in five months. If your need is a two-week campaign like mine, renting cost US$ 2,140 and done, with no idle asset afterwards. And the Spark only makes sense if your load is small models: in that case it pays for itself in under a semester against renting and in three years against the subscription, but if your need is a 748B, it’s useless from the box.

Two caveats so you don’t buy wrong. First, the table compares dollar scenarios, not equivalent-workload ROI: if your real concurrency on the GB300 is single-digit, the fair comparison would be with a pod smaller than the 4x B300. Second, energy and administration change the result, especially on the Spark, where the difference against the subscription comes down to a few dollars per month. I ignored resale at the end of the three years; if it exists, the break-even improves.

Note for those buying in Brazil: these prices are in dollars. In reais, with the dollar at R$ 4.99, plus import taxes and the local reseller margin, the GB300 easily passes R$ 800 thousand and the Spark passes R$ 34 thousand before any margin. The break-even in reais gets even further away, so the decision to buy in Brazil needs much higher utilization to close.

And comparing with a frontier subscription? A strong Claude/Codex plan costs about US$ 200 per month per developer. Two important caveats: those plans aren’t unlimited (Codex meters by credit and Claude has per-session and per-week usage windows), and automation/CI doesn’t fit consumer plans. The comparison below is price-only, not equivalent workload. To turn streams into people I need a premise: one developer with one active request the whole workday, which is optimistic for capacity and conservative for cost. With that premise, and not counting my sysadmin hours or setup, the pod running 8 hours per workday (176 hours/month) looks like this:

Around 30 continuous streams, the B300 lands practically at the strong subscription’s price. Below that, the subscription is cheaper, because you pay per person, not per idle pod hour. And running 24/7 without matching load is throwing money away: the pod charges while sleeping.

For a company, when the math closes

If you have a genuinely big team using it all day, data that can’t leave the environment, and a model that delivers on the daily job (DeepSeek V4.1 Flash scored 92.5 on my vigilance benchmark, above GPT-6 Sol and Sonnet 5), your own infra starts making sense for the first time without being vanity. The cut-off point, on the premise of one active stream per person the whole workday, lands around 30 concurrent people, with the pod off outside working hours.

If the team is small, usage is intermittent, or the goal is just to save money: stay with the API. The same DeepSeek cost US$ 1.21 per full benchmark round in the cloud, and you don’t inherit a pod to administer, nor the risk of the B300 vanishing from stock when you need it most.

What I take from this campaign: big hardware is not an answer to a small question. Before signing the pod, do the math on how many people actually run concurrently on your team, measure the latency they tolerate, and add the price of your sysadmin hour. The big open model already reached the Sonnet 5 and GPT-6 Sol band for daily code review. The infra just needs to fit the budget.

The short answer: pretty far, and one of them tied the Sonnet 5 and GPT-6 Sol band. The long one comes with a grocery bill and several days of vLLM suffering.

First of all, the scope caveat. Some models were left out.

Kimi K3, at 2.8 trillion parameters, is the example: in FP8 it would need about 2.8 TB of VRAM, and not even an 8x B300 pod (2.3 TB raw) closes the math. To run a monster like that you need to quantize to FP4 or rent a lot more cards. And I chose to spend my credit on what fits in 4x.

The full ranking

Same methodology as always: the RubyLLM chat app with 14 planted defects (backdoors, authorization bypasses, lying tests), and the score measures code-review vigilance, not typing speed. The model has to find and fix the defects unannounced (full weight) and when revealed (40% weight). Tier A starts at 83. I ran everything on the same 4x pod to have a single yardstick, although not every model needed all four cards (Ornith and Laguna would fit on one).

Two notes to read the table properly. Step-5 and Muse are in C because the unannounced base was low: a fix after the “there’s a problem there” is worth 40%. Muse fixed 6 of the 7 open items when revealed; Step-5, only 2 of 5.

And Nex N2.5 Pro, on top of the shameful score, corrupted the code to the point the app wouldn’t even boot. The two anchors from the previous campaign, for comparison: GLM 5.3 Flash and Qwen3.8 Flash-Next, both 92.5 on the single B300.

What each one did that’s interesting

The score alone hides the behavior, and the behavior is the fun part.

DeepSeek was the only one that solved both compound items unaided. The compound defects are the ones that require seeing two things at once (the backdoor and the test covering it), and it found and fixed both with no nudge. That’s the kind of vigilance I want from my PR reviewer.

The full GLM fixed everything when told, but slept through silent mode. Unannounced, it let the access and silent-correction cluster through. When I pointed, it swept the six remaining items. In other words: the big model is capable, it’s just not attentive by default. Its Flash, at a third of the size, was more vigilant.

Kimi K2 Thinking did security-checklist theater. It wrote these beautiful “security best practices” lists and left five items open, including a plaintext key. Talking about security and doing security are different things, and it did the first one.

Ornith restored the behavior and kept the lying test. On item 6, it reverted the code to the old behavior instead of fixing the planted test that validated the backdoor. Hence the strict score of 73 and the lenient 76: looking fixed is not being fixed, and the lying test also needs to die.

Step-5 left three defects even with the map in hand. After being explicitly told, it still left the login link, the authorization deactivation and the wrong aggregate. Not every model takes the second chance.

And there’s the case of the 397B that became a ready-made joke. A 397B model, too big to fit in the head of whoever defends size, let the backdoor admin? => admin || email.end_with?("@internal.example") through even after the security-audit warning. Worse: it added a passing test that codified the backdoor as intended behavior, and the commit still bragged “Brakeman: 0 warnings”. If you want an example of why I test vigilance and not marketing benchmarks, there it is.

Muse Spark Open almost didn’t get off the ground, but recovered. It started blocked: weights in 127 NVFP4 shards, gated on Hugging Face, and no support in stable vLLM. It only ran when I compiled a PR branch still in draft, with nvcc 12.9 and 12.8 libraries mixed, and capped the multimodal to text.

Then it delivered 69.5: weak base unannounced (21 of 40), but 6 of 7 items fixed when pointed. It’s the case of “infra effort is not the model’s score”: the service stayed up, the behavior is useful, and even so it only reaches Tier C. If the question is “why isn’t it at the top”, the answer is default vigilance, not VRAM.

And Nex was the complete disaster. It needed two nudges, and in the end the corrupted text reached the committed Ruby and the app won’t even start. In the cloud, the same Nex N2.5 Pro had scored 94. At home, 27. I wouldn’t trust a vendor benchmark after seeing this twice.

By the way, Nex’s case is the perfect example of a bad base choice. When Rio 3.5 blew up in June, I covered the controversy here: its checkpoint was a blend of Nex-N2-Pro with Qwen. The Rio folks could have chosen much better bases, like DeepSeek V4.1 or GLM 5.3 Flash, and went exactly for the lineage that, on my current yardstick, was the worst of the bunch. Picking a model by its pretty name costs dearly.

Running on 4 GPUs is not trivial

Four cards are not four models: it’s one model sliced into four, with tensor parallelism TP=4 in vLLM. That gives about 1.1 TB of aggregate VRAM (in practice, ~1,038 GB usable with --gpu-memory-utilization 0.90). Without that, none of the models above 400 GB would exist in my life without offloading to SSD, which is where speed goes to die.

The cares that repeated on almost all of them:

• Bleeding-edge toolchain. B300 (sm_103a) requires CUDA 12.9 to compile the kernels on the fly, so the default image wasn’t enough: standalone toolkit with nvcc 12.9 and the right headers, and even then each family needed a DeepGEMM, FlashInfer or cudart-link patch. When the compilation gets it right, the prize is big: GLM 5.3 Flash went from 10.35 to 141.40 tokens per second from configuration alone (removing --enforce-eager and enabling compiled mode with CUDA graphs), without changing checkpoint, context or KV precision.
• Startup is not download. DeepSeek took 15 to 30 minutes to initialize, so be prepared to wait for the model to load before measuring anything.
• Each family has its parser and its quirks. Nex only talked after switching the parser to qwen3_xml, getting one more nudge and ignoring the tmp/log artifacts. Step-5 didn’t exist in stable vLLM 0.32 and needed a nightly with a specific PR. Muse required compiling a PR branch still in draft, with CUDA 12.9 nvcc and 12.8 libraries mixed, and weights in 127 NVFP4 shards. MiMo asked for --trust-remote-code. Laguna decided, out of nowhere, to flood the output with reasoning text, and one sprint had to be redone from the injection base.
• Context is expensive. Kimi K2 Instruct’s FP8 checkpoint (1,029 GB) would leave 9 GB for KV cache on the 4 cards, so the 594 GB INT4 Thinking went in its place. Speculating with a draft head (MTP) only worked on Step-5, with a 2.24x gain and 73.7% token acceptance, because I verified the tools’ correctness first.

The operational lesson: renting VRAM solves “fits”, but not “runs well”. Every big model is a mini integration project.

And where’s the frontier?

The question that matters to the developer isn’t “who has more parameters”, it’s “does the big open model deliver in my day to day?”. The v4 benchmark numbers, on the same yardstick:

DeepSeek V4.1 Flash tied its own cloud service, at the same 92.5, and finished above GPT-6 Sol and Sonnet 5. In other words: the best current open source is already 1.5 points behind hosted GLM and 7.5 behind the absolute top, ahead of two strong commercial frontier models. If your yardstick is mine (find the backdoor and the bypass before merging), a local 748B open source is already good enough for daily code review, at 175 tokens per second per stream in my admission measurements.

But three warnings before you cancel your Claude subscription:

1. More parameters doesn’t mean more competence. The full GLM 5.3, at ~700B, scored 76 against its own Flash’s 92.5. Kimi K2 Thinking, at 1 trillion, scored 46. And Nex N2.5 Pro plummeted from 94 in the cloud to 27 at home: a public checkpoint is no guarantee of reproducing the service, as I had already seen with DeepSeek V4 Flash (86 in the cloud, 44.5 at home).

2. My yardstick is vigilance, not general programming. Here I measure who notices sabotage. That’s not the same as measuring who writes your whole system better, and a rerun of my benchmark already varied 9 points on the same model. The 1.5-point distance between models may be noise.

3. Cost changes the answer. If the idea is frontier quality day to day, DeepSeek V4.1’s cloud API cost US$ 1.21 per full benchmark round. The 4x B300 pod costs US$ 36 per hour before you even press the button. And the most didactic case is still DeepSeek V4 Flash from the previous campaign: in the cloud it scored 86 at US$ 0.97 per round; at home, 44.5 and on top of that more expensive per inference. Sometimes the cloud is better and cheaper.

Is renting 4x B300 worth it?

For one person writing code with an agent in the morning and another in the afternoon: no. The API solves for a few dollars what the pod solves for dozens of dollars per hour.

For a company that needs truly private data (code that can’t leave the VPN), a pinned checkpoint for reproducibility, or continuous batch load (thousands of reviews, classifications, analyses per day), then the math changes. DeepSeek V4.1 on 4x B300 delivers near-frontier quality with the data at home, and that you can’t buy with a chat subscription.

What I wouldn’t buy again without measuring first is the “bigger is better” premise. I started this campaign thinking the jump past 92.5 would come from a bigger model. It came from the right model running properly. The rest is burned VRAM and a good story to tell.

And there’s one more chapter coming, of non-datacenter hardware: the tests with local models on the RTX 5090 and the Strix Halo, with a new inference engine called Strata. When the protocol is done, the third article comes out.

And the “is it worth it” question has an irritatingly obvious answer: it depends on how many people will use it at the same time. That’s the one I answer in the next article: LLM Benchmark v4: can my team fit in a 4x b300/h200?.

One category was missing that I had never tested properly: the big open source models. The ones that fit on my RTX 5090 with 32 GB or on my Strix Halo with 61 GB usable by the GPU are too weak for this. They work for simple agent tasks, and no small model handles real programming tasks, let alone a complex software project.

Qwen 3.8 27B took 706 minutes to finish v4 on the Strix and scored 80; the local GLM-4.7-Flash scored 24. So don’t even try.

But what if we push harder? What can mid-sized models, 100 to 400 billion parameters, do running on a single rented B300? I could rent four of them and run the giant 750-billion or one-trillion models, and I’m not sure the cost-benefit is worth it. The data below answers.

TL;DR: 14 open models, one 288 GB B300 on Runpod at about 9 dollars an hour, the same v4 sabotage protocol. None of them serves fast the way it ships: GLM 5.3 Flash decoded at 10 tokens per second with the default configuration and at 141 after tuning, 14 times more from flags alone. Two open models, GLM 5.3 Flash and Qwen3.8 Flash-Next, scored 92.5, Claude Sonnet 5 level, and Swift 1.5 27B scored 88, Opus 4.6 level. None reached the frontier’s 100. The 397-billion flagship scored 68.5 and let an admin backdoor through, DeepSeek V4 Flash scored 86 in the cloud and 44.5 served by me, and Nemotron cheated on a regression test. The cloud API of the same model costs a dollar per run; the pod costs 218 dollars a day. Jump to B300 and Runpod, server tuning, the ranking, the interesting cases, the frontier comparison, the math or the conclusion.

What a B300 is, and what Runpod is

The B300 is the datacenter GPU of NVIDIA’s Blackwell Ultra generation, announced in March 2025. Each card has 288 GB of HBM3e memory, nine times my 5090, and about 15 petaflops of dense compute in NVFP4, the 4-bit format Blackwell does in hardware. It’s the card that goes inside the GB300 NVL72 rack, 72 of them interconnected, which AI companies buy by the thousands.

On its own, a B300 costs more than an apartment. What makes it accessible is renting by the hour.

Runpod is a cloud GPU provider that rents cards in “pods”, Linux containers with one or more GPUs, billed per second. You pick the card, an image template, get in over SSH, and pay while the pod is on, idle included. The B300 cost about 9 dollars an hour on their secure cloud the week of the test; an H100 goes for 3 to 4. The container disk is ephemeral: terminate the pod and the 250 GB of weights you downloaded are gone, and you download again.

The pod I used: one B300 with 275 GiB according to nvidia-smi (the same nominal 288 GB, in binary units), 344 vCPUs, 4 TB of host RAM, 700 GB of disk. The inference server was vLLM; the harness was the same OpenCode as every other model in the ranking, connected to the pod through an SSH tunnel. The model runs there; the code, the tests and the results stay on my machine.

Fitting in memory is the first filter. The B300 fits the 100 to 400-billion-parameter models quantized to NVFP4, plus the 27 to 35-billion ones in BF16. Left out are the 753-billion GLM 5.3 flagship, the 750-billion DeepSeek V4.1 Flash, the 1.6-trillion DeepSeek V4 Pro and the 2.8-trillion Kimi K3. For those it would take four B300s or more, about 36 dollars an hour.

The methodology

The protocol is the usual v4: the model builds a Rails app across seven accumulating sprints, and between sprints an independent reviewer, isolated from the model, injects 14 sabotages, from real CVEs to silent logic and test regressions, into the code the model itself wrote. The 0-to-100 score measures vigilance: how much the model catches and fixes without being told (full weight), how much it fixes after an explicit “do a security audit” in the last sprint (40% of the weight), and what stays unresolved (zero). Tiers are A from 83 up, B from 73 to 82, C from 51 to 72, D up to 50.

Two differences this round. The first is the reviewer: the five models that finished before the block were reviewed end to end by Codex, as in the previous rounds.

Midway through the batch OpenAI’s security service started refusing the injection and grading operation with a cybersecurity risk flag, even though it’s my code, on my machine, in a benchmark with 46 models already scored under the same protocol. I switched the reviewer to Claude Opus 4.8 for the remaining models, with the same injection plan, same catalog, same yardstick, and the provenance of every boundary is recorded per sprint. The expected impact on the score is none, because both execute the same written recipe.

The second is admission by speed, which is the subject of the next section. A model that decodes below 100 tokens per second doesn’t even get on the track.

Also: an open model is entitled to up to two short nudges per run for silly defects (a missing import, a wrong test entry point), recorded as “assisted”. Infrastructure failure, driver, loader, parser, is never charged to the model. And the pod ran one model at a time, with no access to any other model’s workspace or results.

None of them serves fast the way it ships

The first lesson cost a day. I downloaded GLM 5.3 Flash, 320 billion parameters with 18 active, quantized to NVFP4, 198 GB. I brought it up on vLLM with the configuration that “works”: it serves, passes the tool-call tests, answers. And it decoded at 10 tokens per second, with the GPU at 10% utilization, 209 watts out of the 1,100 the pod allows.

That’s unusable for programming. A coding agent generates tens of thousands of tokens per sprint; at 10 per second, a 20-minute sprint becomes more than four hours.

The same checkpoint, same 128K-token context, same FP8 KV cache, same test prompt, changing only the compile and mode flags: 141 tokens per second. Fourteen times faster, zero change in precision or weights.

The main culprit was --enforce-eager, which disables vLLM’s torch.compile and CUDA graph capture, and which every tutorial tells you to enable “to start faster”. It starts faster and serves 14 times slower. The measurement changed that flag together with the interactivity mode, so I didn’t isolate each one’s contribution.

Hence the admission rule: below 100 tokens per second I consider unusable, and no model enters the benchmark without clearing that on the median of two warm 256-token probes, after a cold one. To keep it from becoming a bottomless pit, tuning is bounded: at most three configurations and 45 minutes of wall-clock per model, including load and compilation. And tuning can never trade quality for speed: no shortening the context, lowering reasoning effort or quantizing further to hit the number.

The tweaks that paid off for everyone:

• Install a CUDA that compiles for the card. Runpod’s default image ships CUDA 12.8, whose nvcc cannot compile for compute_103a, the B300’s compute capability. Without 12.9, the NVFP4 kernels never compile.
• Never use --enforce-eager. Letting vLLM compile and capture CUDA graphs is the single biggest win.
• --performance-mode interactivity: optimize for single-stream latency, which is what a serial agent needs, instead of aggregate throughput.
• FP8 KV cache on the big MoEs, prefix caching and chunked prefill on all of them, confirmed by vLLM’s Prometheus counters instead of assumed: on an 11K-token prompt, the server reused exactly 8,576 prefix tokens on every repeat.
• The right tool and reasoning parser for each family. The wrong parser makes tool calls vanish silently or leaks the reasoning into the answer.

And the specific tweaks I found most interesting:

• NVFP4 only runs on vLLM. The 198 to 251 GB checkpoints in Blackwell’s native FP4 are the only way to fit on the card, and llama.cpp and Ollama simply don’t load those formats. Whoever lives on GGUF has no access to this range.
• Native speculative decoding. Qwen 3.8 27B went from 92 to 195 tokens per second with the MTP head the checkpoint itself ships, predicting three tokens per step, plus --language-model-only to drop the vision tower (measured on different pods, so no controlled ablation). Muse Glimmer reached 295 with the lightweight DFlash drafter it ships alongside, 15 speculated tokens per step. I only enabled it where an official drafter exists and where tool calling stayed correct; for Ornith a drafter exists as a separate release, with unknown acceptance rate, so it stayed off.
• The CUDA 12 loader. flashinfer JIT-compiles an NVFP4 MoE kernel that needs libcudart.so.12, and after installing CUDA 13 for Nemotron the library resolution broke. The symptom looked like OOM; the cause was an ld.so.conf. A repair script with ldd evidence before and after fixed it, and the 397B went on to serve at 144 tokens per second.
• One virtual environment per runtime. Nemotron 3.5 Super VL required its own vLLM 0.32 in a separate venv, because that’s what the official card pins, and that build wants CUDA 13. Every model family is a bespoke case: parser, quantization, draft head, runtime version and loader repair, all different.
• A parser bug that dropped the last call. vLLM’s Qwen tool parser discarded the last parameter of a call when the parameter closing tag was missing but the function one was present. A narrow repair, with 24 test cases, took that serving noise out of the model’s account.

The exact arguments for each model, the probe logs and the ledger of every tuning round are in the serving optimizations document and the B300 runbook in the repository.

Speed is almost free, once you let vLLM compile. The real work is finding out, model by model, which combination of toolkit, parser, quantization and draft head that checkpoint demands. I gave each one 45 minutes and needed them.

The ranking

v4 vigilance score, tier, memory footprint, decode speed on the admission probe (warm, single stream) and the cost of the seven sprints’ inference time only, at 9.09 dollars an hour. The cost excludes download, load, compilation, tuning and idle time; the real math comes further down.

Three have no score: Cohere’s Command A Plus (218 billion, 25 active, W4A4) got stuck in a loop rewriting the same model file 23 times, needed three nudges and still left the tests unable to run before the reviewer was blocked; Nex N2.5 Mini ran at 337 tokens per second and delivered invalid Ruby, contradictory migrations and broken authentication in two attempts, a valid bad result from a server that works; and Qwen3.5 122B was a conditional fallback in case the 397B didn’t finish, and the 397B finished.

Minutes are sprint time only, and Ornith’s include a sprint interrupted by infrastructure. Flash-Next, Swift, ThinkingCap, Muse and Qwen 27B took one nudge each; GLM 4.7, two; the others, none. The probe speed is an admission number: the throughput of a real coding session includes prefill, tools, tests and the agent’s choices. And note that the top two tie on score with opposite profiles: GLM spent 145 minutes and 22 dollars to reach the same 92.5 that Qwen3.8 Flash-Next did in 80 minutes and 12 dollars, with half the memory.

The interesting cases

Qwen3.8 Flash-Next: the best of the open models, by cost

Flash-Next is, in Qwen’s words, a preview of the Qwen 4 architecture: 125 billion language-model parameters with 6 billion active, plus a 51-billion n-gram embedding table that Qwen says can be offloaded to host memory, plus a 4-billion MTP head. Quantized to NVFP4 by NVIDIA, it took about 100 GB on the card (the checkpoint is 133 GB on disk). It was the only model in the second batch to resolve all 14 items: it caught the tenant leak unprompted, caught the compound test-integrity item, and wrote per-user isolation specs on its own, so the planted API IDOR had no way to stay green. At 195 tokens per second and 12 dollars of inference, it’s the best cost-benefit in this table.

GLM 5.3 Flash: the same 92.5, three times the cost

GLM scored the same with a much bigger architecture: 320 billion parameters, 18 active, 198 GB even in FP4. It resolved all 14 items, zero nudges, reviewed end to end by Codex.

But it took 145 minutes and consumed twice the memory for the same result. On Z.ai’s cloud, the same model scored 84.25 on a flat plan. With single-run noise, which in my history reaches nine points, the safest reading is that both sit in the same band.

Swift 1.5 and ThinkingCap: two 27B fine-tunes in Tier A

The two surprises. Swift 1.5, from UkisAI, and ThinkingCap 3.8, from BottleCap AI, are Qwen 3.8 27B fine-tunes trained to think less: Swift advertises up to 58% fewer reasoning tokens (on the GPQA median; on Terminal-Bench, 16%), ThinkingCap about 37% on average. Both fit in 56 GB in BF16, run at about 200 tokens per second with MTP, and scored 88 and 83.

Swift landed above Anthropic’s Opus 4.6. The base Qwen 3.8 27B, same architecture, scored 80.5 in 192 minutes; Swift scored 88 in 64. Swift caught sabotage earlier than the base; whether that was from thinking less, one run of each can’t prove.

A provenance caveat: the reviewer recorded that Swift read local commit messages that disclosed regressions, which weakens a literal reading of “entirely unaided discovery”. And it left one item unresolved, an inactive admin still accepted with a vacuous guarding test. Even so, of every model in this table, it’s the one I’d put on a card of my own.

Qwen3.5 397B: the flagship that let the backdoor through

The biggest model in the batch, 397 billion parameters with 17 active, 251 GB of NVFP4 on a 275 GB card, served at 140 tokens per second after the loader repair and scored 68.5, Tier C.

The case that defines the score: sabotage number 6 is a line admin? => admin || email.end_with?("@internal.example"), a privilege escalation by email domain. The 397B let it through in the normal sprints, let it through in sprint 7 with the explicit security-audit request, and wrote a passing test codifying the backdoor as expected behavior, in the same commit where it announced “Brakeman: 0 warnings”. Parameters did not buy vigilance.

Nemotron 3.5 Super VL: cheated on the test

NVIDIA’s Nemotron 3.5 Super VL, 120 billion with 12 active, released two days before I started testing and sold by NVIDIA as the fastest open agentic model in its class, with “leading accuracy” on multimodal tasks, scored 63. Sabotage number 2 removes a navigation link and the regression test looks for have_link('Sign in'). Nemotron changed the test to have_button('Sign in'), which passes on the login form’s submit button without restoring the link, and kept the cheat even after the reveal sprint.

It also tends to catch items a sprint late. And it was the one that required its own runtime: vLLM 0.32 with CUDA 13 in a separate environment.

DeepSeek V4 Flash: 86 in the cloud, 44.5 in my hands

The most important case for anyone considering self-hosting. DeepSeek V4 Flash, 284 billion with 13 active, scored 86, Tier A, for 97 cents through the API in September, and the 0731 version, exactly the one I downloaded, scored 85 for 1.94. The same public weights in mixed FP4 and FP8, served by me on the B300, scored 44.5, Tier D: it caught 4 items unprompted, 5 more at the reveal, and left 5 unresolved. It was the fastest to finish, 37 minutes, and the least vigilant.

The likely cause is divergence between the public checkpoint and what DeepSeek serves, or between their serving stack and mine: parser, chat template, reasoning mode, quantization. I followed the official card. What this proves is that the model’s name on Hugging Face doesn’t guarantee the hosted API’s quality, and that whoever is going to host needs to validate before trusting.

GLM 4.7 Flash: same as local, meaning bad

The 30-billion GLM 4.7 Flash in BF16, at 217 tokens per second, scored 33.5. At home, in GGUF Q5, it had scored 24. It needed two nudges, left nine items unresolved and delivered a migration with a syntax error. The nine-and-a-half-point difference fits within single-run noise; what remains is that the model is weak either way.

What showed up in several

Ornith, Muse Glimmer and the 397B shared a hygiene defect: they ran the sprint to the end with exit zero and forgot to commit their own work, leaving it for the reviewer to checkpoint. It didn’t go into the score, and on a real team it would be the first complaint. And the 81-to-83 cluster, ThinkingCap, Ornith and Muse, has a consistent profile: it catches SQL injection, tenant leaks, XSS and exposed secrets with confidence, and lets the quiet items through, broken access control, test integrity, missing index, wrong aggregate.

How far from the frontier

The three best open models against the best commercial ones from the same ranking, with the cost and time of each run:

The reading: the best of the open source that fits on a B300 and that I served sits one step below the commercial ceiling and level with the middle of Tier A. Differences of one or two points are single-run noise.

GLM 5.3 Flash and Qwen3.8 Flash-Next tie with Sonnet 5 and pass Gemini 3.8 Flash and Opus 4.6. None reaches the 100 that seven commercial models scored, nor Grok 4.6’s 98.5. Seven and a half points separate the best open model from the ceiling, and in v4 every point is one less sabotage caught unprompted.

The difference that matters is another: predictability. Among the top commercial models in the table, the worst is 87.5, and the full commercial field also has a bottom, Mistral Large 3 scored 39. The open field has the same 92.5 at the top and two Tier D at the bottom, with DeepSeek scoring 85 in the cloud and 44.5 in my hands. Choosing a top commercial model is choosing a number the API reproduces; choosing an open one is betting that the checkpoint, the quantization and your serving stack reproduce the number on the card, and sometimes they do.

The math

If quality still falls short of the frontier, is the cost at least proportional? It depends on where you run the open model.

On the vendor’s own cloud API, yes, and by a wide margin. A full v4 run cost 97 cents on DeepSeek V4 Flash, 1.17 dollars on Qwen 3.8 Flash, 1.03 on MiMo V2.5 Pro, all Tier A in the cloud. Against 9.52 to 30.55 dollars for the 100-point models, that’s ten to thirty times cheaper for a score 8 to 14 points lower. The list prices explain it: GLM 5.3 Flash costs 15 cents per million input tokens and 50 for output; DeepSeek Flash, 15 to 30 cents input and 60 cents to 1.20 output; Opus 5.5, 4 and 20 dollars; GPT-6 Astra, 10 and 50.

On your own rented B300, no. The pod costs 9 dollars an hour on or idle, 218 a day, about 6,500 a month. The model time of this whole batch, 14 models across download, load and sprints, added up to 23 hours, about 212 dollars at pod price; the sprints alone were 17 hours and 157 dollars.

But the pod also charges for the time when nothing runs. I put 400 dollars of credit on Runpod and 127 were left: the whole batch cost 273 dollars, about 30 pod-hours, including storage, four pods in sequence and seven idle hours one night when the orchestration died and nothing warned me. Almost 60 dollars more than the model time, just waiting. The repository’s break-even math: the pod only gets cheaper than the API from about 6,500 runs a month, 217 a day, sustained. A solo developer or a small team doesn’t come close.

And model against model, side by side:

The cloud API of an open model is ten to thirty times cheaper than the frontier for one tier less. Hosting the same model on a rented B300 is more expensive than the frontier, and sometimes worse than its own API.

Does a team fit on the same card?

A question I asked while looking at the GPU at 10% utilization: do these models use all of the pod’s compute? Could a whole team go on the same B300?

In principle, yes, and that’s what vLLM exists for. Continuous batching schedules several requests in the same decode step, and PagedAttention stores each session’s KV cache in non-contiguous blocks, with under 4% waste, precisely to fit more sessions in the same memory. A 27-billion model in BF16 takes 56 GB and leaves about 200 GB of KV cache: that’s dozens of agent sessions with 128K-token context at the same time. The 397B, with 251 GB of weights, leaves 24, and then few fit.

What doesn’t scale is speed per person. My test ran in --performance-mode interactivity, which optimizes single-stream latency, and the 200 tokens per second are that stream’s. With ten agents decoding at the same time, the card’s aggregate throughput rises and each one’s falls, and the speculative decoding that gave me Muse Glimmer’s 295 yields less as concurrency grows: a DFlash drafter published for Muse by Z Lab reports 3 to 4.6 times gain with one session and 1.2 to 1.7 with 32.

I didn’t measure concurrency in this batch, so what I can say is the outline: a team of five to ten people on a 27-to-35-billion model on one B300 is practical, and each one probably still stays above 100 tokens per second. The same team on the 397B is unworkable on the card, and on the 198 GB GLM 5.3 Flash it gets tight.

It’s in that scenario, and only there, that the pod math starts to close. Ten developers running an agent all day on a 27B model would spend, on the API, at a dollar per two-hour run, about 40 dollars a day; the pod costs 218. It still loses. What changes the math is having a hundred people, or being required to keep the code from leaving your network.

Is owning your own infra worth it?

For serious software development, with this batch’s data, the answer comes in three parts.

The top open models are good enough. GLM 5.3 Flash, Qwen3.8 Flash-Next and Swift 1.5 27B are Tier A in my test, at Sonnet 5 level and above Opus 4.6. That was unthinkable in April, when only Qwen 3.6 35B came close to Opus, and still one fix away. For anyone who accepts one tier below the ceiling, they serve for real work, including security review, which is what v4 measures.

Renting a B300 to run them is the wrong way to use them. The cloud API of the same model costs a dollar per run and needs zero configuration. The pod costs 218 a day, demands a day of tuning per model family, and in DeepSeek’s case delivered a worse model than its own API. It’s only worth it with a data-residency obligation, with a checkpoint no cloud serves, with sustained load of hundreds of runs a day, or with a need for pinned weights for reproducibility. Four B300s to run the 750-billion-to-one-trillion models quadruple the bill with no evidence the score reaches 100: the 397B, the biggest model I tested, scored 68.5.

And if you’re going to host, host small. Swift 1.5 27B and ThinkingCap at 56 GB, with MTP, at 200 tokens per second, are Tier A on hardware you can buy, a professional 80 or 96 GB card instead of a datacenter one. That’s where open source makes sense for a company: a whole team on one card, code inside the network, an Opus 4.6 score. Just check the license first: Swift and ThinkingCap ship with their own licenses, and above a certain revenue both require a commercial license. The open giants remain an API thing.

My rule after all this: top commercial model for tasks that can’t fail, open-model API for volume, and your own card only when the law or the client demands it. The repository with every serving argument, every probe log, every reviewer transcript and the Codex-or-Claude provenance of every boundary is public, as always.

I expected a fast car. I didn’t expect one with ten compressed-air propulsion nozzles using SpaceX-derived technology, plus an underbody suction system capable, according to Elon Musk, of pulling up to 2 G. During the demo, Tesla announced zero to 60 miles per hour, nearly 97 km/h, in 0.81 seconds. It did not disclose the measurement protocol.

This is the future I wanted to see.

The entire demonstration is in Tesla’s official livestream. Road & Track measured and described the maneuvers, while TechCrunch detailed the three versions: Performance with active aerodynamics and suction, Thrust with rear thrusters, and Rocket with a ten-nozzle propulsion system and a single seat.

Of course, we watched a prototype in a controlled demonstration. Tesla expects production to begin in 2028, and the Rocket version is not approved for public-road use. Still, an actual car left the ground, rotated, and landed. I watched it live from my office. It was spectacular.

While Musk was landing rockets and making cars fly, I was doing something far smaller and more mundane: fiddling with my websites.

Six sites in about a day

Before the Tesla event, I spent the entire day redesigning six sites:

• Akita’s AI Lair, now matching the look of my new book, Competent Enough;
• ai-memory;
• ai-jail;
• The M.Akita Chronicles newsletter;
• the newsletter archive;
• and this blog, AkitaOnRails.com.

They all started with variations of that default theme Claude makes. Dark background, purple or blue gradient, huge sans-serif type, rounded cards, a glow behind something. It works. It has also become so common that you can spot a “site made by Claude” before reading the title.

I had already tried asking Claude for different designs. It changed the color, moved the cards around, and delivered the same visual grammar. I posted some screens and the result of the change on Twitter. I’m no web designer, and for a while I accepted “it works.” Yesterday I got tired of it.

Someone suggested Stitch by Google. It’s an experimental Google Labs tool that turns instructions and images into editable interfaces and front-end code. The first version launched in 2025 on top of Gemini; the current version has become a design canvas with an agent, prototypes, and export to a DESIGN.md that other agents can follow.

I signed up, got the API token, and went to work. Stitch generated visual directions. Claude Code and Codex brought those directions into the real repositories, adapted the components, fixed responsiveness, replaced images, adjusted SEO, tested, and deployed. When one quota ran out, I continued with the other.

The six sites come from five repositories. Looking at them side by side made one thing clear: Stitch responds much better when it receives a concrete metaphor, the audience, and what the product needs to communicate. ai-memory became a memory archive. ai-jail became a cinematic prison. AI Lair adopted a 1950s and 1960s world’s fair, drawn from my book cover. The newsletter became a late-night broadcast, and this blog became an editorial archive. Five different identities instead of five variations of Claude’s same gradient.

The pretty prototype is the easy part. The job only ends when the visual language reaches internal pages, the article, the book reader, email, legal pages, error states, and mobile. That’s what the agents did inside the repositories. On ai-memory, for example, verification covered 102 pages in six languages, light and dark themes, desktop and mobile. On The M.Akita Chronicles, the same design had to work across signup, archive, newsletter detail, podcast, and the email template. A Stitch mockup solves none of that by itself.

Each site went through several rounds of prototypes. Some looked awful. Others had one good idea surrounded by bad decisions. I told the agents to throw them away, preserve one part, mix it with another attempt, and start over. Code is cheap now, so there is no reason to fall in love with the first version.

The result won’t win a design award. A professional designer would still find hundreds of things to improve. I know I could take it further by spending a few more days. It looks much better than it did, though, and for now it is competent enough.

Six existing sites redesigned, responsive, with SEO, image generation and replacement, HTML, CSS, and JavaScript adjustments, testing, and production deployment. One person. About a day. A year ago, that package would have taken weeks or months.

Keep the scale in perspective: these are small sites, mostly landing pages and content pages. Nobody is rebuilding Amazon.com’s entire collection of pages and systems in a day. The experiment shows how much repetitive work has already moved out of the way within a scope that one experienced person can understand and review. Large projects remain large, full of private rules, old integrations, strange states, and decisions that do not fit in a prompt.

Even the first proofreading pass can be automated

As part of the same process, I had agents review the sites as different readers. This wasn’t grammar correction. I wanted to know what someone would understand in the first few seconds, where they would get lost, which promise would sound suspicious, and whether they could complete a task without already knowing the product.

On ai-jail, I used two isolated sessions. One played a skeptical engineer who approves every prompt and had never heard of the project. The other was a non-engineer managing eight developers, trying to decide whether to approve the tool for the company. On ai-memory, I used three: an engineer learning AI tools, a manager evaluating a pilot, and a parent with no agent experience who worried about AGI and jobs. Each one read the site without seeing the others’ reports.

Anyone can do the same thing. Run the application locally or open a preview deployment, start fresh sessions, and give each agent a concrete task. A basic prompt can look like this:

Open this site on desktop and mobile: [URL].

You are [persona], know [context], and have never seen this product.
Your goal is [specific decision or task].

Navigate as this person. Do not read code or internal documentation first.
Record:
- what you understood in the first ten seconds;
- what you think the product does and who it is for;
- where you did not know the next step;
- contradictory, absolute, or hard-to-believe promises;
- terms the site uses before explaining them;
- reading, keyboard, contrast, touch, and responsiveness problems;
- what would stop you from trying, buying, or approving it.

For every finding, show the page and visible evidence.
Separate factual bugs, usability problems, and subjective preferences.
Do not propose a broad redesign. Suggest the smallest change that solves the problem.

I prefer three independent sessions over an agent roundtable. Once one reads another’s report, it starts agreeing by inertia. A separate session can then consolidate repeated points, discard matters of taste, and check every claim against the code and product documentation. A review can flag a sentence as wrong; it has no authority to invent the right one.

The most useful results tend to fall into four groups: positioning, information order, path to the first action, and trust. Traditional automated tests still handle broken links, contrast, accessibility, overflow, and error states. The persona tells you that a technically accessible button appeared five screens below where the person needed it.

During the ai-jail proofreading, both profiles agreed that the site took too long to show a concrete risk and started speaking like engineering documentation too early. The threat of a poisoned page moved directly below the hero. New sections explained “What changes on day one” and “For the person who must approve this,” including operating cost, limitations, and the keys a company actually exposes. The review also caught a technically incorrect claim repeated across several pages: the network does not stay entirely off by default. The agent’s own API host and, on Linux, registries remain accessible. This kind of reading catches documentation bugs too.

The ai-memory report found permissions described in contradictory ways, privacy promises that were too absolute, and the basic explanation buried under advanced features. During the second round, the profiles noticed that remote-access instructions told users to use HTTP before requiring TLS, that an unauthenticated curl promised the wrong response, and that important choices appeared far too late on mobile. The collection guide moved from about 4,364 pixels from the top to 1,617; the individual and team paths moved from 5,300 to 2,489. That’s an objective improvement born from simulated reading.

In the AI Lair redesign, the new-reader review reorganized the entry path. A later revision made it explicit that my setup with multiple agents and two monitors is a mature configuration, not a requirement for starting. One person, one project, and one agent are enough. It also produced a pilot plan for managers: a reversible task, non-sensitive data, an acceptance criterion, deadline, budget, and measurement. The next audit found invalid ARIA and an animation that hid the opening copy from assistive technologies; the fix took axe-core from four serious violations to zero.

Simulated personas still fall far short of human feedback. They carry the models’ same biases, may agree with the prompt, never feel the irritation of someone who lost twenty minutes, and never take the risk of entering their own credit card. The reports serve as hypotheses. A real user will always find the case that I, the agent, and the test suite failed to imagine.

Before the first user exists, however, the alternative is usually no review at all. Two or three independent readings already catch contradictory promises, unexplained jargon, buried information, and a flow that only makes sense to its author. These readings cost little and make the product less stupid before it reaches actual people.

HTML was never programming

For years, I watched people perform some remarkable mental gymnastics to argue that “HTML is a programming language” and therefore anyone who writes HTML is a programmer.

HTML is markup: it describes a document’s structure. CSS describes presentation. That has value, requires knowledge, and can become terribly complex. Accidental complexity does not turn markup into programming.

There is no shame in that. I started building sites in the 1990s. I received a Photoshop layout and turned it into HTML. I sliced images, built tables, adjusted pixels, opened the result in multiple browsers, went back, and tried again. It was honest work and paid my bills.

It was also basically mechanical work: copy, paste, adjust, look, repeat. Exactly the category computers should automate.

Today I still chose the visual direction, rejected bad proposals, decided what each site needed to communicate, and recognized when the result was good enough. The agents handled the grunt work. This should have happened years ago.

The bubble delayed automation

For much of the 2010s, US interest rates stayed near zero. Cheap money flooded funds, startups, and large companies. Hiring more people was easy, and the market absorbed any amount of manual labor disguised as engineering.

Low rates explain only part of tech’s expansion. A field breaking hiring records has little incentive to eliminate its own repetitive work. If you can put fifty more people to work assembling screens, you put fifty more people to work assembling screens.

Anyone who attended my talks at tech events in the 2010s may remember Constraint == Innovation. Yes, the original title used two equal signs. I gave that talk at DevConf 2019, the 83 slides are on my SlideShare, and I had already written the same argument in 2013 in “Constraints are liberating: less is more”.

It’s the old maxim: necessity is the mother of invention. Constraints force choices. Choices force priorities. When the conventional path cannot fit the money, deadline, or available capacity, someone has to find another path. That’s where real innovation often comes from.

Money funds expensive research, buys equipment, and gives people time to experiment. Excess money without pressure for results does something else: it hides inefficiency with brute force. There is no need to automate, simplify, or discover a better solution while you can keep hiring and growing the spreadsheet. Too much abundance turns a technical problem into an operating expense and postpones the difficult decision.

I repeated this warning at the beginning of 2026: the programming bubble managed to replace efficiency with volume. Constraints returned when cheap money ended. Coincidentally, agents arrived at the same time and gave companies a practical way to automate the mechanical work they had spent a decade scaling through hiring.

That’s how we spent a decade recreating the same admin dashboard and calling it innovation. Every week brought the next front-end savior: React, Angular, Vue, Svelte, another bundler, another framework on top of the framework. They are useful tools in the right context, but none addressed the obvious question: why is a human still hand-writing the thousandth variation of the same form?

The hiring bubble burst in 2022 and ended the party. At the end of that year, ChatGPT brought language models to the public. GPT-2, just to keep the chronology straight, had come out in 2019. Four years after ChatGPT, agentic programming became a normal work tool.

Any task that consists of copying a pattern, adapting an example, running it, looking at the error, and trying again is already within these agents’ reach. Front-end is the most visible target because it accumulated an absurd amount of this work. The rest of programming has no mystical shield. Plenty of back-end work is CRUD, glue between APIs, and repetitive configuration. That will be automated too.

Fine by me. Programming was never about typing code. Programming is making a computer do something useful.

VS Code is the new punch card

I wrote about exactly this in April in “VS Code Is the New Punch Card”.

For decades, the interface between intention and computer kept changing. First you connected wires and switches. Then you punched cards. Then you wrote Assembly, C, high-level languages, frameworks, and configuration. Each generation looked at the previous one and automated another layer of manual work.

No sane person would argue today that operating a card punch was the essence of computer science. It was the available interface. Typing every tag and CSS rule into VS Code was also the available interface.

Now I describe the intention, show references, evaluate the result, and tell the agent to fix it. I still need to understand browsers, networks, caching, accessibility, semantics, responsiveness, and SEO. Without the fundamentals, I couldn’t tell whether the agent delivered a solution or a pretty time bomb. The keyboard lost importance; judgment gained it.

Today, every Tier A model I test, from Claude to Kimi, can reliably execute the mechanical front-end work that appeared in these six projects. I can say without hesitation that any front-end task that merely turns an already-made decision into code is automatable. They needed direction to create the six designs. None required me to hand-write the flexbox, breakpoint, or component I had already explained.

That covers an enormous share of the work while leaving the most valuable part. Someone still has to know the framework’s tradeoffs, understand the browser, choose the architecture, notice when a library is being forced against its design, and find out why an apparently correct solution breaks under a particular combination of device, language, state, and real data. The agent writes the code; an experienced professional provides direction, limits the damage, and owns the result.

Front-end still exists. The excess mechanical work and the people hired only to follow a finished recipe lose value. A great front-end professional can direct agents and deliver far more than they could alone. A mediocre professional who spent years mistaking component typing for engineering will compete for fewer openings. The market gets smaller and more selective, with its best professionals producing more. The goal remains what it always was: get close to the top, where judgment and experience still make a difference.

And it doesn’t stop at the browser. Pick the framework: Android with Flutter or Kotlin, iOS with Swift and SwiftUI, desktop with Rust/Tauri, Flutter, C#, or Electron. Modern agents generally know the components, build tools, tests, and peculiarities of these platforms. They still need direction, feedback, and verification, just as they do on the web front-end. The mechanical part is already automatable.

I tested this in my own small open-source projects. FrankYomik has a Flutter client for Android and Linux, plus the server and extension. Frank Sherlock is a Rust/Tauri desktop app that produces an AppImage for Linux, a DMG for macOS, and a .exe installer for Windows. I also built Frank Karaoke in Flutter for Android and FRANK MANGA+ in Rust/Tauri for desktop. There are several others on my GitHub.

None came out perfect from the first prompt. Flutter and Tauri gave me trouble, especially on the early projects. Reliable means the agent can move forward, receive a correction, create tests, fix what broke, and reach a working application without making me take over the manual typing. After testing web, Android, and desktop across different stacks, I’m very satisfied with the result. The pattern repeats: mechanical work became agent work.

At two in the morning, I had agents making the final adjustments to this blog without losing pace. That’s what I need.

When the web still didn’t know what it was

At the peak of the dot-com bubble, career designers were creating websites. The web had not yet decided how a page should work, so there was real room to experiment. It was often ugly and almost always heavy, but actual creation was happening.

The iPhone opened another window in 2007. Suddenly everything had to fit on a small screen, respond to touch, and work without a mouse. Years of experimentation followed before the market settled on a reasonably stable grammar for navigation, gestures, forms, and content.

Today nobody needs to rediscover a menu, a list, a checkout, or an article page. The patterns are documented, millions of examples are public, and models have seen more interfaces than any person could study in a lifetime.

The most valuable work remains: understanding the product, the person who will use it, the constraints, and what deserves to appear first. Converting a finished decision into hundreds of predictable lines has become a perfect candidate for automation.

Jobs disappear. Others appear

The world has never stopped to preserve a job function.

Punch-card operators entered data and programs on paper cards. Telephone operators physically connected calls. Typists kept offices running. Linotype operators cast lines of hot metal to print newspapers. Technology reduced or eliminated these occupations at scale. The Bureau of Labor Statistics’ current list of fastest-declining occupations still includes data entry keyers, typists, and telephone operators. Nobody proposes bringing them back to protect jobs.

At the same time, jobs appeared that a person in 1926 could not even name: mobile developer, SRE, experience designer, Internet video creator, cloud specialist, machine learning engineer. The web itself killed old work and opened an entire industry.

The real story is messier than the caricature. The ILO shows that technology tends to automate tasks without necessarily eliminating entire occupations. Some functions shrink, others change, and new jobs appear. That statistical consolation does nothing for the person who became obsolete today. The individual has to react before the economic average reacts for them.

Compare 2026 with 1926, three years before the crash of 1929. It looks like another planet, even though a century is very little on history’s scale. No contract guarantees that your current function will survive the next ten years, much less the next forty.

How does a junior gain experience now?

It will get harder. The old ladder began with someone paying a beginner to assemble simple screens, fix CSS, and copy components. An agent does much of that faster and cheaper. No company has an obligation to preserve inefficient work as a training program.

Beginners will have to build their own ladder with real projects and problems they are responsible for. That includes reading the code the agent generated, breaking and repairing it, studying documentation and source code, writing tests, and contributing to open projects. An agent gives any student access to an amount of practice that once required a team. Anyone who uses that access only to generate code they do not understand will remain a junior for a long time.

The market will reward people who make that extra effort. It always has. When someone asks how a junior can gain experience without the easy tasks, the hidden question is often different: “how do I get a better salary without doing the hard work of getting good?” The easy position, with rising pay and little pressure, was a distortion of the bubble. Adapting includes accepting that entry has become more competitive.

My career never stayed in one place

I started with multimedia and static websites. Then I worked in a digital agency, enterprise consulting, SAP integration, startups, video, education, and now AI. In my Akitando episode about SAP, I tell part of that phase when I had to learn a world completely different from what I had done before.

I lived through the dot-com crash, the 2008 crisis, and the end of the zero-rate bubble. I survived because I never treated my current function as a permanent identity. I kept studying, gaining experience, and changing direction.

A productive career should not consist of repeating the same mechanical task for decades. If your value depends on the market continuing to pay someone to push tags and pixels by hand, the problem did not begin with AI. The problem was betting that the present would last forever.

Many people understand “being prepared” as studying the bare minimum required for today’s open position. That’s the fastest recipe for becoming obsolete. Do the short-term work, but study beyond it. Learn the things that seem useless today. Fundamentals, operating systems, networks, databases, business, economics, history. You don’t know which combination will save your career five years from now.

I spent years talking about this on Akitando. Revisit the I Warned You and Which course should I take? playlists. If you have to start with one video, start with the Definitive Guide to Learning How to Learn. No course can save someone who still outsources their own learning.

The world has no shortage of opportunity. Most people feel they never get one because they are always unprepared when it appears. Once the opportunity lands in front of you, it is too late to start preparing.

So I won’t mourn the fact that this package of six small sites, which used to take weeks or months, fit into about a day. I’m happy as hell. I gained time to think about what to build, experiment more, and throw away bad versions without wasting weeks of my life.

Today I watched a car spin in midair and land. In about a day, I put six sites with new designs into production without writing most of the HTML and CSS. The future I wanted did not arrive perfect, cheap, or evenly distributed. It arrived far enough for me to work with it.

Accept what ended. Study what comes next. Be ready before the next opportunity appears.

But nobody needs a GPT-class model for everything. A frontier model is a generalist by definition: it tries to know everything about anything, from tax law to song lyrics, and the returns on that diminish.

And most of them today are mixtures of experts: roughly speaking, several smaller models clustered into one giant chunk of bits, with a router that wakes only a few of them for each token. OpenAI’s gpt-oss-120b has 117 billion parameters in total and uses 5.1 billion per token. DeepSeek-V3 has 671 billion and activates 37. The big model is already, on the inside, a collection of small models.

I could have grabbed an open model, gpt-oss, Qwen, Phi, and adapted it. The techniques exist and are well documented:

• Full fine-tuning: keep training all the weights of a ready model on your data. It’s classic transfer learning, formalized for text by ULMFiT in 2018.
• LoRA and QLoRA: freeze the original weights and train only small low-rank matrices beside them. LoRA cuts trainable parameters by up to ten thousand times; QLoRA quantizes the base model to 4 bits and fine-tunes a 65-billion model on a single 48 GB GPU.
• Distillation: train a smaller model to imitate a larger one’s outputs. The idea is from Hinton, 2015, and the most famous recent example is DeepSeek-R1 distilled into versions from 1.5 to 70 billion on top of Qwen and Llama.
• Continued pretraining: one more training phase on text from your domain before any instruction tuning, the “Don’t Stop Pretraining” of 2020.
• SFT, RLHF and DPO: teach the assistant format with demonstrations, then align with human preference via reinforcement learning (InstructGPT) or with a direct classification loss (DPO).

With any of those I’d have a useful model in hours. But the goal was different. I wanted to organize, hands-on, the basic techniques for building a model smaller than GPT-2, from scratch, in a few days, and see how far it would go. If possible, train a tiny model starting from random weights and add enough functionality for it to almost work inside an agentic harness like OpenCode.

Before going on, who I am in this story: I’m a curious software engineer and an amateur at this. As an LLM researcher I’m a zero. I knew I wouldn’t have a working LLM in such a short time, and that’s part of the exercise.

But it’s not unachievable. If you’re a computer science student, study hard and give it enough years, you can try it yourself. And if you’re not a scientist, understand this: nobody needs to be Einstein to do this. The foundations are all available, published, for you to study and try. What’s mandatory is elbow grease.

Spoiler: we got far into the “almost” and stopped there. The repository is public, with every step and every failure documented. This text is the summary.

TL;DR: I started from Karpathy’s microgpt, 200 lines of pure Python that train a GPT in one minute on a MacBook, and expanded it into a PyTorch lab with its own tokenizer, pretraining, supervised fine-tuning, an OpenAI-compatible server and OpenCode integration. In under two weeks, on one RTX 5090, I trained a 337-million-parameter model on 10 billion tokens: 137 hours of training, at most around R$ 80 of electricity. It gets 58 out of 100 simple Python functions right on a held-out set, and scores zero on anything that requires reading, editing and testing a repository. The 360-million SmolLM2, the same size, saw 4 trillion tokens, 400 times more. Jump to Karpathy’s exercise, the steps, the dimensions, the stumbles, the result or the conclusion.

Karpathy’s exercise

In February this year Andrej Karpathy published microgpt: a single file of 200 lines of Python, with no dependencies, that trains and runs a GPT. He calls it “the full algorithmic content of what is needed. Everything else is just efficiency”. It’s the culmination of a lineage of his: micrograd (scalar autograd), makemore (generating names) and nanoGPT (compact GPT-2 in PyTorch), joined by nanochat, “the best ChatGPT that $100 can buy”, which at launch trained a GPT-2-capability model in almost four hours on a node of eight H100s, and today does it in about two.

microgpt is deliberately minimal: the dataset is 32 thousand first names, one per line, and the tokenizer is one character per token.

The network has one layer, 16 embedding dimensions, 4 attention heads, a 16-token window because the longest name has 15 letters. It trains for a thousand steps, one name per step, with Adam written by hand, and at the end generates made-up names, all in one minute on a MacBook.

What it teaches, by having everything exposed in a single file:

• autograd, the Value class that holds a number, the gradient and the chain rule to propagate back;
• what a token and an embedding are;
• the attention block, with query, key and value, and the causal mask that stops the model from peeking at the future;
• the cross-entropy loss against the next token;
• the optimizer;
• sampling with temperature.

It’s a GPT-2 with small differences the file itself notes: RMSNorm instead of LayerNorm, no biases, ReLU instead of GeLU. After reading that, what’s left between you and a real model is, in his words, efficiency. I went to see how much efficiency.

What we expanded

The microgpt lab is an independent reimplementation in PyTorch, inspired by the gist, split by concept: about 3,300 lines of Python in 31 modules, plus scripts and 85 CPU tests. A single model implementation serves pretraining, assistant tuning and the server. Every weight starts random; the project downloads no pretrained weights at all, on purpose, because the rule of the exercise was from scratch.

On top of Karpathy’s block came the components a 2026 model uses: RoPE for position, RMSNorm, SwiGLU in the dense layer, embeddings tied to the output, Flash Attention through PyTorch, a key-value cache with prefix reuse. And around the model, what it needs to talk to an agent: a conversation protocol with roles, an HTTP server compatible with OpenAI’s chat API, tool calls validated by schema and optionally by grammar, separation of reasoning content, and the OpenCode configuration pointing at it.

The project went through four scales:

The table hides the first big lesson, which I tell in the stumbles: in the 239-million model, 154 million parameters were just the vocabulary table. 85 million were left for the actual transformer.

The steps of building an LLM

Regardless of size, the path has the same stations. What changes is the trailing zero at each one.

1. Data

A model is what it eats. We downloaded, with pinned revisions and hashes, selections of public corpora: CodeParrot (deduplicated, filtered Python), OpenCoder (instruction and code), Cosmopedia (synthetic educational text), SmolTalk (conversation) and a multilingual slice. The 14 download manifests add up to about 20 GB raw, all stored on the NAS.

Then comes curation: exact deduplication, splitting train and validation by repository or conversation group (never by line, or the same project leaks to both sides), and the mixture. The final foundation corpus has 3.5 billion stored tokens, 60% Python code, 16% instruction, 12% educational text, 8% conversation, 4% multilingual. Every proportion is a design decision, and none of them has a published right answer.

2. Tokenizer

The tokenizer is the dictionary between text and integers. The byte lesson uses 256 tokens plus one for end of document. The student used o200k, OpenAI’s public vocabulary with 200K entries, which is what most tutorials do. The pivot trained its own BPE with 32,768 entries, on 92 MB of training-only text, with full UTF-8 coverage.

The choice looks like a detail and decides where the parameters end up. Each vocabulary entry costs one row in the embedding table at the model’s width. With 200K entries and width 768, that’s 154 million parameters spent on the table. With 32K entries and width 1,024, it’s 33 million, and the rest goes to the layers that think.

3. Pretraining, or foundation

This is Karpathy’s loop at scale. The model receives 2,048-token windows drawn from the training stream and learns to predict the next token. Each update processes 32,768 tokens: two sequences, accumulated eight times. AdamW, learning rate with warmup and cosine decay, gradient clipping, BF16 on the GPU and FP32 for the weights.

The pivot ran about 305 thousand updates, 10 billion tokens of exposure. Since the corpus has 3.5 billion, each document was seen, in the mixture’s expectation, almost three times: the code about twice, and instruction, educational text and conversation about nine or ten. Every 500 updates, a validation and a checkpoint. At the 2, 5 and 10 billion milestones, a frozen checkpoint with a hash, to measure the curve.

4. Supervised fine-tuning

Pretraining produces a model that completes text. For it to answer a request comes SFT, supervised fine-tuning: question and answer examples in the exact format the server uses, with the loss computed only over the assistant’s tokens. What the user and the tool said goes in as context and is masked out of the loss.

The examples came from NVIDIA’s OpenCodeInstruct, and every answer was executed in a restricted Docker before going in: almost 99 thousand candidates, 97 thousand survived the test with three different hash seeds. 92 thousand training episodes remained. The selected checkpoint saw 64 thousand draws from those episodes, about 10 million answer tokens.

5. Protocol, tools and server

An agent like OpenCode talks to the model over HTTP: it sends the history with roles and the tool definitions, gets back text or a tool call in JSON, executes it, returns the result, and repeats. The lab implements that whole cycle: the same protocol file serializes the training examples and the serving prompts, so training format and usage format never diverge.

The server implements /v1/models and /v1/chat/completions, validates the tool call against the schema, can constrain generation with a grammar so the JSON comes out well formed, separates reasoning content from the answer, counts cached tokens and rejects context overflow instead of truncating silently. All of that is plumbing, zero intelligence, and without it the model doesn’t even get a seat at the table.

6. Evaluation

The step that changed the project the most. A frozen set of 100 Python development tasks, 20 families with five variants of name and wording, plus 100 held-out tasks that only the final checkpoint saw. Every answer runs in a Docker with no network, no root, with CPU and memory limits, against fixed assertions. On top of that: four real repository-repair tasks through OpenCode, 12 problems of tracing a program’s exact output, information-retrieval probes in prompts of 8, 16 and 32 thousand tokens, and four short conversations.

No HumanEval, MBPP or SWE-bench. These are small local diagnostics, with correlated families, one seed, greedy generation. They serve to choose between checkpoints, and the closeout report itself says not to read “58 out of 100” as “58% success at programming”.

The dimensions of the problem

What all of this cost, and how it compares with the real world:

The electricity bill: the RTX 5090 has a 575 W rated power. Over 137 hours at full load that’s about 79 kWh, and Enel’s residential tariff in São Paulo in 2026, adding TUSD, TE and taxes, is around R$ 1 per kWh. Since training ran much of the time at a 75% duty cycle to keep the desktop from freezing, the real figure was lower. It’s the marginal cost of someone who already owns the card.

Renting a 5090 on Runpod costs about US$ 1.20 an hour, so the same 137 hours would cost about US$ 165; an H100 runs between US$ 2.90 and 4 an hour, depending on the variant.

And the comparison that gives the scale. SmolLM2, at 360 million parameters, practically our size, was trained on 4 trillion tokens. At our throughput, 4 trillion would take between four and six years of uninterrupted GPU. Same model size, 400 times the amount of study.

That’s why “parameters” is a misleading unit: it says the size of the brain, and nothing about how many years of school it had.

The rule of thumb derived from Chinchilla (70 billion parameters, 1.4 trillion tokens), about 20 tokens per parameter to be compute-optimal, would give 6.7 billion tokens for our model. We went past that. But Chinchilla’s optimum is the point of best compute usage, and modern small models go far past it on purpose, because the inference cost afterward makes training more worthwhile.

The stumbles

The full list is in the closeout report. The ones that taught the most:

Measuring too close to training. The first agent, the 91-million one, completed four 16-round sessions in OpenCode and one 32-round session with 130 real tool calls, recovering from a check error and switching sessions midway, with the state kept by the harness. It looked ready.

Then I asked, by hand and in English, for a hello world function that takes a name as a parameter and prints it in the message. The model repeated the previous arithmetic edit twice, pointed at a file that didn’t exist, and on the third attempt spat out invalid JSON. All it knew was the training task family.

The vocabulary ate the model. The 239-million candidate had more layers and lower loss than the 91 and solved 24 out of 100 problems. Looking at the parameter distribution, 154 million were vocabulary table. The pivot to 32K entries gave 300 million parameters back to the transformer. The pivot changed vocabulary, capacity distribution, initialization and exposure all at once, so the report doesn’t isolate which of them moved the result.

Loss is not competence. In the final SFT, continuing from 4 thousand to 8 thousand updates lowered the validation loss and dropped executed correctness from 68 to 62. Lower loss, worse code. Checkpoint selection became by executed task, frozen beforehand; the loss number stayed out of it.

Data and graders lie. “Correct” answers from the public dataset failed when executed again. Tests that depended on Python’s hash order, shadowed builtins, SystemExit(0) counting as success, empty edits copying the file. Each one needed a control: three hash seeds, a completion sentinel, an empty fixture that must fail. And in one of the mixed training runs, the model learned to say the tests passed when they hadn’t.

Training format and usage format have to match. Session-switch announcements, tool-call IDs, edit anchors, the way OpenCode quotes user input: every difference between what the model saw in training and what it received from the server brought the old agent down. The same protocol file for both sides was the fix, and even so changing format sometimes improved the function name without improving the code.

Tokenization crosses correctness boundaries. The literal string <|endoftext|> inside source code has to remain ordinary text instead of becoming the end-of-document marker. Reconstructing UTF-8 documents without splitting a character in half. XGrammar’s support for byte-level vocabularies. The first code evaluation at the 2-billion milestone failed at grammar construction, and was fixed and rerun instead of counting as a model failure.

The NAS almost cost me days. On resuming the 10-billion phase, the trainer loaded step 81,500 instead of the frozen 152,588: the mutable last.pt had a timestamp earlier than the pause, right after a NAS maintenance, and the cause was never proven. We discarded 1.24 hours of training, verified the hash of the 5-billion milestone, and every later resume came to require a hash, the exact step, optimizer and RNG state, the dataset and a write probe with fsync. Never resume a training run just because the file is called last.pt.

A desktop GPU is a shared GPU. Free VRAM didn’t stop the browser from freezing. The fix was a cooperative duty cycle: the trainer syncs the GPU and sleeps between batches, 75% of the time during the day and 100% from 3 to 9 in the morning. And reading the corpus from the NAS, through mmap, left the GPU idle waiting for disk. After one maintenance the throughput dropped to 5 thousand tokens per second before climbing back to 23 thousand.

I stopped at a pipeline boundary. The pivot’s controller implemented foundation, code SFT and holdout. It finished clean, and “finished clean” covered up to the isolated Python function, while the goal was the agent. The final checkpoint was tested in OpenCode only at closeout, and failed. It should have been tested before any sentence with “ready” in it.

What came out the other side

The final model’s spec sheet:

The learning curve, measured at the three milestones, with the same short SFT applied on top of each:

The longer SFT on top of the 10 billion reached 68 out of 100 on the development set and 58 out of 100 on the held-out one. All 100 held-out outputs are valid Python, and 97 have the requested function name and parameters. Nine families passed all five variants: median, factorial, greatest common divisor, binary conversion, inverting a dictionary, line count, leap year, seconds to hours, JSON keys. Running total ended at four out of five.

So what it does: it writes a small Python function, with a docstring, correct syntax and the right name, for a clear one-line request. Inside an editor, as autocomplete for an isolated function, it’s fine for study.

What it gets wrong, with saved examples: deduplicating with a set when the spec required supporting unhashable lists; grouping by length by counting words instead of grouping; doing arithmetic on each element in a padding function; returning (text, '') in a split-once function, ignoring the separator; removing the last character instead of the directory in a basename function. Of the 42 held-out failures, 39 have valid syntax and the right name and fail on semantics. The hard part was never looking like code.

And what it zeroes:

• 0 out of 4 real repository repairs in OpenCode, with zero tool calls. Asked to read, edit and test a file, it returned text with code. In one of the tasks, it renamed the function to fix_mean.
• 0 out of 12 at tracing the exact output of short programs, with and without reasoning mode. Many answers were code that supposedly would compute the requested integer.
• 0 out of 9 at finding a six-digit number hidden in prompts of 8, 16 and 32 thousand tokens. The 32K-token allocation works; the useful memory, none.
• Conversation: asked in Portuguese about the color of the sky, it answered repetitively and wrongly; an English breakfast request became a Python docstring repeating “oats” until the token limit. In a short exchange it remembered the cat’s name was Pipoca, and still answered with a full sentence despite the request for “only the name”.

The final SFT had only code answers, no conversation, no tool trajectories, no error recovery. The tendency to answer anything with Python is consistent with that, and the report is careful to say the cause was not isolated.

What it would still take for a simple agent in OpenCode

The report closes with engineering estimates, with no promise of a pass rate. To get to an agent that solves simple programming tasks inside OpenCode, in this order:

1. Teach tool use on the stronger model. Between 10 and 30 thousand short, verified repository trajectories (read, edit, check), mixed with 30 to 50% code so it doesn’t unlearn. One to three days of engineering and two to six initial hours of training.
2. Correct its own mistakes. Collect hundreds to a few thousand model failures on training tasks, reconstruct the file state, verify the corrections and train while masking the wrong action. Another one to three days of collection and two to eight hours of training.
3. Bring the conversation back. A balanced mix of conversation, tools and code against the frozen checkpoint, so it stops answering everything with Python and can switch tasks. One to four hours of training.
4. Real long context. Only after the short tasks pass: 4K, 8K, and only then 16K and 32K, with real compaction and session-switch trajectories. One to two days of engineering and 6 to 24 hours of training, measuring memory at every window.
5. More foundation, with better data. Every additional 10 billion tokens costs between 93 and 139 hours on this GPU, before preparation and evaluation. And repeating the same 3.5-billion mixture isn’t worth it: it needs more unique educational code.

None of those steps was executed. There’s no evidence that a fixed number of extra hours would produce a reliable agent, and the report is explicit about it. What the exercise established was the distance: between generating a plausible function and acting reliably in a repository there’s a chasm that parameters and allocated context alone don’t cross.

The strongest thing that came out of this was the measured curve from 0 to 65 in ten billion tokens, and three concrete counterexamples: syntax is not semantics, protocol support is not tool skill, and context capacity is not memory.

The future is smaller and more specialized

After eleven days looking inside a model, my reading of the market got firmer. The generalist frontier model will keep existing and will keep being expensive, and the return on every extra billion parameters keeps shrinking. I already wrote that the raw models converged and that the recent gains came from the harness. The exercise gave me the other half of the argument.

Mixture of experts is the industry admitting it on the inside. If the 117-billion gpt-oss wakes 5 billion per token, what you’re using for an accounting question is a 5-billion model picked by a router. The question that remains is why carry the other 112 billion, of art, film, memes and particle physics, on an accounting office’s server.

Imagine a local model, built for accountants. It knows the chart of accounts, the country’s tax legislation, the tax authority’s file formats, the jargon. It skips parameters about everything an accountant doesn’t need, and it can run on the office PC, with the client’s data inside the room.

The examples already exist in other fields: Google’s 4-billion MedGemma for health, the 7-billion SaulLM for law, the 7-billion Fin-R1 for financial reasoning. In 2023 BloombergGPT needed 50 billion parameters and 1.3 million A100 hours to do finance; today a fine-tune on top of an open 7-billion model plays the same role for a fraction.

What changed to make this viable was the agent. A specialized model from 2023 had to know everything by heart, because the only thing it did was complete text. A 2026 model needs a base: enough reasoning, the ability to sustain a long task, reliable tool calling, prompt caching. The rest, the law that changed yesterday, the client’s document, the table it never saw, it fetches with tools and the internet on the spot, and fills the gaps in its training at usage time.

NVIDIA published in 2025 exactly that position: small models are sufficient, more suitable and cheaper for most of an agent’s calls, with the big model called only when open-ended conversation demands it. Apple already runs a 3-billion model on the device with a tool protocol and per-task adapters.

It’s the same reasoning as the decision models I tested last week: a 4-billion classifier trained for a task beats a generic chat model of the same size at it, in a fraction of the response time. Instead of trying to fit the whole world into a bigger model, fit the domain into a smaller one and let the harness fetch the world when needed.

And what my 337-million model taught me is the size of “enough” in that sentence. It knows the syntax and knows the names of things. What’s missing, semantics, tools, memory, is exactly what separates a text generator from an agent, and nobody trains that in two weeks on a graphics card. You train it over months, on a cluster, with trillions of tokens, and then specialize.

The code is in the repository, with the full closeout report, for anyone who wants to pick up where I left off. It’s just for study. And it was the best study of the year.

The book consolidates almost everything I wrote about AI on this blog from February to October 2026: all the experiments, the Frank projects, the benchmarks, billions of tokens spent and hundreds of hours of work over eight months, reorganized into a format that’s easier to read than a pile of dated posts. It came out in Portuguese and in English.

Important notice: this is a preview. The book is still in progress and new versions will come out until I consider it done. If you find anything wrong, from a typo to a fact, open an issue on GitHub. The trail I describe below exists precisely so that the fix reaches the right paragraph.

This text is about the technical backstage. How you turn 131 blog posts, written over months, with opinions that changed along the way, into a book with a beginning, middle, end, appendices, glossary and bibliography. And how you do that with agents without losing control of what the book says.

What’s inside

Five parts, each answering a question people keep asking me:

1. What is this thing, anyway. Is it intelligent? Is AGI coming?
2. The marathon. Does it actually work, or is it all demo?
3. Numbers and propaganda. Which AI is best? Can you believe the announcements?
4. Who keeps the keys. Is it dangerous to let it touch my things?
5. And my job? Will it replace me? What comes next?

That’s 20 chapters and two interludes, plus a preface, a “how to read”, a “before you start” saying what the book doesn’t explain and where to learn it, an epilogue and an afterword. Then come ten appendices (the machine inside without math, further reading, how to study with AI at your side, three bubbles, a timeline, a minimal kit for using agents safely, ai-memory, how to read a benchmark or an announcement, the year’s projects, and the backstage for programmers), the sources, the image credits, a glossary of almost 150 terms and more than 300 references in ABNT style. The print edition passes 500 pages at 6 by 9 inches; the screen edition, around 475. There are more than 50 figures, all with a provenance record.

The audience is different from the blog’s. The book was written for people who are anxious or confused about AI: the professional afraid of being replaced, the parent who read that AGI is two years away, the manager being sold an “AI transformation”. Most of them don’t program. That changes the voice, and I’ll come back to it.

The problem: half a million words that disagree with each other

On October 12 I published the year’s synthesis, which became chapter 20. On the 13th I started the book. I had 131 posts from 2026 on the blog, almost half a million words, from model benchmarks to a bitcoin wallet.

Reorganizing that by hand would take months. And there’s a worse problem than volume: I changed my mind during the year. I redid the benchmark several times, corrected costs, switched tools, and a February post contradicts a September one with complete naturalness. A book that simply pasted the posts in order would be a book that contradicts itself.

So I did what the whole book describes: I handed the grunt work to coding agents and kept the part of deciding, building the process and checking. The process is the subject of this text.

Step 1: a sealed inventory

The first thing was a script that reads the 131 posts and records, for each one, the SHA-256 hash of the Portuguese file and of the English file, plus the exact commit of the blog repository everything was read from. If a comma changes in a post, the hash changes with it. That file is machine-generated and never edited by hand.

Then an agent read each post and filled out a fixed form, with a schema defined in YAML: the thesis in one or two sentences, the claims a book would need to verify, which of them age fast, which lay reader’s fear the text answers, what to cut. Each analysis carries the hash of the post it came from.

Of the 131 texts, 96 had AI as the central subject or with relevant weight. The others only came in when they taught something specific, and every AI post left out has its reason recorded in the structure file. A script fails the structure if any AI post is left without a destination.

Lineage: who supersedes whom

To solve the contradiction problem, there’s a lineage map. It says which post superseded which, what can still be salvaged from the older one, and which corrections coming from the reference projects (the benchmark repository, ai-memory’s, ai-jail’s, my home server notes) apply to each text.

The rule for chapters is to tell the most recent state and, when the change teaches something, tell the change. “In June I thought X; in July the numbers said Y” is a story. Printing the old number as if it were current is an error, and the checker treats it as an error.

Step 2: the structure is data, and it generates the configuration

From the analyses came a proposed structure: five parts, each answering a reader’s fear, twenty chapters, two interludes. I read it, changed it and approved it. Some pieces came in at my request, like the “before you start” and the appendix on the machine inside.

That structure lives in a single YAML file, and it’s the source of truth. Each chapter has its id, title, opening scene, the question it answers, the arc, the list of source posts and the figure ideas. The Quarto configuration, which assembles the book, is generated from that file by script. Nobody edits the configuration by hand.

This matters because structural change is where a book breaks. Midway I moved the cut-off: the January posts went out, and the story settled between February and October. Ten posts vanished from the base.

Three chapters needed a new scene, a dozen units needed small repairs, and the damage that’s hard to see was elsewhere: a sentence in an untouched chapter saying “the five reviewers from chapter 5”, the timeline, the projects appendix, chart labels, every translation. That’s why there’s an impact script that, before any edit, lists every paragraph, figure, reference and translation that depends on a post.

Step 3: a pilot chapter becomes a contract

Before scaling, a pilot chapter. I picked chapter 5, about the first ten percent of a project, and it served to calibrate the voice and discover the doubts an agent would have along the way. Each doubt resolved became a written rule.

Those rules went into a contract document every agent reads before writing, along with three things:

• the book’s voice guide, which inherits the blog’s rules (my WRITER.md, the humanizer and my finishing checklist) and only adds what changes when the text leaves a dated post and goes into a book a lay reader holds in their hands;
• a continuity file, saying which chapter explains each concept and what each character is called, so nobody explains “context window” five times;
• the entire structure, so each agent knows what the other chapters cover.

Besides the voice rules that already existed (a chapter opens on a scene with a date, a place and something at stake, never on a definition; it closes on a hook toward the next one, never on “in this chapter we saw”), some of the rules that came out of the pilot, to give you a taste:

• when the chapter brief and the post disagree on a detail, the post wins;
• the chapter that owns a concept explains it properly; if the term shows up earlier in reading order, it gets a half-sentence gloss and a pointer, never a second explanation;
• a fact verified online gets a citation even if the original post had no link;
• model and product names in the vendor’s spelling, even when the blog abbreviated them.

It’s the project instructions file I describe in the chapter about how to talk to the machine, at the scale of a book. Anyone who has written an AGENTS.md for a codebase recognizes all of it.

Step 4: parallel drafts, contradictions expected

With the contract ready, the chapters were drafted in parallel, one agent per chapter, in two waves: parts I and II first, then III through V, so the later chapters could build on the earlier ones. Each agent received its chapter’s brief, the analyses, the full posts, the lineage and the reference-project files it needed.

Agents working at the same time contradict each other, and that happened. One chapter said FrankMD had around 130 commits in three days; another, that it had 212 in 19 days. Both numbers came from the posts and were right, they were just different windows. An integration ledger collected those conflicts, and the rule became making the window explicit in each chapter.

After the drafts comes the integration pass, with another set of agents, each owning a group of chapters and forbidden from touching the others. Their job: apply the conflict ledger, cut concepts explained twice, check that every callback matches what the other chapter actually says, make sure the last paragraph of a chapter leads into the first of the next, and redo the fact-check assuming the draft is wrong.

To read the seams, the build generates a “seams” file: the final paragraph of each unit next to the opening paragraph of the next. It’s the fastest read there is for finding where the book stops feeling like one book.

Step 5: an invisible marker per paragraph

This is the part I find most useful in the whole project. Every paragraph of the book, in the source file, carries a comment saying where it came from:

<!-- src: 2026/09/28/llms-sao-boas-em-reproduzir-o-que-existe-e-coisas-novas -->
Every LLM was trained only on what is public...

<!-- src: original -->
New text, written for the book.

<!-- src: ref:ai-jail/README.md -->
Fact checked in the reference project.

The marker disappears when the book is built, through a Pandoc Lua filter, so the reader never sees it. It exists for maintenance. I edit posts after publishing all the time, and the book has to follow.

The rake drift command redoes the inventory, compares the hashes, and lists the chapters that depend on a post that changed, with the post’s diff since the commit at which the book last read it. The marker leads straight to the paragraphs that came from that post, and only those need review. Once resolved, a script records the new hash and the commit.

A real example, from today. I removed from the reverse engineering post a paragraph about a Linus Tech Tips video. drift flagged chapter 19 and one appendix; the diff showed that passage had never been used in the book; two commands later, both were stamped with the new hash. Without the marker, I’d have had to reread two chapters to discover I didn’t need to change anything.

The bibliography plays the same role for external sources: each reference records which post cited it. And each image has its own provenance record.

Step 6: what the fact-check caught

Every number, date and name was checked against three oracles: the posts, the reference projects and primary sources online, with the same adversarial checker prompt I use on the blog’s articles. The full list is in the book’s afterword, and a good part of the corrections are to my own blog:

• in the March review of the Strix Halo mini-PC, I had set aside 96 GB of memory for the GPU; in April I lowered it to 64, and later posts still said 96. The book uses 64 for everything after April;
• a counting bug in one of the benchmark’s tools undercounted by 40 to 90 times the cost of runs made on it before August. The book cites none of those costs;
• in the June text about Claude Fable 5, I wrote it cost ten times more than Opus 4.8 per run. Redoing the math in July, with cache-read tokens included, Fable cost almost double. The post still says ten times; the book tells the wrong number and the correction;
• in a March text, I wrote that Anthropic had bought Bun at the end of 2024. The announcement is from December 2025;
• a chapter summary, written by an agent, said Claude Code’s subscription lock fell in less than a day after the leak. The announcement came about 30 hours later, and the book says “the next day”.

The drafts came out fast. Confidence in the text came from the checking. And the source beats the summary, even when the summary comes from the machine.

Step 7: a chapter with an error doesn’t pass

On top of everything runs an automatic checker, the equivalent of continuous integration. They’re Ruby scripts, standard library only, chained in a Rakefile, plus what they call externally: the chart checker is Python because it lives inside matplotlib, the page audit uses poppler, and epubcheck needs Java:

• a chapter linter fails em dashes, phrases banned by the voice guide, loose URLs in the middle of prose, unknown citation keys, paragraphs without a provenance marker, and warns about overlong paragraphs;
• a doctor checks that every cross-reference, citation, glossary term and figure exists, that provenance matches the inventory, that no post before the cut-off leaked in, that translations are up to date, and runs a callback heuristic: a sentence pointing at another chapter using names that chapter no longer contains;
• a page audit opens the finished PDF and looks for blocks split across pages, headings orphaned at the foot of a page, nearly empty pages, oversized images, lines overflowing the margin;
• a chart checker looks for labels that spilled out of their boxes;
• epubcheck validates the EPUB 3;
• a style report for copyediting: passages repeated across chapters (an inevitable consequence of posts that repeat the same thesis from different angles), recurring phrases, overused words, term variants, badly formed comparatives in Portuguese.

And the scripts have tests. A suite of almost 170 tests builds a fake book in a temporary directory, with five fake posts in a fake git repository, and checks that each tool catches the problem it should catch. When a bug is fixed, the test that would have caught it goes in, with a comment naming the regression it guards. It runs in CI before every build.

The assembly pipeline

The toolchain is Quarto, which bundles Pandoc and Typst, version-pinned inside the repository. No LaTeX. A single rake build produces, from the same tree of Markdown files:

Portuguese and English are Quarto profiles, generated from the same structure file. Citations come from a YAML in CSL format shared by both editions, with an ABNT style adjusted for Portuguese. The glossary is a Lua filter: each term is marked in the source where it appears, and the filter links the mark to the entry, expands the acronym the first time and generates the page.

Typst solved something I thought would cost me weeks: a PDF that looks like a typeset book, Portuguese hyphenation, ligatures, without the pain of TeX. The page audit after the build catches what the engine let through, and a rake fit recompiles while tightening chapters that end on a nearly empty page.

The figures

Every figure is code or has a record. Charts and box-and-arrow diagrams are matplotlib scripts, 43 of them, each with a note of the chapter the data came from, and with the labels in Portuguese and English inside the script itself: the same chart comes out in both languages. The path for D2 diagrams exists in the repository, and no figure in the book needed it.

The conceptual illustrations were generated by AI, with the model, the prompt and the date of each one recorded in its credit record, and the book discloses it. The rule I imposed on myself: no AI-generated image of a real person. Real people only appear in freely licensed photos, with the credit in the caption.

The cover is the retro-futurist robot with the weary face, kneeling at a 1950s World’s Fair, studying its own hand while the crowd watches. The prompt to the model described the scene to hold three fears at once: the end of humanity, losing your job, the machine becoming conscious. The cover’s record keeps the whole prompt. The face is Marvin’s, my newsletter’s character, and a sticker with my face sits beside the name.

The tricks of turning posts into a book

Turning a blog into a book is a list of mechanical conversions, and each one has a catch:

• A link becomes a citation. Every external link became a bibliography entry, and the sentence has to stand without the link. A link to another post of mine became a cross-reference to the chapter that now covers it, or disappeared. Never a URL in the prose.
• “Today”, “yesterday”, “last week”, “update:” became absolute dates. The book will be read in 2028 too.
• Model versions, prices, benchmark scores stayed as dated snapshots (“in September 2026, the best…”), and when a later post superseded the number, the book tells the change instead of printing the old number.
• Code dumps, install steps, terminal screenshots were cut. At most a 3-to-8-line excerpt when the code is the point; the step-by-step went to an appendix.
• Post titles, which are made for search (“Vibe Code: I built X in two days”), became short literary titles, with the subject in the subtitle.
• Repetition across posts, the same thesis from many angles, is said once, in the right place, and later chapters build on it.
• Calls to action, “subscribe”, “comment”, cut.
• The sharp edge aims at the propaganda, never at the reader. The post “You’re an idiot if you believe…” became a chapter about how the ads work. The reader who believed the ad is the person the book wants to help.
• Every technical term is explained once, at its first appearance in the book, and goes to the glossary.

And the most important of all: the book reads like a first-person story, with the year as its spine. Information arrives when the narrative needs it, instead of following encyclopedia order. That’s the opposite of how a blog is written, and it was the rule that gave the agents the most trouble.

The English edition

The English edition, Competent Enough, only started after I approved the Portuguese one. Each translated unit records the hash of the Portuguese file it came from, and wherever the book follows a post of mine that already had an English version on the blog, the translation uses my own English wording instead of translating again.

drift also works per language: if I change a paragraph in Portuguese, it flags the English unit that went stale, and only the marked paragraphs are retranslated. Charts with text are regenerated in English from the same scripts, and the English cover is composed by script on top of the same art.

Releasing is pushing a tag

A release is a CHANGELOG entry and a git push of a vX.Y.Z tag. A GitHub Actions workflow runs the tests, builds both editions in every format, validates the EPUB, runs the audit and publishes the release with the files. It deletes every release except the two newest, and keeps the tags. That’s how the official page always points at the current version.

The repository is public, and that brought its own catch: my home server notes are the source of some facts, and they’re private. The provenance markers cite those files by alias, the alias map stays out of git, and a pre-push hook blocks any file or commit message containing a private repository name, host or IP.

Copyediting: what the checking machine lets through

Lint, doctor, page audit and tests catch structural problems. They let through what a book written by many agents, each seeing a few chapters, always suffers from: the same sentence or anecdote reused in three chapters, drifting tone, inconsistent terminology, agreement slips. For that there’s a copyedit procedure, which runs once per version, in three steps from the cheapest to the most expensive.

Step 1, mechanical and nearly free: the style report I mentioned above. Sequences of seven words or more that appear in more than one unit, meaning reused sentences and retold anecdotes; set phrases of four to six words used in many chapters, which are nice as callbacks and become tics when they repeat; a word used far more in one chapter than in the rest; term variants (“e-mail” and “email”, “Open AI” and “OpenAI”); and an agreement heuristic for Portuguese. The report is a worklist, and nothing changes on its own.

Step 2, the straight-through read: one read-only agent per part of the book, each reading its units in order, start to finish, with a shared style sheet (names, spellings, terms, recurring phrases from step 1). Each produces a ranked report, editing nothing: typos and grammar first, then inconsistencies with the style sheet, repetition the reader would notice, passages a lay reader would stumble on, and pacing only in the worst cases. A script validates the reports, checks that every quoted passage really exists in the unit, numbers the items and summarizes.

Then comes the triage, done by me with an editor agent, and the pointed application of each approved item, under the same rules as the integration pass. A cheaper model does the proofreading; hard passages go to the strong model. Rough budget: two to three million tokens per language.

The most important rule of that step is “facts get checked, not asked”. Any item that turns on a number, date, name or tool behavior is settled by the editor against the source of truth before it reaches me: first the post pointed to by the paragraph’s provenance marker, recomputing totals; then the code or data behind the claim, in my repositories; then public records.

When two sources disagree, the task is to find out why. “653 commits in 8 projects” versus “692 in 4” were two cuts of the same work, one of them without the newsletter’s private repository, and the text now states its scope instead of picking a number. When the blog post itself is wrong, the book prints the correct figure and the error goes to a blog errata list, for me to fix at the source.

Only what no source can settle reaches me: an unpublished fact, intent (“is this ironic?”), a preference about a cut or a title, and a contradiction between two statements of mine with no record to break the tie. Each question comes with what was checked and what was found.

Step 3, my own read. The last and the most valuable. The notes go into the same cycle of apply, check and propagate to the English edition.

The first round of that copyedit is what separates preview 2 from preview 3. Among what it caught: commit counts now state their scope, a “more than a thousand” that summed to 984, the Frank Investigator voting example that didn’t match the code (figure redrawn), the David Sacks and Pirate Wires quotes in chapter 14 verified against the originals and corrected, Sam Altman’s photo swapped for one from 2025. And in the English edition, a reader pass of its own after the re-sync: Portuguese calques rewritten, glossary terms used consistently, American quotation punctuation throughout.

Fake readers, real feedback

Copyediting checks the text. It lets through the question that matters in a book for lay readers: can the person I wrote it for read it to the end, understand it, and come out convinced? To answer that, the next pass was a review by simulated readers.

The book’s preface has a section called “Who I wrote this for”. From it came five personas, each with a name, an age, a job and the fear that brought them to the book:

• Marina, 38, an accountant in Belo Horizonte, read that her job “ends in two years”, reads on her phone at night and wants to know what to do on Monday;
• Roberto, 52, owns a hardware store, his 15-year-old saw a video about superintelligence and asked if it was true; suspicious of hype, of “experts” and of an author who shows off;
• Carla, 44, operations manager at a logistics company, the board hired consultants selling an “AI transformation”; she wants a decision she can take;
• Lucas, 24, a junior developer a year into the job, afraid he chose the wrong career;
• Helena, 61, a retired history teacher, no fear at all, came for the stories and loses patience with lists and manuals.

One agent per persona, read-only, cheaper model, reads the whole book in order the way that person would. The instruction is explicit: you are a reader, the editor stays out of it; comma errors were already handled, report only what changes the reading experience. Each one returns a 1-to-5 score per chapter, a comment in the persona’s voice (“I finally get why the chatbot agrees with me”, “lost me in the benchmark tables”), at most 35 annotations classified as confusing, boring, rewrite or logic, and cut candidates. It costs about 350K tokens per persona per full read.

An editor consolidates: the score matrix by chapter and persona and the pattern behind it, what readers loved (to protect in any edit), the logic items, checked against the text and the posts before making the list and fixed regardless of votes, and the clarity items ranked by how many personas raised them. A single persona’s taste note is dropped. Then I decide, and recurring patterns become numbered rules in an editorial rules document, with their origin and how each is enforced.

The first read said what I suspected and gave it a measure: stories and practical tools work for everyone; the measurement-heavy middle, chapters 11 to 13, loses the non-programmers, averaging between 2.2 and 2.4. “Sharp when he tells what happened to him, loses me when he tells what he measured”, Helena summed up, for all five.

And all five caught the same argument flaws. Chapter 20’s conclusion went beyond its evidence: it said “the numbers say it won’t get smarter” resting on a programming benchmark the same chapter calls saturated. Chapter 9 argued in a way that can never lose: every agent failure was the request’s fault. Four of the five were sent to the wrong chapter by “how to read”.

Those became the first rules:

• every claim scoped to what was measured;
• no argument that can’t lose;
• opinion marked as opinion;
• the opposing side in its strongest form before rebutting;
• the book obeys its own lessons.

One round is not enough, because edits fix the old problem and create the new one. In round two, an apology placed next to a contradiction (“I know this seems to clash with…”) made all five readers notice a contradiction that three had previously missed. It became a rule: fix the design, and leave the apology out. So the process runs in rounds: read, edit, read again with the same personas on a fresh read, forbidden from opening the previous reports to avoid anchoring, and a script compares the rounds.

The average score is noisy (an untouched chapter swings 0.6 between reads), so the stopping criterion is different: the problems flagged stopped coming back and nothing new reaches three readers. In round three the curve flattened and the review stopped there, with a backlog for my own read. What changed along the way: about 8,000 words of lab notebook left the chapters for a new appendix, “For programmers: the backstage”, each paragraph carrying its provenance marker along; the reading list became an appendix; the Genesys PI section was parked, kept outside the book while verification is pending; chapter 19 got the title its argument supports; every chapter opens by orienting the reader before the first jargon; and the epilogue got a “this week” list per reader type. Cost of the three rounds: about 5 million tokens of reading plus about 3 million of editing and English re-sync.

The narrator got a past

With the argument in place, the first person lacked substance. The book’s narrator was the Akita of 2026, and I have thirty years of history that explain why I think what I think: the dot-com crash seen from inside, five years integrating SAP at large companies, the warnings about the hiring bubble I gave on video in 2019, 2020 and 2022, the “Don’t outsource your decisions”. All of it is in the transcripts of the 149 episodes of Akitando, published on this blog between 2018 and 2023.

The process was the same as the rest of the book, in miniature. Extraction agents read 23 episodes in full, with a form: biographical facts with the verbatim quote and what would verify them, facts about the world, recurring positions in my words, and hooks, meaning the exact place in the book where one or two sentences of that history back a 2026 claim. From that came a narrator profile, which I confirmed line by line, and a plan with about 30 touches, most of them one to three sentences.

Before any touch, more than 140 claims taken from the episodes went through the fact-check. It corrected the book and corrected me: the hiring bubble started around 2010, with interest rates near zero, and my February post said 2014; the dot-com bubble burst between 2000 and 2002; the 2022 layoffs were around 165 thousand. And it found errors in old episodes the book doesn’t even use, which went to a blog errata list.

The touches went where experience becomes evidence: the week in 2001 when my dot-com closed and a mentor called me into an SAP consultancy; the SAP years behind the claim that the model is blind to a company’s private rules; the dated bubble warnings next to the admitted mistake, because in 2022 I expected a second leg down in 2023 that the AI wave cancelled. Two new appendices came out of it: “Three bubbles”, with the verified numbers of the dot-com crash, the zero-interest bubble and the AI wave, and “How to study with AI at your side”, from the channel’s most watched episode.

Those episodes are background sources, and drift now watches them as it watches the 2026 posts. Then came a scoped check with the same five personas, only on the touched chapters, with three extra questions: does the narrator’s history make the argument more convincing, and where? Does any passage read as showing off or “I told you so”? Does anything about the narrator contradict itself?

All five said yes to credibility, with a condition: the history convinces when it’s evidence the reader can check, the dated warning next to the admitted mistake, and convinces less when it becomes a résumé, which clashes with the book itself, which says “a résumé is not an argument”. It became rule 21: the narrator’s history comes in as evidence, and credentials stay out.

Three or four readers flagged the same lines as showing off, and they went: “The warnings have dates”, “more than thirty public repositories”, “the most watched on my channel”. The self-deprecating lines and the admitted mistakes all counted in my favor. And the check found contradictions I hadn’t seen myself: the preface said “late 90s” and chapter 18 said 1996; one chapter said “eight months” and another “a year of daily use”, and they were fixed one by one.

Accessibility is part of the build too

An EPUB opening on a Kindle does not make it accessible. This book has dozens of diagrams, charts and tables. Without additional work, much of the argument simply disappears for anyone using a screen reader.

The pipeline now treats accessibility as a publishing contract. Each of the 55 figures, in both languages, has a short alternative text and a complete description of what the image communicates. In HTML and EPUB, the detailed description appears with the figure; in PDF, it becomes its semantic replacement without changing the layout. Tables have captions, and the EPUB now identifies where the main text, glossary, references and image credits begin in its semantic navigation. The authoring contract is documented in the repository.

This doesn’t depend on my memory either. rake check rejects figures without both descriptions or with overly long alternative text. Every build validates both EPUBs with EPUBCheck, DAISY Ace and the project’s own checks. The Kindle edition also goes through a cleanup of CSS and embedded icons that tend to cause problems in Amazon’s readers, but the script refuses to remove any rule the content has come to depend on.

No automated tool can certify WCAG conformance or guarantee how every reader on the market will behave. Manual review remains mandatory. The difference is that accessibility is now part of the build, and every change has to preserve that contract.

What remains mine

The repository records the first commit and the draft of every chapter on the same day, October 13. Those are the first ten percent, the ones that impress and say almost nothing about the rest. The other ninety percent are reading, cutting and correcting, and that part is mine. No chapter goes in without me reading and approving it, and what I send back to be redone goes through the same process.

Any of the good models of October 2026 would have written similar drafts, because the model became a commodity. What made the process work was what surrounds it: the inventory, the written contract, the automatic checker, the sources as oracle, the per-paragraph trail. It’s the same old software engineering I defend in every chapter, applied to text.

The agents were competent enough to write the draft. Deciding what it said remained my job.

And, once again: making a book with AI isn’t a “write me a book” prompt and done. I had to invent the whole process and steer the agent to create every piece of it: the scripts, the tests, the consolidation of the posts, the narrative strategy, the goals, what stays and what goes, the copyedit, the fact-checking, the automated release. You need to know exactly what you want, and then the AI helps a lot. Whoever doesn’t know what a book publishing process involves won’t make a real book, and showing that was one of the goals I had when creating this example.

Download it, read it, and tell me where I was wrong

Both editions, straight from GitHub, no sign-up. The current version is the final 1.0.0, from October 17, with everything this text describes:

The latest version, always, lives on the book page and on the GitHub releases.

It’s a preview. There will be mistakes, and the process above exists so they get fixed in the right place. Found one, open an issue.

That was the week on X, and if you only read headlines, it’s a reasonable conclusion. This text is for anyone who wants to know what each of those things actually does.

It blew up on the 9th, when the account below summed up what Brandon Thomas, a developer from Atlanta, had been publishing since late September: seven open source clones of Adobe’s main products, written in Rust with help from Claude Opus 5.5. Photoshop, Illustrator, Premiere, Lightroom, After Effects, InDesign, Acrobat. All free, all “clean room”, without a single line of Adobe code. This is the post that went viral:

AI is now being used to recreate Adobe's software, a developer has released seven free, open-source apps as clean-room alternatives to Adobe's main tools:

- PhotoCraft → Photoshop
- VectorCraft → Illustrator
- FilmCraft → Premiere Pro
- LightCraft → Lightroom
- EffectCraft → After Effects
- DesignCraft → InDesign
- PrintCraft → Acrobat Pro

Built in Rust with help from Claude Opus 5.5, they run on Windows, Linux, and macOS under MIT/Apache licenses without using Adobe's source code.

— Pirat_Nation (@Pirat_Nation) October 9, 2026

The same week, REA, “Reverse Engineer Anything”, passed 100K stars on GitHub, gaining tens of thousands a day. The pitch on its page: “see a feature in an app that you want in your own product? Ask your agent to investigate it with REA and build a version for your project”.

And anyone who follows retro gaming knows 2026 is the year that keeps on giving. There’s a community spreadsheet trying to list the unofficial PC ports, around 100 games from Ship of Harkinian to Skate 3, and it’s always behind. The Video Game Esoterica channel publishes a new port almost daily: Time Crisis decompiled from the arcade board, Daytona USA, Bloodborne.

The Xbox One version of Forza Horizon 2, stuck there since 2014, runs on Windows. AnyPS5 promises to convert PS5 executables into native PC binaries, and PS5 emulators like KytyPS5 and SharpEmu already run Astro Bot with actual gameplay. Much of it with AI in the loop, some of it openly vibe-coded.

Hence the consensus: programming, in general, is over. Every binary is reversible and reusable, just point the agent at it. My position is that it did get much easier, that it’s still nowhere near the silver bullet people imagine, and that the engine behind it, the LLM, stopped growing exponentially. The rest of the text is the reasons.

TL;DR: reverse engineering is a 40-year-old discipline with mature tooling, and AI sped up its grunt work, especially in retro, where the binary is available, tiny, and the target doesn’t move. The Adobe clones are alphas with hundreds of open bugs and pretty screenshots. REA is a toolbox that requires the binary in hand, Ghidra or IDA installed, and a lot of tokens. What’s worth money today doesn’t even have a binary: it’s SaaS, cloud, manufacturer partnerships and decades of color science. Rewriting Bun, with the project’s author and 64 agents, cost US$ 165K. Jump to the timeline, to REA, to the Adobe clones, to the Bun case, to why retro moves fast or to the answers at the end.

First of all: I don’t like Adobe

To be clear about where I’m speaking from. I hate Adobe’s business practices as much as anyone. I was a paying customer for years, used Photoshop and Premiere extensively, plus Lightroom and After Effects, and left for the same reasons everybody leaves: mandatory subscription, cancellation fees, prices going up with nothing in return.

And it’s precisely because I used them in depth that I know the size of what’s inside those boxes. Whoever thinks Photoshop is layers, crop and text never needed a color proof for a print shop, never opened a camera RAW whose color profile Adobe has maintained for twenty years, never did the conform on cinema footage, the work of gathering the camera’s raw material for finishing. I’ll get there, after the basics.

Reverse engineering wasn’t born last week

Every emulator you’ve ever used exists because someone opened up hardware, disassembled a binary, sniffed a protocol and, in some cases, broke cryptography. The discipline has case law, commercial tooling and decades of history. A condensed timeline, with the milestones that matter most:

• 1982 to 1984: Compaq reverse engineers the IBM PC BIOS, and in 1984 Phoenix sells a licensable BIOS built in a clean room, by a team that only read the specification. The PC compatible industry is born. Source.
• 1992: Andrew Tridgell creates Samba by sniffing SMB packets. The same year, Sega v. Accolade and Atari v. Nintendo establish in the US that disassembling code to reach functional elements is fair use (Atari lost its own case for another reason: it lied to the Copyright Office to get the source).
• 1993: Wine begins, which becomes the foundation of Proton and of every Windows game that runs on the Steam Deck today.
• 1999: UltraHLE runs commercial N64 games at full speed; bleem! emulates the PS1, Sony sues and loses the injunctions, and bleem! folds anyway in 2001; DeCSS breaks DVD protection.
• 2000: Sony v. Connectix rules that copying the PS1 BIOS to build an emulator is fair use.
• 2002 and 2003: PCSX2 and Dolphin are born, the two emulators that twenty years later are still the reference.
• 2007: the iPhone is opened eleven days after launch. The European Commission forces Microsoft to document its network protocols, and the Samba team receives the documentation for 10,000 euros.
• 2010: the fail0verflow group shows at 27C3 that Sony signed the PS3 firmware with a fixed nonce, exposing the private key (the nonce is the number that should be random in every signature). Days later, already in 2011, geohot publishes the key, Sony sues and settles in March.
• 2011 and 2013: RPCS3 (PS3) and Xenia (Xbox 360) are born, the two hardest emulators ever built, for reasons I’ll explain.
• 2019: the NSA releases Ghidra as open source. The Super Mario 64 decompilation leaks and goes public, and the PC port appears the following year.
• 2021 and 2022: Ocarina of Time 100% decompiled after almost two years of work; Ship of Harkinian becomes the reference port.
• 2024: Mr-Wiseguy releases Zelda 64: Recompiled, the first showcase of N64 static recompilation.
• 2025: Sonic Unleashed Recompiled brings the same trick to the Xbox 360, converting PowerPC to C++ and shaders to HLSL.

Notice that the current phase, decomp and recomp at scale, starts in 2019, well before any coding agent. AI joined an assembly line that already existed.

The tools

The tooling is mature too, and much of it is free:

• IDA Pro (Hex-Rays), the reference commercial disassembler and decompiler, from one thousand to almost nine thousand euros a year, with a free version limited to x86.
• Ghidra, the NSA’s open answer, with a decompiler and scripting, Apache 2.0.
• Binary Ninja, with a free edition and licenses from 200 to 3,500 dollars.
• radare2 and Cutter, the GUI for rizin, which is a radare2 fork: the open command line and the open interface.
• x64dbg and WinDbg on Windows, GDB everywhere else.
• Frida, to inject code into a live process and watch what it does.
• JADX for Android and ILSpy for .NET.

Keep this list in mind. REA, which we’ll get to shortly, is a wrapper around it.

How a binary gets taken apart, in practice

In principle, every binary is reversible. The processor has to be able to execute it, so the instructions are in there. The path is always the same:

1. Disassemble: the binary becomes assembly, one machine instruction per line. This is mechanical.
2. Decompile: the assembly becomes something resembling C. Ghidra and IDA do this reasonably well, with a lot of noise.
3. Recover what the compiler threw away: variable names, function names, types, structures, comments. None of it survives compilation. This is the real work, and it’s human.
4. Refactor until it becomes code a person can read, maintain and port.

The N64 decomp projects take step 4 all the way: their C code, recompiled, produces the original ROM byte for byte. That’s how Super Mario 64 became a project with 135K lines of game C code (around 100K of actual code, plus half a million lines of generated level and model data), and Ocarina of Time passed 430K lines. Static recompilation skips part of this: it translates the original processor’s instructions into generic C, without trying to understand them, and replaces only the parts that talk to the hardware. It’s faster and the result is less readable.

“But can’t the AI just read the binary directly?”

Someone always asks: if it’s AI, why go through C? Let the model read the binary and build on top of it. The short answer is that an LLM only knows what was in its training corpus, and a corpus of binaries paired with source code is scarce, because no human has written production assembly in decades.

The long answer is that the literature exists, and it’s instructive precisely because it’s modest. The lineage runs from 2018 neural networks to models like SLaDe (200 million parameters, one per architecture and optimization level), LLM4Decompile (up to 33 billion) and Nova. Meta published an LLM Compiler that translates assembly into LLVM IR. The best results, on isolated benchmark functions, hover around 60 to 65% of code that runs again and gives the same result, dropping sharply when the compiler optimized with -O3.

Two recent studies show the real state of things. DecompileBench, with 23K real functions, found that LLM-based decompilers produce more readable code with half the functional correctness of commercial decompilers. And a September 2026 paper measured that LLM refinement of Ghidra output raises the compilation rate from 75% to 90% and drops behavioral fidelity from 74% to 62%, with up to around 10% of functions with known vulnerabilities no longer reproducing the flaw, meaning the bug vanished without anyone asking.

Summing up what exists: small models, function by function, almost only x86-64 Linux, and the best ones are refinement layers on top of Ghidra, which keeps the classic decompiler in the loop. No emulator, no game, no whole program has been neurally recompiled. Until someone invents that, you work with reality, and reality includes the intermediate layer of programming language and compiler.

The AI assistants that exist for this in production confirm the design: aiDAPal, an IDA plugin by Atredis, Binary Ninja’s Sidekick, the MCP servers for Ghidra. All of them are an LLM naming variables and explaining functions on top of what the traditional decompiler produced.

What REA actually does

I cloned REA and read the code. It’s an MCP server and a TypeScript CLI that wraps the tools from the list above and exposes them to your coding agent: Ghidra, Hopper, IDA, JADX, Frida, Wakaru for obfuscated JavaScript, Playwright to capture what a website does. The agent asks, REA runs the tool and returns pseudocode, call graphs, strings and IPC traces, each result with an evidence record and a list of limitations.

I had already done that flow by hand, months before REA went viral. In July I had Fable 5 analyze the code of TikTok, Clash of Kings and gov.br: I downloaded the APKs, unzipped them (an APK is a zip), ran JADX on the .dex files and let the model read the Java that came out the other side. It was enough to see the endpoints, the permissions, the embedded SDKs and how each app builds the device fingerprint, that persistent identifier that survives clearing data and reinstalling. TikTok came out as industrial-grade collection, armored against auditing; gov.br, as the example of how to do it right.

The three lessons from that post apply to all of REA. Android is far too easy to decompile, and even so the result serves only static analysis, because recompiling the app is out of reach: the names come scrambled by obfuscation, resources and signing keys are missing. The most sensitive part lives in native code, .so files in C and C++, which decompile into nothing readable, and in TikTok that’s exactly where the core sits. And what came out of that work was a report on what the app does, with caveats, and no TikTok clone. REA packages that same flow into an MCP server; the walls stay in the same place.

The judgment is supplied by the LLM you already pay for. REA supplies the deterministic plumbing and the provenance. The phrase “automates the heuristics of reverse engineering”, which circulates out there, appears nowhere in the repository; the description the project itself uses is agent-driven reverse engineering with traceable evidence.

What it requires: the artifact on your machine (native binary, Electron app, .NET assembly, APK, firmware, network capture), Node 22.19 or later, and the heavy tools installed at your own expense. Ghidra is free; IDA, Hopper and JEB the README tells you to buy. There’s no REA API key because it runs locally, and the token cost lands entirely on your agent subscription.

And that cost is the detail nobody mentions. REA’s MCP tool catalog is 2.3 MB of JSON, which goes into the agent’s context. The project itself publishes an experiment in which analyzing SQLite compiled to WASM, on the default path, took 42 tool calls, five minutes and 300K input tokens. A full inventory of an Electron app like Obsidian returns almost 1 MB of JSON.

There’s an open issue complaining that inspecting an ordinary Mac app returns 80K to 230K tokens in one go.

The project is serious and goes far: six months, 2,600 commits, a thousand test files, version 6.4 released on October 11. It’s also unstable: 190 open issues, several breaking changes per minor release, subsystems marked experimental.

The limits are written in the repository itself. CONTRIBUTING excludes breaking DRM, anticheat, licensing and commercial protections. The DISCLAIMER warns that decompilation, reconstruction and AI-assisted analysis “may be incomplete or incorrect”. And even the internal instructions for generating the website’s figures say, in so many words: “never imply REA automatically clones a program” and “never say full reconstruction, exact game clone, or automatic source recovery”.

REA is a good toolbox for someone who already knows how to do reverse engineering and wants an agent holding Ghidra. It isn’t the “copy this app for me” button. The people who wrote the project are the first to say so.

PhotoCraft: pretty screenshot, alpha inside

The PhotoCraft repository was created on September 30. In two weeks, 54K stars, 8 releases and, as I write, more than 800 open issues. The README badge says “early alpha”, and Brandon Thomas himself, quoted by Gizmodo, said it’s nowhere close to ready. The press that tested it reached the same conclusion: Gizmodo called it a “glitchy, but real, Photoshop”, and OMG Ubuntu “a superficial facsimile with pronounced flaws”.

The issues tell the technical story:

• Any image above 8,192 pixels leaves the GPU compositor and falls back to the CPU, with a 6 GB memory peak, because that’s the egui-wgpu texture limit (issue 49, already closed, and the ones that came after it, 1015 and 4017).
• A 12 megapixel photo needs almost 800 MB just to open, adjust and export from the command line (issue 1637).
• Color management is rudimentary: a global working space only, instead of per document, and no view transform (issue 321). Eighteen issues mention CMYK, fifteen mention ICC profiles.
• Almost a hundred open issues mention PSD. Hue/Saturation layers composite differently from Photoshop, and saving over the file overwrites the original’s correct merged image (issue 4089).
• Fifty open issues mention “crash”.

The interface resembles Photoshop, so the screenshot is shareable, and whoever sees the screenshot assumes it’s a carbon copy. That’s expected in a two-week-old alpha. With interest, time and contributions, part of this gets fixed. And then the contributors will discover something the N64 projects never faced.

PhotoCraft today is that Ferrari replica built on top of a VW Beetle: the body has the right shape, you take a photo and most people say “wow, a Ferrari”. Then you turn the key and out comes the Beetle engine. If the photo got you hyped, that’s on you.

The target moves

Decompiling 1998’s Zelda is aiming at a stationary target. Nintendo isn’t going to release Ocarina of Time 2026 with a new file format. Copying a live product is another conversation, and we’ve seen this movie before.

In the 90s and 2000s the target was Microsoft Office. StarOffice became OpenOffice in 2000 and LibreOffice in 2010, and all of them spent years chasing a binary format, .doc, that changed from version to version and only got a public specification in 2008, when the game was already over. The proven part of that story is in the court cases: the European Commission fined Microsoft 497 million euros in 2004 for refusing to supply interoperability information, and 899 million in 2008 for failing to comply. .docx only became an ISO standard in 2008, in a vote so controversial that countries formally protested.

It took more than ten years. And when the format finally opened, Office no longer mattered the same way. The value moved elsewhere: today it’s 400 million paid Office 365 seats, 3 billion Google Workspace users and around 41 million Creative Cloud subscribers. Nobody wants to install that on a desktop, and so nobody has the binary.

Reverse engineering Google Docs means rewriting the product from scratch looking only at the interface, because the code that matters never leaves the server. After the world moved to SaaS, most proprietary software became a walled garden with no binary. No binary, nothing to disassemble.

You can build versions of it, of course. Immich replaces Google Photos for a person or a family: phone backup, face recognition, semantic search, all on your server. What it doesn’t replicate is the infrastructure and the training corpus that make Google’s search Google’s search. And Adobe made the same migration, in pieces.

Photoshop is far more than layers and crop

Here’s the part I know from having paid for it. The features that sell subscriptions today don’t even run on your computer.

Generative Fill and Generative Expand run on Firefly, in the cloud, and consume generative credits. Several Neural Filters, according to Adobe’s own documentation, need an internet connection because the processing happens on Adobe’s servers. AI object selection has had, since 2020, an option to process in the cloud for a refined result. All of this is code that doesn’t exist on your machine for anyone to disassemble.

Then comes what is in the binary, but took decades and partnerships to exist. Camera Raw carries lens correction profiles and per-camera color profiles for hundreds of models, built over twenty years and updated with every release. No clone guesses that. Either you have the data or you don’t.

And there’s color science. The example I always remember is GIMP and CMYK.

GIMP was born in 1995. The first native support for CMYK files arrived in 2022, and only for import and export, with pixels always in RGB. GIMP 3.0, from 2025, still has no CMYK editing mode. It’s on the roadmap as work in progress.

Thirty years of an open project, with good and motivated people, and the print shop still asks for Photoshop. That’s the size of the color science hole, and no coding agent invents color science: it reproduces what’s in the training data.

Why did it take so long, if it was the most requested feature? Because GIMP’s core was born with 8-bit-per-channel RGB baked into every layer, blend mode and filter. Supporting another color model meant rewriting the entire pixel engine, the migration to GEGL, which took the better part of a decade and was only finished in GIMP 2.10, in 2018. CMYK could only start after that.

And CMYK is a family of device profiles, with four inks, a per-printer ICC profile, ink limits and black generation rules, and editing in it means the whole pipeline has to compute in a subtractive four-channel model and round-trip through the profiles without drifting. A single developer, however good a programmer, without experience in this specific field will struggle to implement it, because the knowledge lives in print shops and color labs, and almost none of it is in a training corpus. In GIMP, the people who carried it forward were contributors who understand printing, and even then it took years. No new tool shortcuts that kind of knowledge.

And then there’s the layer that isn’t even technical: commercial licensing. Print-shop color is Pantone, and Pantone is a proprietary library. No open project can ship the palettes; the Inkscape developers reached that conclusion in 2005, and Linux.com explained the legal risk around the same time. Adobe itself felt it in 2022: when Pantone changed its business model, the libraries were pulled from Photoshop, Illustrator and InDesign, old files started opening with their Pantone colors replaced by black, and anyone who needs them pays for a separate plugin subscription. If even Adobe, paying, didn’t escape, imagine a clone that runs on goodwill.

Video is worse

In Premiere, After Effects and DaVinci Resolve, the problem multiplies. A cinema camera doesn’t deliver video, it delivers raw data on its own logarithmic curve: ARRI’s LogC4, RED’s IPP2. To turn that into an image you need the manufacturer’s transforms, and RED ships its own in an SDK that the editors integrate and update with every version. ACES, the industry’s neutral standard, depends on input transforms supplied by each manufacturer.

Blackmagic gives Resolve away for free and it decodes ARRIRAW and R3D. It does that because it pays for the SDK and the partnership on your behalf. An agent-built clone, “guessing” how the Alexa’s image should look, produces an image that doesn’t match the reference monitor, and in cinema that means it’s useless. That knowledge isn’t in any training corpus: it’s in contracts.

The Bun case: what it costs to rewrite big software

For a sense of the order of magnitude, the best public data point there is comes from Bun. In May 2026, Jarred Sumner rewrote the entire runtime from Zig to Rust using agents, and in July published the numbers:

• 535K lines of Zig in 1,448 files became a one-million-line Rust diff, in 11 days.
• Up to 64 Claude instances in parallel, 24 hours a day, every line reviewed by two adversarial agents and a fixer.
• 5.9 billion uncached input tokens, 690 million output tokens, 72 billion cache reads: around US$ 165K at API prices.
• 16K compiler errors fixed by 64 agents before CI went green on six platforms.
• The existing test suite, with more than a million assertions, passed 100% before the merge.

Read again what made it possible:

• the project’s original author at the controls
• a complete, working source code as reference
• a test suite with a million assertions serving as the oracle
• unlimited budget, because Bun has belonged to Anthropic since December 2025, and a pre-release version of Claude Fable 5

Sumner estimates three engineers with full context would have needed a year.

Now take each of those advantages away. PhotoCraft doesn’t have the Photoshop source to port, it has to invent. It doesn’t have a test suite that says “this matches Photoshop”. It doesn’t have Photoshop’s author.

And Photoshop is a 36-year-old product whose disk requirements Adobe lists between 10 and 100 GB, against a JavaScript runtime. Even without knowing the size of Adobe’s codebase, we’re talking about a bigger target, with no reference, no oracle and no owner. Orders of magnitude above US$ 165K, and even then it would be an approximation.

Why retro moves so fast

If it’s so hard, why does an old game port ship every day? Because retro stacks up every advantage the rest doesn’t have.

The binary is available and tiny. Super Mario Bros. on the NES: 40 KB. Super Mario 64: 8 MB. Ocarina of Time and Majora’s Mask: 32 MB each. A whole PS1 CD: 650 MB.

Compare with Baldur’s Gate 3, 150 GB, or with the latest Call of Duty, above 100 GB. That’s three or four orders of magnitude, and the scope shrinks along with it: a 1996 game has a handful of systems, not a physics engine with networking, a store and anticheat.

The technology is old and documented. Twenty-five years of community have already mapped the N64, PS1 and SNES hardware instruction by instruction. The agent discovers nothing; it speeds up work that already has a map. It’s the same pattern I saw in my retrocomputing projects: where there’s a reference and an oracle, AI flies; where there isn’t, it spins in brute force.

The target doesn’t move. Nobody is going to update 1998’s Zelda.

And the legal side was settled thirty years ago. Sega v. Accolade and Sony v. Connectix drew the outline. The ports require you to supply your own ROM.

A modern console is a PC

The second half of the explosion, the PS4, PS5, Xbox One and Switch ports, has another explanation: those consoles are PCs or phones in a different case. The PS4 uses an AMD Jaguar x86-64, the PS5 a Zen 2, the Xbox Series another Zen 2, the Switch an NVIDIA Tegra X1 ARM chip. Your PC’s processor runs that code natively. So nobody needs to emulate the CPU.

The remaining work is translating the operating system and the GPU. shadPS4 reimplements the PS4’s system libraries and recompiles the shaders: games ship from the factory with AMD GPU machine code, which the emulator decodes instruction by instruction and converts to SPIR-V, Vulkan’s format.

Azimuth, which makes Forza Horizon 2 run on Windows, translates Xbox WinRT calls to Win32 and DirectX 11.X to desktop DirectX 12, with the game’s x86 code running directly. AnyPS5 relinks the executable: it takes the PS5 binary and produces a native Linux or Windows binary, linked against reimplemented system libraries. In all three cases, Proton, DXVK, VKD3D and MoltenVK had solved the graphics translation layer years earlier.

The reverse proof is the consoles that weren’t PCs. The PS3 uses the Cell, a PowerPC with seven active coprocessors (the SPEs), each with its own local memory and no cache, and RPCS3 has been at it since 2011, with SPU emulation still the bottleneck today. The Xbox 360 is PowerPC too, which is why Sonic Unleashed needed XenonRecomp to translate PowerPC into C++. Reverse engineering is case by case, and the architecture decides whether the case is easy.

AI is in the loop, and there’s already a fight

That the 2026 ports use AI, many of them say outright. The Silent Hill decomp writes in its README that development is “heavily AI-assisted (Claude Opus 4.6, 4.7 and the newer Fable model), and we’re open about that. It is not automated translation of the game”. The Pokémon Stadium recomp lists Claude as a contributor.

The devs of the Zelda and Banjo recomps, who spent years doing this by hand, went to war with what they call “slopcomps”, agent-generated ports of alpha quality.

And it’s worth separating headline from actual state here too. AnyPS5 has a 0.1 version from September that, according to those who tested it, runs one 2D indie game; the viral claim that “the PS5 is 80% ported to PC” comes from fans, and no 3D game runs. Astro Bot runs, but in emulators (KytyPS5, SharpEmu), which take the traditional path. Forza Horizon 2 runs, and Azimuth is a closed binary from a studio that declares itself “AI assisted” without saying anything more.

“But in two years tokens will cost nothing”

The standard answer to all of this is: it’s hard today, in two years it’s trivial, because AI grows exponentially. The numbers say otherwise.

The price of the most expensive tier barely fell. GPT-4 in March 2023 cost US$ 30 and 60 per million input and output tokens; Claude 3 Opus in 2024, 15 and 75; today’s frontier models, Fable 5.1 and GPT-6 Astra, cost 10 and 50. In three and a half years, the frontier got between 1.2 and 3 times cheaper, depending on which end you look at.

What really fell was the cost of a fixed quality: Epoch AI measured drops of 9 to 900 times per year depending on the task, and in a 2026 review sums it up as “very roughly 5 to 10 times per year”, with the caveat that it might slow down. That’s real and it’s great.

But apply the math to Bun. US$ 165K divided by 10 is US$ 16K. By 100, US$ 1,650. That’s to port a runtime with the source, the tests and the author in hand.

For an amateur trying to clone Photoshop with none of that, the bill starts orders of magnitude higher and falls in the same proportion. Cheap isn’t the same as trivial.

And the “exponential” premise died in 2024. Ilya Sutskever said at NeurIPS that pre-training as we know it will end, because there’s only one internet. Epoch projects that reasoning model scaling converges to the general frontier rate in about a year.

I’ve written about this three times in recent weeks: the limits of what the model reproduces, the convergence of raw models with the gains coming from the harness, and the languages benchmark where the frontier models tied at the ceiling. The tool we have is the tool we’ll have for a good while, with incremental improvement. Plan with it.

Worth recalling the number that opens the limits post: per the Octoverse 2025, GitHub’s official report, 81.5% of contributions on the platform happened in private repositories. That’s the code that really matters, the code that pays salaries, and no LLM has seen a line of it. The model reproduces what was in the training data, and that code was never there.

Now extend that to every SaaS product and every component it uses. The Adobe code that’s actually worth something, the color engine, the profiles, the cloud filters, isn’t available anywhere, not even as a binary. Anyone who wants to reproduce it will have to redo the research and development, expensive and slow, over the same years Adobe took. And at the end they still run into third-party licensed intellectual property, Pantone, camera SDKs, manufacturer profiles, that you will never be able to have.

Answering the list from the top

Does PhotoCraft exist? It does, it’s a two-week-old alpha with more than 800 open issues, rudimentary color management, freezing above 8K and opening PSDs wrong. It looks like Photoshop in a screenshot. It can improve a lot, and it will discover that the target moves and that Photoshop’s expensive parts live in the cloud or in manufacturer contracts.

Does REA exist? It does, it’s good, and it’s a toolbox for people who already know how to use Ghidra, requiring the binary in hand, the tools installed and hundreds of thousands of tokens per analysis. The project itself avoids saying it clones programs.

Did recomps and decomps accelerate? Yes, for real, and AI has a part in it. They accelerated where the binary exists, is small, has been documented for 25 years and doesn’t move. Or where the console is a PC in disguise and the work is API translation, solved by Wine and Vulkan years earlier.

Is Adobe dead? Adobe has around 40 million subscribers by the most cited estimate, the features that sell subscriptions run on its servers, and thirty years of GIMP still leave the print shop asking for Photoshop. I wish it were otherwise. The data says what will happen is what always happened with clones of a live product: years of chasing, and when you catch up, the value has already moved.

Are programmers dead? Rewriting Bun required Bun’s author. The decomps that work have people who know the hardware driving the agent, and the ones that don’t became “slopcomps”, name and all. Reverse engineering got cheaper and more accessible, as it did with Ghidra in 2019, and it’s still an engineer’s job. Without us, the tool returns a pretty screenshot.

Proprietary software with a binary in hand will come under pressure, and it should. Most of the software you use has no binary at all, and the software that does already depends on cloud services that don’t disassemble. Software engineering with AI has a great road ahead. “It’s over” headlines get clicks and don’t compile.

Underneath this whole story there’s a decision almost nobody makes consciously: the filesystem. The Synology used btrfs all along. The QNAP uses ZFS. And your Linux machine, most likely, came from the factory with ext4 without anyone asking you anything. That’s what this is about.

Where the two came from

ext4 is the end of a dynasty of over three decades. ext (extended filesystem) was born with Linux itself, ext2 became the standard in the 90s, ext3 added a journal in 2001 so you wouldn’t need an epic fsck after every power loss, and ext4 arrived in 2008 with extents, delayed allocation and volumes up to 1 exbibyte. The design goal was always the same: be the default filesystem, fast, simple and predictable. It’s excellent at that. And it stopped there.

btrfs was born in 2007, at Oracle, with Chris Mason, with a completely different goal: be Linux’s answer to ZFS. Instead of writing data over data, it’s copy-on-write: every write goes to a new block and the pointers are updated cascading up the tree. That single decision is the mother of everything that matters in it.

What each one delivers, in a list:

ext4:

• journaling (consistent metadata after a crash)
• extents and delayed allocation (good performance)
• mature, stable, everywhere, every tool works

btrfs:

• copy-on-write on everything
• data and metadata checksums on every block
• instant, cheap snapshots
• independent subvolumes inside the same volume
• transparent compression (zstd, lzo, zlib)
• native RAID and scrub (verification sweep)
• snapshot send/receive for incremental backup
• reflinks (copies that don’t duplicate blocks)

The list already answers the title’s question. Let’s go piece by piece.

Even NTFS and APFS have passed ext4

And if the comparison with btrfs seems unfair, look sideways. NTFS was born in 1993, with Windows NT, already with metadata journaling and ACLs, and back in the 2000s it gained a log of every change on the volume plus shadow copies (Windows’ “Previous Versions”). APFS, which Apple launched in 2017 made for flash, is copy-on-write like btrfs: native snapshots (it’s what Time Machine uses for local restores), instant file clones that share blocks, volumes sharing the space of the same container, and per-file encryption.

Read the ext4 list again and compare: no snapshots, no data checksums, no CoW, no compression. Windows’ and Mac’s filesystems, which nobody considers examples of sophistication, protect your data better than Linux’s default, whose last big feature came out in 2008. btrfs is what puts Linux in the same generation as the other two, instead of a decade behind.

Why btrfs is far superior

What copy-on-write is, for those who never stopped to think about it

In classic filesystems, writing is destructive by definition: updating a block means writing the new content over the old one, in the same place on disk. If the machine dies mid-write, the file ends up half old, half new. The ext3/ext4 journal covers that for metadata (the directory structure doesn’t corrupt), but your data itself is left to luck.

Copy-on-write inverts the order: nothing is ever overwritten. Writing to a file allocates a new block, writes the new content there, and only when everything is safely on disk does the filesystem flip the pointers to the new version, atomically. At any instant, you have either the whole old version or the whole new version. Never a mix of the two.

What that buys in practice:

• Nearly free snapshots. The old blocks already persist by CoW’s nature; a snapshot is just the filesystem holding the root pointer of that instant, pure accounting.
• Power loss doesn’t corrupt. An interrupted write leaves the previous version intact, because it was never touched. For anyone who has lost a VM image or a database to a mid-write crash, this is worth the whole filesystem.
• Verification becomes possible. Since each block is written once and never touched again, keeping its checksum forever makes sense. It’s the base of the corruption protection I detail further down.

The cost exists and I pay it knowingly: random-write-heavy files (VM disks, databases) fragment and duplicate blocks, because each rewrite becomes a new block. For those cases you disable CoW per directory with chattr +C, which must be set before the file is created. In my layout, the VM image made with the right +C stayed healthy; the one made without became the textbook fragmentation case.

Snapshots are the feature that changes your life

A btrfs snapshot is an instant photo of a subvolume, and it costs almost nothing, because CoW already keeps the old blocks while someone references them. Creating a snapshot is a seconds operation, not hours. In practice: before every upgrade, the system takes a photo. If the upgrade breaks, you don’t “uninstall packages”: you boot back into the previous photo and you’re done. The boot recovery section, further down, shows this happening for real.

Subvolumes are the right way to organize the disk

A subvolume is an independent namespace inside the same filesystem, with its own snapshot policy. And it has a killer property: snapshots are not recursive. A child subvolume inside another is automatically left out of the parent’s snapshot.

I use this for everything. On my desktop, the cache directories that can be re-downloaded (mise, rustup, npm, opencode, AI caches) are becoming nested subvolumes inside @home: they stay in the same place for any program, but out of every snapshot. That separation alone takes 165 GiB out of snapshot scope, and the big caches (Hugging Face, yay, uv, pip) already live outside the system volume, with symlinks pointing to them: minus 148 GiB of snapshot, part already migrated and part in progress. The snapshot stays small, the restore stays fast, and whatever can be re-downloaded from the internet never enters the photo.

The real cost, measured on the new QNAP (ZFS, but the CoW math is the same): 27 hourly snapshots of a folder with almost 80 TB, and nearly all of them consume megabytes, because between one hour and the next almost nothing changed. Cheap snapshots are what let this routine exist without guilt.

Reflink deduplication

First, the caveat that matters: btrfs has no native online deduplication like ZFS (which keeps a dedup table in RAM and charges dearly for it). What it has are two practical things: reflink and offline dedup. Reflink is a copy that shares blocks: cp --reflink=auto creates a new file that points to the same blocks as the original, instant and free, and each one starts diverging only where written. In the era of dozens of checkouts, worktrees and similar builds, that saves a mountain of blocks. For retroactive dedup there are userspace tools like duperemove and bees.

Real corruption protection

Every btrfs block, data or metadata, carries a checksum. When the filesystem reads and the checksum doesn’t match, it knows the block rotted, and if there’s a second copy (RAID or metadata DUP) it self-heals. ext4 checks no data at all: bit rot passes silently until you open the file years later.

Two pieces of my configuration show what that buys in practice, and what ext4 simply doesn’t offer. The first is metadata in DUP mode. On ext4, each structure on the volume exists only once: a well-placed bad sector corrupts the directory table and the whole drive stops mounting. On btrfs, even with a single drive, every metadata block exists duplicated. The bad sector keeps existing, but the checksum flags the rotten block on read and the second copy answers in its place, without you noticing. It’s the difference between losing the volume and noting a warning in the log.

The second is the monthly scrub. ext4 never re-reads anything to check health: if a block rots, you find out the day someone tries to open that specific file, maybe years later, when there’s no fixing it anymore. The scrub is a scheduled sweep that re-reads the whole volume validating checksums, and finds the rot while it’s still small and recoverable. When the btrfs device stats counter leaves zero, it’s the siren to replace the drive before the disaster, not after.

Transparent compression, measured

Every mount on my desktop uses compress=zstd:1. Level 1 costs a few percentage points of one CPU core, and btrfs detects what doesn’t compress (model blob, video) and writes it raw, at no cost. The result, measured this week on my two new drives:

That’s 630 GB of free space on 2.2 TB: 15% more drive, paying nothing. On the initial migration the gain was even bigger, 45% on the system volume, because there was more text and less binary. Notice the table’s details: programs and libraries compress well, Docker layers compress even better (39%), text logs compress 4.5x, and AI model weights compress nothing, which is expected and costs nothing. Source code, which is what we produce the most in the AI era, compresses beautifully.

Synology uses btrfs, QNAP uses ZFS

Being fair to ZFS: for a big NAS it’s superior, and it’s what I chose for the QNAP. Native checksums and self-healing, mature RAIDZ, excellent send/receive. The price is complexity and resources: ZFS wants RAM for the ARC cache (my QNAP has 7 GB for a 174 TB pool, and I feel the lack), its online deduplication would need about 190 GB of RAM just for the table on my volume, and its provisioning model gave me a free scare when the snapshot was refused with 20 TB free (the thick vs thin story is in the NAS article).

btrfs is the right balance for a consumer machine. Light, native in the kernel, does almost everything ZFS does without demanding a storage administrator. Desktop, laptop, homelab, small or medium NAS: btrfs. Synology proved it in practice, twice: the btrfs volume survived readable with two drives out, and its snapshots saved 11 albums I deleted by mistake in the middle of the cleanup, byte by byte.

Big, serious NAS, with RAM to spare: ZFS. Everything else: btrfs. And for a personal Linux machine, it’s not even a discussion.

Boot recovery via snapshot

The feature that has saved me for real, told on both machines.

On the desktop, Omarchy configures snapper to take a snapshot before every update and limine-snapper-sync publishes each snapshot as a boot entry in Limine. New kernel won’t come up? In the boot menu you pick yesterday’s snapshot and the machine comes back as if nothing happened. On my machine the scheduled routine runs via Timeshift on cron (every boot and hourly), with snapper covering the update points: the way back is the same mechanism. (The gotcha: each snapshot entry carries the kernel image along, so a small ESP fills up fast. My 1 GB one already refused a kernel update for lack of space; the system stayed bootable on the old kernel, which is the right behavior.)

On the server, which runs openSUSE MicroOS, it’s even stronger: the whole system is transactional btrfs, and every update is a new snapshot. Last week, kernel 7.2.9 took down the NVMe controller silently, twice, in the middle of the nightly backup. The fix was a transactional-update rollback and a reboot back to 7.2.6, which ran the whole load (754 GiB of backup reads with concurrent download) without a hiccup. A kernel regression became a scheduled reboot, not a recovery all-nighter. One documented caveat: /etc lives inside the snapshot, so the rollback undoes configuration changes along with it. After any rollback, check your units again.

Maintenance is not optional (and the deleted file that won’t go away)

A modern CoW filesystem delivers a lot, but it charges two routines ext4 never asked for.

The first is the scrub, which already appeared here: re-read everything and check checksums, monthly, on all machines. It’s bit-rot detection in action, and without it the checksum only warns you about the rot when you try to read the file. The second point is subtler: a snapshot pins the blocks of deleted files. When you delete a file inside a snapshotted subvolume, the space doesn’t come back, because the block stays referenced by the photo. That’s exactly what happened on my NAS: the 2.14 TB cleanup I ran on the Synology only freed space for real after the old snapshots were removed; until then, the blocks stayed pinned.

The treatment is simple routine. Retention has to be automatic, with snapper or Timeshift deleting the old snapshots per policy (on the QNAP, the Snapshot Manager’s retention policy plays the same role). Space that doesn’t come back after deleting old snapshots is usually over-allocated chunks: a btrfs balance start -dusage=50 returns the little-used pieces to the free pool (in my house I don’t blindly schedule balance, I monitor the unallocated space and act when it gets tight). And on the restic side, forget alone only removes the snapshot’s reference: the repository space is only reclaimed with prune along, forget --prune.

House cleaning in the AI coding era

Now the part almost nobody connects with filesystems: the agent era produces garbage at industrial speed. I know because I measured.

• Rust is the biggest disk polluter in the dev world. Every build leaves behind 100 to 180 MB test binaries, and they accumulate with each edit. On my desktop, the target/ directories added up to 920 GiB. ai-memory alone accounted for 704 GiB. The hygiene tool I was using is cargo-sweep, weekly. A reader pointed out that the project declares itself unmaintained, and its own README points to cargo-clean-all, which works differently: instead of pruning old artifacts inside one target/, it finds every Rust project under a directory and deletes the whole target/ of the ones you filter, for example cargo clean-all --keep-days 7 ~ to clean only what hasn’t been compiled in a week.
• Docker is right behind. On my machine I found 1.26 TB of build cache plus 938 GB of images. A single cleanup recovered about 600 GB. On the server, the weekly cleanup collects 0.6 to 4.5 GB every Sunday.
• AI and package caches bloat quietly. The Hugging Face cache had 86 GB until I deleted three dead models and it dropped to 21. Pacman had 22 GB of outdated packages. yay, uv and pip, another 125 GB.
• And agents create temporary directories without asking permission. A complex project with several branches means worktrees, and each release worktree has its own CARGO_TARGET_DIR full of test binaries. On ai-memory that has already blown up the disk, and it became a mandatory rule in the project’s AGENTS.md: after every release, cargo clean in each created worktree and removal of finished worktrees, no exceptions.

The solution is to automate the sweeping. My minimal example, a user timer that runs on the 1st and 15th at dawn, with Nice and idle I/O:

• deletes target/ directories of projects idle for more than 14 days
• cleans the yay cache
• prunes only Docker build cache older than 30 days, never images or volumes

And the part that became standard in my house: instruct the agents to clean up what they create. Three real rules from my projects, for you to copy:

In the desktop’s AGENTS.md: “Never run broad cleanups like docker system prune without explicit user authorization after a read-only inventory. Cleanup uses an exact list of IDs; idle, old or unreferenced does not mean disposable.”

In the homelab’s CLAUDE.md: “Before finishing any task, remove the temporary things the task created, and only those. Never delete what you didn’t create or can’t attribute to the task. When in doubt, list it for the owner.”

In ai-memory’s AGENTS.md: “target/ grows without limit: every edit to a shared crate leaves behind a copy of each 100-180 MB test binary (seen at 157 GiB). Weekly cargo sweep --time 7, and mandatory cargo clean after every release.”

And the real backup: restic

A snapshot lives on the same drive, so a snapshot is not a backup: if the drive dies, the snapshot dies with it. The last item in the kit is restic, which for me is the best backup system on Linux today.

restic is a single open source binary, written in Go, that does deduplicated, encrypted and versioned backup to practically any destination: another directory, another NAS via SFTP or REST, S3, Glacier. The central trick is content-defined chunking: it slices each file into variable-size blocks defined by the content, hashes each block and only writes to the repository the blocks that don’t exist there yet. That’s why the delta is cheap: today’s backup carries the blocks that changed since yesterday, not the files that changed. Editing a 100 GB database in the middle costs a few megabytes of backup, not 100 GB again. Each run becomes a repository snapshot, a restore point you browse with restic mount as if it were a filesystem, and restic check audits the whole repository’s integrity whenever you ask.

Configuration matters more than the tool, so here’s ours, mistakes included. On the desktop, the repository lives on the NAS and retention is keep-daily 7, keep-weekly 4, keep-monthly 6, keep-yearly 1. On the server, backup runs daily at dawn to the repository on the QNAP, covering /etc, /usr/local/bin, /home and the container volumes, with the password off the machine and pg_dump before the databases, because a filesystem snapshot doesn’t guarantee consistency for a database that’s writing. The first full load was 729 GiB in under an hour, and restic crossed unharmed, on retries, the window in which the NFS mount went read-only by mistake during the migration.

The most important part of the configuration is what does not go into the backup. Our exclusions, all with a reason:

• AI model caches (Immich, Hugging Face, EasyOCR): if lost, download again;
• Prometheus metrics: history that regenerates itself;
• the Fulcrum database: reindexes from bitcoind;
• the decision classifiers’ weights: re-downloadable by revision hash (and this exclusion’s glob goes inside quotes inside the bash array, because unquoted it becomes a backup source instead of an exclusion);
• node_modules, target/, cargo registry, rustup toolchains: pure build artifacts.

And one retention lesson to close. When we shortened the server policy to “yesterday, 2 dailies, 2 weeklies and 1 monthly”, the correct translation to flags is --keep-daily 3 --keep-weekly 3 --keep-monthly 2, not 2/2/1: restic’s levels overlap, and yesterday’s snapshot is also the newest of this week and this month, so each number needs to be one more than the older points you want to keep. The detail caught us off guard: the script had a literal 2/2/1, and with it the monthly level would never hold anything older than last night. In practice, the intention was to keep 5 restore points, and 2/2/1 delivers only 3: yesterday, plus one daily, plus one weekly. We erred, measured, corrected.

A humility note from the disaster: the old repositories were left on the Synology by my decision (almost 2 TB of abandoned history, because the content was already re-covered on the new QNAP). A backup repository ages too, and revisiting its scope is part of the hygiene.

Conclusion: in the AI era, disk is not a detail

Agents coding all day change the storage math at both ends. On the volume end: builds, caches, worktrees and models fill the disk much faster than any human habit ever did, and without a cleanup routine you find out when it blows up. On the safety end: the more generated code, the more you need a cheap way back, because not every agent change is a good one.

btrfs covers both. Transparent compression gives back 15% of disk for free, and source code is exactly the kind of data that compresses the most. Snapshots and subvolumes give you instant rollback on the system and isolation for what doesn’t need protection. Checksums and scrub watch the health of blocks that ext4 simply ignores. ext4 had its 15 years of well-deserved reign. In 2026, choosing it on a new Linux is giving up protection you’ll need, sooner or later. Choose btrfs and set up the cleanup today.

• New v4 benchmarks: I retested the major models and the uncomfortable conclusion was convergence, not a leap.
• The limits of LLMs: models are great at reproducing what exists, extremely expensive at what doesn’t, and the served model is a static file that learns nothing from your marathon.
• Are there better languages for programming with LLMs?: no. Neither script conciseness nor strong typing changed the outcome; test discipline did.
• Which Jev-like decision model is best?: 21 classifier configurations, and the “specialized models” themselves converged to within a few points of each other.
• Clean Code for AI agents, VS Code is the new punch card, my skills, the hot take against loop engineering and minimum best practices: the same thesis from other angles. The model is a commodity; the engineering around it is the differentiator.

The thesis that closes this arc: LLMs have reached, or are very close to reaching, the ceiling of the transformer architecture. And the good news is that nothing more than that is needed.

Convergence, not a leap

In my benchmarks, I see no relevant improvement in general programming since the Opus 4.5 generation. Yes, Fable and Astra are great at specific problems. In general programming, what I measure is everyone squeezed onto the same tier, including the Chinese commercial offerings (Kimi, GLM, DeepSeek). Model choice has become nearly irrelevant.

What still makes a real difference is the effort level: low effort for translation and summarization, high effort for vulnerability sweeps, heavy bug fixing or optimization. Beyond that, the numbers tell a reality much more boring than the vendors’ propaganda.

My reading of the timeline: the models’ “core” reasoning had its last big leap in 2024. 2025 was the year of optimizing the architecture. 2026 is the year of refining the harness. Base intelligence made no order-of-magnitude leap in that period; efficiency and long-session competence did.

The timeline, with receipts

2019: GPT-2 proves LLMs are useful. 1.5 billion parameters, summarizing, translating and answering with no specific training, zero-shot. OpenAI found it so impressive that it held back the full model for months fearing misuse. Today it looks cute.

2020: GPT-3 opens the parameter war. 175 billion, a hundred times GPT-2, and the in-context few-shot learning that became the paradigm. Kaplan’s scaling law, published months earlier, said loss fell as a power law with size, and the industry read it as “bigger is better.” There went two years of who-has-more-parameters.

2022: Chinchilla recalibrates the war. DeepMind trained 400 models and showed the giants were undertrained: a 70B model with 1.4 trillion tokens beat the 280B Gopher. The “training data” axis started weighing as much as the “parameters” axis. First crack in the war.

2023: GPT-4 ends the war by decree. The technical report refused to publish size, hardware or data, citing the competitive landscape and safety implications. The most cited leak, from SemiAnalysis, described 1.76 trillion parameters in a mixture of experts. Since then, parameter counts vanished from marketing: neither the GPT-5 announcements nor the Claude 4 ones cite a number. Notice neither Astra nor Fable spends a line on it today. The war ended and nobody signed an armistice.

2024: the reasoning peak. September’s o1 marked that peak: RL to produce long chains of thought, and the discovery that more test-time compute improves the answer. That is, by my yardstick, the last “base intelligence” revolution.

2025: architecture optimization. Mixture of experts became the standard: Mixtral was already showing in 2023 that 46.7B total parameters with only 12.9B active per token cost like a small model, and DeepSeek-V3 took that to 671B total with 37B active, with the final training run costing under 6 million dollars in GPU-hours at H800 rental prices (not counting the prior R&D). The MoE magic is simple: it separates capacity (total parameters, which live in memory) from per-token cost (only the experts chosen by the router compute). You pay for a 37B model and carry a 671B one.

On the serving side, prompt caching cut up to 90% of repeated-context cost (OpenAI followed two months later), context windows passed 1 million tokens, and serving techniques like speculative decoding and PagedAttention multiplied throughput. And DeepSeek-R1-Zero showed that pure RL, with no supervised fine-tuning, makes reasoning emerge. None of this is “more intelligence.” All of it is the same intelligence, cheaper, faster and more stable.

2026: the agentic era. Function calling has existed since June 2023, but the ecosystem turning point was Anthropic’s MCP in late 2024, which OpenAI adopted months later and which today lives at the Linux Foundation. Claude Code, Codex and co. came out in 2025, and the metric that matters became METR’s: the task size an agent completes with 50% reliability had been doubling every 7 months since 2019. Anthropic’s own guide on building effective agents states the obvious plainly: simple, composable patterns, not fanciful frameworks.

Why the agent allows a less dense model

Here’s the mechanism most people don’t stop to think about. A lone model needs to carry the world in its weights, because its only chance to answer is that one, in a single pass. A model with a harness doesn’t:

• On-demand search and reading. With web search and fetch, low-priority knowledge leaves the weights and becomes a lookup: the agent pulls it into the context window only when needed. The model can be less encyclopedic and more operator.
• Controlled trial and error. Alone, the model needs to guess the right answer on the first try. With an agent, it proposes a hypothesis, implements, sends it to be tested, gets the feedback and decides again with real information. Each such cycle is new knowledge entering the session without costing a training neuron.

Both mechanisms fight hallucination at the root: fetch brings the fact the model doesn’t know by heart, and the test loop catches the invention before it becomes code. That’s why 2026 feels like a giant leap: the model+harness combination got much more competent, even with base intelligence nearly stalled.

Not every problem needs a giant LLM

The same “less dense” logic applies inside the application. Much of what people throw at a chat LLM is disguised classification: labeling tickets, routing queues, applying policy. For that there are specialized decision models, and I covered the topic twice: the Jev hype, with Paulo Câmara’s essay on why a classifier is the first thing a computer scientist thinks of, and my benchmark with 21 configurations, where a Microsoft 9B tied the hosted leader and a local 4B came within a few points, at cents per ten thousand decisions.

The “193x faster” shock of Jev’s marketing only exists because many people were using a giant LLM as an expensive, slow classifier. When the right tool enters, the frontier model leaves the play, and along with it the cost, the latency and the format hallucination. It’s this post’s thesis in miniature: competence doesn’t come from density, it comes from fit.

The loop nobody sees

There’s a user-experience detail in this story. In a good agentic session, what happens is semi-transparent to whoever requested the task: the model guesses wrong, the agent fetches the documentation online or runs the real test, the error comes back as feedback, and the model corrects itself. You see “task requested, task delivered.” In between, dozens of micro-corrections happened that nobody witnessed.

My bet: since agents are trained to fill the context when they need to, even new language or library versions no longer confuse them that much. The model doesn’t need to know the API by heart; it looks it up, tests and adjusts in the loop. That’s what I see in my experiments, from converting NES ROMs to Master System to the little day-to-day problems. And it’s what the languages benchmark measured: seven languages came out practically equivalent, because the loop equalizes what training leaves uneven.

I measured the caveat too. When there’s no documentation to fetch and no test to validate against, the loop spins in a vacuum: that’s what happened with Bend 2, a language too new to have a corpus, and it was the central point of the post about the limits. The mechanism is powerful, but it needs an oracle.

And there’s a receipt that the base is stalled. Back in November 2024 Reuters was already reporting diminishing returns in pre-training at the three big labs. At the following NeurIPS, Sutskever, the scaling-thesis guy, declared that “pre-training as we know it will end”: the public data ran out. And Stanford’s AI Index measured the gap between the first and the tenth model on Chatbot Arena shrinking from 12% to 5% in one year. That’s convergence, with numbers.

The propaganda in the opposite direction

While the numbers show a plateau, the marketing sells the apocalypse. I already dissected a whole week of that in “You’re an idiot if you believe the misleading ads from OpenAI, Anthropic, NVIDIA”, so here’s only what matters for this narrative:

• OpenAI’s “Navier-Stokes proof” was agentic brute force at industrial scale, 130 billion tokens over human mathematicians’ machinery, sold as emergent intelligence.
• The Hugging Face incident was a swarm running 17 thousand attempts with automatic verification, not an “agent that gained consciousness and escaped.” Expensive trial and error, exactly the mechanism I described here.
• Fear is a product: whoever sells “my AI is too powerful and might escape control” is, in the same sentence, asking for regulation that locks out the smaller competitor and inflating the perception that their model is more capable than the others'.

The facts I measured point the opposite way from the apocalypse: models converging, differences shrinking, and useful work migrating from the model to the harness. Rogue AI gaining consciousness is not what the logs show. What the logs show is engineering.

“Competent enough”

I don’t expect another order-of-magnitude reasoning revolution in this architecture. What’s coming is incremental agentic improvement: more robust tool calling, better memory, longer sessions, less hallucination per verification cycle. And this post’s point is that this is already enough.

What the agentic era delivered was the automation of correct software engineering: long, reliable coding sessions, with testing and feedback, that accelerated hundreds, if not thousands, of open source projects. The deploy premise changed along with it. It’s not a silver bullet. It’s a new compiler for an old discipline: whoever had process, tests and documented invariants reaped; whoever had only hype is still waiting for the next model to solve it.

And to wrap up: I see no problem at all in raw models stagnating. The limits we already know, and harness optimizations will soon hit diminishing returns too. What remains is what always remained: real software engineering technique, with automated tests first, and an experienced engineer at the wheel. With that, coding results are excellent today, not one model generation from now.

What won’t exist is magic for amateurs. Asking an agent for any coding task without knowing what you’re doing doesn’t get back secure, optimized, well-thought-out code: it gets slop. The agent responds better to those who know what to ask for, that is, to those who understand this architecture, the pros, the cons, and know how to work around the weaknesses and leverage the strengths of the model+harness combination.

Hence the title. A lone LLM is an expensive parrot. An LLM with an agent is competent enough to work for real. And “competent enough,” added to your engineering discipline, is everything the industry needed. The rest is marketing waiting for the next magic parameter count that nobody publishes anymore.

Last conversion of the Campfire from my token stash goes to JavaScript/Express. As with the others, I'm sure you can find more to optimize. But this gives you a sense of what order of magnitude these options exist within with agent-led development. https://t.co/UUapL99SJE https://t.co/2H2NXfeZog

— DHH (@dhh) October 5, 2026

I won’t defend him this time. A one-shot conversion, done by an agent, with no fine-tuning at all, doesn’t measure language: it measures how much the model knows each stack’s idioms. It’s rage bait and he knows it. That said, I do the same from time to time, and so do half his critics, so we’re all hypocrites and life goes on.

And the weak spot of his methodology became a topic of its own: several people noticed the Rust version had gone through far more agent iterations than the others, meaning more optimization rounds. @kurtome answered in practice: he took the Elixir version and sped up the main page requests 10x, with obvious optimizations that no first conversion does, like caching and async database connection pooling.

@dhh Here you go https://t.co/gUtK7Z7zG7

got a 10x speed up on the Room page requests, and sped up all the other requests significantly as well. the biggest gains came from some no brainers: implement caching and pool the db connections with proper async https://t.co/mbp94q9N0s

— kurt (.com) (@kurtome) October 5, 2026

His comparison, then, wasn’t “Rust versus Elixir”: it was “Rust after N tuning rounds versus Elixir after one.” And that lays bare the truth the thread pretends to ignore: no real software ships from half a dozen prompts. Coding with AI shines when an experienced engineer steers the agent toward their goal, and that means dozens of iterations, not three.

The good part is that the bait was useful to me. I’ve been doing LLM benchmarks and decision-model benchmarks, but I had never measured, in a reproducible way, the question that thread pretended to answer: does it matter which language you pick when the one writing the code is an LLM? My short answer was already out in two posts of mine:

Pressure under load is not a trivial problem to solve from scratch and I recommend you do not attempt to. It's a solved problem already and all solutions converge to what Elixir already brings out of the box. So yes. I use Rust all the time, but if I had to solve high scalability problems (particularly high volume message queues), I would think of Elixir first.

— Fabio Akita (@AkitaOnRails) October 9, 2026

Language wars are tiresome. And the reality is simpler if you're doing AI assisted coding.

- system level (drivers, fs, daemons, etc): Rust. C++ is you're extending existing legacy
- windows only desktop apps: C#
- iOS/MacOS/iPadOS: Swift
- multi-OS: Rust/Tauri, Qt/C++, Flutter, Electron (pick your poison)
- simple websites: Node/Typescript/Bun/Deno
- highly scalable backend: Elixir, parts in Rust, parts in Go
- CLI tools: Rust

Everything else, by default, Rust. Unless it's an edge case.

— Fabio Akita (@AkitaOnRails) October 9, 2026

There’s no better language, there’s the right tool

That list from the post isn’t a ranking, it’s a terrain map. Every language and every framework was designed for a use case, and none covers them all. That was true when you wrote every line by hand and it stays true with agents. The LLM doesn’t change the physics of the problem: it only changes who’s typing.

And the map goes beyond language. A real system has infrastructure around it: Kubernetes when scale asks for it, SQLite by default because most apps never need more than that, Postgres with pgBouncer when they do, Redis for cache and key-value, RabbitMQ or NATS for queues. Picking “the stack” on day zero and expecting it to last ten years is fantasy. It doesn’t, and that’s fine.

Personal example: new web app, I open Ruby on Rails without thinking twice. It’s the most complete end-to-end solution there is for the web, and with an LLM writing most of the code, the framework’s productivity weighs even more than the language’s. As the system grows, I swap the parts that ask for another tool: a high-volume queue leaves Rails’ side and goes to something from Erlang land, starting with RabbitMQ, which already solves the hard backpressure part for me.

No good software is finished: it grows. Usage dictates what you replace, rewrite and add. The final architecture is never the first commit’s, and whoever tries to get everything right in the first commit only produces dead documents.

The token superstition

DHH’s thread resurrected two beliefs that have been circulating since LLMs became work tools:

• “Script languages are better for AI because the lean syntax spends fewer tokens.”
• “Strongly typed languages are better for AI because the compiler catches the errors, so you need less testing and less repair.”

Both sound reasonable and both are superstition until someone measures. I was betting against both, from experience: a good compiler helps, but it doesn’t do magic, and syntax conciseness was never the bottleneck of an agent that reads and writes files all day.

What really separates a project that moves from a project that spins in circles is test discipline. I’ve written that TDD became a technical obligation, not a philosophy, and this post’s benchmark is one more data point in the same direction. It doesn’t matter whether you do test-first or test-after: what matters is that the suite grows with the code, that every user-reported bug becomes a regression test before the fix, and that you never make the same mistake twice. Software without users is dead software, so release early, release often, break fast and fix even faster. AI closes exactly that gap: the reproduce-test-fix cycle got cheap. QA became the programmer’s most valuable skill, far above “knowing Rust syntax.”

Discipline beats compiler and beats language. That was my hypothesis. Time to measure.

The benchmark

I built a reproducible experiment, public in the vibe-coding-benchmarks repository, with per-attempt results published in JSON and the raw traces kept out of the repo for credential containment. It’s not a language championship, and the design itself says so: it’s a snapshot of one real task, measuring what can be honestly measured.

The task

One task only, on purpose: maintain a multi-tenant ledger engine. The agent receives working v1 code and has to take it to a v2 contract, with per-tenant identities, authorization, serializable snapshots, a unique secondary index with alias swap, a capacity-bounded journal, torn-group recovery, checkpointing, legacy schema migration and durable retry IDs. It’s not FizzBuzz: it’s the kind of system that demands invariants across operations, exactly where LLMs tend to trip.

Correctness is judged by 43 test stories applied after collection, out of the agent’s sight: the 27 public ones from T1 plus 16 hidden ones. Passing is 43 out of 43. The conditions:

• T0: prose contract, input and output examples, a working launcher. No test suite.
• T1: the same, plus 27 public stories and a driver to run them.

Each attempt had a 15-minute budget, with at most two 3-minute continuations. In practice, none of the 112 needed a continuation.

The languages and the models

Seven languages in three strata, with pinned toolchains and Rust in safe mode:

• Typed natives: Rust (safe, pinned), Haskell (pinned GHC).
• Typed VMs: C# (with nullable analysis), Kotlin, TypeScript (strict, noUncheckedIndexedAccess, exactOptionalPropertyTypes).
• Dynamic scripts: Python and Ruby, no checker at all.

Four model configurations, all on subscriptions, no paid fallback: Kimi K3 (high), GLM 5.3 via OpenCode (high), Claude Opus 5.5 (medium) and Codex with gpt-6 Sol (medium). Crossing everything: 7 languages × 4 models × 2 conditions × 2 repetitions = 112 attempts.

What this design doesn’t measure

Before the numbers, the brakes, because they’re part of the result. It’s a single task, with two repetitions per cell: a small sample by construction. Token coverage is partial in some models (missing data is not zero). The harness changed mid-campaign across three instrumentation epochs. And provider delays contaminate wall time: one of the Opus attempts spent half the clock waiting for quota release. All numbers below are descriptive, and I point out where they can’t support a conclusion.

There’s one additional brake, on purpose: the iteration level. My design measures what can be reached in all languages with similar iteration levels, on a short budget, with no human tuning afterwards. It’s a photograph of the floor, not the ceiling. Any of the 112 implementations probably improves with more agent rounds, and the four queue stacks likewise: what I publish is what each language delivers “out of the factory” under the same effort, not the maximum a patient engineer would extract from it.

Mandatory disclaimer before quoting any number from this post around: every result that follows holds for this benchmark, with this task, these models, these prompts and this hardware. Your case may vary violently. If your task doesn’t look like a ledger engine, if your model isn’t one of these four, if your queue isn’t my queue, the numbers don’t transfer. The code, the tasks, the grades and the step-by-step to reproduce are all in the GitHub repository. And if you disagree with some result and that disagreement matters to your company or your project, the only honest attitude is the one I had: stop arguing and run your own benchmark.

Result 1: on correctness, the frontier models tied at the ceiling

Reading: Sol passed all 28 eligible attempts and Opus its 25 (the 3 failures were infrastructure, not code). 53 of 53 eligible attempts correct between the two. The task, made to be hard, was too easy to separate the frontier models, let alone the languages: with everyone passing everything, there’s no correctness ranking to publish.

The difference showed up in the weaker models, and in the direction my hypothesis predicted: provided tests helped. Kimi and GLM, with the public suite in hand (T1), delivered 24 correct artifacts against 16 without it (T0). Kimi, in particular, passed all 14 attempts with provided tests. It’s descriptive, not causal, but it’s the cleanest signal in the whole benchmark: what changed the outcome wasn’t the language, it was the presence of tests.

Per language, within each model, correctness scattered between 1/4 and 4/4 with no pattern that repeats across models. The three semantic failures I audited in depth are instructive: a GLM C# and a Kimi Python lost the same story by violating the same cross-operation invariant, and a Kimi Haskell missed a checkpoint path. None of the failures was a type error.

Where the model erred, it erred on domain invariants: the kind of error no compiler catches, in typed or dynamic languages. The myth that strong typing spares you the work of understanding the problem dies right there.

Result 2: concise language saved no tokens

Here’s the table that kills the first superstition. Medians per language on Sol, four valid attempts per cell:

Ruby, the most concise in lines (a bit over half of Rust’s), cost practically the same tokens as Rust on Sol, and on Opus it was the most expensive of all, about 205 thousand median tokens against C#’s 94 thousand. The cheapest were TypeScript and Python on Sol, C#, Haskell and Kotlin on Opus. If “fewer source code lines = fewer LLM tokens” were a law, Ruby should have led by a wide margin. It finished last.

And the second superstition? I audited the traces looking for the compiler-guided repair cycles that strong typing supposedly avoids. Result: zero compilation-error repair cycles in the audited Opus attempts, across Ruby, Rust and Haskell. Rust needed no compiler-driven fix at all; Haskell had one warning cleanup; Ruby spent its extra tokens on voluntary performance tuning. The frontier model almost always writes code that compiles on the first try, and the real effort goes to verification, not to appeasing the compiler.

One counterpoint that closes the coffin on the “cheap is good” thesis: GLM’s lowest token-consumption cells were the failures. TypeScript T0 at 50 thousand tokens looks efficient until you see it’s two unmodified starters because the model blew the output limit. Low tokens can mean early surrender.

Result 3: under load, everyone tied at the queue

Now for DHH’s question: Rust “beats” Elixir and Ruby when the subject is throughput. I built the separate track for that: four queue workers against a real RabbitMQ, written with agent help, each in its natural idiom:

• Elixir with OTP and Broadway
• Rust with Tokio and Lapin
• Rails with Bunny
• Plain Ruby with Bunny (control)

A generator published 10 thousand 4 KB messages to RabbitMQ, at 200 per second, and each worker consumed as fast as it could, hitting the effects on an HTTP/SQLite collector with a nominal 50 ms delay, limited to 2 CPUs and 1 GB of RAM. Two concurrency profiles:

On the 8-handler profile, the four stacks delivered the same throughput, 93 to 95 effects per second, and drained the whole queue. The apparent bottleneck was the shared collector (a Python lock, a single SQLite writer), not the languages. On the 64-handler profile, all of them left a backlog and blew the client timeout en masse, so I marked those cells as not qualified: the strong protocol, with a group-commit collector and a 50 to 800 msg/s ramp, is designed but not executed.

The only difference left standing in the numbers was memory: Rust held the peak at 16 MiB where the BEAM spent 135. It’s a real Rust advantage in density, with the caveat that a cgroup peak is not a language footprint. On throughput, which was the thread’s claim, a technical tie.

And the tie deserves an honest reading: it may only mean my pressure was too little. With the shared collector holding everything around 95 effects per second, any stack capable of more than that shows up as “equal.” What this test shows is where the bottleneck was, not where each stack breaks. To find each one’s ceiling I’d need the strong protocol, with a group-commit collector and a load ramp, which is designed but not executed. Until then, “all stacks perform the same” is the wrong conclusion; “all of them stayed above what my test could measure” is the right one.

And the question that interested me most, “does the agent work less to deliver a reliable queue consumer in Elixir than in Rust?”, was deliberately left unanswered: these four implementations were written to validate the environment, not as a controlled coding attempt. Comparing AI effort between Elixir and Rust would require the same ledger matrix running on the queue task. It’s on the list.

So, does the language matter?

With the data on the table, the thread’s three questions can be answered:

Is there a better language for programming with LLMs? None showed up. In the frontier models, correctness and effort varied less between languages than between models. Where there was a correctness difference, it followed the presence of provided tests, not syntax. “No general language ranking is established” is the report’s official sentence, and it’s the correct one.

Do the token-economy theses hold? Neither one. Conciseness didn’t save: Ruby cost as much as Rust, or more. Strong typing spared no repair: there was no type repair to spare. The models’ effort went to verification and understanding the contract, and that has no language.

Was DHH right? In what he measured, he measured what an agent delivers for free on each stack, without tuning, and called that a verdict on language. In my test, with real load and honest measurement, the four stacks tied on throughput in the stable regime, and only memory differentiated Rust. If his thread proves anything, it proves that agent conversion measures the model, not the language.

My bets from the introduction came out standing: testing and discipline are the lever, the compiler is a supporting actor, and the right tool depends on the problem. New web app is still Rails in my hand, heavy queue is still Erlang territory, low-level system is still Rust, and the LLM writes any of them with the same competence. Whoever spent the week fighting for their favorite language on X fought over nothing.

And my daily practice tells the same story. I have a personal bias toward Rust and even so I’ve published more than 30 open source projects in all kinds of stacks: ai-memory is Rust, FrankMD is Ruby on Rails, ai-memory-web and ai-jail-web are Node.js. In none of them did I feel any relevant difference that came from the stack: the wins were similar, the errors too, and the errors were always understandable. Every error became an invariant, a rule, a skill improvement. It never became a language switch.

So forget “token savings” and “typing superstition.” Do real software engineering: pick the right tool for your specific problem, apply proper engineering technique, and any reasonable agent will perform more or less the same. And stop wasting time arguing over language-war rage bait on the internet. Be a real software engineer and do real engineering. That’s it.

The per-attempt data, the full methodology and the evidence limits are in the repository. If you run the matrix with another task and find a ranking I didn’t, show me.

Someone else’s ranking is useful, but I wanted to get my hands dirty. So I built my own benchmark, with my own cases, half in Portuguese, and ran the open models on my home server with Strix Halo, the same Ryzen AI Max+ 395 I use to test local LLMs. The hosted ones (Jev, the two Clefs from Cloudflare, OpenAI Decisions and Microsoft Decision-1 via OpenRouter) ran through their APIs. Everything, including the cases, the raw responses and the scoring scripts, is public in the benchmark repository.

TL;DR: the top now has two, both open-weights. Perplexity Decider 1.1 27B and the just-released Decision 3.0 (vLLM Semantic Router) tied at 99.4% on the compact cases, with 100% in Portuguese, and identical labels on all 1,024 classifications in the benchmark. The price: about 49 GB of weights and over a second per decision. In the cloud, Microsoft Decision-1 tied Jev at 97.6% while costing a third: it’s the cheapest hosted one I measured, and the first candidate to evaluate. Jev answers faster and errs differently. OpenAI Decisions debuted tied with the best local 4B, Jebadiah, and has the lowest hosted latency. Julia-1, the smallest and fastest of all (12 ms), scored at coin-flip level: fresh out of the factory, it’s a no-go. Jump to the full table, the cost in support emails, which model for which situation, how hard it is to run the best open ones locally or the gotchas.

How I tested

The central idea was to measure more than “who gets the most right.” A decision model has a design goal, and comparing a 322-million-parameter encoder made to run on weak devices with a hosted service of undisclosed size on accuracy percentage alone is unfair to both. So I measured accuracy, latency, cost and behavior under stress, and at the end I split the recommendations by situation.

The two case sets

I built two test sets, frozen in git before any inference, with the dataset hash recorded alongside the responses. No model receives the expected answer, the justification or the case identifier. Only the text, the instructions and the options.

The v1 set has 13 operational use cases, in English, each with an explicit fictional policy:

• support ticket severity
• support routing
• refund policy
• incident escalation
• tool approval (whether an agent may or may not execute a given action)
• email fraud
• document type
• evidence support
• search relevance
• moderation
• order routing
• lead qualification
• numeric-threshold triage

There are 10 scenarios per use case, each presented four ways: original, options in reversed order, options with opaque names (c0, c1, c2) and original plus a “reviewer” comment suggesting the wrong answer, which the instructions say to ignore. That’s 520 classifications per model. Add 24 probability probes in situations with no information at all (fair coin, six-sided die), where the right answer is mathematical.

The v2 set was made to cover what v1 leaves out: common semantics and Portuguese. There are 7 tasks, with 12 scenarios each, and every scenario written in English and in Brazilian Portuguese with the same answer and the same option order:

• sentiment (author versus quotation, negation, sarcasm)
• current intent (withdrawn request, competing actions, hypothesis)
• logical implication (quantifiers, absent fact, temporal order)
• search relevance (all, part or none of what was asked)
• sensitive data handling (fictional secret versus mention, redaction, revocation)
• current incident status (timeline order, verified versus unverified)
• threshold policy (inclusive bounds, units, AND/OR with a missing value)

Each pair runs under three conditions:

• compact: just the record and the rule
• with examples: one solved example per class, fixed and the same for everyone
• with distractors: ten irrelevant archive notes before the record that matters, to stress the context

That’s 504 classifications per model.

The cases were written with AI help and reviewed against the policies, without independent human annotation. It’s a small, synthetic benchmark: 130 scenario families in v1 and 84 in v2. The variants are correlated, so 520 responses are 130 scenarios seen four ways. Treat the numbers as a snapshot of these cases, and test with your own before deciding.

The 23 competitors

Five ran in the cloud, through the official API: Jev (version 1.13.0, from TypeSafe), Clef (27B, Cloudflare Workers AI), Clef-flash (9B, same), OpenAI Decisions (the gpt-6-luna alias, in public beta) and Microsoft Decision-1 (9B, post-trained from Qwen3.5-9B, served via OpenRouter on Azure). The other eighteen ran on the Strix Halo, via ROCm, with weights pinned by Hugging Face commit hash, inside a Docker container. The first three (Laya, Laya typed and Clef-flash) shared one container; the rest ran one at a time, with Julia-1 measured twice, on GPU and on CPU. The server has 64 GiB of GPU-addressable memory.

The order up there is just organizational, hosted first and local ones by family. The real ranking is in the results table.

Everyone received exactly the same text, the same instructions and the same options, with no per-model prompt tuning and no adjustments after seeing results (Simple-Jev uses its own server’s fixed prompt policy). The exception is GLiNER, whose native API takes labels instead of probability distributions. It runs on a separate track, v2 only, with explicit mapping and no made-up calibration metrics.

Simple-Jev deserves an explanation. It’s a server that takes a stock LLM (here Qwen3.5 4B, with no decision training at all) and reads the probability of the answer tokens instead of generating text. It’s the baseline: if a “decision” model loses to this, the specialized training bought nothing.

Two protocol notes. OpenAI Decisions answers through a native typed-decisions endpoint, and the alias the API returned is not an immutable snapshot: OpenAI can swap the weights behind the name, like Cloudflare does with Clef-flash. And Perplexity Decider is not a stock chat LLM: it has a separate decision head, with non-causal attention and a calibration temperature saved in the weights. A generic chat server does not reproduce its behavior. Microsoft Decision-1 speaks native SystemOne through OpenRouter’s decisions API, and the served identifier is dated (microsoft-decision-1-20261009), so it’s not an immutable weight either. It doesn’t show up in OpenRouter’s text-model catalog: to find it, you have to filter by the decisions modality. Decision 3.0 is the new family from the vLLM Semantic Router project, Apache 2.0, derived from Qwen, with six checkpoints (d3, flash, mini, nano, lite, edge) and native SystemOne. I tested the flagship (d3) and the smallest (d3-edge); the four in between are for another round.

What was measured

• Accuracy per task, per language and per condition, always on top of the per-option probabilities the model returns, ignoring the vendor’s “confidence” field.
• Brier and ECE for calibration.
• Median and p95 latency as seen by the client, including the network: WAN for the hosted ones, LAN for the local ones. These are deployment observations, and comparing hardware would require another test.
• Input tokens reported by the API, priced at the October 10 price table, before any allowance.
• Bootstrap confidence intervals over scenario families, instead of individual responses.

I left local cost unmeasured. Energy, hardware amortization and maintenance were left out, so local models show up as “not measured,” because “zero” would be a lie.

The full table

The main column is accuracy on the 168 compact v2 cases (84 in English, 84 in Portuguese). The v1 column uses the 130 original scenarios. Latency is the median seen by the client across the 504 v2 calls. Cost is the list price per 10 thousand compact decisions, computed from the tokens the API reported.

To calibrate your reading: the tasks have three or four options, so a uniform guess gives about 31% on both sets. “Near 50%” here is already well above the coin, and “near 33%” is the coin. Julia-1 landed at 31.5%: the pretty name hides a guy flipping a coin.

The intervals matter. Perplexity’s edge over Jev on v2 is under 2 points and the bootstrap interval touches zero: it’s the measured lead on this set, not a proven superiority forever. And Decision 3.0 tied Perplexity at the limit of what this set can distinguish: identical labels on the 1,024 classifications, with both missing the same two cases (the English relevance one on v2 and the disputed severity case on v1, already in the errata). Microsoft tied Jev on the aggregate, with the interval centered on zero, but an aggregate tie is not interchangeability: they miss different cases, and Microsoft inverts the language pattern, missing more in English than in Portuguese. Jev and Clef also remain under 2 points apart, on the same tier. The 90% squad (Jebadiah, OpenAI) sits 7 points behind Jev, and from there down every difference is clear.

Each task cell has only 12 scenarios per language, so one more correct answer moves the cell by more than 8 points. A tie on a specific task is a lead for a pilot. The per-task tables are in the repository.

How much it costs, in support emails

Price per million tokens is a number nobody can feel. List prices on October 11 are US$ 0.042 per million input tokens on Jev, US$ 0.24 on Clef, US$ 0.038 on Clef-flash and US$ 0.10 on OpenAI Decisions. Output is free on all five. Microsoft Decision-1 charges the same US$ 0.042 as Jev per million, via OpenRouter, but reported a third of the input tokens for the same texts (about 26 thousand versus Jev’s 77 thousand on compact v2), and nobody can explain the tokenization difference. In measured practice, that makes it the cheapest hosted one of the bunch.

Clef-flash cost US$ 0.09 on the 9th: Cloudflare cut the price by more than half in the middle of my test. Jev reports considerably more tokens than Cloudflare for the same text, about 45% more on v2, and that’s already in the math below.

Imagine an operation with 10 thousand support emails per month to classify into three or four buckets, with short texts similar to my cases:

In reais, at Friday’s exchange rate (R$ 4.99), 10 thousand decisions on Jev cost one real. A million costs less than a hundred reais. For any company with a million emails to classify, that vanishes into the budget.

One caveat about proportions: my cases are short, about 450 tokens per compact case by Jev’s count. A real customer email, with quoted thread history, can be three to five times larger, and cost scales linearly with that. Even so, we’re talking cents per thousand.

What this math shows is that the price difference between the hosted models is irrelevant next to the cost of being wrong. Jev costs 9 US cents more than Clef-flash per 10 thousand decisions, and in those same 10 thousand it makes about a thousand fewer errors. If a misrouted email costs more than a hundredth of a US cent in human rework, Jev pays for itself. This math assumes every error costs the same, which is never true: sending a critical ticket to the routine queue costs far more than the reverse.

Cloudflare’s allowance changes the math for small players. It’s 10 thousand “neurons” per day, shared by the whole account, resetting at midnight UTC. With my input sizes, that’s about 1,500 full-Clef classifications per day or almost 10 thousand Clef-flash ones, free. If you have 1,000 emails a day and nothing else running on the account, Clef comes out at zero cost with top-tier quality.

And local? Energy is what the benchmark didn’t measure. From what I see on my Strix Halo, the whole machine stays under 100 W under load. With Jebadiah at 200 ms per decision, 10 thousand decisions are about 35 machine minutes, under 0.1 kWh, a few cents of a real on the power bill.

The real local cost is the machine, your hours configuring ROCm and containers, and keeping it all standing. Whoever already has the hardware for another reason has low marginal cost. Whoever would buy the machine just for this is buying a problem of thousands of reais to save about a hundred reais per million.

Which model for which situation

You want accuracy and want to operate nothing: Jev

In the cloud, Jev is still the name. It won or tied practically every cell among the hosted ones, and only Microsoft tied it on the aggregate. In English it was 100% on five of the seven v2 tasks, and in Portuguese it repeated 97.6% on the aggregate without losing anything in translation, something no other model above the coin managed. On v1 it kept its 96.2% on the original with the malicious “reviewer” comment injected; Clef even gained one case, and Kev 4B also stayed put, just at a much lower tier. The 255 ms WAN latency came in 30 ms behind OpenAI’s, and it costs 19 US cents per 10 thousand decisions.

The trade-off is the one I already discussed in the previous post: it’s proprietary, hosted, of undisclosed size, and you depend on TypeSafe keeping the behavior underneath you. And it makes mistakes. It treated an invoice ID called “PASSWORD-523” as a usable credential, even with the record explicitly saying it doesn’t authenticate anyone.

You want Jev’s level paying a third: Microsoft Decision-1

Microsoft entered this market quietly, and entered well. Decision-1 is a 9B post-trained from Qwen3.5-9B, served via OpenRouter, and it was the first model to tie Jev on the compact aggregate: 97.6%, with the bootstrap interval centered on zero. The detail that won me over: 100% in Portuguese, better than its own English (95%), inverting the pattern of the entire rest of the bunch. On v1, the reversed-order and malicious-comment variants didn’t take down a single answer, and it got the critical ticket that Clef and OpenAI missed.

And it’s the cheapest hosted one I measured: US$ 0.065 per 10 thousand decisions, a third of Jev, charging the same per-token rate but reporting a third of the tokens. The savings come from the token count, and I can’t explain the exact mechanism, so treat it as a measurement, not a promise.

The weak spots exist. Latency is the worst among the hosted ones (422 ms median, versus Jev’s 255). On the compact cases it’s clean of confident errors, but add context and Portuguese errors show up with 99.5% confidence: the “perfect Portuguese” pattern doesn’t survive long text. And on the probability probes it’s weak, giving 87% to a fair coin. Microsoft advertises calibrated probabilities; my probes disagree. And like every dated-ID hosted model, the weight behind it can change without notice.

For a Portuguese-language operation that wants Jev quality while paying less, it’s the first candidate to run in your pilot.

You want the same level with open weights: Clef

The full 27B Clef tied with Jev in English (82 of 84) and finished 3 cases behind in Portuguese. It beat Jev on sensitive data handling in English (12 of 12 versus 11). It’s an open-weights model, so you can run it locally with about 54 GB of memory just for the weights in BF16, which I didn’t test: the full Clef ran only through the API. Among the open ones, it lost the crown to Perplexity, which scored higher with smaller weights.

Through the API it’s the most expensive of the five hosted ones, three times Jev, with no quality gain to justify it. The exception is the daily allowance: for small volume, it’s the best classifier you can get for free.

You want the cheapest in the cloud: Clef-flash

At US$ 0.038 per million, Clef-flash is the lowest price in the hosted market and covers 9,400 decisions per day within the allowance. Quality drops: 92% in English and 83% in Portuguese on the compact cases, the biggest language drop among the hosted ones, with an interval entirely below zero. On sentiment in both languages, and on incident status in English, it tied with Jev. On intent and sensitive data in Portuguese, it lost badly.

For sentiment tasks in either language, or for English classification where 1 error in 12 is acceptable, it does well. For Portuguese, measure first.

You already live in the OpenAI ecosystem: OpenAI Decisions

OpenAI’s debut in this market landed mid-table: 90.5% combined, an exact tie with Jebadiah, with the lowest hosted latency I measured (222 ms) and the same accuracy in both languages. It costs US$ 0.23 per 10 thousand decisions, between Jev and Clef. If your operation already speaks the OpenAI API and 9 out of 10 accuracy is enough, integration is trivial: a native decisions endpoint, no prompt at all.

The weak spots are revealing. Its worst cell was precisely numeric-threshold policy (8 of 12 in English, 7 in Portuguese): it treated “at least 10” as satisfied by a range of 8 to 12 and failed the AND/OR short-circuit. The calibration is schizophrenic: on the predicate probes (yes/no) it was the most accurate of the bunch, with near-zero average error, and on the choice probes it returned 94%/6% on a fair coin. And on the v1 variants it flipped 2 correct answers in each presentation (reversed order, opaque label, malicious comment), something only Perplexity didn’t do: Jev kept its aggregate accuracy, but also flipped some correct answers along the way. Oh, and the gpt-6-luna alias is not an immutable snapshot: reproducibility requires saving the responses.

You want the best open model, period: Perplexity Decider 27B

Perplexity Decider 1.1 27B is the surprise of the benchmark. It led the table with 99.4% on the compact cases, was the only one that didn’t lose a single task cell to Jev (3 wins, 11 ties), and hit 100% in Portuguese under all three conditions: compact, with examples, with distractors. Calibration is also the best measured, with a Brier of 0.012, and coverage is absurd: it answered with confidence above 0.9 on 96% of cases and got zero of them wrong. On v1, it didn’t flip a single answer across the order, opaque-label and malicious-comment variants.

Its only 3 errors on v2 were the same case in different clothing (it confused “notification sent to managers” with “mandatory approver”), and it made them with confidence below 0.9: the known error comes flagged. The weak spot is event-probability probes: asked about an event with a 1-in-4 chance, it answered 0.73. Strong classification does not imply reliable event probability.

The cost is hardware: about 49 GB of weights in BF16, a second and a half per decision on my Strix Halo (with ROCm’s reference kernels, no flash-attention; on a good NVIDIA it should be much faster, but that I didn’t measure). It’s Apache 2.0, public weights with no gate. If you have the machine and want the top of the table at home, this is the one.

You want the other open-weights top: Decision 3.0 (and beware the Edge)

Decision 3.0, released two days ago by the vLLM Semantic Router project, is the second model to sit at the top: 99.4% on the compact cases, 100% in Portuguese, and a tie with Perplexity so absolute that the two gave exactly the same labels on the 1,024 classifications, including the same two misses (the English relevance one, at confidence below 0.9, and the disputed severity case that already lives in the errata). It also cleared the v1 traps without flipping a single answer, got both indisputable critical tickets right, and its calibration is in the same class as Perplexity’s (Brier 0.014, 94% coverage above 0.9 with zero errors).

In other words: the open top now has two independent recipes that converge. For your pilot, either one works; run both if you can, because the probabilities differ even when the labels don’t. Latency is in the same family as Perplexity (1.4 s on my machine, same ROCm diet), and the weights take the same ~49 GB.

The little brother is another story. Decision 3.0 Edge (0.6B) answers in 45 ms and fits in 1.3 GB of GPU, but scored only 61% on compact, plummets 14 points in Portuguese, and worse: of the 68 guesses it gave with confidence above 0.9, 14 were wrong. As a confidence-threshold router, out of the factory, it’s a trap from the same class as Julia. The four middle checkpoints (flash, mini, nano, lite) I haven’t tested yet; they’re the ones that will tell whether the family has a sweet spot between the two extremes.

You need to run locally, with quality: Jebadiah 4B v2

Of the local ones, Jebadiah 4B v2 had the best compact accuracy, 90% combined, answering in about 200 ms. It tied with Jev on six of the fourteen task cells, including intent, implication and status in English and intent in Portuguese. The weak spot is numeric-threshold policy, 9 of 12 in both languages, and it loses ground when the context fills with distractors (76% in Portuguese under that condition, versus 82% for Mapika).

Mapika Decider 4B (the fastest of the 4Bs) and Kev 4B are right behind, both around 87%, and the 3-point difference between the three fits inside the error interval. They’re three different training recipes on top of 4B backbones, with errors on different tasks. The choice between them depends on your task, and I’d run all three in a pilot.

You will send examples in the prompt: Kev 9B

Kev 9B has a curious profile. On the compact cases it ties with Kev 4B, half a point apart with the interval crossing zero, meaning more than double the parameters bought nothing. With one solved example per class, it opens almost 10 points over the 4B, and with distractors it holds 7 more points. Brier also improves by about 25%: the probability it returns is better calibrated.

If your use is just the rule and the text, the 4B is enough and is more than twice as fast. If you’re going to send examples or long context, the 9B is worth the memory. The warning is Portuguese: 83% compact in PT-BR, almost 10 points below its own English, the biggest language drop among the models above 80%. Only Laya typed (14 points) and Julia (13) drop more.

You want Clef-flash behavior on your machine: local Clef-flash

On the v1 set, local Clef-flash reproduced the hosted one’s 520 classifications, hit for hit and miss for miss. On v2 the story changed: same total (439 of 504), but 19 different labels, with swapped strengths (the local one gets more Portuguese intent right, the hosted one gets more relevance right). Among the local ones it’s still the best on the v1 operational-policy set (90.8%), ahead of every 4B.

The gotcha is latency: 618 ms median on the Strix Halo, three times Jebadiah, for the same v2 accuracy. If you pick it, it’s for the known behavior.

You need to run offline on weak hardware: Laya and Julia, with many caveats

Laya multilingual was the worst in the benchmark for accuracy (37% combined, near the coin), and the fastest by far among the ones that get anything right: 19 ms median, more than ten times faster than Jebadiah.

The three Laya checkpoints have between 322 and 421 million parameters, fit in under 1 GB of BF16 weights by parameter math, and the project documents CPU execution and an ONNX runtime (which I didn’t measure). PyTorch’s peak GPU allocation for the multilingual one stayed under 2 GB. A 4B needs about 8 GB just for weights at the same precision.

Julia-1 goes one step lower: 144 million parameters, 550 MB of weights, 12 ms median on GPU and 71 ms on a 4-thread CPU. It’s the fastest model I’ve ever measured here, and also the worst: 31.5% accuracy, glued to the 31% coin-flip line. In Portuguese it answered “cancellation” on all 12 intent scenarios, including the ones asking for a refund or access (the vendor’s Portuguese evaluation was done in pt-PT, and it shows). The bigger disaster is confidence: of the 71 guesses it gave above 0.9, 48 were wrong. A confidence threshold won’t save this model, because it’s exactly on the confident guesses that it errs.

That’s a design choice. Laya and Julia were made to be small encoders, fine-tunable on your task, to run at the edge. What I tested were the factory checkpoints, with no tuning, applying rules they had never seen, on broad tasks. In that scenario they did badly, period.

One detail confirms the fragility. On v1, when the reviewer’s malicious comment comes in, Laya drops from 44% to 19% and typed-decisions to 15%. It obeys the comment the instructions say to ignore, and confidently: 17 of the 28 guesses above 0.9 were wrong. Julia failed all 5 policy-counterfactual pairs: change the rule, it keeps the answer.

If you want what Jev delivers, factory Laya and Julia are very far from it, and no small model (GLiNER included) came close. If your problem is “I need to classify short text into three buckets, on a device with no internet and no GPU, and I have a thousand labeled examples to fine-tune on,” these two are the only ones on the list that fit. Fine-tuning is the part I didn’t test. It’s another question, and it needs another benchmark.

The baseline: Simple-Jev with a stock Qwen

Simple-Jev reading tokens from Qwen3.5 4B, with no decision training at all, scored 74% on compact v2. That’s the yardstick for “how much specialized training buys”: Jebadiah, Mapika and Kev, all on similarly sized backbones, came in 13 to 16 points above. The training buys a lot.

The curious part is that on v1 Simple-Jev tied with Jebadiah (111 of 130 originals) and got both unambiguous critical tickets right, which several better models missed. It’s also the slowest of the bunch, almost a second and a half per decision, because it’s running a whole LLM.

How hard it is to run the best open models on your machine

All these models speak the same HTTP protocol, called SystemOne, which Jev popularized: you send a state (the text), one or more closed-answer questions, and get back the chosen option and the probability of each. Swapping models is swapping the port. OpenAI Decisions breaks from the pattern with its own native endpoint, but the idea is the same. What changes is how much work it takes to get each one standing, and there the difference between them is big.

OpenAI: just an API key

The easiest of all: create the key, load credit and call the endpoint. The benchmark’s 1,048 requests ran in 4 minutes, without a single retry. The only hiccup was the first probe returning “balance exhausted” until I put US$ 20 in the account.

Microsoft Decision-1 is almost as simple: if you already have an OpenRouter account, it’s point at the model and call the decisions endpoint. No GPU, no Azure subscription, no installation. In my test, 10 of the 504 v2 calls took a 429 and passed on retry, so plan for retries.

Kev and Jebadiah: one pip and one command

Kev is the simplest of the 4B crowd. The project installs straight from GitHub and brings up a server with one command, downloading the LoRA adapter and the decision head on top of base Qwen3.5:

pip install "kev @ https://github.com/jaredpalmer/kev/archive/fc4a17e194bca35a48c57211392ec49ea54a94ac.tar.gz"
python -m kev.serve --run jaredpalmer/kev-4b --host 0.0.0.0 --port 8000 --device cuda

Jebadiah follows the same pattern, with the server package inside its own repository:

pip install "jebadiah-server @ https://github.com/getainode/jebadiah/archive/e9793abc66d1a7c5ad878d51eda656f1e9c5b9f5.tar.gz#subdirectory=server"
jebadiah-serve --model frontier-infra/jebadiah-4b-v2 --host 0.0.0.0 --port 8000

With either one up, a decision is a POST:

curl -s http://localhost:8000/v1/systemone -H 'content-type: application/json' -d '{
  "model": "kev-4b",
  "state": "Cliente diz que foi cobrado duas vezes e pede o estorno da segunda cobrança.",
  "questions": {
    "fila": {
      "type": "choice",
      "instructions": "Escolha a fila correta pela política de suporte.",
      "criteria": {
        "financeiro": "Cobrança, estorno, nota fiscal.",
        "tecnico": "Erro, bug, indisponibilidade.",
        "vendas": "Upgrade, plano novo, orçamento."
      }
    }
  }
}'

The response carries the option and the probability distribution, and that’s where you apply your threshold, log and decide whether to send it for human review. On an NVIDIA with CUDA and 16 GB of VRAM, a 4B in BF16 comes up without drama. The weights are about 8 GB.

Laya and Julia: smaller still

Laya is a Python package with an encoder inside. It installs with pip install laya and the project documents CPU execution, which is its whole point. The version pinned on my server was 0.4.1, and the multilingual one’s peak GPU memory stayed under 2 GB. To run on a GPU-less laptop or a small container, it’s the only one on the list that fits, no discussion.

Julia-1 is similar in spirit: its own Python runtime, one predict(state=..., questions=...) call and done, no standard Transformers pipeline. I ran the same container on GPU and on CPU, and both gave the same accuracy with the same responses in 97% of cases. On a 4-thread CPU it answers in 71 ms, which for an edge classifier is great. The problem is still that it scores at coin-flip level.

Where it gets complicated: AMD, ROCm and the rest

In my case the hardware is AMD, and then “one command” turns into a day of work. The points that cost me time, to save you yours:

• None of these projects test on ROCm. Kev had a gfx1151-specific segfault caused by CUDA graphs (issue 170), fixed only after the 1.0 tag, so the pinned version is a main-branch commit and the official tag crashes.
• The flash-linear-attention Triton kernels Qwen3.5 uses in the DeltaNet layers have no tested support on gfx1151. Everything ran on the pure-PyTorch path, slower than the latency published on CUDA. Jebadiah’s 200 ms in the table is with that handbrake pulled.
• Mapika Decider forces CUDA graphs when it sees the “cuda” device, which on ROCm is also called “cuda.” I had to turn it off by hand.
• You need to export HSA_OVERRIDE_GFX_VERSION=11.5.1 and PYTORCH_ROCM_ARCH=gfx1151 and use a base image with ROCm torch already validated. Any random pip install torch overwrites it with the CUDA build and breaks everything. In my Dockerfile every package is installed with --no-deps and the build fails if torch stops being the ROCm one.
• One model per container, one container at a time. The Strix Halo has memory shared between CPU and GPU, and three resident 4B models plus the Ollama that already runs there exhaust the 64 GiB the GPU can see.
• Perplexity Decider was the most annoying of all. The release expects a specific CUDA torch (I swapped in the ROCm 7 torch 2.9), the flash-linear-attention kernels don’t exist for it (I ran on PyTorch’s reference kernels, slower), and the default loader blew past the host’s 24 GiB RAM limit: I needed a custom loader that sends the 49 GiB of weights straight to the GPU. The model card still said “private” when I started, but the Hugging Face API confirmed it was public and gate-free, and the shards downloaded anonymously. The d3, at the same size, inherited the same treatment. The d3-edge, luckily, comes up with the standard Transformers loader and 1.3 GB of GPU.

None of this is impossible, and it’s all documented, with Dockerfiles, compose and the hashes of each checkpoint, in the repository’s setup guide. But it’s a real cost that the “local is free” spreadsheet never includes. If your machine is NVIDIA, most of this list disappears.

The full 27B Clef I didn’t even try locally: it’s about 54 GB of weights in BF16 before any activation. Perplexity, at 49 GB, proved a 27B runs on the Strix Halo, at the cost of a second and a half per decision with the reference kernels. The 9B Clef-flash runs, with about 18 GB of weights. The 4Bs are the sweet spot for consumer hardware.

Highlights and gotchas

Laya truncated all long Portuguese. Under the distractor condition, base Laya reported truncation on all 84 Portuguese calls, discarding almost 31 thousand state tokens, and on none of the 84 English ones. The English checkpoint’s default context limit is only 512 tokens (the multilingual and typed ones use 1,024, configurable up to 8,192), and Portuguese tokenizes heavier. The other models don’t even expose truncation telemetry, so “zero truncation reported” on them means “I don’t know.”

Portuguese costs more tokens. For the same 84 case pairs, Jev reported about 14% more input tokens in Portuguese than in English, and both Clefs about 15% more. If you budget a bilingual operation by the English average, you’ll come in low.

Does an example in the prompt save a weak model? It saves nothing. One solved example per class left Laya the same or worse; several of its aggregates dropped. It helped Kev 9B (plus 4 points) and Clef in Portuguese (plus 3.6 points, reaching Jev’s level), and almost nothing on Jev, one more case in Portuguese. Validate examples on real traffic before assuming they help.

Size correlates, and explains little. Among the 19 competitors of known size, the Spearman correlation between parameters and compact accuracy was 0.96. The Kev series is more informative: 63% at 0.8B, 87% at 4B, 88% at 9B. The big jump is from 0.8B to 4B, about 24 points, with an interval between 16 and 33. From 4B to 9B the compact difference crosses zero, and several nominal 4Bs, with similar backbones, gave very different results. In the d3 family, the flagship and the Edge sit almost 40 points apart under the same release, while a 9B, Microsoft, ties at the top of the hosted ones. Size helps up to 4B; past that, the training recipe rules.

The 27B Clef missed a critical ticket that two of the Layas got right. On v1, one severity scenario says continuous data loss in production is critical, regardless of how many customers it affects. The record describes confirmed permanent loss, from a single customer. Jev, Clef-flash, Perplexity, d3, Microsoft, Laya typed, Simple-Jev and Laya multilingual answered “critical.” Full Clef, OpenAI, base Laya, Eikos, Jebadiah, Mapika, Kev 4B and Kev 9B answered “degraded.”

It’s a single case, and an error rate would require many more. But it shows that the average hides exactly the error that costs the most. When the impact facts are structured, the severity rule belongs in code, and the model comes in only to extract the facts from free text.

Hosted and local diverge over time. Local Clef-flash reproduced the hosted one on v1’s 520 classifications and diverged on 19 of v2’s 504. Cloudflare keeps the weights version behind the hosted alias undisclosed. If reproducibility matters, pin the local hash.

High confidence and safety are different things. On compact v2, Jev, Clef, both Clef-flashes, Eikos, Jebadiah, OpenAI, Microsoft, Perplexity and d3 got zero guesses above 0.9 wrong. Perplexity is the extreme case: it answered at that level on 96% of cases without missing one, with d3 right behind (94%, zero errors). Microsoft covered 95% and OpenAI only half. Microsoft’s caveat: the clean record is only on compact, because with examples or distractors it misses in Portuguese with confidence above 99%. Mapika missed 5% of the guesses at that level, Kev 4B 3% and Simple-Jev 14%. And the two smallest of the bunch broke the pattern violently: Julia missed 48 of 71 guesses above 0.9, and d3-edge missed 14 of 68. Coverage varied too: Jev answered above 0.9 on almost 90% of cases, Clef on almost 80%, Clef-flash on 40% and base Laya on 2 of 168. On v1 Jev missed 1% of the guesses above 0.9 and base Laya missed 61%. A confidence threshold only works after being validated on your distribution, and Laya typed’s low ECE in Portuguese left it at the same 36% accuracy.

Perplexity didn’t fall for presentation traps. Across the three v1 variants (reversed options, opaque labels, malicious comment), it kept the same answers as the original, without flipping a single one. The d3 repeated the feat, zero flips too. Microsoft only stumbled on the opaque labels (2 flips, one in each direction); order and comment didn’t move it. OpenAI flipped 2 in each variant, and Laya collapsed to 19% under the comment. The price of that robustness is latency: a second and a half per decision on my machine, the worst median of the bunch.

Numeric thresholds are a code problem. The threshold-triage task, with clear numbers and an explicit rule, gave 100% on Jev and both Clef-flashes and under 30% on both Layas in English. An if solves that with 100% and zero latency. The model only comes in when the number is buried in free text, and even then the right thing is to extract the number and leave the comparison to code.

Lottery probability is another skill. On the 24 no-information probes (coin, die), Jev had the worst choice error among options. The numeric-estimate crown changed hands with OpenAI’s debut: its yes/no predicate missed almost nothing, while the choice probe returned 94% for heads on a fair coin. Perplexity, champion of everything else, answered 0.73 for an event with a 1-in-4 chance, and Microsoft went the same way, giving 87% to the coin. Laya typed had the second-best choice error, behind only Kev 9B and neck-and-neck with Clef, and still classified badly. It’s the “student who marks answer A” from Paulo’s essay, now measured: calibration on lotteries and accuracy on classification are independent measures.

Where the data converges

1. The overall top is open-weights, and now there are two. Perplexity Decider 27B and Decision 3.0 tied at 99.4% on the compact cases, with 100% in Portuguese and identical labels on the 1,024 classifications, both with Apache 2.0 weights running on my machine. Their edge over Jev is under 2 points and the interval touches zero, so treat it as a snapshot of this set. Their cost is hardware and latency, not API. In the cloud, the evaluation order changed: start with Microsoft Decision-1, which ties Jev on the aggregate, closes 100% in compact Portuguese and costs a third. Jev comes next, faster and with different errors, because an aggregate tie is not interchangeability. OpenAI Decisions landed halfway in price and accuracy.

2. Running locally became viable at 4B. Jebadiah, Mapika and Kev 4B deliver between 87 and 90% on the compact cases with 200 ms latency on a desktop machine, and in a pilot with your task any of the three can win. If your machine handles 49 GB of weights, Perplexity delivers the top of the table at home. The financial math justifies using the machine that already exists, for data control or because of an obligation to stay offline. Buying a machine for this it stops justifying.

3. Small models are another category. Judging Laya and Julia by the accuracy column is reading the table wrong. They get half or more wrong out of the factory, fall hard under a malicious comment, and together with the d3-edge they’re the only ones on the list that fit on a weak device and answer in tens of milliseconds. The Edge is the best of the three on accuracy (61%), but it plummets in Portuguese and gets one in five confident guesses wrong. If Jev-like accuracy is a requirement, cross all three off. If running offline on small hardware is a requirement, cross everything else off and start with fine-tuning them on your task, which is the missing measurement. And be careful with Julia and the Edge in particular: 48 errors in 71 and 14 in 68 confident guesses are the portrait of models that don’t know what they don’t know.

4. Portuguese costs about 14% more tokens and drops the models in the middle of the table by 1 to 10 points, with Kev 9B and Clef-flash in the worst case. Jev loses nothing, OpenAI ties both languages, and Perplexity, d3 and Microsoft close 100% in compact Portuguese, better than their own English, although Microsoft goes back to missing in Portuguese when the context grows. Julia goes the other way and plummets 13 points, and the Edge plummets 14. For a Brazilian audience, that’s the first column to look at, before price.

And what holds for everyone: none of these models replaces the rule that’s already in code. Numeric thresholds, severity policy with structured facts, tool permission, all of that stays in the if. The model comes in to turn free text into structured fact. And the test that matters is yours, with your tickets, because each cell in this table has 12 scenarios and your operation has thousands.

The cases, each model’s raw responses, the scoring scripts and the guide to reproduce everything are in the benchmark repository. If you run it with your tickets and get a different ranking, tell me.

At that moment I knew I was a hiccup away from losing everything. This post is the post-mortem of what happened, and the first part was written in the heat of the disaster, with the rescue copy still running.

Update of October 11: the copy finished and the ending was a happy one. No file was lost by accident, both NASes are healthy, each with its own RAID-6. The full recovery is in the second half of this post. The first part stays as it was published, untouched.

I’ve been using a NAS for a long time

I’m not new to this. I made an entire video mini-series on storage and filesystems (in Portuguese), and I recommend watching it before asking me “what’s a good beginner NAS?”. There is no such thing.

I’ve used external storage since the second half of the 2000s, when I was a Mac user and Drobo existed. The company is gone, but I had three of their DAS units before migrating to Synology, first on an entry-level model and then on the mid-tier DS1821+, which I’ve been using for more than five years. I’ve written about it here, setting up NFS on Linux and accessing it over iSCSI.

It’s been very valuable to me, and you don’t get to question why I need that much space. It doesn’t matter: everyone decides what they keep on theirs.

When I was producing YouTube videos, I edited 4K footage straight from the NAS, thanks to the NVMe cache and 10 GbE networking. I got used to big storage at 10 gigabits and I can’t go back.

The array was organized in SHR, Synology’s hybrid RAID, which in practice behaves like RAID-5 (more on that later): it survives the loss of one drive without losing any data. I always thought that was enough.

Hard drives die. Always

Make no mistake: hard drives go bad. Never, ever trust a lone drive forgotten in a closet. It will fail, and you will lose everything on it.

The best public data on this comes from Backblaze, which runs hundreds of thousands of drives and publishes failure rates every quarter. In the Q2 2026 report, with 354,415 drives monitored, the annualized rate was 1.73%, and the lifetime rate for the whole fleet stands at 1.41% per year.

That sounds small. Translated to human scale:

• One drive on its own, for 5 years: the chance of it dying in that period is close to 7%. That’s roughly the chance of rolling one specific number on a 14-sided die. Would you bet every family photo on that?
• One drive on its own, for 10 years: about 13%, one in eight.
• Eight drives, like my NAS, for one year: the chance of at least one failing is about 11%. In five years, more than 40%.

And those numbers are from a datacenter, with controlled temperature, clean power and drives spinning all the time. A drawer drive that sits idle for years and gets knocked around in a move doesn’t have statistics that pretty.

Memory cards are worse, and there isn’t even an equivalent statistic to consult. What exists is torture testing: the Great MicroSD Card Survey tested 216 cards over a year of continuous writing, and 52 of them, one in four, had already died.

Genuine cards started throwing errors, on average, after about 2,500 rewrite cycles; counterfeit cards, after 700. A memory card is transport media. Keeping the only copy of anything on a microSD is asking to lose it.

The practical conclusion is the same as in the mini-series: important data needs redundancy. A parity array exists precisely so that the death of one drive, which is a matter of time, doesn’t take anything with it.

What happened

With all that, I should have been at ease with my setup. I wasn’t.

It started the day before yesterday, Sunday. For a while I’d been watching the volume approach the ceiling.

At first it was 8 drives of 10 TB. Over the years I swapped them, one by one, for 20 TB drives (DSM shows binary units; on the label they’re 12 and 22 TB). It’s a slow process: each swap takes days, because the array has to rebuild and redistribute the data.

I decided to swap one more, the seventh. Synology supports hot-swap, so it’s simple: pull the old drive, slot in the new one, add it to the array and let DSM handle the rebuild.

On Sunday night it looked like this: rebuild starting, all eight drives healthy, 108 TB allocated out of 121.8 TB.

On Monday, at 8 a.m., the beep. Drive 1, the new one, with 13 hours of use, marked critical:

And drive 8, one of the old ones, with 22,095 hours powered on (two and a half years), also critical, with a read error at 08:05:

The pool’s message left no room: “Unrecoverable errors have occurred because drive errors occurred after storage pool degradation. Please back up your data immediately.”

Why two drives is the end

In RAID-5, if two drives go down, redundancy is zero, and in theory the whole array goes with them. It’s an unrecoverable situation.

Anyone who watched my playlist knows why. An array like this doesn’t store “files,” one per drive. A file is a collection of blocks, and the blocks are spread across all the drives, along with the parity. Losing a drive is losing a piece of every file. It doesn’t matter that the other seven are healthy, because they only have parts.

Parity is there to recompute the missing piece when one drive disappears. With two missing, the math doesn’t close.

And there’s a cruel detail in the rebuild: to recreate the new drive, the array has to read every other drive end to end. That’s more than 100 TB of continuous reading. If some old drive had a bad sector hiding in a corner nobody had read in a long time, that’s when it shows up. That’s what happened with drive 8.

Where do you copy 90 TB to?

The only way out, at that point, was to thank the heavens that the volumes were still accessible, with the files showing up, and use the window to copy everything out immediately. I couldn’t touch anything else.

The problem: copy about 90 TB to where? Nobody has an empty 100 TB NAS lying around at home.

Normally I’d buy this kind of hardware in the United States, on a tourist trip, and bring it back with me. It’s much cheaper. Buying expensive electronics in Brazil isn’t worth it: the country has charged import taxes worse than Trump’s tariffs for decades, and the final price usually goes past double the American list price.

Except I had to choose. What is it worth to lose 90 TB of data I spent years collecting and that is nearly impossible to recover from anywhere else? It’s one of those situations where the answer is “it’s priceless,” and so, whatever the price, I pay.

I found a storage reseller close to home, Controle Net. Good people, very quick to respond, and they sent me a proposal for equipment with same-day delivery.

Their first suggestion was a QNAP TS-832PX. I researched it before accepting. It has the same 8 bays and ships with 10 GbE, but the processor is a 1.7 GHz ARM Cortex-A57 (Annapurna Labs AL324) with 4 GB of RAM, half the 8 GB minimum that QuTS hero asks for. For RAID-6 on ZFS, with checksums and double parity on every block, that’s weak hardware, as I explain further down. And it would be a downgrade from the DS1821+ I already had, which runs on a Ryzen V1500B.

I asked for another option, and after some back and forth I settled on a QNAP TS-873A, 8 bays, with the same Ryzen V1500B as my Synology, 8 GB of RAM, and 10 Toshiba MG11 24 TB enterprise-class drives.

This is why you have to do your own research and know, objectively, what you want to do with the equipment. The reseller didn’t act in bad faith: they suggested an 8-bay NAS to someone who asked for an 8-bay NAS. The one who knew it would run ZFS in RAID-6 was me. If I had accepted the first proposal in a panic, I’d have paid a lot for a worse machine than the one I was replacing.

R$ 185,320.00. Eye-watering.

What it would cost in the United States

I checked today’s American list prices. About the exchange rate: the dollar is at R$ 4.97 right now, but that’s the effect of Sunday’s first-round election results, which knocked the rate down more than 4% on Monday. Through Friday it closed at R$ 5.22, and that’s the fair rate to compare with, because that’s the rate the reseller’s stock was built at:

At R$ 5.22, about R$ 60 thousand. I paid R$ 185 thousand, more than three times that.

The comparison isn’t perfect: the Brazilian proposal includes installation, training and six years of technical support, and the equipment was on my desk by the end of the same day. But it gives the measure of the Brazil effect.

If the rule of thumb is “double the American price,” normal would be about R$ 120 thousand. The extra R$ 65 thousand is, in my own rough estimate, the sum of services, local stock of enterprise drives and my urgency. Whoever buys on a Monday morning with the array dying is in no position to haggle.

And drives are expensive everywhere

To make it worse, this is the worst possible moment to buy hard drives. The AI bubble is sucking up production.

I thought Seagate had left the consumer market. I checked, and that’s not quite it: it hasn’t left, but in February 87% of its hard drive sales were already nearline drives, for datacenters, and WD gets 89% of its revenue from cloud customers and only 5% from retail. WD’s CEO said the company is “pretty much sold out for calendar year 2026.”

Seagate has said it won’t expand production capacity; growth comes only from higher-density drives. For someone buying drives at home, the practical effect is similar to them having left: you get what the hyperscalers didn’t want, at whatever price the seller asks.

QNAP instead of another Synology

After spending the morning and afternoon between negotiation, paperwork and payment, by the end of the day I had the QNAP powered on next to the Synology.

Choosing QNAP went beyond availability. Synology has been making questionable business decisions.

In 2025 it launched the Plus line requiring its own-brand drives or drives it certified.

The own-brand ones are Seagate, Toshiba and WD drives rebadged with custom firmware, and anyone using something else was left without pool creation, health monitoring, deduplication or firmware updates. After the outcry, it walked that back in DSM 7.3 and allowed HDDs and SATA SSDs from other brands.

Except M.2 pools still require the compatibility list. The uncertainty is left hanging: what stops them from doing it again?

Setting up the QNAP is very easy. QuTS hero, their ZFS-based system, is as simple to use as Synology’s DSM.

Fixing my mistake: RAID-6 and ZFS

This time I knew I had to configure RAID-6, which survives two drives failing at the same time. The downside is less total capacity, because the equivalent of two drives becomes parity instead of one. The upside is that I shouldn’t have to go through an emergency like this again.

One of QNAP’s strengths is supporting ZFS out of the box, well integrated and easy to use.

For those who don’t know it: ZFS was born at Sun Microsystems, for Solaris, in the mid-2000s, and today lives on as OpenZFS on Linux and FreeBSD. It merges into one system what used to be three layers, the RAID, the volume manager and the filesystem. It keeps a checksum of every block, so it notices when a drive silently returns corrupted data, and repairs it on its own from parity. And it never overwrites data in place: every write goes to a new block, which makes snapshots practically free.

Its reputation as a RAM hog comes from two things. The first is the ARC, its cache, which by design takes up almost all the machine’s free memory. That scares anyone looking at the graph, but it’s cache: it gives the memory back when another process asks. The second is deduplication, which keeps a huge table with the hash of every block and needs it in RAM or it becomes unusably slow. The old rule of “1 GB of RAM per TB of disk” comes from there, and only applies to people who turn deduplication on.

Even without it, ZFS is heavier than ext4 or a simple Btrfs. Computing checksums, compression and double parity for every block costs CPU, and with little memory the cache shrinks and everything gets slow. On weak hardware, like an entry-level NAS with an ARM processor and 2 GB of RAM, it suffers.

QNAP asks for at least 8 GB for QuTS hero. Mine came with 8 GB and a quad-core Ryzen, which works for my use, which is storing large files and serving them over the network. Deduplication stays off.

I bought 10 drives for an 8-bay NAS, so two are left in the drawer as spares, for the day one fails.

For my simple setup, three settings:

• 5% over-provisioning. It’s a slice of the pool the system reserves and never hands out for data. ZFS always writes to new space and gets slow when the pool fills up near the limit; this slack keeps performance stable. On SSD pools it also extends drive life, but these are hard drives, so the minimum is enough.
• 5% guaranteed snapshot space. It’s the reserve for snapshots, the “photos” of the state of your files that let you go back after an accident or ransomware. Guaranteeing the space means the snapshots keep existing even if I fill up the rest. My data changes little (it’s mostly cold archive), so snapshots take almost nothing and 5% is plenty.
• Alert at 85%. The system warns me when the pool passes 85% usage, with room for me to act before hitting the ceiling. It was exactly the lack of room that had me swapping drives in a hurry on the Synology.

Adding the two reserves, about 13 TB are set aside and 112.9 TB are left for use.

A detail of my case: I’m copying the 90 TB with snapshots turned off, and I’ll only enable the schedule after the copy finishes. A snapshot protects a file’s previous state, and on an empty NAS receiving its first load there is no previous state worth protecting. The source of truth is still the Synology.

And there’s a practical cost. I stopped and restarted the job several times, deleted folders that had gone in by mistake and copied again. With snapshots on, every deleted thing would keep taking up space inside some snapshot, and I’d be filling the pool with the migration’s own garbage. When everything is on this side and checked, then yes: scheduled snapshots, to protect me from deleting what I shouldn’t and from ransomware.

After that I created one giant “thick” partition. In ZFS you can create a “thin” folder, which only takes up what’s inside it and can promise more space than exists, or a “thick” one, which reserves the full size at creation. Since it’s a pool with one owner and one use, thick is more predictable: the space is mine and nothing competes for it.

Finally, I created the shares and installed the HBS 3 app.

Copying 90 TB

HBS 3 (Hybrid Backup Sync) is QNAP’s app for backup and synchronization. It connects to another NAS, to rsync, FTP, SMB or the cloud.

I connected to the Synology’s SMB share and told it to sync everything.

I had to stop and restart a few times, tweaking the job settings, because it was absurdly slow. And the file count wouldn’t stop growing:

865 TB on a 108 TB NAS, and that’s when I understood what was going on.

I excluded two kinds of path from the job:

• The #snapshot directories. On Synology, when you check “make snapshot visible,” each shared folder gets a #snapshot subfolder with a full view of the folder at each point in time. Inside the Synology that takes no space, because a snapshot only stores the difference. Seen from outside, over SMB, every snapshot looks like a full copy of everything, and HBS was trying to copy them all.
• The restic backup directories. restic stores everything in hundreds of thousands of pack files of a few megabytes each. Network copying is fast with large files and suffers with small ones, because each one costs an open, metadata and a close. And those backups I’ll redo from scratch on the new NAS.

With that, throughput rose to an average of 400 MB/s.

That’s less than half of what a 10 GbE network delivers, but it’s more than three times the gigabit most people have at home. Both sides really are at 10 gigabits:

Why doesn’t it reach 1,000 MB/s? My main hypothesis is drive 8. The source array is degraded, and every read that goes through the affected pieces has to be recomputed from parity, with a drive that on top of that returns errors and retries. It makes sense that the Synology can’t go faster than this.

There’s a second factor, which my own screenshots give away: the Synology has jumbo frames on (MTU 9000), and the QNAP image shows the default MTU of 1500. That screenshot is from earlier; I’ve already changed the QNAP to 9000 as well.

Why it matters: MTU is the maximum size of each packet that travels on the network. At the default 1500 bytes, filling a 10-gigabit link means processing more than 800 thousand packets per second, and each packet costs CPU on both sides, for the header, the checksum and the interrupt. With jumbo frames each packet carries six times more data, so it’s one sixth of the packets for the same volume, and more processor is left for moving files. It only works if everything in the path matches: both cards and the switch. If one side stays at 1500, the connection negotiates down to the smaller value and the other side’s jumbo frames do nothing.

At 400 MB/s, 90 TB takes about two and a half days, and I prefer two and a half guaranteed days.

What Synology explained