Overview

Strata is an open-source inference engine that runs one model architecture: Qwen 3.8 Flash and its code-focused variant and fine-tunes. The project is MIT licensed, about two weeks old, has roughly 9,000 GitHub stars, and is written largely by one developer. The author tested it on a single 12 GB GPU with 64 GB of system RAM. The model has about 125 billion parameters.

The author reports two measured results on this hardware:

• Generation (decode): about 40 tokens per second.
• Prompt processing (prefill): up to 900 tokens per second, compared with about 120 tokens per second for llama.cpp running the same model file on the same card.

The author also argues that prefill, not decode, determines agent responsiveness, and that a single-architecture engine can outperform a general engine on both.

Why Prefill Matters for Agents

Decode speed is the rate at which a model writes its answer. Most speed charts report this figure. Agents, however, spend most of their time on prefill, which processes the new input before the model writes.

The author tested this with a grocery order placed through an agent:

Each step prefills only the new content, such as the updated page state, tool results, and the latest message. The prefix cache makes this possible. Without it, the model would reprocess the full history on all 26 calls.

The author's conclusion: when selecting a model or engine for agents, measure prefill speed rather than decode speed.

Architecture and Constraints

The model has the following structural properties, which Strata uses:

• Experts: 512 experts per layer. Each token routes to 10 of them.
• Sparse attention: An indexer selects about 2,000 positions for each query.
• Storage: Most of the model consists of experts, which do not fit in 12 GB of VRAM. They reside in system RAM and cross the PCIe link to the GPU as needed.

Strata treats the GPU as a small working area (VRAM), system RAM as the main store, and PCIe as the transfer path between them.

Prefill Optimizations

1. Larger Prompt Chunks

llama.cpp processes prompts in ubatches of 512 tokens by default. With experts offloaded to RAM, it either computes them on the CPU, which is slow, or copies CPU-side expert tensors to the GPU for every ubatch. Each long prompt therefore requires many transfers.

Strata borrows nearly the entire expert cache during prefill and processes prompts in chunks of 6,144 tokens. Each non-resident expert crosses PCIe once per 6,000 tokens rather than once per 500.

Strata's own measurements:

• 4,000-token chunks: about 790 tokens per second.
• 8,000-token chunks: about 970 tokens per second.

After prefill, the cache refills.

2. Overlapped Transfers

Before each chunk begins, Strata builds the complete list of non-resident experts for every layer, in order. A dedicated thread copies them directly from pinned RAM into a ring of 384 GPU slots. While layer n computes, layer n + 1 is already loading. The GPU does not wait for copies.

Strata also skips routing decisions during prefill. Above about 1,000 tokens, nearly every expert is routed anyway, so Strata streams all experts.

3. Expert Computation

The expert matrix multiply uses llama.cpp's own MMQ kernel, taken directly from upstream. Strata changes how the work is delivered to it:

• It fuses the gate and up projections into a single product.
• It launches experts in groups of 16, sizing each launch to the group.

Fewer small launches leave more GPU time for computation.

4. Attention

Stock llama.cpp builds a full mask of size context × tokens and runs dense flash attention over it. Strata does not build this mask. Its own kernel selects the required blocks and reads only those rows of the KV cache.

Kernel Ownership

The MMQ expert kernel is the only major kernel Strata borrows from llama.cpp. Attention, the indexer, the GDN scan, and the hyper-connections are Strata's own kernels. According to the author, several of these outperform stock llama.cpp.

The author's summary: the expert computation is identical in both engines. Strata differs in how often weights cross the PCIe link, how rarely the GPU stays idle, and in its kernels for all other operations.

Decode Optimizations

Decode processes one token at a time, and each token activates 10 experts per layer. No large batch exists to hide transfer latency.

Expert Cache

• Dense weights occupy about 3.5 GB of VRAM.
• The remaining VRAM holds a cache of about 2,860 individual expert slots.
• This cache covers under 12% of all 24,000 experts.
• On this card, the cache serves about two-thirds of all expert lookups.
• Strata starts from a routing profile and periodically replaces cached experts with those the current conversation uses.

Three-Way Layer Split

llama.cpp assigns whole expert tensors to either the CPU or the GPU at load time. Each transition between backends requires a copy and a synchronization, during which the GPU is idle.

Strata splits each layer three ways, and all three run concurrently in the same layer:

1. The GPU computes cache hits immediately.
2. About a quarter of the cache misses transfer over PCIe to the GPU.
3. The CPU computes the remaining misses directly from pinned RAM.

Handoffs use flags in pinned memory within a single CUDA graph, with no driver synchronization between steps.

GPU Utilization

The author notes that part of Strata's figure reflects spin-wait kernels polling the handoff flags. This time is not all productive computation.

Speculative Decoding

The model includes a multi-token prediction (MTP) head. Strata keeps it entirely in VRAM and drafts up to three tokens per step. The full model then verifies the drafts in one pass. Verification is exact, so output matches decoding without speculation. On this card, each pass yields slightly more than two tokens on average. Mainline llama.cpp has no MTP support for this architecture.

Decode Results

• Tuned llama.cpp build with the same model file: about 26 tokens per second.
• Strata: about 40 tokens per second.

Setup

Model Options

The choice depends on system RAM:

The author used the GSQRCO quantization.

The model's N-gram table occupies about 28 GB of the file. It is a plain lookup table that Strata streams from SSD as needed, so it does not need to fit in RAM.

Installation

• Linux: Run setup.sh.
• Windows: Run the "start here" batch file.

The installer checks the GPU and RAM, recommends a model, and downloads it. If a model file already exists, the user can point the installer to its folder. No prebuilt Linux engine existed for this version at the time of testing. The installer therefore installed the CUDA toolkit and compiled the engine. It also downloads a separate MTP head of about 5 GB.

First start: The engine loads tens of gigabytes into RAM and locks part of it for the GPU. The machine can freeze for one to two minutes. The process should not be interrupted.

Context length: The author set the context to 128k. This adds no measurable cost, because the expert cache resizes to the VRAM remaining after the KV cache.

Running Strata and llama.cpp Behind One Endpoint

Strata provides an OpenAI-compatible server for this one model. The author's other models run in llama.cpp's router as presets. The author placed llama-swap in front of both.

llama-swap is a proxy on a single port. It starts the backend that matches the requested model name. The author notes that this was the first time using llama-swap. The llama.cpp router already met the author's needs before, but llama-swap now serves as the single entry point for every AI endpoint.

Configuration

The configuration includes two types of entries:

• llama.cpp router: Each preset within it is listed by name, so clients see each model separately.
• Strata: Two entries, one using both GPUs and one using a single GPU.

All entries belong to one group set to swap and exclusive. Only one entry holds the GPUs at any time. The image model also belongs to this group. Clients see no change.

Swap Behavior

Requesting the 35B model starts the router. Requesting Qwen 3.8 Flash stops the router and starts Strata. After 10 minutes without Strata requests, it shuts down and releases its pinned RAM.

Known Limitation

llama-swap does not pass through progress updates. The author's harness displays progress while a model loads and while it processes a prompt. llama-swap can insert a loading message, but only as text inside the response. During a swap, the harness therefore shows no progress.

Limitations

Single request at a time. Strata runs one model and processes one request at a time. It does not batch. Serving several users or parallel agents is outside its current scope.

Pinned RAM. The IQ3 XXS quantization pins about 40 GB of RAM. Pinned memory is locked, so no other process can use it while Strata runs. This is why the configuration shuts Strata down after 10 idle minutes.

Platform constraints:

• Windows: multi-GPU pinning is capped at 8 GB.
• WSL: pinning is limited to about 1 GB.
• Linux is the recommended platform.

Output reproducibility. The same expert produces slightly different numerical results depending on whether the GPU or the CPU computes it, because the two round differently. The cache moves experts between the two, so even greedy decoding is not byte-for-byte repeatable by default. Several switches can force repeatability, at a cost in speed.

Assessment of the divergence. Strata's paper compares its output distribution with llama.cpp. The average difference is smaller than the difference llama.cpp itself shows between CPU and GPU execution. When a token changes, the change occurs at a near tie. The author characterizes this as noise.

Experimental speed projection setting. This setting is off by default. Despite its name, it does not increase speed. It functions as a refusal-removal control vector. When enabled:

• The top-ranked token changes at about one in 10 positions.
• Code perplexity increases by 15%.

The author recommends keeping it disabled.

Project maturity. The project is two weeks old. One developer wrote most of it, and most published benchmarks come from one PC. The documentation states this and labels its estimates. Breaking changes between versions should be expected.

Porting the Techniques to llama.cpp

The author tested whether llama.cpp could reach the same prefill speed. Over two days, the author ported Strata's prefill techniques into a personal llama.cpp fork:

• Expert cache used for larger chunks.
• Experts streamed ahead on a second CUDA stream.
• Sparse attention computed over only the selected cells.
• A multi-token kernel for the GDN layers.
• The MTP prompt pass writes only its KV cache.

On a 17,000-token prompt on one card, the fork's prefill rose from about 120 tokens per second to about 620. Strata reaches about 900 tokens per second on the same test.

A few members of the author's Discord community are testing the same approach.

Analysis: Two Types of Engines

The author's view is that a single-model kernel is much easier to optimize than a general tool that supports many architectures. Most of Strata's code is AI-generated under human direction. About two-thirds of its commits are co-authored by an AI coding agent. The author expects that writing kernels for a new LLM will become much faster.

Based on this, the author proposes that two engine types are needed:

1. Single-architecture engines that run one model as fast as possible.
2. General engines that run most models and accept some speed penalty.

Over time, optimizations from single-architecture engines move into general engines, which makes the single-model engine unnecessary for that model. When a new model arrives, a new single-architecture engine can be built first and its techniques moved into the general engine afterward.

Verdict

The author used the 35B MoE model as a daily driver because of its speed, not its capability. The author describes Qwen 3.8 Flash as in a different category and close to Claude Opus 4.8, a comparison based on the author's own assessment.

With Strata, the author reports that the operations an agent waits on are now fast enough. The grocery order completed in under two minutes on one card. Generation is still slower than the 35B model, but the author's measurements show that agents spend less time generating than prefilling.

The recommendation depends on available RAM:

• 64 GB: The full model fits. The author would switch to it.
• 32 GB: The coder-pruned variant is available. It is worth testing if the agent mainly writes code.

With llama-swap in front, the user can keep the 35B model and all other models on llama.cpp and request Qwen 3.8 Flash by name only for tasks that need it.