Why ninfer-ext’s smaller EXL3 model can run more slowly

A local documentation assistant needs room for more than its model weights. The input and growing answer also consume memory, so a smaller model file can leave space for a configuration bundle that otherwise will not fit. With ninfer-ext’s two new Qwen3.8-27B downloads, that saving has a cost: in the project’s measurements, the smaller artifact generates answers more slowly. The choice depends on whether the saved space enables useful work, or simply makes ordinary answers take longer.

There is an earlier decision, too: whether these files can run on your machine at all. The EXL3 builds in ninfer-ext’s September 29 announcement target ninfer-ext on 64-bit Linux with an RTX 5090. They cannot be dropped into exllamav3, vLLM or llama.cpp. Understanding why takes us through three things that are easy to conflate: how the weights are compressed, which program reads them, and how an application talks to that program.

One familiar label, three compatibility questions

Start with the weights, the numbers a model uses to calculate its answer. Quantization stores approximations of those numbers using fewer bits, reducing the data that must be held in memory and read during execution. But an approximation is useful only if the engine knows how to interpret it. ninfer-ext, a fork of Neroued’s NInfer C++/CUDA engine, pairs its own compact, trellis-based EXL3 encoding with GPU kernels that decode those weights during computation. Its quantizer begins with full-precision tensors; it does not import an exllamav3 checkpoint.

That pairing is packaged as a .ninfer artifact. As the model card explains, ninfer-ext is the loader for these files; Transformers, vLLM, llama.cpp and exllamav3 cannot substitute for it. The shared EXL3 name prompted confusion in the Reddit discussion, but it does not establish file compatibility. Nor does an engine-specific format mean the software is closed: the repository uses Apache-2.0. The license tells you what you may do with the code. The loader determines whether that code can interpret this particular model file.

An application sits one layer above all of this. Once the server is running, ninfer-ext offers OpenAI- and Anthropic-compatible request interfaces. A documentation assistant might keep much of its client code and simply contact a different server. From the assistant’s point of view, little has changed; underneath it, the server still needs the matching artifact, decoding routines and GPU. API compatibility can make replacing a server convenient without making its model files interchangeable.

Check what the model needs to run

Read diagram description

First match the .ninfer model file to ninfer-ext. Check its current Linux and RTX 5090 requirements. Account for model weights, the conversation cache and temporary working memory. Then compare acceptable output quality, required context and concurrency, and latency under the same workload. A smaller file does not guarantee faster work.

Why fewer bits can mean more waiting

Consider a hypothetical documentation assistant on a Linux workstation with an RTX 5090. Its everyday job is answering short reference questions, but engineers occasionally supply a long configuration bundle and ask where two settings disagree. The team wants room for those larger inputs without making every ordinary lookup noticeably worse. This is a more useful question than asking which file is smallest, because it names the work that the saved space must enable.

The published 4.0-bits-per-weight artifact is 15.68 GiB; the 3.5-bpw alternative is 14.24 GiB. Those labels describe the main weight encoding, with some components using other rates. Both include text, vision and multi-token prediction support. Yet neither download size is a complete GPU-memory budget. Execution needs working memory, and the KV cache retains attention state from the input and generated text for reuse as the answer grows. The model file occupies only part of the space the request will need.

Reducing the weight footprint can leave more room for the cache and other working memory, but the GPU still has to interpret the encoded weights while calculating an answer. The project’s explanation attributes the slower 3.5-bpw results to a heavier decoding path for half-bit rates. In these measurements, the reduction in data movement does not compensate for that extra work. The published comparisons make the tradeoff visible:

“Decode” here means producing the answer token by token. MTP is multi-token prediction, used for speculative decoding in the second speed column. These are single-request spot measurements reported by the project, using one RTX 5090, CUDA 13.3, greedy generation, 64–256 output tokens and a 1,024-token prefill chunk. Prefill processes the prompt before answer generation begins. These details matter because the table describes that experiment, not the documentation team’s mix of short lookups, large bundles and simultaneous requests.

For the short lookups, the larger artifact might fit comfortably and finish sooner. For a bundle that cannot otherwise fit, the smaller artifact might be the only useful option of the two. The same speed penalty means something different in each case. The comparison therefore has to follow a request all the way through: did it fit, did it produce an acceptable answer, and how long did the engineer wait?

Comparing short lookups with long configuration bundles

Before comparing answers, establish the supported runtime. The repository calls for an RTX 5090 with sm_120a support and a suitable CUDA toolkit, listing CUDA 13.3 as validated. Its README supplies the Linux dependencies and build instructions. Community reports about another GPU or a port do not expand that stated support. Once ninfer-ext is built and the matching 4.0-bpw file is downloaded, the model card’s starting example is:

build/apps/ninfer-serve qwen3_8_27b_exl3_4bpw.ninfer \
  --port 8099 --spec mtp --draft-tokens 3 --fixed-draft

The command is a documented launch example, not a runtime test performed for this article. Save the engine revision, artifact and settings when using it. A successful load answers the compatibility question; it does not yet answer whether the smaller artifact would help. That requires returning to the short questions and long configuration bundles that made the choice interesting in the first place.

For the hypothetical assistant, the hypothesis is specific: the smaller artifact should admit the required long inputs without imposing an unacceptable delay on ordinary answers. Compare both candidates under the same questions, input lengths, output limits and arrival pattern. Record cache settings and speculative-decoding mode as part of each result, so that a change in those settings cannot masquerade as a benefit of the format. This is the bottleneck-led experiment described in our Linux performance guide, applied to the memory-versus-speed choice.

Read time to first token and total completion latency alongside completed requests and resource-limit rejections. Otherwise, refusing the hardest requests can make a service appear faster. Answer quality belongs in the same comparison: an engineer who receives a quick answer that misses the configuration exception still has the original problem. Use the same known references to judge both artifacts, and keep the request population and measurement window with the result, as explained in our metrics guide.

If both artifacts already handle the required inputs, saving download space may buy this team little, and the faster candidate may be the useful one. If the smaller footprint turns an otherwise rejected bundle into a good answer, slower generation can be a reasonable price. Keeping the short-lookup and long-bundle results with the chosen configuration makes it possible to revisit that tradeoff when the assistant’s workload changes.