TL;DR – Strata and HyperQwen are amazing! I got Qwen3.8 Flash Next to be my daily driver.

In the first post I promised to explain why BeeLlama and which knobs I turned. While tuning BeeLlama, I went down a rabbit hole of optimisations. Features that work for one model don’t work for another, and similarly, what works on one piece of hardware doesn’t work on another. So the community started building their own forks of llama.cpp, vLLM and SGLang, sometimes combining features from all three.
This left me with a question:

Are general-purpose engines like llama.cpp and vLLM the future of local inference, or will model-specific, hardware-specific engines win?

Every big jump in speed* on my box came from something narrower than the general engine.

Stock llama.cpp: works, but caps out

Qwen3.8-27B Q4_K_M on two 3060s with MTP: about 30 tok/s at 32K Q4 context. It runs any GGUF on day one, which is its real strength. It just doesn’t go far. I wanted larger context, and for MoE models (specially the Qwen3.8 Flash Next) it doesn’t support MoE cache.

beellama: KVARN AND MOE CACHE

For 256K context I moved to beellama.cpp, a fork built around the KVarN cache format. Result: 38 tok/s.

GSQ-RCO: NEARLY LOSSLESS QUANTISATION (Claimed)

ISTA-DASLab’s GSQ-RCO IQ3_S is calibrated per tensor for this exact model. They crunched a 50GiB model to nearly 12GiB and also claim it’s lossless. It’s 4 GiB smaller than Q4_K_M, and those 4 GiB went straight into context: 256K at 635 tok/s prefill, 42.7 tok/s decode. (I didn’t face any drop in quality day-to-day)

ExLlamaV3 + TabbyAPI: the best 27B on two cards

ExLlamaV3 another fork of llamacpp takes it to another level. With Swift 1.5 in EXL3 it was the fastest dense 27B I ran on two GPUs: 758 tok/s prefill, ~50 tok/s decode at 262K, three times llama.cpp’s prefill. It also said to support paged attention like vllm but I haven’t tested it yet.

tensor_parallel: true
tensor_parallel_backend: nccl
cache_mode: Q4
gpu_split: [10.5, 11.5]
draft_mode: mtp

It supports fewer models and only NVIDIA cards. That focus is exactly why it wins.

Strata: a 125B MoE at chat speed

Strata is written for one model family, Qwen3.8-Flash-Next. It keeps experts in RAM and caches the hot ones in VRAM. Same model, same two cards:

• llama.cpp: 19.5 tok/s, 161 tok/s prefill, 128K
• Strata: 48.5 tok/s, 1,335 tok/s prefill, 256K

This is massive. This means I can use the bigger model as my daily driver. And just for a comparison, this model is better than DSV4.1 Flash and GPT5.6 Luna and even within reach of Kimi K3(literally has 1 trillion parameters).

vLLM on three cards: 20 to 55 tok/s with patches

I got tempted and bought a third RTX 3060. Tensor parallel across three cards is impossible for qwen (4 KV heads), so I used pipeline parallel. Stock vLLM did 20 tok/s. With ExTV‘s patches, one HyperQwen patch and a HyperQwen-prepared checkpoint, MTP works across pipeline stages: 55 tok/s, 38 tok/s after a 35K prompt, 1,023 tok/s prefill, 250K context.

VLLM_PP_LAYER_PARTITION=24,24,16 \
vllm serve Swift-1.5-Qwen3.8-27B-W4A16-HyperQwen \
  --pipeline-parallel-size 3 --max-model-len 250000 \
  --kv-cache-dtype fp8 --enable-prefix-caching \
  --speculative-config '{"method":"mtp","num_speculative_tokens":3}'

Still vLLM, but the speed came from patches written for this model’s exact shape on this GPU generation. A second fine-tune of the same architecture with a different checkpoint layout couldn’t fit MTP at all.

So, are general engines the future?

I don’t think llama.cpp and vLLM go away. New models land there first, and everyone forks from them. But the last 2–3× on consumer hardware came from code that knew the model’s shape, and the GPU generation. The future might be a general engine as the base, with thin model- and hardware-specific layers on top.

My daily drivers

• Strata with Flash-Next IQ3_S on two cards.
• vLLM + HyperQwen Swift 1.5 on three cards.

ExLlamaV3 with TabbyAPI is still the fastest for the 27B on two cards, but streaming tool calls and aborting requests weren’t smooth enough for agent work. I found TabbyAPI to be buggy.

Swift1.5 GSQ-RCO Quant: https://huggingface.co/ukisai/Swift-1.5-Qwen3.8-27B-GSQ-RCO-GGUF
Qwen3.8-Flash-Next GSQ-RCO Quant: https://huggingface.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-GGUF

*Prefill is how fast the model reads your prompt; decode is how fast it writes the answer, both in tokens per second. MTP (multi-token prediction) lets the model draft a few tokens ahead and check them in one pass.