RTX 3090 pushes Qwen3.8-27B from 82 to 382 tokens per second with a custom AI inference engine
A single NVIDIA GeForce RTX 3090 is still proving that older high-end GPUs can be surprisingly capable for local AI workloads when the software is properly optimized. Despite being limited by 24GB of VRAM, the card’s massive 936.2GB/s memory bandwidth gives it enough muscle to run large language models at impressive speeds under the right conditions.
One developer recently reported running Qwen3.8-27B on an RTX 3090 and increasing performance from 82 tokens per second to as much as 382 tokens per second. That is close to a 5x improvement without changing the GPU or switching to a different model. Instead, the gain came from a custom inference engine and several clever decoding optimizations.
Why AI text generation can be slow on GPUs
Large language models generate text one token at a time. Each new token depends on the previous one, which means the GPU must repeatedly access the model’s weights from memory. In many cases, the bottleneck is not raw compute power but memory movement.
The RTX 3090 has plenty of bandwidth, but running a large model such as Qwen3.8-27B still puts pressure on its 24GB VRAM. As a result, the GPU may spend a significant amount of time waiting on data rather than fully using its processing power.
To solve this, the developer focused on reducing how often the main model has to perform expensive work during generation.
Speculative decoding played a major role
The biggest improvement came from speculative decoding. This technique uses a faster “drafter” to predict several upcoming tokens. The main model then checks those guesses in a single pass, accepts the correct ones, and rejects the rest.
In simple terms, instead of asking the main model to confirm one word at a time, the system asks it to verify a larger chunk of text at once. Since the main model still makes the final decision, the output should remain the same as normal generation.
According to the developer, accuracy on a math benchmark remained steady at 96.5 percent, suggesting that the speedup did not come at the cost of quality in that test.
Performance improved in several stages
The reported speed increase did not happen all at once. The developer first tuned Qwen3.8-27B’s built-in multi-token prediction feature, also known as MTP. This allows the model to predict multiple upcoming tokens by itself.
That change raised single-user performance from 82 tokens per second to around 114 tokens per second.
Next, the developer added another drafting method called DFlash2, which proposes seven tokens at a time. A second helper method, described as a lookup drafter, was also introduced. This lookup system scans text already present in the conversation or document and predicts that repeated phrases may continue in the same way.
That lookup method pushed performance to around 138 tokens per second.
The largest boost appeared during repeated-text tasks
The most dramatic gain came when the model was working with text that repeated or quoted existing content. The developer found that the verification step did not have to be limited to the same seven-token size used by DFlash2. The lookup drafter could fill additional slots at almost no extra cost.
As a result, the custom engine began checking 16 tokens per step instead of only seven.
When Qwen3.8-27B quoted from a document, around 15 of those 16 guesses were reportedly accepted. This pushed performance on a 25,000-token document task from 260 tokens per second to 382 tokens per second.
For ordinary chat, the same approach was less dramatic but still useful, producing about 133 tokens per second. That remains a strong improvement over the original 82 tokens per second baseline.
Context window also increased significantly
The developer also used a more compact format for storing the model’s working memory, allowing the context window to expand to 138,696 tokens. That is a major advantage for long-document analysis, research workflows, and question-answering over large files.
However, this improvement comes with trade-offs. The compact memory format reportedly doubles the time needed to load a document at the beginning. That means it is better suited for situations where a user loads a document once and then asks many questions about it, rather than repeatedly loading new files.
There are some limitations
The 382 tokens-per-second result is not the expected speed for every task. It appears to be most effective when the model is repeating, quoting, or continuing text that already exists in the context. In more typical chat scenarios, performance is closer to 133 tokens per second.
The longer verification step also uses half of the available request slots and consumes around 8,000 tokens of context, so it is optional rather than something that should always be enabled.
Outside repeated-text workloads, this mode can run at about half the speed of plain multi-token prediction, meaning users would need to choose the best mode depending on the task.
A strong example of software optimization for local AI
The reported jump from 82 to 382 tokens per second on a single RTX 3090 highlights how much performance can still be unlocked through inference optimization. Rather than relying on a newer GPU, the developer improved the decoding pipeline, used smarter token prediction, and reduced wasted work during generation.
For local AI enthusiasts, this is an encouraging example. The RTX 3090 may no longer be NVIDIA’s newest flagship, but its 24GB VRAM and high memory bandwidth continue to make it a popular choice for running large language models at home.
That said, these results have not been independently verified, so they should be treated with some caution. Even so, the reported performance gains show how custom inference engines, speculative decoding, and memory optimization could help make large AI models faster and more practical on consumer graphics cards.
