You can also read the article here on Substack and view the code on GitHub.
This documentation is fairly thorough, simply because I wanted to take the time to practice writing about the internals of transformer systems. The README for this project is much shorter and objective.
The input text fed into an LLM first goes through a process known as tokenization.
For example, "stemming" is a known facet of tokenization, in which a word
like running gets reduced to run.
After the input text is split into tokens, each token is mapped to a specific integer ID, then looked up in an embedding table. This embedding can be thought of as the model's initial, standalone interpretation of the word, similar to looking up its definition in a dictionary, but in the model's internal vector space.
This embedding is a high-dimensional vector, in this case 3,584 dimensions.
The encoded input is then passed into the transformer architecture itself. This project uses Qwen2.5-7B: a causal, decoder-only transformer.
Each of the 28 layers performs exactly two operations, in sequence:
The goal of attention is to examine every token, determine which other tokens are relevant to it, and pull information from those tokens proportionally to how relevant they are.
This is done by taking each token's vector (3,584 numbers) and projecting it into three separate vectors, via three separate learned weight matrices:
The concrete sequence of one token figuring out what to attend to:
This whole mechanism is captured in one line:
This doesn't happen just once — it runs in parallel across several heads. Qwen breaks from the usual 1-to-1 head matching: there are 28 query heads, but only 4 key/value heads. Every token still produces 28 separate Query projections, but those query heads are split into 4 groups of 7, and all 7 query heads within a group share the same Key and Value vectors. This is Grouped-Query Attention (GQA) — understanding this mechanism carefully matters, since it's central to this whole project.
Feedforward takes one token's vector and transforms it in place, without involving any other tokens. It's a two-layer network, though in Qwen's SwiGLU version there are actually three weight matrices, not two:
The result is then multiplied by a third weight matrix, projecting it back down to 3,584 dimensions. Empirically, this gating mechanism lets the network learn more selective, expressive transformations for the same parameter budget.
As a token's vector passes through the 28 stacked layers, the raw numbers tend to drift, growing very large or shrinking. To counter this, Qwen uses RMSNorm as a "reset step" before real computation in each layer, rescaling the numbers back into a consistent, well-behaved range.
Finally: after the 28th layer and one last RMSNorm, a linear projection maps the 3,584-dimensional vector back up to vocabulary size, producing a probability distribution over every possible next token.
Consider this question:
Glucose is transported into the muscle cell: A. via protein transporters called GLUT4. B. only in the presence of insulin. C. via hexokinase. D. via monocarbylic acid transporters. Answer: A
Chances are this triggers a minor flashback to every multiple-choice test
you've ever taken. Notice the shape: the question, four labeled options,
then Answer: followed by the correct letter.
Many eval datasets are built the same way. In particular, MMLU (Massive Multitask Language Understanding) is a multiple-choice benchmark comprised of 57 subjects.
MMLU, and most eval benchmarks, test the model on hundreds or thousands of separate questions. To keep the test comparable, every single question gets the same five examples in front of it — that consistency is what makes it a controlled benchmark rather than a moving target. For example:
[Question 1 + A/B/C/D + "Answer: A"] [Question 2 + A/B/C/D + "Answer: C"] [Question 3 + A/B/C/D + "Answer: B"] [Question 4 + A/B/C/D + "Answer: D"] [Question 5 + A/B/C/D + "Answer: A"] Photosynthesis primarily occurs in which part of a plant cell? A. Mitochondria B. Nucleus C. Chloroplast D. Ribosome Answer:
While this repetition is necessary for control and reproducibility, it's computationally wasteful in large benchmarks. The model repeatedly sees the same block of text, computes attention over it token by token, and only then moves on to the actual new information.
This project tests the usefulness of prefix caching. The logic: if two different requests happen to start with the exact same sequence of tokens (e.g. tokens 1 to 800, the fixed 5-shot block), then the K and V vectors for those first 800 tokens will come out identical both times, same input tokens, same model weights.
Prefix caching keeps the KV cache from the first request around even after that request finishes. When the next request comes in, vLLM checks: does this new prompt start with a token sequence I've already computed? If so, it reuses the cached prefix instead of recomputing it.
With the motivation for efficiency established, here's the actual experiment. This multiple-choice format is a familiar one, our brains don't stumble over it since it's the same testing format most of us grew up with. With that context, the specific configuration being tested makes more sense.
| Variable | Values tested |
|---|---|
--enable-prefix-caching |
on / off |
--max-num-seqs |
32, 128, 256 |
| Prompt order | sorted by length / shuffled |
| Client concurrency | matched to each run's max_num_seqs |
2 × 3 × 2 = 12 total runs.
max_num_seqs was tuned alongside caching because, in an initial
experiment that didn't use concurrent request handling, caching showed no
measurable benefit, not because caching doesn't work, but because with only
one request in flight at a time, there was barely any "waiting" for the GPU
to fill with useful cached work. Testing multiple max_num_seqs
values was necessary to find out where caching's benefit actually shows up
across different levels of concurrent load.
Throughput (req/sec), averaged across sorted/shuffled prompt order, one run per configuration:
| max_num_seqs | caching off | caching on |
|---|---|---|
| 32 | ~282 | ~245 |
| 128 | ~85 | ~353 |
| 256 | ~76 | ~80 |
Accuracy was 0.736–0.742 in every configuration, essentially unchanged throughout, as expected, since these flags affect serving performance, not model correctness.
max_num_seqs=128, consistent with a measured 97.9% prefix
cache hit rate in the server logs at that setting. All 500 requests
share the same 5-shot prefix, so this is the regime the optimization is
designed for.
max_num_seqs=256, throughput for both caching on and off
collapsed to roughly the same low value. This points to GPU memory
pressure from holding that many concurrent KV caches becoming the
dominant bottleneck, large enough to swamp any benefit from caching.
max_num_seqs=32, caching off actually outperformed
caching on. This is the one result that runs against the overall trend,
and with only a single run per configuration at low concurrency, it's
plausible this specific data point is noise rather than a real effect.
It's reported as-is rather than smoothed over.
max_num_seqs is
a concurrency batching control, so it can't show an effect unless
multiple requests are actually in flight at once. The results above come
from a rebuilt, concurrent version of the harness.
max_num_seqs=256 was not investigated further to find the
exact point where throughput starts degrading; the sweep only shows that
it happens somewhere between 128 and 256 for this model/GPU combination.
My motivation for beginning this benchmark is to improve my capabilities as a potential research engineer. I believe in science, research, and discovery, and want to use my skills as an engineer to support such avenues.