Full-dataset accuracy falls from 72.40% with FullKV to 8.50%. The accuracy loss at this budget is substantial; the scores alone do not identify its mechanism.
Full-dataset evaluation
Progress and results
5 models, 4 KV configurations, and the complete GSM8K and MATH test sets. This report combines current scores, scoring coverage, and remote job status, with the calculation basis for every result.
Every configuration, in detail
Score = correct answers ÷ scored questions. Each result shows both generated and scored counts, so generation progress is distinct from completion.
| Model | KV configuration | GSM8K generated / scored Full dataset: 1,319 questions | GSM8K score Correct / scored | GSM8K FullKV avg output tokens | MATH generated / scored Full dataset: 5,000 questions | MATH score Correct / scored | MATH FullKV avg output tokens | Configuration status Based on both test sets | Details |
|---|
Read scores alongside coverage
Compare budgets within the same model. Changes from FullKV are shown in percentage points only when both configurations have completed the dataset.
KV budget and accuracy
Fixed 0–100% scaleSolid bars indicate full-dataset results; hatched bars indicate partial samples.
Evaluation progress by model
25,276 evaluations per model4 configurations × (1,319 GSM8K questions + 5,000 MATH questions).
A total of 2,393 generated records await scoring, and another 36,240 question evaluations await generation. Each model and KV configuration counts as a separate evaluation.
GSM8K scores 89.92%, close to FullKV’s 90.07%; MATH scores 75.78%, below FullKV’s 87.30%. The two tasks differ in their sensitivity to compression.
MATH scores 85.30% and GSM8K scores 90.07%. Among the completed Gemma4 compression configurations, this one retains accuracy best.
Gemma1B complete; 4 workers still running
Job status comes from remote read-only checks at 16:54:28 PDT on October 15, covering the original 5 models in this report. This report does not start, stop, or reschedule jobs.
Key events and queue interruption
Qwen0.8 awaiting restartReal model weights passed native token parity and full-attention-only compression checks. Native linear state / SWA is preserved; Gemma4 shared KV is compressed at the producing layer. The session records report 58 CPU checks passed, with code committed and pushed.
Qwen27 FullKV was the first complete configuration at that point. This is a historical progress record; the main evaluation session resumed on October 15.
After rank2 / rank1 finished, N4-1 GPU7 / GPU6 respectively detected other compute processes and raised “Target card is not strictly idle.” The controllers exited with returncode=1 before launching the next child process. There was no automatic wait-and-retry for an idle GPU.
In that historical snapshot, all four Gemma4 configurations were complete. Qwen27 was generating MATH for R-KV 512, and later Gemma3-270M and 1B shards were producing output on N1-1. Those process counts and stages do not describe the current state.
All four Gemma3-1B configurations are now complete, and Gemma4 remains fully complete. Qwen27 and Gemma270M each have 2 workers on N1-1; Qwen27 R-KV 512 has generated 2,027 MATH answers, with scoring pending. The 3.8/9B runs have no verified scores yet and are outside this report’s 20 configurations.
Ranks 0, 1, and 2 are still the only completed shards, and no generation processes were observed. The original GPU6 queue was [1, 3, 5, 7], and the GPU7 queue was [2, 4, 6]. Ranks 3–7 have not started and await coordination by the main evaluation session.
Observed generation processes by node
4 generation workers| Node | Generation jobs in this report | Count |
|---|---|---|
| N1-1 | Qwen27 × 2 Gemma270M × 2 | 4 |
| N1-3 | No generation processes found for these models | 0 |
| N4-1 | No generation processes found for these models | 0 |
| N4-2 | No generation processes found for these models | 0 |
These counts include only generation child processes for the original 5 models in this report. They exclude queue controllers and do not indicate that a node is idle. All 4 observed processes are on N1-1; none were found for these models on the other three nodes.
The main R-KV evaluation session resumed on October 15 and is progressing the 3.8/9B runs. That session manages the runs; this page only summarizes existing results through read-only collection and does not start, stop, or reschedule jobs.
Protocol and verification evidence
The score denominator, completion criteria, and limits of the conclusions supported by this snapshot.
Fixed generation protocol
- Models and configurations
- 5 models × FullKV / R-KV 128 / 512 / 1024
- Evaluation datasets
- GSM8K test: 1,319; MATH test: 5,000. Neither MATH-500 nor pilot subsets were used as substitutes.
- Inference settings
- BF16 · greedy · batch = 1 · seed = 42
- Attention
- Native SDPA; cuDNN SDPA disabled
- Compression scope
- No compression during prefill. Full attention only, compressed every 128 decode steps. Native SWA and linear state are preserved.
- Adaptation parameters
- window_size = 8 · mix_lambda = 0.1 · retain_ratio = 0.2
- Output limit
- GSM8K 8,192 tokens;MATH 16,384 tokens
- Lengths retrieved at
- FullKV output length
- Actual generated continuation tokens, excluding the prompt and including thinking and generated special / EOS tokens, not just the final answer text.
- FullKV length statistics
- Covers all generated FullKV samples in the snapshot, including unscored samples. For even n, the median averages the middle pair; P95 uses the nearest-rank value at sorted position ceil(0.95 × n).
- Thinking
- Enabled for Qwen3.6-27B; disabled for Qwen0.8 / Gemma3 / Gemma4. Compare configurations within the same model first.
- Sharding and merging
- Each model uses 8 fixed shards. Results are merged across nodes by question ID, with source fingerprints verified.
How to read this report
- Dataset completion and configuration completion differ. A dataset result is complete once every GSM8K or MATH question has been generated and scored. A configuration is complete only when both datasets are finished. This snapshot has 21/40 dataset results and 10/20 configurations complete .
- Partial scores may cover different questions. Scores marked * cover only the samples scored so far. A partial score from a different configuration or coverage set should not be read directly as a gain or loss against a full-dataset FullKV result.
- Generation and scoring can progress separately. A total of 2,393 records await scoring. Results without scores show “Awaiting scoring”; those without output show “Not started.”
- Questions that reach the output limit remain in the test. An output-cap hit means the output token count reaches the cap, including cases where EOS lands exactly at the limit. Its association with accuracy changes does not establish causation.
- Compression triggers do not measure GPU memory savings. “Samples triggering compression” counts generated samples with at least one eviction. It is neither the fraction of tokens removed nor a measure of throughput or GPU memory gains.
- This report makes no claims about performance gains. This snapshot lacks comparable latency, throughput, and peak GPU memory measurements from exclusive GPU runs, so it cannot quantify speed or memory gains.
Expand sources, versions, and audit records
4-node aggregation · 0 verification errors · Fixed generation protocol and dataset SHA256
Expand sources, versions, and audit records
4-node aggregation · 0 verification errors · Fixed generation protocol and dataset SHA256
Source data and pinned versions
Code branch: codex/hybrid-full-attention-math-20261001 ↗bdff2101133f97896bae84fcbb1dc1468268cf5d
Session and observation scope
Main session: R-KV Qwen
Sampled at 2026-10-15 23:54:28 UTC. Generation records, scores, FullKV lengths, and process status were collected in one read-only round across N4-2, N4-1, N1-1, and N1-3. Checks passed for models, generation parameters, actual dataset hashes, question-source fingerprints, and cross-node duplicates. Complete groups have IDs that exactly cover the expected range. Coverage counts show the missing questions in incomplete groups.
This read-only process check found 4 in-scope generation processes, all on N1-1: Qwen27 × 2 and Gemma270M × 2. None were found for the original 5 models on N1-3, N4-1, or N4-2. This observation does not describe resource use by other jobs.
Original run records verified the earlier Qwen0.8 queue interruption: the controllers exited during idle-GPU checks before launching the next shards. Original run evidence is retained locally.
Snapshot fingerprint
SHA256 of the original complete aggregate JSON:
All displayed data is embedded in this HTML, which can be saved, shared, and opened offline on its own. The public page consists only of this file; original run evidence is retained locally. This read-only update refreshes scores, coverage, FullKV output lengths, and process observations together at the displayed snapshot time. No evaluation jobs were launched, stopped, or rescored.