R/KV Evaluation Monitor RESEARCH NOTE / 013
R-KV · Full benchmark audit

Full-dataset evaluation
Progress and results

5 models, 4 KV configurations, and the complete GSM8K and MATH test sets. This report combines current scores, scoring coverage, and remote job status, with the calculation basis for every result.

R-KV Qwen 20 configurations · 126,380 question evaluations Offline snapshot · Not live monitoring
Qwen3.5-0.8B: 5 remaining shards have not started. On the evening of October 10, the original queues detected other compute processes on the target GPUs. The idle-GPU check caused the controllers to exit. Ranks 0/1/2 are complete; ranks 3–7 have not started. View queue interruption evidence and next steps
01 / RESULTS

Every configuration, in detail

Score = correct answers ÷ scored questions. Each result shows both generated and scored counts, so generation progress is distinct from completion.

20 / 20 configurations
Full dataset scored * Partial-sample score / length — No score yet
Scroll horizontally · Open “Details” for correct counts, output length, output-cap hits, and compression triggers
Model KV configuration GSM8K generated / scored Full dataset: 1,319 questions GSM8K score Correct / scored GSM8K FullKV avg output tokens MATH generated / scored Full dataset: 5,000 questions MATH score Correct / scored MATH FullKV avg output tokens Configuration status Based on both test sets Details
02 / COMPARISON

Read scores alongside coverage

Compare budgets within the same model. Changes from FullKV are shown in percentage points only when both configurations have completed the dataset.

KV budget and accuracy

Fixed 0–100% scale

Solid bars indicate full-dataset results; hatched bars indicate partial samples.

0255075100%

Evaluation progress by model

25,276 evaluations per model

4 configurations × (1,319 GSM8K questions + 5,000 MATH questions).

Scored Generated, awaiting scoring Not yet generated

A total of 2,393 generated records await scoring, and another 36,240 question evaluations await generation. Each model and KV configuration counts as a separate evaluation.

QWEN3.6-27B · MATH · R-KV 128
−63.90 percentage points

Full-dataset accuracy falls from 72.40% with FullKV to 8.50%. The accuracy loss at this budget is substantial; the scores alone do not identify its mechanism.

GEMMA4-E2B · MATH · R-KV 512
−11.52 percentage points

GSM8K scores 89.92%, close to FullKV’s 90.07%; MATH scores 75.78%, below FullKV’s 87.30%. The two tasks differ in their sensitivity to compression.

GEMMA4-E2B · MATH · R-KV 1024
−2.00 percentage points

MATH scores 85.30% and GSM8K scores 90.07%. Among the completed Gemma4 compression configurations, this one retains accuracy best.

03 / OPERATIONS

Gemma1B complete; 4 workers still running

Job status comes from remote read-only checks at 16:54:28 PDT on October 15, covering the original 5 models in this report. This report does not start, stop, or reschedule jobs.

Key events and queue interruption

Qwen0.8 awaiting restart
Qwen and Gemma hybrid-attention adaptation completed

Real model weights passed native token parity and full-attention-only compression checks. Native linear state / SWA is preserved; Gemma4 shared KV is compressed at the producing layer. The session records report 58 CPU checks passed, with code committed and pushed.

Scaled to 18 generation workers; first configuration completed

Qwen27 FullKV was the first complete configuration at that point. This is a historical progress record; the main evaluation session resumed on October 15.

Strict idle-GPU checks failed before the next shards started

After rank2 / rank1 finished, N4-1 GPU7 / GPU6 respectively detected other compute processes and raised “Target card is not strictly idle.” The controllers exited with returncode=1 before launching the next child process. There was no automatic wait-and-retry for an idle GPU.

At that time: 7 configurations complete, 10 generation workers active

In that historical snapshot, all four Gemma4 configurations were complete. Qwen27 was generating MATH for R-KV 512, and later Gemma3-270M and 1B shards were producing output on N1-1. Those process counts and stages do not describe the current state.

10 configurations complete; 4 generation workers active

All four Gemma3-1B configurations are now complete, and Gemma4 remains fully complete. Qwen27 and Gemma270M each have 2 workers on N1-1; Qwen27 R-KV 512 has generated 2,027 MATH answers, with scoring pending. The 3.8/9B runs have no verified scores yet and are outside this report’s 20 configurations.

Remaining Qwen0.8 queues
Ranks 0, 1, and 2 are still the only completed shards, and no generation processes were observed. The original GPU6 queue was [1, 3, 5, 7], and the GPU7 queue was [2, 4, 6]. Ranks 3–7 have not started and await coordination by the main evaluation session.

Observed generation processes by node

4 generation workers
Node Generation jobs in this report Count
N1-1Qwen27 × 2
Gemma270M × 2
4
N1-3 No generation processes found for these models 0
N4-1 No generation processes found for these models 0
N4-2 No generation processes found for these models 0

These counts include only generation child processes for the original 5 models in this report. They exclude queue controllers and do not indicate that a node is idle. All 4 observed processes are on N1-1; none were found for these models on the other three nodes.

Remaining work

  1. Have the main evaluation session coordinate the remaining 5 Qwen0.8 shards.
  2. Follow the active Qwen27 and Gemma270M jobs; Gemma1B and Gemma4 are complete.
  3. Score pending records, finish the remaining configurations, then merge and verify IDs across the full datasets.
  4. The evaluation session will manage the remaining runs and close out once the original 20 configurations are complete.
Evaluation session resumed

The main R-KV evaluation session resumed on October 15 and is progressing the 3.8/9B runs. That session manages the runs; this page only summarizes existing results through read-only collection and does not start, stop, or reschedule jobs.

04 / PROTOCOL & EVIDENCE

Protocol and verification evidence

The score denominator, completion criteria, and limits of the conclusions supported by this snapshot.

Fixed generation protocol

Models and configurations
5 models × FullKV / R-KV 128 / 512 / 1024
Evaluation datasets
GSM8K test: 1,319; MATH test: 5,000. Neither MATH-500 nor pilot subsets were used as substitutes.
Inference settings
BF16 · greedy · batch = 1 · seed = 42
Attention
Native SDPA; cuDNN SDPA disabled
Compression scope
No compression during prefill. Full attention only, compressed every 128 decode steps. Native SWA and linear state are preserved.
Adaptation parameters
window_size = 8 · mix_lambda = 0.1 · retain_ratio = 0.2
Output limit
GSM8K 8,192 tokens;MATH 16,384 tokens
Lengths retrieved at
FullKV output length
Actual generated continuation tokens, excluding the prompt and including thinking and generated special / EOS tokens, not just the final answer text.
FullKV length statistics
Covers all generated FullKV samples in the snapshot, including unscored samples. For even n, the median averages the middle pair; P95 uses the nearest-rank value at sorted position ceil(0.95 × n).
Thinking
Enabled for Qwen3.6-27B; disabled for Qwen0.8 / Gemma3 / Gemma4. Compare configurations within the same model first.
Sharding and merging
Each model uses 8 fixed shards. Results are merged across nodes by question ID, with source fingerprints verified.

How to read this report

Accuracy = correct answers ÷ scored questions × 100%
  • Dataset completion and configuration completion differ. A dataset result is complete once every GSM8K or MATH question has been generated and scored. A configuration is complete only when both datasets are finished. This snapshot has 21/40 dataset results and 10/20 configurations complete .
  • Partial scores may cover different questions. Scores marked * cover only the samples scored so far. A partial score from a different configuration or coverage set should not be read directly as a gain or loss against a full-dataset FullKV result.
  • Generation and scoring can progress separately. A total of 2,393 records await scoring. Results without scores show “Awaiting scoring”; those without output show “Not started.”
  • Questions that reach the output limit remain in the test. An output-cap hit means the output token count reaches the cap, including cases where EOS lands exactly at the limit. Its association with accuracy changes does not establish causation.
  • Compression triggers do not measure GPU memory savings. “Samples triggering compression” counts generated samples with at least one eviction. It is neither the fraction of tokens removed nor a measure of throughput or GPU memory gains.
  • This report makes no claims about performance gains. This snapshot lacks comparable latency, throughput, and peak GPU memory measurements from exclusive GPU runs, so it cannot quantify speed or memory gains.

Expand sources, versions, and audit records

4-node aggregation · 0 verification errors · Fixed generation protocol and dataset SHA256

Source data and pinned versions

Code branch: codex/hybrid-full-attention-math-20261001 ↗bdff2101133f97896bae84fcbb1dc1468268cf5d

Session and observation scope

Main session: R-KV Qwen

Sampled at 2026-10-15 23:54:28 UTC. Generation records, scores, FullKV lengths, and process status were collected in one read-only round across N4-2, N4-1, N1-1, and N1-3. Checks passed for models, generation parameters, actual dataset hashes, question-source fingerprints, and cross-node duplicates. Complete groups have IDs that exactly cover the expected range. Coverage counts show the missing questions in incomplete groups.

This read-only process check found 4 in-scope generation processes, all on N1-1: Qwen27 × 2 and Gemma270M × 2. None were found for the original 5 models on N1-3, N4-1, or N4-2. This observation does not describe resource use by other jobs.

Original run records verified the earlier Qwen0.8 queue interruption: the controllers exited during idle-GPU checks before launching the next shards. Original run evidence is retained locally.

Snapshot fingerprint

SHA256 of the original complete aggregate JSON:

All displayed data is embedded in this HTML, which can be saved, shared, and opened offline on its own. The public page consists only of this file; original run evidence is retained locally. This read-only update refreshes scores, coverage, FullKV output lengths, and process observations together at the displayed snapshot time. No evaluation jobs were launched, stopped, or rescored.

Configuration detail