{
  "title": "Algorithm Deep-Dive: RMM — TopK Column-Norm Slicing: Formulas, 1B–70B Results, and the Attention/MLP Asymmetry",
  "url": "/en/posts/deep-code-rmm/",
  "permalink": "https://hackcv.com/en/posts/deep-code-rmm/",
  "date": "2026-08-23",
  "lastmod": "2026-08-23",
  "author": "hackcv",
  "description": "RMM full breakdown: contraction-dim TopK column-norm selection, minimax optimality proof, retention-ratio knob; 8 benchmarks × 4 retention levels, attention vs MLP asymmetry data, A100 end-to-end 1.40× speedup.",
  "categories": ["Research Brief"],
  "tags": ["AI","Inference Optimization","Matrix Multiplication","RMM","Algorithm Deep-Dive"],
  "cover": "https://picsum.photos/seed/algorithm-deep-dive-rmm-topk-column-norm-slicing-formulas-1b70b-results-and-the-attention/mlp-asymmetry/1200/675",
  "readingTime": 3,
  "wordCount": 806,
  "content": "\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eOne-line takeaway\u003c/strong\u003e: RMM selects \u003cstrong\u003eTopK slices by column L2 norm along the contraction dimension\u003c/strong\u003e of matrix multiplications and computes only what\u0026rsquo;s kept — no training, no weight changes, one retention-ratio knob for a predictable accuracy-efficiency trade-off. Measured: \u003cstrong\u003e70B is nearly lossless at 80% retention\u003c/strong\u003e, Llama3.1 8B gets \u003cstrong\u003e1.40× end-to-end speedup\u003c/strong\u003e on long sequences, 4096-token runs avoid \u003cstrong\u003eOOM\u003c/strong\u003e on 70B; mechanistically, \u003cstrong\u003eattention is far more reducible than MLP\u003c/strong\u003e (Q projection drops only 2pp at RR=0.5 vs 29.5pp for whole-MLP).\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"background--motivation\"\u003eBackground \u0026amp; Motivation\u003c/h2\u003e\n\u003cp\u003eTransformer inference cost is dominated by high-dimensional matmuls (QK^T, PV, three FFN projections), but much of it is redundant: attention scores are sparse, FFN activations are highly sparse in high dimensions. Existing approaches face a dilemma:\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eRoute\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eRepresentative\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eFlaw\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eTrained sparsity\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eSparseGPT/Wanda/SliceGPT\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eChanges weights, costly\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eStatic pruning\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eMagnitude et al.\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eInput-independent; degrades sharply across distributions\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eRMM fills the gap: \u003cstrong\u003eno weight changes + input-adaptive dynamic pruning\u003c/strong\u003e.\u003c/p\u003e\n\u003ch2 id=\"core-approach-formula-level\"\u003eCore Approach (Formula Level)\u003c/h2\u003e\n\u003ch3 id=\"contraction-dimension-topk-selection\"\u003eContraction-Dimension TopK Selection\u003c/h3\u003e\n\u003cp\u003eFor matmul \u003ccode\u003eY = A·B\u003c/code\u003e (A∈ℝ^{n×d} activations, B∈ℝ^{d×m}), select index set ℐ ⊆ [d] (|ℐ| = ⌈ρd⌉) along the contraction dim:\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eRMM_ρ(A,B) = A[:,ℐ] · B[ℐ,:]\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003e\u003cstrong\u003eImportance score = activation column L2 norm\u003c/strong\u003e: \u003ccode\u003es_j = ||A[:,j]||₂\u003c/code\u003e, take TopK-largest ⌈ρd⌉.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eProperties\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003eDeterministic for a given input; input-adaptive per layer/head/token\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMinimax optimal\u003c/strong\u003e (Theorem 1): TopK by column norm minimizes worst-case approximation error over any B under the retention budget\u003c/li\u003e\n\u003cli\u003eError bound: \u003ccode\u003e||AB − A[:,ℐ]B[ℐ,:]||_F ≤ Σ_{j∉ℐ} ||A[:,j]||₂·||B[j,:]||₂\u003c/code\u003e\u003c/li\u003e\n\u003cli\u003eComplexity: O(n·ρd·m) vs dense O(n·d·m); column-norm O(n·d) + TopK overhead is small\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eComponent mapping\u003c/strong\u003e: QK^T selects along head feature dim (score = Q column norm), PV optionally along token positions, MLP/linear projections along activation hidden dim; under GQA, selection is done on Q per head and K/V gather the corresponding dims.\u003c/p\u003e\n\u003ch3 id=\"the-retention-ratio-ρ-knob\"\u003eThe retention-ratio (ρ) Knob\u003c/h3\u003e\n\u003cp\u003eρ∈(0,1] directly controls ⌈ρd⌉ retained dims — a smooth, predictable trade-off. \u003cstrong\u003eComponent-differentiated\u003c/strong\u003e: attention can be aggressive (RR as low as 0.5), MLP must be conservative and split by projection type. With no labeled data, scan ~100 unlabeled samples for consistency (Llama-3.1-8B at RR=0.7: 87/100 Wikipedia paragraphs \u003cstrong\u003esequence-identical\u003c/strong\u003e to the dense model).\u003c/p\u003e\n\u003ch2 id=\"results\"\u003eResults\u003c/h2\u003e\n\u003ch3 id=\"scaling-law-8-tasks--rr-0905\"\u003eScaling law (8 tasks × RR 0.9→0.5)\u003c/h3\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eModel\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eRR=0.8\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eRR=0.5\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eLlama3.1 \u003cstrong\u003e70B\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003enear-full (MMLU 75.0→72.6)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003estill usable (GSM8K 53.7→19.9 but most tasks gentle)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eQwen3 32B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003ealmost lossless (MMLU 80.8→78.6)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003egentle degradation\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eLlama3.1 8B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003emild drop\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eGSM8K 26.2→5.9 noticeable\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eQwen3.1 7B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eclear drop\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eGSM8K 39.9→1.7 collapses\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eLarger models tolerate more reduction\u003c/strong\u003e; small models show an inflection around RR=0.7 (WikiText ppl: Llama3.2-1B 20.04→31.29 at RR=0.7).\u003c/p\u003e\n\u003ch3 id=\"vs-static-pruning-rr05-llama31-8b-avg-5-qa\"\u003evs Static pruning (RR=0.5, Llama3.1 8B, avg 5 QA)\u003c/h3\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eMethod\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eAvg\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eFull model\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e69.8\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eRMM\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e59.8\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eSparseGPT\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e56.1\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWanda\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e52.7\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eMagnitude\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e39.3\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eSliceGPT\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e37.0\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch3 id=\"attention-vs-mlp-structural-asymmetry-table-16-8b-avg-5-qa\"\u003eAttention vs MLP: structural asymmetry (Table 16, 8B, avg 5 QA)\u003c/h3\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eTarget\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eRR=0.9\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eRR=0.7\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eRR=0.5\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eQ projection\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e69.60\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e70.01\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e67.80\u003c/strong\u003e (nearly flat)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eQKV projections\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e68.92\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e67.35\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e59.79\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eAttention-internal (QK^T+PV)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e69.45\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e66.98\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e59.56\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eWhole MLP\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e63.06\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e55.93\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e40.28\u003c/strong\u003e (collapses)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eMLP Up\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e65.69\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e59.88\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e52.44\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eMLP Down\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e67.43\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e65.75\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e61.36\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eSupplementary (ARC-Easy RR=0.7 normalized): attention drops 3.52pt (retained energy 89.69%), MLP Up 16.32 (82.24%), MLP Down 3.51 (99.02%), whole MLP 18.78 (87.85%) — \u003cstrong\u003eDown is most robust, Up most sensitive, errors accumulate across projections\u003c/strong\u003e.\u003c/p\u003e\n\u003ch3 id=\"long-context-ruler-rr05-still-flat\"\u003eLong context (Ruler, RR=0.5 still flat)\u003c/h3\u003e\n\u003cp\u003eCWE 5K/15K/30K: 98.0/94.0/28.9 vs baseline 98.2/94.0/29.6 — \u003cstrong\u003epruning does not amplify long-context degradation\u003c/strong\u003e.\u003c/p\u003e\n\u003ch3 id=\"a100-measurements-ρ08-batch1\"\u003eA100 measurements (ρ=0.8, batch=1)\u003c/h3\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eSeq len\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eQK^T\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eAV\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eE2E (8B)\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eE2E (70B)\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e1024\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.36×\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.67×\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.05×\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.03×\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e2048\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.29×\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.81×\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e1.27×\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e1.41×\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e4096\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.56×\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e1.89×\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e1.40×\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eOOM→runs\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eLonger sequences, bigger gains\u003c/strong\u003e (selection overhead dominates at short lengths); 70B goes from OOM to runnable at 4096 — memory savings and latency wins together.\u003c/p\u003e\n\u003ch3 id=\"compatibility--generalization\"\u003eCompatibility \u0026amp; generalization\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eOrthogonal to INT8\u003c/strong\u003e: INT8 + RMM (attention RR=0.8) COPA 81.40→77.40 — lower precision × fewer FLOPs stack\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eVLM generalization\u003c/strong\u003e: Qwen2.5-VL-7B nearly lossless at RR=0.8 (POPE 83.7→82.0); InternVL3-8B flat at 92.33 even at RR=0.5\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003evs TEAL (activation sparsity)\u003c/strong\u003e: TEAL only prunes projection inputs, cannot shrink QK^T/PV internal matmuls; RMM\u0026rsquo;s matrix-product view covers a broader operation space\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"engineering-notes\"\u003eEngineering Notes\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eIntegration\u003c/strong\u003e: wrap attention/FFN operators — prototype in PyTorch; production needs custom kernels to realize actual speedups\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eConfig\u003c/strong\u003e: aggressive attention (RR 0.5–0.7), conservative MLP (0.8+, Down can be lower); tune prefill (prune FFN) and decode (prune attention) separately\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGotchas\u003c/strong\u003e: short sequences gain little; strong-reasoning tasks like GSM8K are most sensitive (fastest to degrade) — be careful with math workloads\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eValidation\u003c/strong\u003e: scan ~100 unlabeled samples for consistency to pick ρ quickly, no annotation needed\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"scope--trade-offs\"\u003eScope \u0026amp; Trade-offs\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eFits\u003c/strong\u003e: long context, batch generation, lowering cost on deployed models, memory-constrained 4096+ runs; stacks with quantization\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDoesn\u0026rsquo;t fit\u003c/strong\u003e: short-sequence high-concurrency small batches (gains washed out by GEMM libraries); strict-accuracy workloads\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTrade-offs\u003c/strong\u003e: vs static sparsity (dynamic robustness but needs kernels); vs quantization (orthogonal, stackable); vs activation sparsity TEAL (broader matmul coverage)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"reproduction-notes\"\u003eReproduction Notes\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003earXiv: 2608.13426 (8-13, 24 pages); authors Zixuan Lan et al.; no repo noted\u003c/li\u003e\n\u003cli\u003ePath: implement the column-norm TopK slicing operator → run the RR curve on an 8B model → long-sequence A100 benchmark\u003c/li\u003e\n\u003cli\u003ePer-component RR (attention vs MLP) is the key engineering decision\u003c/li\u003e\n\u003c/ul\u003e\n",
  "summary": " One-line takeaway: RMM selects TopK slices by column L2 norm along the contraction dimension of matrix multiplications and computes only what\u0026rsquo;s kept — no training, no weight changes, one retention-ratio knob for a predictable accuracy-efficiency trade-off. Measured: 70B is nearly lossless at 80% retention, Llama3.1 8B gets 1.40× end-to-end speedup on long sequences, 4096-token runs avoid OOM on 70B; mechanistically, attention is far more reducible than MLP (Q projection drops only 2pp at RR=0.5 vs 29.5pp for whole-MLP).\n"
}