{
  "title": "Paper Review: Co-RL — Peer Rewards Replace RLHF: Formulas, Mechanism, and 7+4 Benchmark Results",
  "url": "/en/posts/deep-read-co-rl/",
  "permalink": "https://hackcv.com/en/posts/deep-read-co-rl/",
  "date": "2026-08-23",
  "lastmod": "2026-08-23",
  "author": "hackcv",
  "description": "Co-RL full breakdown: peer-reward pseudo-label formulas, ring-topology majority voting, three diversity dimensions, GRPO integration; Qwen2.5-3B +8.6% avg across 7 text benchmarks, VLM +7.2% across 4, label-free parity with supervised methods, code open-sourced.",
  "categories": ["Research Brief"],
  "tags": ["AI","Reinforcement Learning","Multi-Agent","Co-RL","Paper Review"],
  "cover": "https://picsum.photos/seed/paper-review-co-rl-peer-rewards-replace-rlhf-formulas-mechanism-and-7\u0026#43;4-benchmark-results/1200/675",
  "readingTime": 3,
  "wordCount": 656,
  "content": "\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eOne-line takeaway\u003c/strong\u003e: Co-RL makes parameter-independent models judge each other — rewards come from \u003cstrong\u003emajority-voted pseudo-labels of peer answers\u003c/strong\u003e, not one\u0026rsquo;s own. Cohort diversity (heterogeneous families/sizes/sample rephrasings) breaks the correlated-error feedback loop of self-rewarding. Measured: Qwen2.5-3B averages \u003cstrong\u003e+8.6%\u003c/strong\u003e across 7 text benchmarks (49.3 vs 40.7 base), 5 VLMs average \u003cstrong\u003e+2.3–7.2%\u003c/strong\u003e, matching or beating supervised methods \u003cstrong\u003ewith zero ground-truth labels\u003c/strong\u003e.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"the-problem\"\u003eThe Problem\u003c/h2\u003e\n\u003ch3 id=\"the-supervision-dependence-dilemma-of-reasoning-rl\"\u003eThe \u0026ldquo;Supervision Dependence\u0026rdquo; Dilemma of Reasoning RL\u003c/h3\u003e\n\u003cp\u003eRL\u0026rsquo;s strongest gains for LLM/VLM reasoning rely on \u003cstrong\u003everifiable rewards\u003c/strong\u003e (code tests, math answers) — but such annotations are costly and deplete as reasoning capability exceeds what humans can reliably evaluate.\u003c/p\u003e\n\u003ch3 id=\"the-mechanistic-flaws-of-self-rewarding\"\u003eThe Mechanistic Flaws of Self-Rewarding\u003c/h3\u003e\n\u003cp\u003eLetting a model grade itself is the common shortcut, but the paper identifies three failure paths:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eCorrelated-error loop\u003c/strong\u003e: feedback from oneself (or same-source models) reinforces the same mistakes\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResponse homogenization\u003c/strong\u003e: a single reward signal collapses the policy\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTraining collapse\u003c/strong\u003e: diversity exhaustion → loss oscillation/divergence\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003e\u003cstrong\u003eCore insight\u003c/strong\u003e: the problem is not \u0026ldquo;unsupervised\u0026rdquo; but \u0026ldquo;\u003cstrong\u003efeedback correlated with the learner\u003c/strong\u003e\u0026rdquo; — cut the correlation and the unsupervised signal becomes clean.\u003c/p\u003e\n\u003ch2 id=\"method-breakdown-formula-level\"\u003eMethod Breakdown (Formula Level)\u003c/h2\u003e\n\u003ch3 id=\"cross-agent-supervision-peer-rewards\"\u003eCross-Agent Supervision (Peer Rewards)\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003ePseudo-label construction\u003c/strong\u003e (ring topology; agent n is supervised by peer n−1):\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eâ₋ₙ(x) ∈ argmax_b Σⱼ₌₁ᴷ 𝟙[aₙ₋₁ʲ = b]      # majority vote over K sampled answers\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003e\u003cstrong\u003eReward assignment\u003c/strong\u003e (hard 0/1):\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003erₙᵏ = 𝟙[aₙᵏ = â₋ₙ(x)]                     # 1 iff matches peer pseudo-label\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003e\u003cstrong\u003eGRPO integration\u003c/strong\u003e (in-group relative advantage):\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eÂₙᵏ = (rₙᵏ − mean{rₙʲ}) / std{rₙʲ}        # normalized within rollout group\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003eKey point: \u003cstrong\u003ean agent never contributes to its own supervision target\u003c/strong\u003e — the essential difference from self-rewarding.\u003c/p\u003e\n\u003ch3 id=\"three-dimensions-of-cohort-diversity\"\u003eThree Dimensions of Cohort Diversity\u003c/h3\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eDimension\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eImplementation\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eMechanism\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eDecoupled optimization\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eIndependent params/optimizers, no shared gradients, interact only at reward time\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eIndependence itself is a diversity source\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eCross-family \u0026amp; size\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003ePair Qwen×Llama (different tokenizers/pretraining), 3B×1.7B mixes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eDifferent inductive biases → orthogonal error patterns\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eInput formation\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eDeepSeek-V3 rephrases MATH problems; agents train on different phrasings\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eBreaks phrasing-correlated errors\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eQuantified evidence\u003c/strong\u003e (error-decoupling analysis): cross-family pairing (Qwen2.5-3B × Llama-3.2-3B) Cohen\u0026rsquo;s κ=0.38, complementarity c=31.2%; same-family (×Qwen3-1.7B) κ=0.52, c=24.2% — \u003cstrong\u003ecross-family errors overlap far less\u003c/strong\u003e, the root reason peer grading works.\u003c/p\u003e\n\u003ch3 id=\"training-loop\"\u003eTraining Loop\u003c/h3\u003e\n\u003cp\u003ePer step: sample prompt batch → all agents \u003cstrong\u003eparallel-sample\u003c/strong\u003e K responses → build pseudo-labels/rewards from peers → \u003cstrong\u003esynchronous\u003c/strong\u003e GRPO update of all policies. Symmetric design: every agent is both learner and supervisor.\u003c/p\u003e\n\u003ch2 id=\"results\"\u003eResults\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eText (Table 1, 3B-class, avg of 7 benchmarks)\u003c/strong\u003e:\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eModel\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eBase\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eGT-Reward (supervised)\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eBest self-reward TTRL\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003e\u003cstrong\u003eCo-RL\u003c/strong\u003e\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eQwen2.5-3B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e40.7\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e47.4\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e47.3\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e49.3\u003c/strong\u003e (+8.6%)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eLlama-3.2-3B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e38.7\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e43.0\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e43.1\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e43.9\u003c/strong\u003e (+5.2%)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e7 benchmarks = GSM8K/MATH-500/AMC/HumanEval/GPQA/MBPP/LiveCodeBench. Co-RL gains 3.0–8.6% on average, \u003cstrong\u003e0.8–2.0% above the strongest self-reward baseline\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eMultimodal (Table 4, 2B–12B VLMs, MathVision/MathVerse/MathVista/We-Math)\u003c/strong\u003e:\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eModel\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eBase\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eTTRL\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003e\u003cstrong\u003eCo-RL\u003c/strong\u003e\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eQwen2.5-VL-3B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e37.24\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e42.54\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e43.89\u003c/strong\u003e (+6.65%)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eInternVL-3.5-2B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e43.11\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e45.04\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e45.40\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eOn larger models (7B–12B) Co-RL consistently beats TTRL; \u003cstrong\u003eon Gemma-3-12B it exceeds GT-Reward (supervised)\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKey ablations\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eTraining stability\u003c/strong\u003e: Co-RL keeps reward variance/length stable throughout; TTRL shows reward collapse / length degradation / divergence\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eBudget-fair comparison\u003c/strong\u003e: two individually TTRL-trained models + inference ensemble vs Co-RL\u0026rsquo;s two agents — Co-RL still wins, proving gains come from \u003cstrong\u003ecross-supervision\u003c/strong\u003e, not ensembling/compute\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eThree-agent scaling\u003c/strong\u003e: Qwen2.5-3B + Llama-3.2-3B + Qwen3-1.7B jointly trained — all three improve +7.8%/+6.0%/+8.2%, averaging parity with GT-Reward\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMulti-agent baselines\u003c/strong\u003e (CoMAS setting): Co-RL 62.97, well ahead of MAPoRL(58.22)/TTRL(58.18)/CoMAS(58.94), with half the agents and no external judge\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"limitations--commentary\"\u003eLimitations \u0026amp; Commentary\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eL1\u003c/strong\u003e: cohort size = training cost ×N; diversity hyperparameters (model mix, rephrase ratio) need tuning\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eL2\u003c/strong\u003e: emergent reasoning is implicit behavior — no quality ceiling guarantee, no interpretability\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eLineage\u003c/strong\u003e: self-rewarding (self-grades) → self-play (self-opponent) → \u003cstrong\u003eCo-RL (heterogeneous peer grading)\u003c/strong\u003e — in the \u0026ldquo;no external supervision\u0026rdquo; spectrum, it trades diversity for signal quality, complementing verifier-based routes\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFor practitioners\u003c/strong\u003e: when doing reasoning RL without labels, Co-RL is the most promising unsupervised route; infra teams can borrow the \u0026ldquo;cohort architecture + diversity scheduling\u0026rdquo; design pattern directly\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"reproduction-notes\"\u003eReproduction Notes\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003earXiv: 2608.17253 (v2); code open-sourced at \u003cstrong\u003egithub.com/DrStranded/Co-RL\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003eBar: multi-model parallel training (start with two 3B models to validate the mechanism); temperature/sampling K per paper; ring-topology pseudo-labels via majority vote\u003c/li\u003e\n\u003cli\u003eStart with a \u003cem\u003ecross-family\u003c/em\u003e model pair (biggest error-decoupling gains), then scale size\u003c/li\u003e\n\u003c/ul\u003e\n",
  "summary": " One-line takeaway: Co-RL makes parameter-independent models judge each other — rewards come from majority-voted pseudo-labels of peer answers, not one\u0026rsquo;s own. Cohort diversity (heterogeneous families/sizes/sample rephrasings) breaks the correlated-error feedback loop of self-rewarding. Measured: Qwen2.5-3B averages +8.6% across 7 text benchmarks (49.3 vs 40.7 base), 5 VLMs average +2.3–7.2%, matching or beating supervised methods with zero ground-truth labels.\nThe Problem The \u0026ldquo;Supervision Dependence\u0026rdquo; Dilemma of Reasoning RL RL\u0026rsquo;s strongest gains for LLM/VLM reasoning rely on verifiable rewards (code tests, math answers) — but such annotations are costly and deplete as reasoning capability exceeds what humans can reliably evaluate.\n"
}