{
  "title": "Paper Review: SemComp-Bench — Video Generation Evaluation Moves from 'Looks Right' to 'Task Done'",
  "url": "/en/posts/deep-read-semcomp-bench/",
  "permalink": "https://hackcv.com/en/posts/deep-read-semcomp-bench/",
  "date": "2026-08-23",
  "lastmod": "2026-08-23",
  "author": "hackcv",
  "description": "The hottest paper on HF this week (153 upvotes): Semantic Task Completion video generation + the six-domain SemComp-Data + VLM-driven OA/GR dual-metric protocol, with full benchmark tables.",
  "categories": ["Research Brief"],
  "tags": ["AI","Video Generation","Benchmark","SemComp-Bench","Paper Review"],
  "cover": "https://picsum.photos/seed/paper-review-semcomp-bench-video-generation-evaluation-moves-from-looks-right-to-task-done/1200/675",
  "readingTime": 3,
  "wordCount": 842,
  "content": "\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eOne-line takeaway\u003c/strong\u003e: SemComp-Bench redefines video generation as \u003cstrong\u003eoutcome-oriented semantic task completion\u003c/strong\u003e — success = achieving the intended outcome × semantic grounding against a reference image — and ships a six-domain dataset with a VLM-based auto-evaluation protocol (OA/GR dual scores). Measured across 7 mainstream models: \u003cstrong\u003ethe best OA is only 37.8%\u003c/strong\u003e, I2V consistently beats T2V, and within-scene spatiotemporal consistency is the universal bottleneck — the \u0026ldquo;quality ceiling, task completion is the next frontier\u0026rdquo; claim is now backed by data.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"the-problem\"\u003eThe Problem\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eBackground\u003c/strong\u003e: Video generation evaluation has long been dominated by \u0026ldquo;quality\u0026rdquo; metrics — FVD, CLIP similarity, human preference. They measure \u0026ldquo;how realistic the video looks,\u0026rdquo; and are \u003cstrong\u003ecompletely blind to task goals\u003c/strong\u003e.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eThe flaw\u003c/strong\u003e: For a request like \u0026ldquo;move the coffee cup from the desk to the windowsill,\u0026rdquo; a model that generates a gorgeous close-up of the cup may still score high FVD — because FVD compares frame distributions, not \u0026ldquo;did the cup move?\u0026rdquo; As controllable video generation goes mainstream, \u0026ldquo;was the goal achieved\u0026rdquo; becomes the only acceptance criterion that matters, and legacy metrics fail.\u003c/p\u003e\n\u003ch2 id=\"method-breakdown\"\u003eMethod Breakdown\u003c/h2\u003e\n\u003ch3 id=\"task-redefinition-semantic-task-completion\"\u003eTask Redefinition: Semantic Task Completion\u003c/h3\u003e\n\u003cp\u003eSuccess = \u003cstrong\u003eOutcome Achievement\u003c/strong\u003e × \u003cstrong\u003eSemantic Grounding\u003c/strong\u003e. Evaluation looks only at the final result — no requirement for a complete intermediate step sequence, and no traditional appearance consistency with the reference image.\u003c/p\u003e\n\u003ch3 id=\"data-semcomp-data-six-domains-21-subcategories\"\u003eData: SemComp-Data (Six Domains, 21 Subcategories)\u003c/h3\u003e\n\u003cp\u003eSampled ~20K videos from Koala-36M → \u003cstrong\u003e1,273 instances\u003c/strong\u003e. Each instance = \u003ccode\u003e(reference frame, instruction pair {brief + detailed}, outcome-centric clip)\u003c/code\u003e, with the reference frame and clip from the \u003cem\u003esame\u003c/em\u003e source video (guaranteeing real feasibility). Four-stage curation:\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eStage\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eWhat it does\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eCandidate Filtering\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e45 keywords (news/movie/entertainment groups) drop narration-dependent videos; VLM classifies into six domains, discards low-evidence items as \u003cem\u003eUncertain\u003c/em\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eState Mining\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eFrame-level timestamp localization (State Grounding) + QA check: VLM picks \u0026ldquo;which frame is the outcome\u0026rdquo;; mismatches are conservatively dropped\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eVideo Extension\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003ePanda-70M shot detection + same-scene merging, anchored at the outcome timestamp, extended to 3–4s (mean 4.03s)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eInstruction Structuring\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eBrief instruction ≤30 words (\u003ccode\u003everb + subject + preposition + outcome state\u003c/code\u003e); detailed instruction adds alignment type from a 17-item attribute vocabulary (object_category / person_identity / spatial_relation / pose …)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eSix domains: Food \u0026amp; Cooking, Beauty \u0026amp; Fashion, Sports \u0026amp; Fitness, Crafts \u0026amp; DIY, Gardening \u0026amp; Pets, Arts \u0026amp; Precision.\u003c/p\u003e\n\u003ch3 id=\"evaluation-protocol-vlm-answers-structured-binary-questions\"\u003eEvaluation Protocol: VLM Answers Structured Binary Questions\u003c/h3\u003e\n\u003cp\u003eGenerated videos are sampled into \u003cstrong\u003e27 frames\u003c/strong\u003e; each video is scored by \u003cstrong\u003e3 independent VLM calls\u003c/strong\u003e (Doubao-Seed-1.8), averaged.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eOA dimension\u003c/strong\u003e (4 yes/no questions, \u003cstrong\u003econjunctive\u003c/strong\u003e — all must pass):\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eAᵢ = a_or × a_sg × a_gec × a_gvc ∈ {0,1}     OA Score = (1/N)ΣAᵢ\n\u003c/code\u003e\u003c/pre\u003e\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eQuestion\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eCriterion\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003ea_or Outcome Realization\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eDoes the video clearly reach the instructed completion state at coarse semantics?\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003ea_sg Semantic Grounding\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eDoes the result preserve/modify task-relevant entities per the reference-instruction pair?\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003ea_gec Entity Consistency\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eKey entities stay identifiable — no unexplained disappearance/replacement/drift? (reversed, no → 1)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003ea_gvc Global Continuity\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eNo abrupt global switch in scene/view/layout/background? (reversed, no → 1)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eGR dimension\u003c/strong\u003e (5 yes/no questions, \u003cstrong\u003earithmetic mean\u003c/strong\u003e):\u003c/p\u003e\n\u003cpre tabindex=\"0\"\u003e\u003ccode\u003eGᵢ = (g_pp + g_vc + g_afr + g_wsc + g_ti)/5     GR Score = (1/N)ΣGᵢ\n\u003c/code\u003e\u003c/pre\u003e\u003cp\u003ePhysical plausibility / visual clarity / artifact-free rendering / within-scene spatiotemporal coherence / text \u0026amp; interface integrity.\u003c/p\u003e\n\u003ch2 id=\"results-semcomp-core-60-stratified-instances\"\u003eResults (SemComp-Core: 60 stratified instances)\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eTable 1: OA Score (detailed instruction, I2V)\u003c/strong\u003e\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eModel\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eA_or\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eA_sg\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eA_gec\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eA_gvc\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003e\u003cstrong\u003eOA\u003c/strong\u003e\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eHY†-1.5-720P-I2V\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.878\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.706\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.583\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.794\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e37.8%\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWan2.2-I2V-A14B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.800\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.528\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.628\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.789\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e28.3%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eWan2.2-TI2V-5B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.589\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.400\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.689\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.922\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e23.3%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eSkyReels-V2-14B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.733\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.489\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.522\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.772\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e22.8%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eSeedance 2.0\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.839\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.744\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.444\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.594\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e20.0%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eCogVideoX1.5-5B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.550\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.389\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.506\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.744\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e14.4%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003ePhantom-1.3B\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.539\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.356\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.322\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e0.511\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e3.9%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e(†HY = HunyuanVideo. GR leaderboard: Seedance 2.0 tops at 91.8%, Wan2.2-A14B 89.0%.)\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eKey findings\u003c/strong\u003e:\u003c/p\u003e\n\u003col\u003e\n\u003cli\u003e\u003cstrong\u003eBest OA is only 37.8%\u003c/strong\u003e — achieving the outcome while preserving reference grounding is genuinely hard today\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eI2V beats T2V across all three model families\u003c/strong\u003e — gains come from grounding/entity consistency/global continuity; outcome realization rates are comparable\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eInstruction specificity trade-off\u003c/strong\u003e: detailed instructions give higher OA but are harder to generate; brief ones are more coherent but far lower task completion (CogVideoX T2V-brief OA = 0.6%)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWithin-scene spatiotemporal coherence is the global bottleneck\u003c/strong\u003e: all models G_wsc ∈ 0.328–0.739\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eGR and OA do not correspond\u003c/strong\u003e: Seedance tops GR (91.8%) but scores only 20% OA — \u0026ldquo;renders steadily\u0026rdquo; and \u0026ldquo;gets it done\u0026rdquo; are different things; both metrics are needed\u003c/li\u003e\n\u003c/ol\u003e\n\u003cp\u003e(3-run std dev: OA 0.96–4.41pp, GR 1.35–7.20pp; Seedance OA most volatile.)\u003c/p\u003e\n\u003ch2 id=\"limitations--commentary\"\u003eLimitations \u0026amp; Commentary\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eL1\u003c/strong\u003e: The VLM judge is a double-edged sword — it may be insensitive to physical plausibility / precise spatial relations; manual spot-checks are needed (paper reports per-question std anomalies up to 15.84pp on G_wsc)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eL2\u003c/strong\u003e: Six domains skew toward \u0026ldquo;everyday operations\u0026rdquo;; real productivity task space is much larger\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFor practitioners\u003c/strong\u003e: the evaluation mindset transfers directly — any pipeline whose output gets \u003cem\u003eused\u003c/em\u003e should move acceptance from \u0026ldquo;similarity\u0026rdquo; to \u0026ldquo;goal achievement rate.\u0026rdquo; Rollout path: small task set + VLM judge (Doubao/Claude/GPT), replace human acceptance first, then scale\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"reproduction-notes\"\u003eReproduction Notes\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003earXiv: 2608.17426; dataset + eval scripts released with the paper\u003c/li\u003e\n\u003cli\u003eEval cost: 3 VLM calls per video + 27-frame sampling — budget-friendly\u003c/li\u003e\n\u003cli\u003eMetric gotcha: OA must use the \u003cstrong\u003econjunctive\u003c/strong\u003e formula (all 4 must pass); GR uses arithmetic mean — different semantics, don\u0026rsquo;t mix\u003c/li\u003e\n\u003c/ul\u003e\n",
  "summary": " One-line takeaway: SemComp-Bench redefines video generation as outcome-oriented semantic task completion — success = achieving the intended outcome × semantic grounding against a reference image — and ships a six-domain dataset with a VLM-based auto-evaluation protocol (OA/GR dual scores). Measured across 7 mainstream models: the best OA is only 37.8%, I2V consistently beats T2V, and within-scene spatiotemporal consistency is the universal bottleneck — the \u0026ldquo;quality ceiling, task completion is the next frontier\u0026rdquo; claim is now backed by data.\n"
}