{
  "title": "Algorithm Deep-Dive: SkillForge — Synthesize Issues in 4 Steps, Distill a Dual-Layer Skill Library, +5.8% on SWE-bench",
  "url": "/en/posts/deep-code-skillforge/",
  "permalink": "https://hackcv.com/en/posts/deep-code-skillforge/",
  "date": "2026-08-23",
  "lastmod": "2026-08-23",
  "author": "hackcv",
  "description": "SkillForge full breakdown: strict-mask 4-step issue synthesis, dual-layer entity-grounded skills (diagnostic + intervention), BM25+JIT two-phase retrieval; SWE-bench Verified 72.2% (+5.8%), Pro 34.1% (+5.8%).",
  "categories": ["Research Brief"],
  "tags": ["AI","Agent","Skill Distillation","SkillForge","Algorithm Deep-Dive"],
  "cover": "https://picsum.photos/seed/algorithm-deep-dive-skillforge-synthesize-issues-in-4-steps-distill-a-dual-layer-skill-library-\u0026#43;5.8-on-swe-bench/1200/675",
  "readingTime": 3,
  "wordCount": 853,
  "content": "\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eOne-line takeaway\u003c/strong\u003e: SkillForge doesn\u0026rsquo;t wait for real issues — it \u003cstrong\u003esynthesizes project-specific issues by re-implementing test-covered core functionality\u003c/strong\u003e, distills \u003cstrong\u003eentity-anchored skills\u003c/strong\u003e (diagnostic + intervention layers) while solving them, and injects skills on-demand at interaction time. SWE-bench Verified: DeepSeek-V3.2 hits \u003cstrong\u003e72.2%\u003c/strong\u003e (baseline 66.4%, +5.8%), GPT-5-mini \u003cstrong\u003e60.6%\u003c/strong\u003e (+5.6%) — and ablation shows both knowledge layers are necessary.\u003c/p\u003e\n\u003c/blockquote\u003e\n\u003ch2 id=\"background--motivation\"\u003eBackground \u0026amp; Motivation\u003c/h2\u003e\n\u003ch3 id=\"the-project-knowledge-bottleneck\"\u003eThe Project-Knowledge Bottleneck\u003c/h3\u003e\n\u003cp\u003eLLM coding agents fail on specific repos because they lack project knowledge — module layout, coding style, implicit constraints. Existing self-evolving methods each have a hard flaw:\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eRoute\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eApproach\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eFlaw\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eHistory learning (SWE-Exp/EvoCoder/MemGovern)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eDistill from past fixes\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eDepends on historical fix signals\u003c/strong\u003e; cold-start fails on new repos\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eOnline exploration (SAGE/SWE-Debate/Live-SWE)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eLearn on real issues\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003ePer-issue exploration cost\u003c/strong\u003e is high\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eSkillForge takes a third path: \u003cstrong\u003econstruct knowledge gaps from the repo\u0026rsquo;s tests\u003c/strong\u003e — tests are the spec, with a built-in verifier.\u003c/p\u003e\n\u003ch2 id=\"core-approach-4-step-synthesis--dual-layer-distillation--two-phase-retrieval\"\u003eCore Approach (4-Step Synthesis → Dual-Layer Distillation → Two-Phase Retrieval)\u003c/h2\u003e\n\u003ch3 id=\"-issue-synthesis-four-steps\"\u003e① Issue Synthesis (Four Steps)\u003c/h3\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eStep\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eWhat it does\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e1. Test-driven scope\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eCoverage-instrumented execution of each \u003cstrong\u003epassing test\u003c/strong\u003e → execution trace → covered source files/line ranges, sliced into coherent segments\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e2. Critical-segment selection\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eLLM picks top-k key segments (test purpose + segment summary); \u003cstrong\u003ekey differentiator\u003c/strong\u003e: can select multiple segments across components → synthesizes issues that expose \u003cstrong\u003ecross-component interactions\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e3. Code rewriting (strict-mask)\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eNo original implementation given\u003c/strong\u003e — only surrounding lines, position/indent, and a high-level test goal; the LLM rewrites a plausible implementation preserving the API but simplifying logic → induces \u0026ldquo;general vs repo-specific knowledge\u0026rdquo; gaps (i.e., real developer mistakes)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e4. Instance assembly\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eRewrite breaks the test → buggy snapshot + buggy/reference patches → LLM turns failure evidence into a problem statement \u003cstrong\u003ewithout fix hints\u003c/strong\u003e → standard SWE-bench format\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e577 synthetic issues were produced on SWE-bench Verified (time-isolated: rollback to pre-golden-patch snapshot).\u003c/p\u003e\n\u003ch3 id=\"-dual-layer-skill-library-entity-grounded\"\u003e② Dual-Layer Skill Library (Entity-Grounded)\u003c/h3\u003e\n\u003cp\u003e\u003cstrong\u003eGlobal diagnostic skills M_ext\u003c/strong\u003e (3 fields, answering \u0026ldquo;where to look\u0026rdquo;):\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eField\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eContent\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003epurpose\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eThe entity\u0026rsquo;s functional role in issue resolution (is it a debug entry point?)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003eplaybook\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eReusable, repeatedly-validated reasoning strategy (repo-specific, not generic advice)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003ccode\u003erelated_apis\u003c/code\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003eAPIs frequently co-involved and why (repo interaction patterns)\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eLocal intervention skills M_int\u003c/strong\u003e (answering \u0026ldquo;how to change\u0026rdquo;): distilled from \u003cstrong\u003esuccessful trajectories\u003c/strong\u003e (correct fix strategies) and \u003cstrong\u003efailed trajectories\u003c/strong\u003e (pitfalls exposed by diffing wrong patch vs reference patch), shaped as \u003ccode\u003e{api_path, intervention_skills[]}\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eSkills are aligned to \u003cstrong\u003ereal code entities\u003c/strong\u003e by parsing shell commands (grep/sed/cat) in trajectories + an AST-derived structural index — preventing the LLM from hallucinating nonexistent interfaces.\u003c/p\u003e\n\u003ch3 id=\"-two-phase-retrieval-context-aware-injection\"\u003e③ Two-Phase Retrieval (Context-Aware Injection)\u003c/h3\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eMacro initialization\u003c/strong\u003e: new issue description → \u003cstrong\u003eBM25\u003c/strong\u003e top-5 from M_ext → prepended as project prior in the initial prompt\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eMicro JIT injection\u003c/strong\u003e: M_int is NOT injected all at once — the agent\u0026rsquo;s \u003cstrong\u003eshell commands are monitored\u003c/strong\u003e; when an accessed file hits an M_int entry, the intervention hint is attached as an auxiliary observation in real time. Skills stay \u003cstrong\u003estrictly aligned with current code interaction\u003c/strong\u003e, avoiding semantic-retrieval ambiguity\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"results\"\u003eResults\u003c/h2\u003e\n\u003cp\u003e\u003cstrong\u003eSWE-bench Verified (Table I, Pass@1)\u003c/strong\u003e:\u003c/p\u003e\n\u003ctable\u003e\n\t\u003cthead\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003cth\u003eMethod\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eDeepSeek-V3.2\u003c/th\u003e\n\t\t\t\t\t\u003cth\u003eGPT-5-mini\u003c/th\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/thead\u003e\n\t\u003ctbody\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003eSkillForge\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e72.2%\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e\u003cstrong\u003e60.6%\u003c/strong\u003e\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eMini-SWE-Agent (baseline)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e66.4%\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e55.0%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eMemGovern (best history baseline)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e69.2%\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e58.0%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eSAGE / SWE-Debate (online baselines)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e67.2% / 68.2%\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e56.0% / 56.4%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eSkillForge w/ SWE-Smith (single-function rewrite)\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e68.0%\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e56.4%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\t\t\u003ctr\u003e\n\t\t\t\t\t\u003ctd\u003eSkillForge w/ LLM Summary\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e68.7%\u003c/td\u003e\n\t\t\t\t\t\u003ctd\u003e54.4%\u003c/td\u003e\n\t\t\t\u003c/tr\u003e\n\t\u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003e\u003cstrong\u003eSWE-bench Pro\u003c/strong\u003e (731 instances, Python/JS/TS/Go): 34.1% / 51.7% (+5.8% / +4.1%).\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eAblations \u0026amp; hyperparameters\u003c/strong\u003e:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eComponent ablation\u003c/strong\u003e: removing M_ext ↓3.8%/↓3.0%; removing M_int ↓4.4%/↓3.4% — \u003cstrong\u003eboth necessary; intervention skills matter slightly more\u003c/strong\u003e\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCross-LLM transfer\u003c/strong\u003e: GPT-5-mini using DeepSeck-distilled knowledge scores 55.0% \u0026lt; 60.6% self-distilled — \u003cstrong\u003eskills bind to the distilling model\u003c/strong\u003e (different coding priors → different exposed mismatches); clear diagonal pattern\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRetrieval count\u003c/strong\u003e: k_r peaks at 5 (69.7%); full injection drops to 67.5% (low-ranked skills crowd the context window)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRewrite count\u003c/strong\u003e: k_s peaks at 5 (multi-entity interactions expose richer knowledge), slight drop at 7\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCross-repo\u003c/strong\u003e: improvement on all 7 largest repos, zero regressions (DeepSeek up to +13.6% Sphinx, GPT-5-mini +15.6% scikit-learn), vs SWE-Exp regressing on 3 (Matplotlib −11.8%)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003e\u003cstrong\u003eCase study (Django #11206)\u003c/strong\u003e: formatting a tiny Decimal — \u003ccode\u003eformat(Decimal(\u0026quot;1e-200\u0026quot;), \u0026quot;.\u0026quot;, decimal_pos=2)\u003c/code\u003e should yield \u0026ldquo;0.00\u0026rdquo; but returns \u0026ldquo;1.00e-200\u0026rdquo;. Baseline agent used an exponent heuristic → FAIL_TO_PASS 0/2; SkillForge agent, guided by retrieved knowledge to preserve the existing formatting pipeline and reason about numeric equivalence with the repo\u0026rsquo;s precision semantics → 2/2.\u003c/p\u003e\n\u003ch2 id=\"engineering-notes\"\u003eEngineering Notes\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eTest quality = synthesis quality\u003c/strong\u003e: weak-assertion tests are bad synthesis material; add key tests first if a repo lacks them\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eSkill granularity\u003c/strong\u003e: entity-anchored (file/function level) beats generic experience — retrieval hit rate and injection alignment are the key levers\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRollout path\u003c/strong\u003e: validate the synthesis-distill loop on a medium repo (a few hundred files) first; reference action budget 250 steps, temperature 0\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eCost note\u003c/strong\u003e: synthesis + distillation have extra inference overhead — best for high-frequency, homogeneous issue flows that amortize the skill library\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"scope--trade-offs\"\u003eScope \u0026amp; Trade-offs\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eFits\u003c/strong\u003e: repos with test suites, cold-start on new repos, high-frequency homogeneous issues\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eDoesn\u0026rsquo;t fit\u003c/strong\u003e: testless repos that won\u0026rsquo;t add tests; one-off issue flows (library grows with low reuse)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eTrade-offs\u003c/strong\u003e: vs history learning (no cold-start dependency but needs tests); vs online exploration (no per-issue cost but needs upfront synthesis budget); vs agent-skills (auto-generated + entity-anchored vs human-curated)\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"reproduction-notes\"\u003eReproduction Notes\u003c/h2\u003e\n\u003cul\u003e\n\u003cli\u003earXiv: 2608.18933; code/data \u003cstrong\u003egithub.com/cslsolow/SkillForge\u003c/strong\u003e (SJTU, Haibing Guan\u0026rsquo;s group)\u003c/li\u003e\n\u003cli\u003ePipeline: coverage-instrumented test runs → strict-mask rewriting → failure-evidence-to-statement → Mini-SWE-Agent + BM25 retrieval + JIT injection\u003c/li\u003e\n\u003c/ul\u003e\n",
  "summary": " One-line takeaway: SkillForge doesn\u0026rsquo;t wait for real issues — it synthesizes project-specific issues by re-implementing test-covered core functionality, distills entity-anchored skills (diagnostic + intervention layers) while solving them, and injects skills on-demand at interaction time. SWE-bench Verified: DeepSeek-V3.2 hits 72.2% (baseline 66.4%, +5.8%), GPT-5-mini 60.6% (+5.6%) — and ablation shows both knowledge layers are necessary.\nBackground \u0026amp; Motivation The Project-Knowledge Bottleneck LLM coding agents fail on specific repos because they lack project knowledge — module layout, coding style, implicit constraints. Existing self-evolving methods each have a hard flaw:\n"
}