A three-phase pipeline that turns any codebase into structured, auto-synchronized documentation — cutting agent tool calls by 56% and scope bloat by 52%.
When an AI coding agent gets a behavioral request like "add a retry mechanism to the API client," it has no map of the codebase. It brute-forces through grep, file reads, and directory listings — burning tokens on navigation instead of reasoning. The result: plans that touch too many files, miss hidden dependencies, and hallucinate edit sites.
The core issue is that codebases have implicit structure — execution stages, data flows, cross-file dependencies — that keyword search can't recover. The Handbook paper from Carnegie Mellon and Stanford proposes a structured solution: generate a living document tree from static analysis, then auto-sync it whenever the code changes.
Handbook is a three-phase pipeline that transforms a repository into a navigable document tree with three levels of detail:
The pipeline takes a repository as input and produces a handbook — a packaged set of structured documentation, a program graph, stage assignments, and a machine-readable resynchronization state. When code changes, the handbook updates incrementally rather than regenerating from scratch.
The pipeline starts with deterministic static analysis — no LLM calls. It parses every source file into an Abstract Syntax Tree (AST) and extracts:
Functions that span multiple regions (like Python decorators or Rust trait implementations) are split into contiguous segments. Unparseable files are recorded as unmapped rather than silently dropped. The output is a repository ℛ (the source code) paired with a program graph 𝒢 (the extracted facts).
This phase is fully deterministic — the same repository always produces the same graph. It serves as the ground truth that constrains all later LLM-generated content.
Now the pipeline needs to group the flat list of functions or files into meaningful execution stages. This is where the LLM comes in — but its role is tightly constrained.
In function-as-leaf mode (for smaller repos), the pipeline starts from a seed skeleton — either hand-written or inferred — and assigns each function to one or more stages. The model sees the function's signature, source code, and its caller/callee context. It decides which stage(s) the function belongs to. A reviewer then checks each assignment against the source, the stage definitions, and the call graph evidence.
In file-as-leaf mode (for large repos), the pipeline generates a "file card" for each file — a summary that includes the file's purpose, its functions, and its call relationships. Then it assigns files to stages using the same generate-review loop.
The organization can run in three variants:
Within each stage, a file-call graph provides a caller-before-callee ordering, then the LLM refines it into thematic groups. The result is a stage skeleton 𝒮 and organization state 𝒰.
The pipeline now generates the actual document tree 𝒟. Both modes produce L1, L2, and L3 entries — but they differ in direction:
Function-as-leaf builds top-down. The known stage skeleton provides the outline. Generation flows from L1 overview → L2 component overviews → L3 function entries. Each L3 entry represents one function in that stage. Generation uses a bounded generate-review-revise loop: the model drafts, a critic scores against a rubric, and the model revises until the rubric passes or the budget is exhausted.
File-as-leaf builds bottom-up. File cards become the L3 entries directly. Stage summaries are generated from child summaries and file descriptions, rolling up from leaves to the L1 system overview. This single-pass approach is cheaper at repository scale.
Critical constraint: the model can add behavioral explanations, but it cannot alter file paths, function identities, signatures, source ranges, or resolved calls. Static facts from Phase I remain authoritative throughout.
The paper evaluates on two harnesses that differ by roughly two orders of magnitude:
Terminus-2 is a Python terminal agent with 6 source files but behaviorally rich — multi-stage iteration loops, context management, cross-iteration state. It uses function-as-leaf mode because a reliable seed skeleton is available and function-level entries stay within budget.
Codex is the Rust monorepo behind the Codex coding agent — CLI, TUI, app-server, configuration, and sandboxing across many crates. It uses file-as-leaf mode to infer the hierarchy at repository scale.
The handbook is useless if the agent doesn't know how to navigate it. BGPD is the policy that guides a read-only agent through the handbook from coarse to fine:
The agent starts at L1 and L2 to select stages whose behavior matches the request. It follows the state-register view to find stages coupled through shared state — recovering behavior that is structurally distant but mutually dependent (like a value written in one stage and consumed several stages later).
Within selected stages, the agent opens L3 entries and expands along call relations in the program graph. Only when it has candidates does it open the actual repository, resolve each locator, and verify against live source. The handbook guides the search; the repository remains the authority for the edit plan.
The killer feature: when code changes, the handbook updates automatically. The pipeline computes Δ = Diff(R, R') and only regenerates affected parts.
In function-as-leaf mode, functions are matched using body fingerprints that ignore line numbers. A function that only moves is treated as unchanged — its locator shifts by the line offset. A rename is detected by matching the body below the signature line.
In file-as-leaf mode, file-set differences identify added/removed files, while content hashes identify changed files. A path rename appears as one removal and one addition.
The pipeline checks if the current stage skeleton can accommodate the change. If yes, it updates only the affected entries. If the skeleton is no longer valid, it reruns the full organization and synthesis. Unaffected content is reused from a generation cache ℬ — so a small change might only regenerate one L3 entry and its parent L2 summary.
The evaluation compared a baseline agent (no handbook, read-only file exploration) against a handbook-assisted agent (BGPD navigation) on 30 modification requests per harness, balanced across three types:
Both arms used the same planner (DeepSeek-V4-Pro), same repository snapshot, same planning contract, and same decoding settings. The only difference: handbook access.
The handbook-assisted planner made dramatically fewer tool calls, produced more focused plans with less scope bloat, and scored 27% higher on plan quality. The gains were consistent across all three request types, with the largest improvements on Search-Hostile requests — precisely the cases where keyword search fails.
Plan quality was scored by an LLM judge in two steps. First, the judge built a leakage-safe answer key from the request and pristine source — before seeing any plan. Then it scored each plan independently on three dimensions:
Requests were additionally labeled Easy, Medium, or Hard by localization difficulty — from single-behavior changes to discovering dependencies along indirect execution paths.
Most codebase-aware agents today rely on brute-force search. They work on small repos but degrade quickly as codebases grow. Handbook shows that structured documentation — generated from static analysis and auto-maintained — gives agents the same kind of codebase intuition that human developers build over months of working with a codebase.
The approach is also practical. It doesn't require training new models or fine-tuning existing ones. It's a pipeline you can run on any repository, and the resulting handbook is a navigable skill that any agent can use. The auto-resynchronization means the documentation stays fresh without manual effort.
The paper includes full prompt templates in the appendix — the classification, review, file card generation, and document synthesis prompts are all reproduced. This makes it directly implementable.
"Handbook: A Living Documentation Pipeline for AI Codebase Understanding" — Carnegie Mellon University, Stanford University. arXiv:2607.13285