Summary explanation: Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review

Contents

• Summary
• 4. Task definition
• 5. Method
  • 5.2. Refinement cycles
  • 5.3. Implementation
  • 5.4. Feedback loop
  • 5.5. Verification cycles
• 6. Results
  • 6.3. Functional validation
• 7. Discussion
• 9. Limitations
• Appendix
• Brief Thoughts

This article explains the key points of Specification-first convergence with an AI coding agent: a case study of dismantling a core architectural invariant across 189 files in a 717k-line codebase with no test oracle and no human code review.

• 2026-08-12 (arXiv)
• Abenhaim, Joel.
• AI Sovereign Labs, Paris, France
• Paper
• Project Page

Read this article in Korean

Summary

• This single-case study reports a specification-first refactor of a proprietary 717,725-line TypeScript VS Code extension, performed without human review of generated patches and without a pre-existing test oracle for the new behavior. The task removed the invariant that an AI-request panel remains open throughout a request: after a panel is closed and reopened, it must reattach to the same live stream without losing or duplicating content. In contrast to coding-agent benchmarks based on held-out tests and to loop-engineering workflows with independently checkable work units, the paper constructs and refines a written specification as the reference for a previously unavailable behavior. For pre-existing test oracle, see this post.

4. Task definition

The initial natural-language request required extraction and generalization of streaming logic before applying it to standard AI-code sessions. A closed panel must leave generation running and visible in the activity sidebar; while detached and running, it exposes a stop control that becomes disabled after clicking. Reopening must reconstruct response text, reasoning traces, tool calls, and code blocks while coordinating replay with newly streamed tokens.

The screenshot identifies the two interface surfaces affected by the refactor: the activity sidebar retains the session entry, while the central session panel can be closed and later restored.

5. Method

AICode, using OpenAI ChatGPT 5.6 Sol in max reasoning mode, followed five separately sessioned phases: ideate, specify, refine, code, and verify. Refinement audited the specification against source code; implementation targeted the frozen specification; and verification audited code against that fixed reference. The empirical stopping rule was two consecutive verification cycles with zero findings.

The diagram presents the ideate-to-verify workflow and its two audit loops. Refinement compares the specification with source code, whereas verification compares generated code with the frozen specification.

The table specifies the artifact, comparison reference, and operator decision for each phase. It records that code patches were applied without inspection and that verification stops after two consecutive zero-finding passes.

5.2. Refinement cycles

Fourteen refinement cycles expanded the planned scope from 110 to 160 files and produced approximately 85 specification corrections. Several correction counts are estimates of five because a platform-side migration issue prevented recovery of their lists. Cycle 14 reported zero findings, so the cycle-13 specification was frozen.

The table records 14 refinement sessions, with scope increasing from 120 files in cycle 1 to 160 files by cycle 13 and no findings in cycle 14. Several correction counts are explicitly approximate because their original correction lists were unavailable.

5.3. Implementation

The agent initially declined to make a partial implementation, stating that a migration spanning 160 files, including 31 new files, could not safely be left incomplete. It proposed a ten-step decomposition, but implementation proceeded in three human-confirmed segments. The reported agent time was 2 hours 21 minutes 44 seconds.

5.4. Feedback loop

Compilation, typing, and unit-test failures were deliberately handled in a separate feedback-loop session rather than automatically corrected during implementation. The paper reports that residual syntax, typing, and unit-test errors were corrected in under one hour. This loop was intended to preserve build-validity signals separately from later audits of architectural conformity.

5.5. Verification cycles

Seventeen verification cycles compared the repository with the frozen specification and produced 116 reported code corrections. Unlike the feedback loop, these passes targeted architectural deviations and subtle specification mismatches. Cycles 16 and 17 each returned zero findings, after which the program was manually executed for the first time.

The verification record shows corrections continuing through cycle 15, including 21 in cycle 4 and 10 in cycle 15, before cycles 16 and 17 each produced zero findings. Its reported total is 116 code corrections.

6. Results

Across 31 refinement and verification passes, the study reports 201 corrections before manual execution: approximately 85 specification corrections and 116 code corrections. Slice 2 touched 189 files in three days; together with the extraction slice, two commits changed 288 files with 34,770 insertions and 16,422 deletions. The reported model-inference cost was USD 2,430, and a post-release interface adjustment reportedly required one source-file change plus its tests.

6.3. Functional validation

On first manual execution, the author reports correct panel close/reopen behavior during streaming, display restoration, the detached stop control, and no observed regression in the pre-existing subsystem. The pre-existing unit-test suite reportedly showed no regression, and roughly thirty subsequent sessions across use cases produced no observed bug. Version 2.3.0 was publicly released, allowing users to inspect the claimed behavior, although not the closed-source implementation or process.

The activity-sidebar screenshot illustrates the intended detached state: a session remains marked as running after its panel is closed, displays elapsed time, and exposes the stop control.

The bar chart reports inference spending over the operation and supports the stated rounded cost of USD 2,430. The interface display shows a total of $2,430.83.

7. Discussion

The paper argues that repeatedly auditing intent before generation and auditing implementation afterward can make a frozen written specification the primary quality-control reference instead of line-by-line human review. Its claim is narrow: the operation concerns an original maintenance task without a pre-existing oracle for the new behavior, rather than a greenfield project, transpilation, documented migration, or open-source reimplementation. The reported zero-finding condition establishes convergence under this protocol, not independent proof of functional correctness or general reliability.

9. Limitations

The evidence is limited to one self-reported operation by the author, whose company designed and distributes the agent, with no control condition using other agents. The closed-source codebase prevents third-party replay; “no bug observed” is explicitly not proof of no latent defects, and performance with weaker models is uncharacterized. Raw French logs and the 55-page frozen specification are published, but the code itself is not.

Appendix

• No appendix section is provided in the paper. The data-availability section links the frozen specification and logs for ideation, refinement, implementation, the feedback loop, and verification, totaling more than 1,500 pages.

Brief Thoughts

The report provides unusually detailed process artifacts for a large agent-mediated refactor and usefully separates specification conformity from compilation and test validity. However, the same agent family produced, refined, and verified the reference artifact, while the repository remains unavailable for replay. Replication by independent operators on public codebases, including comparisons with alternative agents and review protocols, is needed to assess reliability beyond this case.