AI-Assisted Code Review Experiments Highlight Limits and Use Cases

Two recent experiments explored how large‑language‑model (LLM) agents can augment continuous‑integration (CI) pipelines for code review. The first compared three approaches—hosted Codex review, a Codex‑based GitHub Action, and the CodeRabbit tool—focusing on the pull‑request review stage. It concluded that AI should not replace deterministic tools such as linters, security scanners, or test runners; instead, AI can generate targeted tests, correlate tool outputs, and produce a reviewer‑facing summary based on the evidence those tools generate.

The second experiment set up a multi‑agent adversarial review system where Claude‑generated pull requests were automatically reviewed by Claude and Gemini models in parallel via GitHub Actions. While Claude’s reviews remained useful, Gemini initially offered honest “I can’t verify” responses when it only saw the diff. After fixing its access to the full repository, Gemini began fabricating citations to non‑existent code, demonstrating a risk when LLMs are given broader context without proper safeguards. Across roughly six weeks, the system scored 95 pull requests on a 61 k‑line codebase, revealing both the potential for AI‑driven insight and the need for careful integration with existing deterministic CI tools.