AI Coding Echo Report - 2026-10-25

25 new studies · 4 disputed

In this report
Grade	Study	Skeptic	Red flags	Original
Tier B	Developers ready to let AI write application logic: 18% in 2025 → 50% in 2026	—	none	no primary source
Tier B	Survey of 15,000+ developers: ~47% of their code is fully written by AI agents; 90% use agents weekly	—	none	no primary source
Tier B	Gemini 4 Argon on Sonar's Java benchmark: 85.66% pass rate; vulnerabilities are 1.9% of findings but 28% of blocker-level ones	—	no baseline, vendor metric	no primary source
Tier B	GPT-6.1 Sol vs GPT-6 Sol on Sonar's Java benchmark: pass rate 85.29% vs 83.09%, 39.4% more code, total findings +31.7%	—	vendor metric	no primary source
Tier B	Qodo survey of 800 U.S. developers and leaders: 89% of organizations report an AI-related production incident	—	headline overclaim	no primary source
Tier B	Coding agents on SWE-Bench Pro: splitting each issue into 3 messages made runs 2.0–2.6× more expensive per task	—	none	no primary source
Tier A	Qodo A/B test of a GEPA-optimized review prompt: cost per run $0.24 → $0.12, miss rate 7.83% → 4.46%	—	none	no primary source
Tier B	13 LLMs as code reviewers on 703 real bugs: Claude Opus 5.5 leads at 56.8% F1; cost per PR spans more than 100×	—	vendor metric	no primary source
Tier C	CodeRabbit open-source reviews: 1,049,817 in September 2026, up from 421,336 in April	—	vendor metric	no primary source
Tier B	AI security scanners on 100 known vulnerabilities: CodeRabbit 86.7%, Devin 73.3%, Codex Security Plugin 62.0%, Claude Security 40.0%	—	cherry pick, vendor metric	no primary source
Tier C	Claude Sonnet 5.5 vs Sonnet 5 in CodeRabbit review: 6 vs 4 of 13 hard bugs caught, reviews about twice as fast	Tier B	headline overclaim	no primary source
Tier A	Cursor harness changes, A/B tested: user token costs −7%; dynamic MCP tool loading −46.9% tokens	—	vendor metric	no primary source
Tier B	Survey of 238 GitHub and Microsoft developers: perceived output quality and usefulness drive trust in generative AI	—	correlation causation, selection bias	not checked
Tier B	Microsoft study of 860 developers: AI is wanted for coding, testing and toil, much less for mentoring	—	none	not checked
Tier B	SWE-Lancer benchmark of over 1,400 real Upwork tasks worth $1M: frontier models fail most tasks	—	none	not checked
Tier B	Survey of 791 Microsoft developers maps what they want from AI and their worries about its reliability	—	none	not checked
Tier B	Survey of 410 developers: AI assistants are used to save keystrokes and time, and avoided when output misses requirements	—	none	not checked
Tier B	Copilot on past human-introduced C/C++ vulnerabilities: repeated the vulnerable code about 33% of the time, the fix 25%	—	headline overclaim, selection bias	faithful
Tier A	Controlled Copilot experiment: developers finished an HTTP-server task 55.8% faster than the control group	—	none	faithful
Tier D	One Sonar developer's time with coding agents: writing/reviewing split moved from 90/10 to about 60/40	—	anecdote generalisation, confounded window	no primary source
Tier C	Claude Opus 5.5 vs CodeRabbit's production reviewer: 51 vs 49 of 80 open-source bugs caught, with 49% more tokens	Tier B	anecdote generalisation	no primary source
Tier C	Bug-fixing agent at a 4-engineer startup: about 70% of reported issues fixed within 30 minutes	—	no baseline	no primary source
Tier B	Cursor Router on live traffic: Fable-level user satisfaction at 68% lower cost	—	vendor metric	no primary source
Tier B	Microsoft survey of 500+ developers: AI is seen as raising productivity on routine tasks, with little effect on collaboration	—	none	not checked
Tier C	LLM-assisted code migrations at Google: 74.45% of 595 changes AI-generated; developers estimate 50% less migration time	—	headline overclaim	not checked
Tier C	Adding Jev to Qodo code review across 24 cases: 1 extra required test gap, 0 additional confirmed defects	A	selection bias	no primary source
Tier A	Controlled user study: developers with an AI assistant wrote less secure code and were more confident it was secure	A	none	not checked
Tier C	GitHub Copilot vs human programmers: humans' solutions are correct more often, but Copilot's bugs are easier to fix	D	none	not checked
Tier D	Early Copilot study reports developers spend more time reviewing suggestions than writing code (no numbers given)	D	no baseline	not checked

How to read this table

Tier A Tier B Tier C Tier D strongest to weakest evidenceTier ? ungradedSkeptic: the blind second grade, — when it agrees≠disputed#numbers not found in the original∅original not reachable

New studies

Tier Bvendor

Developers ready to let AI write application logic: 18% in 2025 → 50% in 2026

JetBrains' Developer Ecosystem and AI Pulse surveys show readiness to delegate application-logic code to AI rising from 18% (April–June 2025) to 50% a year later. Readiness for test writing rose from 50% to 75%, and 79% expect employers to require AI-tool proficiency (59% in 2024). The surveys are reweighted for global representation, but this post gives no sample size and compares different survey waves. JetBrains sells AI coding tools.

Conflict of interest: JetBrains sells AI coding tools and runs the survey.

Measured: Adoption & usage share · share of developers ready to delegate tasks to AI

Baseline: 18% in April–June 2025

Research: June 2024 - July 2026

Part of: JetBrains Developer Ecosystem Survey (editions 2024, 2025, 2026)

Subtopic: Adoption & usage

Question: What share of developers are ready to delegate coding tasks to AI?

Published: 2026-10-19

Coverage: JetBrains Research

Tier Bvendor

Survey of 15,000+ developers: ~47% of their code is fully written by AI agents; 90% use agents weekly

JetBrains' Developer Ecosystem Survey 2026 (May–July 2026, over 15,000 professional developers, reweighted) finds developers report that about 47% of their code is fully agent-written, 38% AI-assisted and 27% manual (the figures are bucket midpoints, so they can exceed 100%). About 22% get over 80% of their code from agents. From the same survey, 90% use AI coding agents at least weekly and Claude Code adoption at work rose from 18% in January to 39%. All figures are self-reported, and JetBrains sells competing AI tools.

Conflict of interest: JetBrains sells AI coding tools and runs the survey.

Measured: Adoption & usage share · share of code generated by agents

Baseline: January 2026 AI Pulse figures

Research: May–July 2026

Part of: JetBrains Developer Ecosystem Survey (editions 2026)

Subtopic: Adoption & usage

Question: What share of their code do developers let AI agents write? · 1 study · 2 articles

About: Claude Code, Codex, Cursor

Published: 2026-08-26

Coverage: JetBrains Research · JetBrains Research

Tier Bvendor

Gemini 4 Argon on Sonar's Java benchmark: 85.66% pass rate; vulnerabilities are 1.9% of findings but 28% of blocker-level ones

Sonar ran Gemini 4 Argon (high reasoning effort) on 4,444 Java tasks from HumanEval, MBPP and ComplexCodeEval and analysed the output with SonarQube. The model passed 85.66% of the 544 test-backed tasks. The 619,852 generated lines produced 10,166 findings (9,626 code smells, 347 bugs, 193 vulnerabilities), and 1.55% of tasks returned no usable code. This was a single run scored with the vendor's own analyzer, with no comparison model, and the run date is not stated.

Red flags: no baseline, vendor metric

Conflict of interest: Sonar sells SonarQube, the analyzer used, and concludes by recommending it.

Measured: Benchmark score · functional pass rate and issue density

Research: unstated

Part of: Sonar LLM evaluation

Subtopic: Benchmarks & evaluations

Question: What pass rate do LLMs reach on Sonar's Java coding benchmark? · 1 study · 2 articles

About: Gemini 4 Argon, SonarQube

Published: 2026-10-20

Coverage: Sonar Blog

Tier Bvendor

GPT-6.1 Sol vs GPT-6 Sol on Sonar's Java benchmark: pass rate 85.29% vs 83.09%, 39.4% more code, total findings +31.7%

Sonar compared GPT-6.1 Sol with GPT-6 Sol (both at medium effort) on the same 4,444-task Java benchmark. The pass rate rose 2.20 points and every per-line density fell (vulnerability density −32.4%). But the model wrote 39.4% more code, so total findings grew from 9,859 to 12,982; only the vulnerability count fell (137 to 129). These are single runs scored by Sonar's own analyzer, with no variance reported, and the run date is not stated.

Red flags: vendor metric

Conflict of interest: Sonar sells SonarQube, the analyzer used for the evaluation.

Measured: Benchmark score · functional pass rate and issue density

Baseline: GPT-6 Sol: 83.09% pass rate, 9,859 findings

Research: unstated

Part of: Sonar LLM evaluation

Subtopic: Benchmarks & evaluations

Question: What pass rate do LLMs reach on Sonar's Java coding benchmark? · 1 study · 2 articles

About: GPT-6.1 Sol, GPT-6 Sol, SonarQube

Published: 2026-10-09

Coverage: Sonar Blog

Tier Bvendor

Qodo survey of 800 U.S. developers and leaders: 89% of organizations report an AI-related production incident

Qodo's 2026 State of AI Code Quality Report, a Censuswide survey of 500 U.S. developers and 300 U.S. engineering leaders, finds 89% of organizations report an AI-related production incident. Only 3.7% of leaders say their current processes are enough. Reviewing AI code was the top named constraint, but only for 26% of each group, and 35% of developers say agents always follow standards. These are self-reported answers, the fieldwork dates are unstated, and the report promotes Qodo's own platform.

Red flags: headline overclaim

Conflict of interest: Qodo sells an AI code quality and governance platform that the report promotes.

Measured: Defect & bug rate · share reporting AI-related production incidents

Research: unstated (2026)

Part of: State of AI Code Quality Report (editions 2026)

Subtopic: Code quality & maintainability

Question: What share of organizations report AI-related production incidents?

Published: 2026-09-23

Coverage: Qodo Blog

Tier Bindependent

Coding agents on SWE-Bench Pro: splitting each issue into 3 messages made runs 2.0–2.6× more expensive per task

JetBrains Research analysed 4,782 agent sessions in JetBrains IDEs and found bug fixes and features make up under half of developer requests, against 93.8% of SWE-Bench Pro tasks. In a pilot on 700 rewritten SWE-Bench Pro tasks, with four models and three runs per setup, delivering the requirements over three messages raised cost per task 2.0–2.6×, while resolve rates moved only 0–5 points. A Concat control suggests the extra cost comes from the sequential delivery rather than the rewording. The data collection dates are not stated.

Conflict of interest: JetBrains sells IDE coding agents, but the study evaluates third-party models on a public benchmark.

Measured: Cost per task · cost per task and resolve rate

Baseline: Single-message SWE-Bench Pro issue

Research: unstated

Subtopic: Benchmarks & evaluations

Question: Do coding agents cost more per task when requirements arrive across several messages?

About: SWE-Bench Pro, GPT-5.6 Terra, GPT-5.4 Mini, Gemini 3.7 Flash

Published: 2026-10-20

Coverage: JetBrains Research

Tier Avendor

Qodo A/B test of a GEPA-optimized review prompt: cost per run $0.24 → $0.12, miss rate 7.83% → 4.46%

Qodo ran a 50/50 production A/B test, assigned by pull request across all customers, that changed only the system prompt of its code-review recommendation agent. Over 27,582 review runs, mean cost per run halved and mean latency fell from 40 to 30 seconds, with no statistically significant change in precision. The miss rate on implemented findings dropped from 7.83% to 4.46%, measured on the subset of PRs with attribution labels. Qodo sells the product tested, and the test dates are not stated.

Conflict of interest: Qodo sells the code review product being optimized.

Measured: Cost per task · miss rate and cost per review run

Baseline: existing production prompt: 7.83% miss rate, $0.24 per run

Research: unstated (2026)

Subtopic: AI code review

Question: Does an optimized system prompt cut AI code review cost?

About: Qodo, GEPA

Published: 2026-10-22

Coverage: Qodo Blog

Tier Bvendor

13 LLMs as code reviewers on 703 real bugs: Claude Opus 5.5 leads at 56.8% F1; cost per PR spans more than 100×

Qodo ran 38 model and reasoning-level configurations on its proprietary benchmark of 125 pull requests containing 703 known bugs traced back from later fixes, scored by an LLM judge (September–October 2026). The top four models (Claude Opus 5.5, Sonnet 5.5, Fable 5.1, GLM 5.3) fall within 4.7 F1 points. Cost ranged from about $0.02 to $3.63 per PR. Some configurations ran only once, the post's figures for Sonnet 5.5 are not fully consistent, and Qodo sells an AI code review product.

Red flags: vendor metric

Conflict of interest: Qodo sells an AI code review product and uses a proprietary benchmark.

Measured: Benchmark score · F1 of review findings vs known bugs

Research: September – October 2026

Subtopic: AI code review

Question: Which LLMs score best as AI code reviewers on known bugs?

About: Claude Opus 5.5, Claude Sonnet 5.5, GLM 5.3, GPT-6 Sol, Gemini 3.8 Flash

Published: 2026-10-12

Coverage: Qodo Blog

Tier Cplatform · judge: B

CodeRabbit open-source reviews: 1,049,817 in September 2026, up from 421,336 in April

CodeRabbit reports completing 1,049,817 reviews on open-source repositories in September 2026, 14.4% more than in August, up from 421,336 in April. The counts include repeat reviews of the same pull request. This is the vendor's own usage figure and says nothing about the effect of the reviews.

Red flags: vendor metric

Conflict of interest: CodeRabbit reports usage of its own product.

Measured: Adoption & usage share · completed AI reviews per month

Baseline: 917,439 in August

Research: April – September 2026

Subtopic: Adoption & usage

Question: How many AI code reviews run on open-source repositories per month?

About: CodeRabbit

Published: 2026-10-08

Coverage: CodeRabbit Blog

Tier Bvendor

AI security scanners on 100 known vulnerabilities: CodeRabbit 86.7%, Devin 73.3%, Codex Security Plugin 62.0%, Claude Security 40.0%

CodeRabbit tested its own security agent and three other systems on 100 known vulnerabilities from 94 open-source repositories in 11 languages. Each system got the vulnerable code with the advisory, patch and identifiers withheld, and the scan ran offline. The figures come from CodeRabbit's internal development slice, scored by CodeRabbit, which also sells the top-ranked product. Training-data exposure to these public vulnerabilities is acknowledged as a limit, and the run dates are not stated.

Red flags: cherry pick, vendor metric

Conflict of interest: CodeRabbit sells the top-ranked product in its own internal benchmark.

Measured: Benchmark score · vulnerability detection rate

Research: unstated (2026)

Subtopic: Security of AI-generated code

Question: Can AI security reviewers detect known vulnerabilities in real repositories?

About: CodeRabbit Security, Devin, Codex Security Plugin, Claude Security

Published: 2026-10-06

Coverage: CodeRabbit Blog

Tier Cvendor · skeptic: B

Claude Sonnet 5.5 vs Sonnet 5 in CodeRabbit review: 6 vs 4 of 13 hard bugs caught, reviews about twice as fast

CodeRabbit replayed identical recorded inputs through its review pipeline in September 2026. On 13 hard known-bug cases, Sonnet 5.5 caught 6 bugs against Sonnet 5's 4, at similar actionable precision (41.2% vs 40.0%). On 44 open-source PRs, it averaged 6:33 per review against 13:31, posted 24% fewer comments, and cost about $0.46 per review against $1.16 in Claude calls. The catch comparison rests on only 13 cases, judge scoring of the 44-PR set was still pending, and CodeRabbit sells the pipeline.

Red flags: headline overclaim

Conflict of interest: CodeRabbit sells the review pipeline the models were tested in.

Measured: Benchmark score · known issues caught and time per review

Baseline: Sonnet 5: 4/13 caught, 13:31 per review

Research: September 2026

Subtopic: AI code review

Question: Does a newer LLM catch more known bugs as a code reviewer than the previous model? · 1 study · 2 articles

About: Claude Sonnet 5.5, Claude Sonnet 5, CodeRabbit

Published: 2026-09-28

Coverage: CodeRabbit Blog

Tier Aplatform

Cursor harness changes, A/B tested: user token costs −7%; dynamic MCP tool loading −46.9% tokens

Cursor reports that harness changes it A/B tested on its user base over recent months cut users' token costs by 7% without lowering agent quality. The changes were a trimmed system prompt (about 66% shorter), tools loaded on demand, cache breakpoints (20% fewer cold cache misses) and sparser line numbers. Loading MCP tools dynamically had earlier cut total tokens by 46.9% in sessions that called one. Sample sizes are not given, and 'quality' is Cursor's own internal measure of its own product.

Red flags: vendor metric

Conflict of interest: Cursor measured its own product.

Measured: Cost per task · token cost per session

Baseline: previous harness

Research: mid 2026 (past few months before Sep 2026)

Subtopic: Cost & efficiency

Question: Does trimming an agent harness's context cut token cost?

About: Cursor

Published: 2026-09-23

Coverage: Cursor Blog

Tier Bindependent

Survey of 238 GitHub and Microsoft developers: perceived output quality and usefulness drive trust in generative AI

Researchers surveyed 238 developers at GitHub and Microsoft and modelled the answers with PLS-SEM. Perceived system and output quality, functional value and goal maintenance were linked to trust, which together with cognitive styles was linked to adoption. The data is correlational from two companies, the summary gives no effect sizes, and the survey date is unstated (DX's research archive lists the paper under November 2025).

Red flags: correlation causation, selection bias

Original: not checked (could not be fetched)

Measured: Developer satisfaction & trust · trust and adoption intention

Research: unstated

Subtopic: Developer experience & trust

Question: Does perceived AI output quality go with higher developer trust in generative AI?

Primary source: https://arxiv.org/pdf/2505.17418

Coverage: DX Research

Tier Bindependent

Microsoft study of 860 developers: AI is wanted for coding, testing and toil, much less for mentoring

Researchers at Microsoft ran a mixed-methods study of 860 developers. How developers appraise a task predicted how open they were to AI for it and how much they used it: strong use in coding and testing, high demand to reduce toil such as documentation, and clear limits for mentoring. The summary gives no effect sizes, and the data date is unstated (DX's research archive lists the paper under October 2025).

Original: not checked (could not be fetched)

Measured: Adoption & usage share · AI use and openness by task

Research: unstated

Subtopic: Adoption & usage

Question: Which development tasks do developers want AI support for?

Primary source: https://www.microsoft.com/en-us/research/publication/ai-where-it-matters-where-why-and-how-developers-want-ai-support-in-daily-work/

Coverage: DX Research

Tier Bvendor

SWE-Lancer benchmark of over 1,400 real Upwork tasks worth $1M: frontier models fail most tasks

OpenAI researchers built SWE-Lancer from over 1,400 freelance tasks posted on Upwork, worth $1 million in payouts. Coding tasks are graded with end-to-end tests that experienced engineers verified three times; managerial tasks are graded against the original hiring managers' decisions. Frontier models could not solve most of the tasks, but this summary gives no per-model scores. The benchmark's authors work for a model vendor, and DX's research archive lists the paper under February 2025.

Conflict of interest: OpenAI sells frontier models evaluated on the benchmark.

Original: not checked (could not be fetched)

Measured: Task success rate · task success rate

Research: unstated

Subtopic: Benchmarks & evaluations

Question: Can frontier LLMs solve real-world freelance software engineering tasks?

About: SWE-Lancer

Primary source: https://arxiv.org/pdf/2502.12115

Coverage: DX Research

Tier Bindependent

Survey of 791 Microsoft developers maps what they want from AI and their worries about its reliability

Researchers surveyed 791 Microsoft developers on where they want AI help and what concerns them, finding key areas for productivity gains alongside serious worries about AI's practicality and reliability. The sample comes from one company, the summary reports no figures beyond sample size, and the survey date is unstated (DX's research archive lists the paper under July 2024).

Original: not checked (could not be fetched)

Measured: Developer satisfaction & trust · developer desires and concerns

Research: unstated

Subtopic: Developer experience & trust

Question: What do developers most want from AI assistance, and what worries them?

Primary source: https://www.microsoft.com/en-us/research/publication/towards-effective-ai-support-for-developers-a-survey-of-desires-and-concerns/

Coverage: DX Research

Tier Bindependent

Survey of 410 developers: AI assistants are used to save keystrokes and time, and avoided when output misses requirements

Researchers surveyed 410 developers about AI programming assistants such as GitHub Copilot. The main reasons for using them were fewer keystrokes, faster task completion and recalling syntax. The main reasons for not using them were output that misses functional or non-functional requirements and difficulty steering the tool. The summary gives no percentages, and the survey date is unstated (DX's research archive lists the paper under February 2024).

Original: not checked (could not be fetched)

Measured: Developer satisfaction & trust · reasons for use and non-use

Research: unstated

Subtopic: Code completion & chat assistants

Question: Why do developers use or avoid AI programming assistants?

About: GitHub Copilot

Primary source: https://dl.acm.org/doi/abs/10.1145/3597503.3608128

Coverage: DX Research

Tier Bindependent

Copilot on past human-introduced C/C++ vulnerabilities: repeated the vulnerable code about 33% of the time, the fix 25%

Researchers prompted GitHub Copilot with C/C++ scenarios in which human developers had introduced a vulnerability. Copilot reproduced the vulnerable code about 33% of the time and the fixed code 25% of the time, with rates varying by vulnerability type. The authors conclude Copilot is 'not as bad as humans', though the sample was drawn only from cases where humans had failed. The study date is unstated (DX's research archive lists it under September 2023).

Red flags: headline overclaim, selection bias

Measured: Vulnerability rate · rate of reintroducing vulnerabilities

Baseline: original human-introduced vulnerabilities

Research: unstated

Subtopic: Security of AI-generated code

Question: Does AI-generated or AI-assisted code contain more vulnerabilities than human-written code? · ⇅ contested (2 independent studies)

About: GitHub Copilot

Primary source: https://link.springer.com/article/10.1007/s10664-023-10380-1

Coverage: DX Research

Tier Avendor

Controlled Copilot experiment: developers finished an HTTP-server task 55.8% faster than the control group

Researchers ran a controlled experiment in which recruited developers implemented an HTTP server in JavaScript as fast as possible. The group with GitHub Copilot finished 55.8% faster than the control group. It tested a single task, the sample size is not given in this summary, GitHub co-authored the study of its own product, and the paper dates from February 2023. The figure matches the original paper.

Conflict of interest: GitHub co-authored the study of its own product.

Measured: Task completion time · task completion time

Baseline: control group without Copilot

Research: unstated (published Feb 2023)

Subtopic: Developer productivity

Question: Do AI coding tools cut the time developers need to complete a task? · ✓ consistent (2 independent studies) · corroborated by 1 other independent study

About: GitHub Copilot

Primary source: https://arxiv.org/abs/2302.06590

Coverage: DX Research

Tier Dvendor

One Sonar developer's time with coding agents: writing/reviewing split moved from 90/10 to about 60/40

In a Sonar podcast recap, a front-end developer reports that after about a year of moving to Claude Code with layered SonarQube and Gitar review, their time split shifted from about 90% writing and 10% reviewing to roughly 60/40. Their pull requests went from about 80/20 code/tests to 40/60. This is a single-person anecdote with no controls, and the developer's squad doubled from four to eight people over the same period. Sonar sells SonarQube.

Red flags: anecdote generalisation, confounded window

Conflict of interest: Sonar sells SonarQube, which is part of the described workflow.

Measured: share of time writing vs reviewing

Baseline: 90% writing, 10% reviewing before AI

Research: about one year up to 2026

Subtopic: Developer productivity

Question: Does agentic coding shift developer time from writing code to reviewing it?

About: Claude Code, SonarQube, Gitar

Published: 2026-10-02

Coverage: Sonar Blog

Tier Cvendor · skeptic: B

Claude Opus 5.5 vs CodeRabbit's production reviewer: 51 vs 49 of 80 open-source bugs caught, with 49% more tokens

CodeRabbit compared two Opus 5.5 configurations with its production model mix in September 2026. On 80 known bug patterns, the Standard setup caught 63.8% against the baseline's 61.3%, at similar precision. On 13 harder cases, the Max setup caught 10 against 5. Reported token usage rose about 41–60%, and each reviewer missed bugs the other caught. The sets are small, the verdict also leans on informal coding demos, and CodeRabbit sells the pipeline.

Red flags: anecdote generalisation

Conflict of interest: CodeRabbit sells the review pipeline the model was tested in.

Measured: Benchmark score · actionable recall of known bugs

Baseline: production baseline 49/80 (61.3%)

Research: September 2026

Subtopic: AI code review

Question: Does a newer LLM catch more known bugs as a code reviewer than the previous model? · 1 study · 2 articles

About: Claude Opus 5.5, CodeRabbit

Published: 2026-09-22

Coverage: CodeRabbit Blog

Tier Cindependent

Bug-fixing agent at a 4-engineer startup: about 70% of reported issues fixed within 30 minutes

Swarmia reports a demo by the CTO of Taito.ai, an eight-person company with four engineers. After introducing a bug-fixing agent, the reported-bug backlog reached zero within a week, and about 70% of reported issues now get a fix within half an hour. The share of pull requests flagged for human review fell from about 90% to 30% after the review rules changed. This is one small company's self-reported case, with no controls and no stated dates.

Red flags: no baseline

Conflict of interest: Swarmia sells engineering analytics, not the tools used; the subject is a former Swarmia engineer.

Measured: Task completion time · time to fix reported issues

Baseline: growing bug backlog before the agent

Research: unstated (2026)

Subtopic: Coding agents

Question: Does a bug-fixing agent shorten the time to fix reported issues?

Published: 2026-09-21

Coverage: Swarmia Blog

Tier Bplatform

Cursor Router on live traffic: Fable-level user satisfaction at 68% lower cost

Cursor reports that its model router, trained on hundreds of thousands of live turns and checked on a held-out set and then on live traffic (July–August 2026), delivers above-Fable user satisfaction at 68% lower cost in Auto Intelligence mode. In Auto Balance mode it beats Opus 4.8 at 41% lower cost. Satisfaction is Cursor's own metric, inferred from what the user does next, and the live comparison design is not fully described.

Red flags: vendor metric

Conflict of interest: Cursor measured its own routing product.

Measured: Cost per task · user satisfaction and cost per turn

Baseline: single frontier models (Fable, Opus 4.8)

Research: July – August 2026

Subtopic: Cost & efficiency

Question: Does routing turns between models cut cost per turn vs one frontier model?

About: Cursor Router, Claude Opus 4.8, Claude Fable

Published: 2026-08-06

Coverage: Cursor Blog

Tier Bplatform

Microsoft survey of 500+ developers: AI is seen as raising productivity on routine tasks, with little effect on collaboration

Researchers at Microsoft surveyed over 500 developers and added interviews and observation, framed by the SPACE dimensions. Developers reported higher efficiency and satisfaction with AI, mostly on routine tasks, and less evidence of impact on collaboration. These are self-reported perceptions, not measured output, and the data date is unstated (DX's research archive lists the paper under August 2025).

Conflict of interest: Microsoft builds AI developer tools (GitHub Copilot).

Original: not checked (could not be fetched)

Measured: Developer satisfaction & trust · perceived productivity and satisfaction

Research: unstated

Subtopic: Developer productivity

Question: How do developers rate AI's effect on their productivity and satisfaction?

Primary source: https://www.microsoft.com/en-us/research/publication/the-space-of-ai-real-world-lessons-on-ais-impact-on-developers/

Coverage: DX Research

Tier Cplatform

LLM-assisted code migrations at Google: 74.45% of 595 changes AI-generated; developers estimate 50% less migration time

Researchers describe a year-long Google case study: 39 migrations by three developers produced 595 code changes (93,574 edits). The LLM generated 74.45% of the changes and 69.46% of the edits. The 50% time reduction is the developers' own estimate against earlier manual migrations, not a controlled measurement, and the exact dates are not given (DX's research archive lists the paper under April 2025).

Red flags: headline overclaim

Conflict of interest: Google evaluated its own internal tooling.

Original: not checked (could not be fetched)

Measured: Task completion time · share of code changes generated by LLM; migration time

Baseline: prior manual migration efforts

Research: twelve months (year-long case study)

Subtopic: Developer productivity

Question: Do AI coding tools cut the time developers need to complete a task? · ✓ consistent (2 independent studies) · corroborated by 1 other independent study

About: Google

Primary source: https://arxiv.org/pdf/2504.09691

Coverage: DX Research

Disputed (verdict and skeptic disagree; both grades shown)

Tier Cskeptic: A · vendor

disputed

Adding Jev to Qodo code review across 24 cases: 1 extra required test gap, 0 additional confirmed defects

A Qodo blog author compared workflows (selected checks, plus Qodo, plus Jev, and all three) on 24 evaluation cases with independently established labels. Jev added one required test gap and no additional confirmed defects, and raised three unsupported concerns; fix requests rose from 18 to 21. All cases had been seen before and the same analyst built and adjudicated them. The run dates are not stated, and the grades were disputed (judge C, skeptic A).

Disputed: grades two tiers apart.
Skeptic's doubts: A small comparison across 24 cases with arms for checks, checks+Qodo, checks+Jev and all three found one extra required test gap and three unsupported concerns, but all cases had been seen before and the same analyst built and adjudicated them, as the author notes.

Red flags: selection bias

Conflict of interest: Published by Qodo, whose code review product is one arm of the comparison.

Measured: Defect & bug rate · required gaps and defects found

Baseline: checks plus Qodo without Jev

Research: unstated (2026)

Subtopic: AI code review

Question: Does adding a second AI check to code review find more defects?

About: Jev, Qodo

Published: 2026-10-05

Coverage: Qodo Blog

Tier Aindependent

disputed

Controlled user study: developers with an AI assistant wrote less secure code and were more confident it was secure

Researchers ran a user study on security-related coding tasks. Participants with access to an AI code assistant wrote significantly less secure code than those without, and were more likely to believe their code was secure. The summary states neither the sample size nor effect sizes, and the study date is unstated (DX's research archive lists the paper under November 2023). The item is disputed because no quoted number was found in the text.

Disputed: no quoted number found in the text.
Skeptic's doubts: User study comparing participants with and without an AI assistant, finding the assistant group wrote significantly less secure code and was more overconfident; the summary does not give sample size.

Original: not checked (could not be fetched)

Measured: Vulnerability rate · code security

Research: unstated

Subtopic: Security of AI-generated code

Question: Does AI-generated or AI-assisted code contain more vulnerabilities than human-written code? · ⇅ contested (2 independent studies)

Primary source: https://dl.acm.org/doi/abs/10.1145/3576915.3623157

Coverage: DX Research

Tier Cindependent

disputed

GitHub Copilot vs human programmers: humans' solutions are correct more often, but Copilot's bugs are easier to fix

Researchers report that Copilot produced solutions for almost all fundamental algorithmic problems, though some were buggy or could not be reproduced. On programming tasks compared with human solutions, humans had a higher share of correct answers, while Copilot's buggy solutions needed less effort to repair. The summary gives no numbers or scale, the study date is unstated (DX's research archive lists it under September 2023), and the grades were disputed (judge C, skeptic D).

Disputed: no quoted number found in the text.
Skeptic's doubts: Compares Copilot's solutions with human ones on algorithm and programming tasks, but the summary gives no scale or numbers, only directional findings.

Original: not checked (could not be fetched)

Measured: Task success rate · correctness of solutions

Baseline: human programmers solutions

Research: unstated

Subtopic: Code completion & chat assistants

Question: Are Copilot's solutions correct as often as human programmers' solutions?

About: GitHub Copilot

Primary source: https://www.sciencedirect.com/science/article/abs/pii/S0164121223001292

Coverage: DX Research

Tier Dvendor

disputed

Early Copilot study reports developers spend more time reviewing suggestions than writing code (no numbers given)

Researchers at GitHub and Microsoft report that developers using Copilot spent more time reviewing code than writing it, and expect developer roles to shift toward assessing AI suggestions. The text gives no numbers, method details or baseline. The paper dates from December 2022, and the item is disputed because no quoted number was found.

Disputed: no quoted number found in the text.
Skeptic's doubts: The claim that developers spend more time reviewing than writing code comes with no numbers, method details or comparison baseline in the text.

Red flags: no baseline

Conflict of interest: GitHub/Microsoft study their own product.

Original: not checked (could not be fetched)

Measured: time spent reviewing vs writing

Research: unstated

Subtopic: Code completion & chat assistants

Question: How much of their time do Copilot users spend reviewing vs writing code?

About: GitHub Copilot

Primary source: https://dl.acm.org/doi/abs/10.1145/3582083

Coverage: DX Research