AI-generated code quality and security

Confidence 0.85 · 8 sources · last confirmed 2026-10-25

What AI-written code is actually like once it exists — measured under test conditions and traced in production repositories. The corpus supports three claims here, and the third is the one that changes planning.

1. A large fraction is insecure, and it is not evenly distributed

2025-07-30-veracode-2025-genai-code-security-report: 100+ models, 80 coding tasks, four languages, each task designed so a secure and an insecure completion are both plausible. 45% of samples introduced OWASP Top 10 vulnerabilities.

Language	Failure rate
Java	72%
C#	45%
JavaScript	43%
Python	38%

Nearly a two-fold spread within the same models on the same tasks — most plausibly training-corpus composition, since decades of public Java predate modern secure-coding norms. The risk an agent carries depends on which language it is writing, which is a variable almost no AI policy accounts for. By flaw class, XSS (CWE-80) fails in 86% of relevant samples; models do materially better on SQL injection and weak cryptography — the classes with the longest history of tooling and education, i.e. the ones where the public corpus was already cleaned up.

2. Hallucinated dependencies are a supply-chain attack surface

2025-06-12-spracklen-package-hallucinations-code-generating-llms: 16 models, Python and JavaScript, 2.23 million code samples in the USENIX camera-ready, of which 440,445 (19.7%) contained at least one hallucinated package — 205,474 unique fictional package names. Rates: ≥5.2% commercial models, 21.7% open-source.

The mechanism is what matters. A one-off invented import is a harmless error. The same model inventing the same name repeatedly turns a random error into a predictable one — and a predictable name is one an attacker can register in advance. The authors call it “a novel form of package confusion attack”; the security community’s name is slopsquatting, by analogy with typosquatting, except the developer never made a typo. 205,474 names is the size of the attack surface, and it is freely derivable by anyone who can run the same models.

Mitigations work: the authors show hallucination-reduction strategies that preserve code quality. The operational answer is available today and unglamorous — verify every dependency an agent adds against the registry before it merges.

3. The defects persist — this is debt, not error

2026-03-30-liu-debt-behind-the-ai-boom traces what happens after the code lands. 302.6k verified AI-authored commits across 6,299 repositories, five assistants, with static analysis run before and after each change to attribute issues precisely, then each issue tracked to the latest revision:

• 484,366 distinct issues introduced
• 89.3% are code smells (maintainability)
• >15% of commits from every assistant introduce at least one issue
• 22.7% of AI-introduced issues are still present at the latest revision

One in five defects is never fixed. The 89.3% smell share cuts both ways: reassuring that most of it is maintainability rather than security, unreassuring because maintainability debt has no forcing function — nothing breaks, no test fails, no incident fires. That is the most plausible explanation for the survival rate, and it means the debt concentrates in the category least likely to be paid down voluntarily.

The finding that governs planning

Security is not on the capability curve. Veracode’s models spanned varying sizes, release dates and training sources over several years, and performance was “flat, regardless of model size or training sophistication.” Newer and larger models were “no better.” Spracklen et al. point the same way: the four-fold hallucination gap is between model classes (commercial vs open-source), not along a capability gradient.

Every other measure in this corpus improves with model generation — SWE-bench resolution ran from 1.96% in 2023 to 87.1% in 2026. Security does not move. There is no version of “wait for the next model” that resolves this, because scaling optimises for benchmarks that do not price a vulnerability.

The consequence: security and quality review are a permanent cost of agentic development, not a transitional one. That is DORA’s “fortify your safety nets” capability, Merge Mommy’s data security dimension, and the reason mature repositories decline auto-merge policies.

Related concepts

attack-surface-management, agentic-pull-requests, agentic-engineering, ai-coding-productivity-evidence, responsible-ai, foundation-models, vibe-coding.

Debates and supersession

• Vendor interest. Veracode sells application security testing, and “AI writes insecure code and always will” is good for that business. The flat-scaling finding is corroborated in shape by independent academic work, which is the main reason to take it seriously. Open: an independent replication of the 45% figure.
• Static analysis defines the defect. The 484,366 figure is “issues a static analyser flags” — over-counting style, under-counting logic. And “verified AI-authored” depends on commit attribution, missing AI-assisted code committed under a human’s name.
• Model vintages are 2024–early 2025 across all three studies. The mechanisms should hold; the specific rates will have moved and nobody has re-measured. Open.
• Is the flat security curve a capability limit or an incentive one? Veracode’s data cannot tell whether models are unable to generate secure code or simply were never trained to put it first. Alim argues the latter, on incentive grounds: “secure by generation does not scale financially, but remediations do pay forever.” That is a claim about vendor motives with no pricing evidence behind it. But it predicts something checkable: security would move only if a buyer paid for it at generation time. Open.
• No source here reports a human baseline. 45% insecure and 22.7% unfixed are alarming, but the corpus contains no comparable measurement of human-authored code under the same instruments. This is the single largest interpretive gap in the concept.

Over-generation as a practitioner’s tip, and the commit filter (added 2026-09-16)

The evidence on this page concerns defects. YK Sugi contributes a different failure mode, from the independent-practitioner side: volume. His claude-code-tips tip 38 — simplify overcomplicated code — rests on the observation that code-generation models “have a bias towards writing more code than needed”, with the qualifier that it “really depends on the particular model.”

His remedy is not a prompt technique, and that is what makes it worth recording here:

“don’t necessarily commit all the code that’s generated. Make sure that you only commit and merge the ones that are important and valuable to your code base — and you need to be the decider of that.”

“If you generate 100,000 lines of code, you don’t have to commit 100,000 lines of code… Just commit what you think is right.”

The commit boundary is treated as the quality gate, not the generation step. That is consistent with this page’s finding that AI-introduced defects persist — debt rather than error — since anything that reaches the trunk is what will still be there later. It also reframes the throughput claims the corpus collects: lines generated and lines merged are different quantities, and practitioners who report the first are not reporting the second.

Two caveats on the source. It is vendor co-marketing with zero measurements, and Sugi’s own demo is a greenfield toy game where he explicitly does not read the code — his two-regime split is that casual one-offs need no review (vibe-coding) and “a serious production code base” gets a draft PR and a manual or AI review. The discipline is asserted, not demonstrated.

The view from inside a product-security team (added 2026-09-19)

Alim (PyData, Sep 2026) is the first source here from a large software company’s own security function rather than a benchmark or a repository study. Ammar Alim leads product security engineering at Adobe. He adds three things to the measurements above.

1. The flat curve, confirmed from practice. “We don’t see any improvement in secure code generation,” because models are “trained on good code and bad code” alike. That is the same mechanism Veracode proposes for the Java/Python spread, reported by someone who sees the output every day.

2. The token tax: who pays for the loop. The economic structure the measurements imply but do not state: the same vendors sell generation, then scanning, then AI-driven fixes, each billed in tokens. “The disease and the cure are sold by the same hand … No one paid for the code to be secured at the same moment as generated.” Read together with Liu et al.’s 22.7% never-fixed residue, the loop does not close. His image is “digging a hole in the sand.” This is his argument, not a pricing analysis; see Debates.

3. Guidance competes for the context window, and loses. The mitigation most teams reach for is to tell the agent to write secure code. Alim reports what happens when every function does the same thing:

“The SRE team has some guidance for the agent, the security team has guidance for the agent, the compliance team has guidance for the agent … So the main point of the agent building software is gone, because the agent is now focused on security, compliance, reliability, performance … if you put too much security guidance it dominates the context window and it degrades [other] aspects of the software.”

This is the organisational counterpart to Gloaguen et al.’s finding that repository context files raise cost by more than 20% without improving success, and it gives a cause the paper does not name. The guidance is written by separate teams, each optimising its own concern, so nobody owns the total. His conclusion is that prompt-level mitigation has a ceiling: “we need models that generate as much secure code out of the box, not models that need a lot of babysitting.” If that is right, the permanent cost this page concludes with cannot be engineered away at the prompt layer either, only moved between review and context.

Security tests before the code, and agents’ side quests (added 2026-10-25)

Pettit, Eidelman & Galloway (Google Cloud Tech, Oct 2026) describes how Google builds security into agent-written code as it is written. Aron Eidelman’s loop: a threat model at planning, failing tests that include security edge cases (“if I’m a normal user and not an admin and I try to access this admin endpoint, do I get a 403”), and a skill that names the project’s own helper methods so the agent does not go on “side quests,” rewriting helper libraries in code that “pass[es] all the tests, but… creates more technical debt.” A deterministic hook scans every push (Semgrep and regex rules, for example for hard-coded API keys) before an agent review, and the agent that wrote code is never the one that reviews it. Asked naively to find vulnerabilities, agents return hundreds of findings of which “maybe a few percent” are real; Google’s Mantis uses a threat model to remove the rest.