Skip to content

What AI changes and what it does not

~7 min

In short

AI assistants change how some programming tasks are done, and the measured effects differ from task to task. Some studies found faster work. One found slower work. Assistants also produce wrong answers in ways that are hard to spot. This page lists what the published evidence supports, says where it is thin, and does not predict what will happen to jobs.

What the evidence shows

Some tasks got faster. In a 2023 randomized experiment, developers who had GitHub Copilot finished a task of writing a small HTTP server in JavaScript about 56% faster than the control group (arXiv:2302.06590). The task was short and self-contained. The abstract page I read did not give the sample size, so check the full paper before you rely on the size of the effect.

Other work got slower. A 2025 randomized trial by METR had 16 experienced open-source developers complete 246 tasks in mature projects. Some tasks were randomly allowed to use early-2025 AI tools, mostly Cursor with Claude models, and some were not. With AI allowed, completion took 19% longer. Before the study, developers had expected AI to make them about 24% faster, and after it they still estimated about a 20% speedup (METR, 2025). The result is for experienced developers on their own large projects, which is not the same work a first-year student does.

Taken together, the two studies say that the effect depends on the task, the person, and the setting. Neither one says AI is always faster or always slower.

Wrong answers can look right. A 2024 study (arXiv:2406.10279) found that about 5.2% of the packages recommended by commercial code-generating models did not exist, rising to about 21.7% for open-source models. The authors also found 205,474 unique invented package names across 16 models and two programming languages. A recommended package name is not evidence that the package exists or is safe to install.

Assistants can agree with you when they should not. A 2023 study of five AI assistants found consistent sycophancy across four free-form tasks, and suggested that human preference data rewards agreeable answers (arXiv:2310.13548).

Hallucination is a known, studied problem. A survey of hallucination in natural language generation describes models producing text that is fluent but not supported by the source or by facts, and reviews methods to detect and reduce it (Ji et al., 2022). The survey is about text generation in general, not code.

What the evidence does not show

  • It does not show the long-term effect on how much students learn. The learning study in using AI to learn was short and used high school maths.
  • It does not show how most students use these tools, or how often their work is correct.
  • It does not show what happens in the job market. No source on this page supports a claim about future jobs, salaries, or what employers will ask for. If you read such a claim, ask where its numbers come from.

What you can say with confidence

  • An assistant can give you an explanation that is clear and wrong. Check it.
  • The speed you feel while using a tool may not match the time the task takes. The METR study found that developers predicted and perceived speedups that were not in the measured times.
  • Code you did not write or read is code you cannot debug. This is a plain consequence of how debugging works, and the python course is where you build that skill. Reading the official documentation, as in the Python documentation, is part of it too.

Where the uncertainty sits

Every study here is recent and small by research standards, and tools change faster than papers are written. Treat each number as a snapshot of one setting. When you read a new claim, check whether it has a named method, a sample size, and a link to the data. If it does not, treat it as a guess.

Try this

Pick one small task from capstone 1. Time yourself doing it with no assistant. Then do it again with an assistant, and time that too. Write both numbers down, and add a note on how much time you spent checking the assistant's output. Compare your result with the METR finding. Your numbers are one data point, not evidence about everyone.

Sources