OpenAI published an internal report on how coding agents are changing the way its own researchers work: Research acceleration: The view inside OpenAI. The material they made public talks about agent usage, experimentation speed, task complexity and research acceleration. Simon Willison picked it up the same day on his blog, which is the usual sign that something is worth reading.
Let me clarify something before going on: what's circulating from the announcement is the summary and the axes OpenAI says it measures. The concrete figures aren't in the material I have at hand, so I'm not going to quote them or make them up. What I can analyze is what they chose to measure, which often says more than the number itself.
Why it matters
The signal is in the metrics they chose, not in the result. OpenAI isn't counting generated lines of code or merged pull requests: it's counting experimentation speed and task complexity. That's a change in the unit of measurement.
When a company measures lines or commits, it's treating the agent as a faster typist. When it measures experiments per unit of time, it's treating the agent as a way to lower the cost of testing a hypothesis. Those are two completely different product theses, and the second one is far more uncomfortable for our profession, because it implies the bottleneck was never writing the code.
The company that sells the models publishing its own internal data is, obviously, marketing. But marketing also reveals where they're pushing the product. And here they're pushing toward agents that can sustain long tasks, not toward faster autocomplete.
What changes in practice
If you take seriously the idea that the metric is "experiments per week" and not "code per hour", several concrete decisions change in a normal team, with no GPUs and no research lab.
| Dimension | Agent as typist | Agent as experiment accelerator |
|---|---|---|
| What you measure | Code produced | Hypotheses tested and discarded |
| Bottleneck | Writing | Verifying and deciding |
| Key investment | Better prompt | Tests, reproducible environments, observability |
| Dominant risk | Buggy code | Poorly grounded conclusions |
| Senior's role | Reviewing diffs | Designing the success criteria |
The most direct practical consequence: value moves from generation to verification. If an agent lets you test five approaches in the time you used to test one, your constraint becomes how fast and how reliably you can decide which one worked. A team without a decent test suite, without a reproducible environment and without product metrics accelerates nothing: it multiplies branches with no criteria for pruning them.
The flow that ended up working for me looks more like this than like a chat:
┌────────────────────┐
│ Hypothesis + succes│
│ criteria │
└─────────┬──────────┘
│
▼
┌────────────────────┐ ┌──────────────────┐
│ Agent implements │──►│ Throwaway branch │
└─────────┬──────────┘ └──────────────────┘
│
▼
┌────────────────────┐
│ Auto verification │ tests, types, lint,
│ (no human) │ benchmark, build
└─────────┬──────────┘
│ passes
▼
┌────────────────────┐
│ Human review │ did it solve the
│ of the criteria │ right problem?
└────────────────────┘
What matters about the diagram is where the human sits: at the start defining what counts as success, and at the end judging whether the criteria were met. In the middle, reading every line doesn't add much. If your process forces you to read the whole diff under a magnifying glass to sleep at night, the problem isn't the agent: it's that you don't have enough automated verification.
Another thing that changes: branches become cheap and disposable. Discarding two days of work used to hurt. Now discarding a twenty-minute attempt is part of the method. That reorders the work toward "try three, keep one" instead of "let's discuss which one is better".
When NOT to use it
Here's the part corporate announcements don't write.
- When you have no way to verify. An agent that produces ten times more code in a codebase without tests gives you ten times more debt with the same confidence as before. The acceleration is only real on the side where there's an automated oracle. At OpenAI that oracle exists: a research experiment produces a number. In a business CRUD with ambiguous rules, it doesn't.
- When the problem is one of decision, not implementation. If the team doesn't know what to build, generating five versions of the undefined doesn't help. It multiplies the work of throwing them away.
- When the data is OpenAI's about OpenAI. Their researchers have context, internal tooling, models without quota limits and problems with measurable success criteria. Extrapolating those results to a five-person team with a 2018 legacy codebase is a leap the report doesn't authorize. I still haven't seen independent public evidence replicating those numbers outside the house.
- When the domain is regulated or sensitive. If you touch personal data, payments or health, experimentation speed isn't the metric; traceability is. There, a slower and more audited flow is the way to go.
- When the legal cost of context is still uncertain. The same day as the report, The Verge reported that the Seattle Times and Newsday sued OpenAI and Microsoft for copyright infringement, adding to earlier suits from the New York Times, Ziff Davis, Merriam-Webster and Britannica. It doesn't change whether the agent works for you technically, but it is a risk factor if you're deciding whether to rest your entire product on a single provider.
That last point connects to something bigger. South Korea launched the "AI for All" initiative precisely to avoid depending on foreign providers: it requires that at least 50% of the service use sovereign national models and another 30% Korean models from companies other than the ones providing the service. A state is treating provider dependency as a structural risk. At the scale of your architecture, the reading is the same: abstract the provider behind an interface of your own before a lawsuit or a price change makes the decision for you.
What I'd do today
Three things, in order. First, I'd invest in automated verification before prompts: tests, strict types, a reproducible environment. That's what turns the agent's speed into something usable instead of noise.
Second, I'd change the team's metric. If today you measure closed tickets, start measuring how many hypotheses you tested and discarded per sprint. That's the practical reading of OpenAI's report for a team that doesn't do research.
Third, I'd keep model access behind a layer of my own, with at least one alternative provider already tested. Not out of ideology: for the same reason you don't hardcode a database.