Agentic software development is an approach in which AI coding agents write, test and review code while humans work on the system instead of in it. Quality comes from deterministically triggered feedback loops: Git hooks, pipeline tests, accessibility checks and architecture reviews run automatically while the agent works and correct it on the spot.
Key Takeaways
- If you work on the system instead of in it, you give agents rules, hooks and skills rather than reviewing every piece of generated code by hand.
- Agent hooks fire deterministically on specific events, for example an accessibility check after every frontend change, so the agent gets immediate feedback before it moves on.
- Non-determinism in AI agents isn’t purely a risk: a model can recognize a flaky test as a symptom, where a deterministic test only reports “red”.
- Pitting two models against each other, such as Claude as the implementer and Codex as the detail-oriented reviewer, produces better results than either model alone.
Working on the System, Not in It: How AI Coding Agents Change the Developer’s Role
When you build software with AI coding agents, you stop working on individual pieces of code and start working on the system that produces them. Benedikt Stemmildt sees this shift as the core of how he works. Instead of implementing and then reviewing, he watches where the agent keeps heading in the wrong direction and steps in at the level of the rules.
The picture behind this is a new junior colleague in pair programming. You notice he doesn’t use hotkeys or ignores an important architecture pattern, and you point it out. A person remembers that. An agent doesn’t: it has no memory that keeps the correction automatically.
That’s why the correction has to go into the system, not just into the conversation. If you notice the agent skipping a documented architecture pattern, you add it as a rule to the rule set. A one-off remark becomes a permanent guardrail. This kind of onboarding is specific to the project and the task and hardly carries over between projects, much like every new employee has to be onboarded from scratch.
“And then you’re building the system rather than being inside it. I don’t do reviews at all anymore. You’re no longer the one doing reviews, you’re working on the system.”
(Benedikt Stemmildt)
Supervised or Unsupervised: The First Fork in the Road for Quality
Before you talk about testing and quality, you need to settle whether the code is produced supervised or unsupervised. That distinction gets lost in a lot of AI debates, yet it shapes every later decision about feedback and safeguards.
Supervised means the developer is watching, following the running terminals and stepping in when something starts to tip the wrong way. Unsupervised means the code is produced without anyone looking, much like a deployment pipeline with nobody sitting next to it.
In supervised mode, you can follow the agent and work out from what you see what it keeps getting wrong. That’s exactly where rules come in. The further you move toward unsupervised, the more the feedback loops have to be secured deterministically, because no human is stepping in anymore.
Classic Quality Mechanisms Still Apply
Building with agents differs less from classic development than many people assume. The mechanisms that stop defects at different levels stay the same: Git hooks, linters, Terraform plans on push, tests in the pipeline, smoke tests in a development environment before the production deployment.
You should have this baseline anyway, with or without AI. Benedikt sees it missing in many projects. If you don’t have it yet, you can use agents to set it up, but then you have to check carefully that the hooks and the pipeline are clean. No YOLO mode here.
The difference is speed. If a pipeline runs badly or defects slip through five times, you don’t notice after five weeks, you notice after thirty minutes. Improving the pipeline works the way it does for any team, just on steroids.
Tests Come from the Live System, Not from the Spec
The agent writes the tests, not the developer. Where the test basis comes from depends on the approach. Many teams go for spec-driven development, where acceptance criteria are defined up front and the tests build on them.
Benedikt prefers another route. He gathers requirements from the live system, meaning the software as real end customers use it. Their feedback becomes a new requirement: a bug that needs fixing, or a behavior they want added.
The same feedback improves the tests. Observability data, click behavior and real interviews reveal paths the end-to-end tests don’t cover yet. A click path customers take again and again belongs in the tests, because it obviously matters.
The Test Pyramid Applies to Coding Agents Too
A good agent works along the test pyramid because it mirrors the developer’s way of working and experience. Principles and patterns from years of software development carry over to the agent, as long as you give them to it.
Left to themselves, agents tend to test every feature manually: they open a browser, take a screenshot, analyze it visually and click their way through. That’s slow and expensive. Testing every feature this way throws away the whole point of the pyramid.
Each level has a different job:
| Test Level | Purpose in the Agent Context |
|---|---|
| Unit Tests | Enforce a structure and architecture that are easy to test; warn about unintended changes to production code |
| Integration Tests | Make sure surrounding systems don’t break |
| Contract Tests | Prevent breaking changes to interfaces by testing the producer the way a consumer would |
| End-to-End Tests | Cover real usage paths, used sparingly |
Benedikt writes unit tests first on purpose, so the architecture of the real code becomes testable. If the agent later changes production code, a test that turns red tells it that it may be touching a part it never meant to touch.
Contract tests catch a typical agent mistake. An agent quickly starts tinkering with its own interface and introduces breaking changes along the way. A separate test from the outside, run by a different agent, flags that the consumer will break before any damage is done.
Non-Functional Requirements Need Different Modes
Performance, security and accessibility each call for a different mode. Not every quality requirement can be handled the same way, and the supervised/unsupervised split plays a part in deciding how.
Performance works very well unsupervised, because you can set hard targets. You define performance budgets per endpoint and let the agent optimize the code until it stays within them. A performance test in the pipeline checks that no budget is exceeded. There are limits: data volume scenarios in the database can be tested, but no ordinary pipeline tests for a DDoS attack.
Security is harder to automate and belongs in supervised mode. A regular security audit, daily or weekly, works through attack vectors and threats together with the agent. Licensing, package dependency updates and accessibility, on the other hand, run as normal parts of the pipeline, the last one for example with a tool like Axe.
How Hooks Give Coding Agents Feedback While They Work
Quality with agents comes from having the right feedback available in many places. A person writing code gets constant internal feedback: this isn’t accessible, this won’t perform. The agent doesn’t have that inner dialogue.
That’s why these checks don’t only go into the pipeline. Hooks put them directly into the agent’s work. When the agent changes a frontend component, a hook fires the accessibility test while it’s still working. The agent learns right away that it forgot about accessibility and fixes it before anything is pushed.
Two kinds of hooks work together. The Git hook fires on commit. The hook in the agent harness fires on other events, for example when a tool runs or a file is read. A hook on the “implementation done” event can tell the agent to simplify the code three times before it’s allowed to move on.
Deterministic or Not: Both Have Value
Choosing between deterministic and non-deterministic feedback is a spectrum, not an either/or. Hooks are deterministic, because an event triggers code. Rules and skills are softer and non-deterministic.
An example makes the difference concrete. Whether the architecture follows a Domain-Driven Design principle can be checked deterministically by an ArchUnit test or judged non-deterministically by a skill. The test might miss something outside its scenario. The model might judge it wrong. Often the best answer is both together.
People tend to underestimate non-determinism here. It adds value when it comes to recoverability. A red test, deterministically, is just red. A model asks back: is the test red because something is broken, because the test is badly written, or because it’s flaky? That assessment can lead to the insight that flaky tests should be fixed as a matter of principle. You use each mode where it helps.
Tracking and Regular Reviews Keep the Architecture Clean
Good architecture comes about the same way with agents as without them: through documented decisions and regular reviews. Agents can help document architecture decisions and keep arc42 documentation up to date. To people that can look redundant, but for agents it’s their own reference knowledge about interfaces and structures.
The second building block is a scheduled review. With schedules, you can have the agent carry out an action once a day, the way a team looks at its architecture once a week. An architecture review skill then assesses whether development follows the principles.
A concrete case shows the pattern. In a large frontend, a global CSS file was growing into a monolith. At around four thousand lines, the agent noticed during a review that this no longer held up, proposed modularizing it, moved components out and made each one self-contained. The next iteration will probably uncover the next case of coupling that’s too tight.
These schedules don’t appear on their own. The developer sets them up during the supervised phase, while guiding the agent. One recurring instruction is to look through the latest conversations, collect your own corrections and derive rules, skills and schedules from them, so the agent sticks to them from then on.
How AI Coding Agents Remember Corrections: Retrospectives on Steroids
With agents, continuous improvement becomes a daily routine. Instead of one retrospective a month, it runs several times a day and takes seconds. One of the first things Benedikt builds in is a skill that runs a retro on its own.
This skill looks at old sessions, searches for things that often went wrong and had to be corrected, and updates its own rules. Whether that holds up in the long run is open. It’s possible that you’re just duplicating knowledge the model already has as rules. The value still lies in experimenting with the system.
For many developers this role is unfamiliar. For years they were deep inside the system, building for others, with no time to work on the system itself. There aren’t even criteria for what “working on the system” means, or for how to make quality goals like security measurable. How secure, exactly? That question often goes unanswered because people rarely had the distance from the system to ask it.
Why Having Codex Review Claude’s Code Raises Quality
A review by a second model catches weaknesses in the first. Benedikt has code written by Claude reviewed by Codex and makes deliberate use of how differently the two models behave.
Claude acts like a full-stack developer who pragmatically finds a way to implement something, as long as the business value is there. Codex looks harder at the details and sticks more strictly to the rules. That strictness helps in a review, but it can get in the way when an implementation needs flexibility, because Codex takes rules too literally.
The two work together much like two colleagues. Codex delivers the list of issues, Claude fixes some of them and explains why others can’t be done. Many people treat agents consistently like employees, probation period included: if a configuration doesn’t work, the agent is discarded and set up again.
Past a Certain Level, Model-Hopping Isn’t Needed
You don’t have to chase every new model. Benedikt doesn’t think it’s necessary to switch models with every release. From Opus 4.5 onward, the available models are more than good enough for the work at hand, and the benchmarks back that up.
What’s left to decide are other questions: cost optimization, whether a subscription may be used, and sovereignty. Open-weights models or local models via vLLM matter for GDPR concerns, but they require real depth. A bare terminal with a chat and an inference endpoint offers no features and only works the way you want after a lot of configuration.
For getting started in companies, Codex and Claude Code remain practical, because the user experience of a ready-made desktop app makes the switch easier. Sovereign setups are powerful, but rarely the easiest first step.
When Code Is Still Written by Hand
Benedikt writes code by hand only in the most extreme on-the-system cases. He writes architecture rules and similar specifications himself, because otherwise reviewing generated versions and boiling them down to what he wants would take more effort.
Sometimes that includes hooks, which he looks at closely and corrects by hand. That’s far removed from the software people actually end up using. He no longer writes product code by hand at all.
One open question remains: does writing rules in English still count as programming? The work is moving from the line of code to the system, from implementation to steering the feedback loops.
Frequently Asked Questions
Does code developed by agents require a different approach to quality assurance than hand-written code?
The mechanisms remain the same: Git hooks, linters, Terraform plans on push, tests in the pipeline, and smoke tests in a development environment before production deployment. This baseline should be in place anyway, but it’s missing in many projects. The difference lies in the speed. If a pipeline isn’t working properly or if errors slip through multiple times, it becomes apparent after thirty minutes instead of five weeks.
Why isn’t it enough to correct an AI agent during a conversation?
An agent has no memory that automatically retains the correction. A junior developer remembers the feedback; the agent does not. That’s why the correction belongs in the rule set: Anyone who notices that a documented architectural pattern is being ignored should define it as a rule. This training is project- and task-specific and can hardly be reused across projects.
Where do test cases come from if you’re not working in a spec-driven way?
Requirements can be gathered from the live system, that is, from the software in use with real end customers. Their feedback becomes a new requirement: a bug that needs to be fixed, or a desired behavior that needs to be added. Observability data, click behavior, and interviews reveal paths that are missing from end-to-end testing. A click path that customers follow repeatedly needs to be mapped.
How do agents test if they aren’t given guidelines on test strategy?
Left to their own devices, agents tend to test manually: opening a browser, taking a screenshot, analyzing visually, and clicking through. This is slow and expensive and negates the advantage of the test pyramid. If you provide them with principles and patterns, the agent mirrors the developer’s workflow. Enforcing unit testing first ensures testability and alerts the agent if it unintentionally modifies production code.
Can non-functional requirements like security be verified automatically?
Only partially. Security is harder to automate and belongs in supervised mode, for example as a daily or weekly audit that works with the agent to uncover attack vectors and threats. Performance works well in unsupervised mode because budgets can be set per endpoint and verified in the pipeline. Licensing, dependency updates, and accessibility run as normal pipeline components.
How do you prevent the architecture from eroding during agent-based development?
Through documented decisions and regular reviews, just as you would without agents. An arc42 documentation, which the agent keeps up to date, serves as a reference for the agent itself regarding interfaces and structures. In addition, there is a scheduled review, evaluated by an architecture review skill. In a large frontend project, for example, it became apparent from approximately four thousand lines of global CSS that modularization was necessary.
Do you have to switch models with every new model release?
No. Starting with Opus 4.5, the available models are more than sufficient for the task at hand, as verified by the benchmarks. Other issues remain to be decided: cost optimization, whether a subscription may be used, and sovereignty. Open-weights or local models via vLLM are relevant for GDPR issues but require a great deal of configuration.
