Agent-based software development refers to an approach in which AI agents write, test, and review code, while humans work on the system rather than within it. Quality is achieved through deterministically triggered feedback loops: Git hooks, pipeline tests, accessibility checks, and architecture reviews run automatically while the agent is working and correct it immediately.
Key Takeaways
- Those who work on the system rather than within it provide agents with rules, hooks, and skills, rather than manually reviewing every piece of generated code.
- Agent hooks fire deterministically in response to specific events—for example, an accessibility check after every front-end change—so that the agent receives immediate feedback before proceeding.
- Non-determinism in AI agents isn’t purely a risk: A model recognizes flaky tests as a symptom, whereas a deterministic test simply reports “red.”
- Using two models in tandem—such as Claude as the implementer and Codex as the detail-oriented reviewer—yields higher quality results than either model alone.
Working on the System Instead of in the System: How the Developer’s Role Is Shifting
Those who build software with AI agents no longer work on individual pieces of code, but rather on the system that generates the code. Benedikt Stemmildt describes this shift as central to his approach. Instead of implementing the code himself and then reviewing it, he observes where the agent repeatedly goes in the wrong direction and intervenes at the level of the rules.
The analogy here is a new junior colleague in pair programming. You notice that he isn’t using hotkeys or is ignoring an important architectural pattern, and you point it out to him. A human remembers that. An agent does not: it has no memory that automatically retains the correction.
That’s why the correction must be incorporated into the system, not just into the conversation. Whoever notices that the agent is bypassing a documented architectural pattern records it as a rule in the rule set. A one-time comment becomes a permanent guideline. This training is project- and task-specific and can hardly be reused across projects, much like how you have to train every new employee from scratch.
And then you’re building the system rather than just being part of it. I don’t do any reviews at all. You’re no longer the one doing reviews; instead, you’re working on the system. Benedikt Stemmildt
Supervised or unsupervised: the first turning point for quality
Before discussing testing and quality, it must be clarified whether the code is generated in a supervised or unsupervised manner. This distinction often gets lost in many AI debates, but it shapes every subsequent decision regarding feedback and validation.
“Supervised” means the developer is watching, monitoring the running terminals, and stepping in if things start to go in the wrong direction. “Unsupervised” means the code is generated without human oversight, similar to a deployment pipeline where no one is sitting there watching.
In supervised mode, you can follow the agent and deduce from what you observe what it is systematically doing wrong. That’s exactly where the rules come into play. The further you move toward unsupervised mode, the more deterministically the feedback loops must be safeguarded, because no human is intervening anymore.
How Traditional Quality Mechanisms Are Preserved
Building with agents differs less from traditional development than is often assumed. The mechanisms that prevent errors at various levels remain the same: Git hooks, linters, Terraform plans on push, tests in the pipeline, and smoke testing in a development environment before production deployment.
You should have this baseline in place anyway, regardless of AI. Benedikt observes that it’s missing in many projects. If you don’t have it yet, you can use agents to set it up, but you’ll need to pay close attention to ensure that the hooks and pipeline are clean. There’s no “YOLO” mode here.
The difference lies in the speed. If a pipeline is running poorly or errors slip through five times, you won’t notice it after five weeks—you’ll notice it after thirty minutes. Improving the pipeline works just like it does for any team, only on steroids.
Tests Are Created on the Live System, Not from the Spec
The agent writes the tests, not the developer. Where the test basis comes from depends on the approach. Many rely on spec-driven development, where acceptance criteria are defined in advance and the tests are built on top of them.
Benedikt prefers a different approach. He gathers requirements from the live system—that is, from the software in use with real end customers. Their feedback becomes a new requirement: a bug must be fixed, or a desired behavior is added.
This feedback can also be used to improve the tests. Observability data, click behavior, and real interviews reveal paths that are still missing from the end-to-end testing. A click path that customers follow repeatedly needs to be mapped because it is clearly important.
The test pyramid also applies to test agents
A good test agent works according to the test pyramid because it mirrors the developer’s workflow and experiential knowledge. Principles and patterns from years of software development carry over to the test agent, provided they are instilled in it.
Left to their own devices, agents tend to test every feature manually: they open a browser, take a screenshot, analyze it visually, and click their way through. This is slow and expensive. Anyone who tests every feature this way squanders the very advantage of the pyramid.
The individual levels serve different purposes:
| Test Level | Purpose in the Agent Context |
|---|---|
| Unit Testing | Enforce a structure and architecture that are easy to test; warn of unintended changes to production code |
| Integration Testing | Ensure that surrounding systems do not break |
| Contract Tests | Prevent breaking changes to interfaces by testing the producer as if it were a consumer |
| End-to-End Testing | Map real usage paths; used sparingly |
Benedikt deliberately writes unit testing first so that the architecture of the actual code becomes testable. If the agent later makes changes to the production code, a test that turns red alerts them that they may be modifying a part of the code they didn’t intend to touch.
Contract tests catch a typical agent error. An agent quickly tinkers with their interface and inadvertently introduces breaking changes. A separate test run from the outside by another agent flags that the consumer will fail before any damage occurs.
Why Non-Functional Requirements Must Be Treated Differently
Performance, security, and accessibility require different approaches. Not every quality requirement can be treated the same way, and the distinction between supervised and unsupervised testing plays a key role.
Performance works very well in an unsupervised setting because hard targets can be set. You define performance budgets per endpoint and let the agent optimize the code to ensure it stays within those limits. A performance test in the pipeline verifies that the budgets are not exceeded. Limitations remain: volume patterns in the database are testable, but a DDoS attack is not something a normal pipeline would test for.
Security is more difficult to automate and belongs in supervised mode. A regular security audit—daily or weekly—works together with the agent to detect attack vectors and threats. Licensing, package dependency updates, and accessibility, on the other hand, run as normal pipeline components—the latter, for example, via a tool like Axe.
Feedback at the Right Level Is the Key Issue
Quality with agents is achieved where the right feedback is available at multiple points. When writing, a person constantly receives internal feedback: “This isn’t accessible,” “This isn’t performing well.” The agent doesn’t have this internal dialogue.
That’s why these checks don’t just go into the pipeline, but via hooks directly into the agent’s work. If the developer modifies a front-end component, a hook triggers the accessibility test while the developer is still working. The developer immediately receives a notification that they’ve overlooked accessibility and makes the necessary corrections before even pushing the changes.
Two types of hooks work in tandem. The Git hook fires on commit. The hook in the agent harness fires on other events, such as when a tool is run or a file is read. A hook on the “Implementation complete” event can instruct the agent to refactor the code three times before it’s allowed to continue.
Deterministic or Not: Both Approaches Have Value
The choice between deterministic and non-deterministic feedback is a spectrum, not an either/or situation. Hooks are deterministic because an event triggers code execution. Rules and skills are more flexible and non-deterministic.
An example makes the difference tangible. Whether the architecture follows a Domain-Driven Design principle can be deterministically verified by an ArchUnit test or non-deterministically evaluated by a skill. The test may overlook something outside its scenario. The model may evaluate it incorrectly. Often, the best solution is a combination of both.
Non-determinism is often underestimated in this context. It adds value in terms of recoverability. A red test is, deterministically speaking, simply red. A model asks: Is the test red because something is broken, because the test is poorly written, or because it’s flaky? This evaluation can lead to the realization that flaky tests should generally be fixed. You use both modes where they’re helpful.
Architecture Stays Clean Through Tracking and Regular Reviews
Good architecture emerges with or without agents: through documented decisions and regular reviews. Agents can help document architectural decisions and keep arc42 documentation up to date. While this may seem superfluous to humans, for agents it serves as their own reference knowledge regarding interfaces and structures.
The second building block is a scheduled review. Using schedules, you can configure the agent to perform an action once a day, just as a team reviews its architecture once a week. An architecture review skill then evaluates whether the development follows the principles.
A specific case illustrates this pattern. In a large frontend project, a global CSS file was growing into a monolith. When it reached around four thousand lines, the agent noticed during the review that it was no longer sustainable, proposed modularization, extracted components, and made each component self-contained. The next iteration will likely uncover the next instance of tight coupling.
These schedules don’t emerge on their own. The developer sets them up during the supervised phase while guiding the agent. A recurring instruction is to review recent conversations, collect one’s own corrections, and derive rules, skills, and schedules from them so that the agent adheres to them in the future.
Retrospectives on Steroids
With agents, continuous improvement becomes a daily routine. Instead of holding a retrospective once a month, it runs several times a day and takes only seconds. As one of his first steps, Benedikt builds in a skill that conducts a retrospective on its own.
This skill reviews past sessions, looks for things that frequently went wrong and had to be corrected, and updates its own rules. Whether this will be sustainable in the long run remains to be seen. It’s conceivable that knowledge already embedded in the model is simply being duplicated once more in the form of rules. Nevertheless, the value lies in experimenting with the system.
This role is unfamiliar to many developers. For years, they were deeply immersed in the system and built for others, without time to work on the system itself. There’s even a lack of criteria for what “working on the system” means and how to make quality goals—such as security—measurable. How secure, exactly? This question often goes unanswered because people rarely had the necessary distance from the system.
Why Two Models Working Against Each Other Improve Quality
A review by a second model catches weaknesses in the first. Benedikt has Claude’s code reviewed by Codex and makes strategic use of the models’ different characteristics.
Claude acts like a full-stack developer who pragmatically finds a way to implement the solution, as long as the business value is there. Codex pays closer attention to details and adheres more strictly to the rules. This rigor helps during the review but can be a hindrance when an implementation requires flexibility, because Codex takes rules too literally.
Their collaboration works much like that between two colleagues. Codex provides the list of errors; Claude addresses some of them and explains why certain changes aren’t feasible. Many consistently treat agents like employees, including a probationary period: if the configuration doesn’t work, the agent is discarded and set up anew.
Model-hopping becomes unnecessary at a certain level
You don’t have to chase after every new model. Benedikt doesn’t think it’s necessary to switch models with every release. Starting with Opus 4.5, the available models are more than good enough for the task at hand, as verified by the benchmarks.
What remains to be decided after that are other issues: cost optimization, whether a subscription can be used, and sovereignty. Open-weights models or local models via vLLM are relevant for GDPR considerations but require in-depth knowledge. A basic terminal with chat and an inference endpoint offers no features and only works the way you want it to after a lot of configuration.
For getting started in companies, Codex and Claude Code remain practical options because the user experience of a ready-made desktop app makes the transition easier. Sovereign setups are powerful, but rarely the simplest first step.
When Code Is Still Written by Hand
Benedikt only writes manual code in the most extreme “on-system” cases. He formulates architectural rules and similar specifications himself because it would otherwise be more work to review generated versions and refine them to meet the desired specifications.
Sometimes this also applies to hooks, which he examines closely and corrects manually. This is far removed from the actual software that is ultimately used. He no longer writes the product code itself by hand.
One question remains: Does formulating rules in English even count as programming anymore? The work is shifting from the line of code to the system, from implementation to controlling the feedback loops.
Frequently Asked Questions
Does code developed using an agency require a different approach to quality assurance than hand-written code?
The mechanisms remain the same: Git hooks, linters, Terraform plans on push, tests in the pipeline, and smoke tests in a development environment before production deployment. This baseline should be in place anyway, but it’s missing in many projects. The difference lies in the speed. If a pipeline isn’t working properly or if errors slip through multiple times, it becomes apparent after thirty minutes instead of five weeks.
Why isn’t it enough to correct an AI agent during a conversation?
An agent has no memory that automatically retains the correction. A junior developer remembers the feedback; the agent does not. That’s why the correction belongs in the rule set: Anyone who notices that a documented architectural pattern is being ignored should define it as a rule. This training is project- and task-specific and can hardly be reused across projects.
Where do test cases come from if you’re not working in a spec-driven way?
Requirements can be gathered from the live system—that is, from the software in use with real end customers. Their feedback becomes a new requirement: a bug that needs to be fixed, or a desired behavior that needs to be added. Observability data, click behavior, and interviews reveal paths that are missing from end-to-end testing. A click path that customers follow repeatedly needs to be mapped.
How do agents test if they aren’t given guidelines on test strategy?
Left to their own devices, agents tend to test manually: opening a browser, taking a screenshot, analyzing visually, and clicking through. This is slow and expensive and negates the advantage of the test pyramid. If you provide them with principles and patterns, the agent mirrors the developer’s workflow. Enforcing unit testing first ensures testability and alerts the agent if they unintentionally modify production code.
Can non-functional requirements like security be verified automatically?
Only partially. Security is harder to automate and belongs in supervised mode—for example, as a daily or weekly audit that works with the agent to uncover attack vectors and threats. Performance works well in unsupervised mode because budgets can be set per endpoint and verified in the pipeline. Licensing, dependency updates, and accessibility run as normal pipeline components.
How do you prevent the architecture from eroding during agent-based development?
Through documented decisions and regular reviews, just as you would without agents. An arc42 documentation, which the agent keeps up to date, serves as a reference for the agent itself regarding interfaces and structures. In addition, there is a scheduled review, evaluated by an architecture review skill. In a large frontend project, for example, it became apparent from approximately four thousand lines of global CSS that modularization was necessary.
Do you have to switch models with every new model release?
No. Starting with Opus 4.5, the available models are more than sufficient for the task at hand, as verified by the benchmarks. Other issues remain to be decided: cost optimization, whether a subscription may be used, and sovereignty. Open-weights or local models via vLLM are relevant for GDPR issues but require a great deal of configuration.
