AI security testing spans the whole life cycle of an AI model: from training, where manipulated input data can skew the model, to day-to-day use, where attackers pull protected data out through carefully crafted queries. The OWASP Top 10 for AI and the OWASP AI Security & Privacy Guide give teams a framework to start from.
Key Takeaways
- AI models can be talked into revealing protected data through indirect queries, even when direct queries are blocked, because the model doesn’t recognize a question about a job title as a workaround.
- Security risks in AI-based systems run through the entire life cycle: from manipulated training data and theft of the model to misused queries in production.
- Output filters often work better than input filters, because they check directly whether the generated answer contains sensitive content such as salaries or personal data.
- The OWASP Top 10 for AI-based software is a structured starting point for security requirements, but putting it into practice takes expertise, either in-house or from outside specialists.
What Makes AI Security Testing Different
With AI-based software, you don’t know in advance exactly how it will behave. That is the whole idea. A machine learning model learns on its own instead of being explicitly programmed, and at best it solves the task better than code ever could. This is where AI security testing gets harder than testing conventional software.
That property creates the first security problem. If behavior isn’t fixed beforehand, it is hard to establish whether the system holds up against attacks. Security is one part of what Jan Jürjens calls trustworthy AI.
Most companies don’t build AI themselves. They buy a model or plug it into their own infrastructure as a component. So the question becomes: how do you make an IT infrastructure that uses AI secure against attacks and trustworthy overall?
Attack Points Span the Entire Life Cycle
Security risks in AI run from training all the way to use. There isn’t one single weak spot, but a chain of phases, each with its own requirements.
During training, an attacker may slip in data that influences the model in ways nobody intended. If you train your own model, you have to secure this phase. Once manipulated training data is in, it is very hard to identify cleanly later.
Once the model exists, classic security requirements apply, just to a new kind of asset. Nobody should be able to copy or steal the model, and nobody should be able to tamper with it. An AI model is often a business asset and needs the same protection as other critical data.
Use is the third area. Here the concern is that queries to the model don’t undermine the security rules and that no protected data leaks out.
Why a Chatbot Can Reveal Personal Data
Filtering the direct question isn’t enough, because the same information can be requested by a detour. That is the core problem in securing queries.
Jan Jürjens explains the pattern with a salary example. If a model was trained on employee data, a direct question about a specific person’s salary should be blocked. Ask instead about the salary of a particular role, say the compliance officer, and the system may happily answer. Anyone who knows who holds that role now knows that person’s salary.
The lack of explainability makes this worse. You get an answer, but no explanation of how the model arrived at it.
“AI doesn’t usually work like a search engine. It aggregates the information from its knowledge store, so that in the end nobody can say what the answer is made of.”
(Jan Jürjens)
Because the answer comes from aggregated information, it is almost impossible to prove afterwards whether something that shouldn’t be there went into it. That is exactly why protection has to start during training.
AI Security Testing Works like Penetration Testing
The most effective defense against manipulated queries is intensive testing from an attacker’s point of view. You think about how someone would go about it and send those queries to the system yourself.
This is the same approach as penetration testing for conventional software. You take on the attacker’s role and try to get a reaction or a piece of information out of the system that it shouldn’t give away. If you find a gap like that, you then block that kind of query.
You can never be sure you have found everything. It is a race, but that holds just as true for security testing of conventional software.
Output Filters Often Work Better Than Input Filters
Checking the output often beats trying to catch every nested input. The reason is practical: right before the answer goes out, you can check directly whether, say, a salary appears in it.
Both places make sense. At the input, you block problematic requests. At the output, you check the result before it leaves the system. Indirectly phrased queries are hard to catch completely, while an output check works no matter how clever the question was.
Limiting requests makes sense anyway. Even with models trained on public data, you don’t want someone to extract the entire model through mass queries and rebuild it. Rate limiting should therefore be standard.
Reputation is another reason for filters. Nobody should be able to get an application to produce offensive output, because that reflects badly on the model vendor and on the company running it.
OWASP Provides the Checklists for AI, Too
There are already established resources for AI security, backed by the OWASP community. The organization started out as the Open Web Application Security Project and renamed itself the Open Worldwide Application Security Project, because its work has long gone beyond web applications.
For AI, there are specific resources, including an OWASP guide on security and privacy and an OWASP Top 10 for AI. The Top 10 format is familiar from web security and has been carried over to AI-based software.
These documents are easy to follow and a good place to start. The real hurdle isn’t reading them. It is putting them into practice.
Easy to Read, Hard to Implement
As a developer, you can follow the OWASP material, but assessing your infrastructure takes someone who knows how to judge it. That is the honest answer if your assignment is to add a chatbot quickly.
To decide whether your infrastructure really is secure, you need either trained people of your own or outside support. Here, too, AI is nothing fundamentally new. It is the same situation as in classic security.
If You Run AI, You Own Everything around It
As soon as you use AI professionally, you count as a deployer under the EU AI Act, and that comes with obligations. The regulation distinguishes between those who develop AI, those who deploy it and those who use it.
In most cases, you build AI into your architecture as a black box. You can’t look inside the model or its internal processes. But you are responsible for everything that happens around it.
That covers several questions at once:
- What data goes in, and is it protected?
- How does data come out, and is it protected at the interfaces?
- What happens with the results, and is that even allowed?
The EU AI Act also limits what AI may be used for. Following those rules is part of the deployer’s duty. As far as you can, also make sure that your vendor complies with the regulations.
The Biggest Weak Spot Is Still the Person at the Keyboard
People often use AI carelessly, and that is an underestimated risk. Documents get uploaded and analyzed without anyone checking what data leaves the building.
Many users trust a chatbot’s answer blindly. That is dangerous whenever something depends on the answer being right, at home or at work. AI isn’t one hundred percent reliable, and hallucinations appear as soon as you dig a little deeper.
Jan Jürjens gives an everyday example. Asked about a trampoline park in Koblenz, a chatbot confidently named one that doesn’t exist there. The system had probably found the name for another city and simply moved it to Koblenz. For a museum, it even gave an address in Aachen, a street that doesn’t exist in Koblenz at all.
Ask whether the answer is correct, and the system sticks to it at first. Only after pushing several times does it admit the mistake.
AI Gives You Hypotheses, Not Facts
Use AI to generate hypotheses, and do the checking yourself whenever the answer matters. That split is the practical consequence of its limited reliability.
Some tasks suit it well, such as quickly summarizing a long document. Even there, a statement may show up that wasn’t in the document, but you can check that against the original. You should always have that kind of fallback.
Be careful with answers you can’t verify yourself. With security topics in particular, a wrong answer delivered with confidence carries a lot of weight. Making users aware of this matters as much as the technical filters.


