Quality from and with Prompt Engineering

ChatGPT can generate impressive test cases - but correctness remains a critical problem. Systematic prompt engineering can significantly improve the quality of generated tests, but even with few-shot learning and embedded specifications, AI produces flawed tests. Software testing requires more than the wow effect: patterns such as React and Chain of Thought increase robustness, but the technology is not yet mature enough for safety-critical areas.

Podcast Episode: Quality from and with Prompt Engineering

As a deep learning enigneer, David explores the possibilities of using AI. It’s about his approach to AI-generated test cases, the limitations and the possibilities. We also talked about prompt patterns, what the smallest changes in the prompt do and how they can be designed to increase the quality of answers. Finally, David assesses the development of AI and has interesting tips for testers who want to start integrating AI into their workflow.

“Testers are great for prompt engineering.” - David Faragó

David is a deep learning engineer at mediform, a company that develops communication solutions such as telephone bots for medical practices. His work there includes fine-tuning large language models, prompt engineering and data science, as well as software engineering, e.g. microservices, Kubernetes and quality assurance. He also has his own company, QPR Technologies, which offers consulting and development services for innovative quality assurance, e.g. using static code analysis or AI. He founded this company with colleagues after completing his doctorate in symbolic AI, formal methods and model-based testing. He is a member of the steering committee of the GI specialist group Test, Analysis and Verification.

Highlights der Episode

• Great language models hallucinate: Even convincing answers can be subtly wrong and mislead for days.
• Prompt engineering needs tester vision: Noticeably improve correctness with few-shot samples and specification in the prompt.
• React pattern massively increases robustness: Reasoning plus external API calls reduce hallucination almost completely.
• Test generation is partially successful: Eleven tests, two incorrect - can only be used productively with reviewer mindset.
• Testers become prompt engineers: Quality assurance for AI output is classic testing knowledge, applied in a new way.

How Prompt Engineering is transforming software test quality

Today we enter the world of prompt engineering and how it is revolutionizing the quality of software testing. David shares his experiences and insights on how artificial intelligence and prompt engineering can be used to create and optimize test cases.

The potential of AI in test design

Artificial intelligence has the potential to eliminate the infamous writer’s block when creating test cases. However, despite the enormous possibilities, he makes it clear that complete automation is not realistic. His main message was clear - AI can do a lot, but it is not a panacea.

The challenge of correctness

One critical issue is the question of the correctness of generated test cases. David emphasized that AI-generated tests must be viewed particularly critically. Despite the impressive ability of AI to generate useful content, errors and inaccuracies cannot be ruled out. This is particularly relevant in the context of test generation, where accuracy is essential.

AI-supported commit messages

David shared a story about his experiment with commit messages using AI for quality assessment. This experience revealed both the potential and the limitations of AI - a single small change could completely reverse the AI’s judgment. This experience taught him to always critically question the results.

The evolution of prompt engineering

We then talked about David’s journey to becoming an expert in the field of prompt engineering. By experimenting with different patterns, he found ways to improve test quality. But despite significant progress in test coverage and structuring, challenges in correctness remained.

Future prospects

David is optimistic that further developments in the field of AI-supported testing are imminent. Nevertheless, a certain skepticism remains regarding the reliability of these methods. As always, it is important to strike a balance between innovation and critical observation.