Quality from and with Prompt Engineering - David Faragó 

 July 4, 2023

"But then I didn't classically improve this one prompt yet, but then moved on to a conversation. It doesn't have to stay with a single prompt" - David Faragó

As a deep learning enigneer, David explores the possibilities for using AI. It's all about his approach to AI-generated test cases, the limitations and the possibilities. We also talked about prompt patterns, what the smallest changes in the prompt do, and how they can be designed to increase the quality of responses. Finally, David assesses the evolution of AI and has interesting tips for testers who want to start integrating AI into their workflow.

David is a deep-learning engineer at the company mediform, which develops communication solutions such as telephone bots for medical practices. His work there includes fine-tuning large-language models, prompt engineering and data science, as well as software engineering, e.g. microservices, Kubernetes and quality assurance. On the side, he still has his company QPR Technologies, which offers consulting and development services on innovative quality assurance, e.g. using static code analysis or AI. He founded this company with colleagues after his PhD on Symbolic AI, Formal Methods and Model-Based Testing. He is a member of the steering committee of the GI specialist group Test, Analysis and Verification.

Highlights in this episode:

  • David talks about the use of artificial intelligence for quality assurance
  • Prompt engineering can help to overcome the writer's block and develop test cases
  • David emphasizes that one should not rely entirely on the results of artificial intelligence, especially in terms of correctness
  • Experiments with different prompt patterns can improve the quality and robustness of the test cases
  • David shares his experience with the React pattern, which can improve the robustness and correctness of results
  • Testers are well suited for prompt engineering as they already have an understanding of quality and requirements
  • David is currently researching the possibilities of fine-tuning models to further improve the results.

 

Contact David:

More links:

 

How Prompt Engineering is transforming software test quality

TLDR: Today we enter the world of prompt engineering and how it is revolutionizing the quality of software testing. David shares his experiences and insights on how artificial intelligence and prompt engineering can be used to create and optimize test cases.

The potential of AI in test design

Artificial intelligence has the potential to eliminate the infamous writer's block when creating test cases. However, despite the enormous possibilities, he makes it clear that complete automation is not realistic. His main message was clear - AI can do a lot, but it is not a panacea.

The challenge of correctness

One critical issue is the question of the correctness of generated test cases. David emphasized that AI-generated tests must be viewed particularly critically. Despite the impressive ability of AI to generate useful content, errors and inaccuracies cannot be ruled out. This is particularly relevant in the context of test generation, where accuracy is essential.

AI-supported commit messages

David shared a story about his experiment with commit messages using AI for quality assessment. This experience revealed both the potential and the limitations of AI - a single small change could completely reverse the AI's judgment. This experience taught him to always question the results critically.

The evolution of prompt engineering

We then talked about David's journey to becoming an expert in the field of prompt engineering. By experimenting with different patterns, he found ways to improve test quality. But despite significant progress in test coverage and structuring, challenges in correctness remained.

Future prospects

David is optimistic that further developments in the field of AI-supported testing are imminent. Nevertheless, a certain skepticism remains regarding the reliability of these methods. As always, it is important to strike a balance between innovation and critical observation.