Acceptance Test-Driven LLM Development

The development of Large Language Models rarely follows structured processes - and this is where clear test strategies are needed. When building an AI-supported telephone bot for medical practices, Acceptance Test Driven Development was transferred to LLM training: dialogs serve simultaneously as acceptance tests and training data, while the CPMAI process combines agile development with machine learning. The challenge lies in the semantic verification of non-deterministic systems - classic unit tests do not work here, instead dialog-based tests, metamorphic testing and automated stress testing dialogs are used.

Podcast Episode: Acceptance Test-Driven LLM Development

First of all: Sorry for the poor audio quality, unfortunately we only noticed it afterwards. I hope the content will make up for it :-) The development of Large Language Models (LLMs) and the role of Acceptance Test Driven Development (ATDD) are central topics in AI development. David, an expert in the development and quality assurance of AI-based telephone bots for medical practices, shares his experience and insights into this process. The challenges and approaches to training and testing LLMs, including the use of prompt engineering and fine tuning, will be highlighted. Of particular note is the approach of applying ATDD methods to LLM developments to improve the quality and effectiveness of the models. Another focus is on the CPMAI process, which represents a modern approach to the development and implementation of AI projects.

“It’s now the first time where it’s really naturally possible to develop applications linguistically.”- David Faragó

David is a deep learning engineer at Mediform, specializing in fine-tuning large language models, prompt engineering and microservices. He also runs QPR Technologies, a consultancy for innovative quality assurance, and is a member of the steering committee of the GI specialist group Test, Analysis and Verification.

Highlights der Episode

• Acceptance Test Driven Development also works for LLMs: write tests before training.
• LLM outputs need semantic instead of string comparisons - otherwise you are testing past the problem.
• CPMAI combines agile software development with machine learning better than outdated CRISP-DM processes.
• Integrate pilot customers directly into iteration cycles - real dialogs beat erroneous business requirements.
• Template-based test dialogs enable metamorphic and combinatorial testing for LLM quality assurance.

Wie Acceptance Test Driven Development LLMs neu definiert

When developing AI-based telephone bots for medical practices, there are a lot of new quality assurance challenges to overcome. One solution approach in the testing process is Acceptance Test Driven LLM Development

The new fields of application for AI development

David Faragó, a renowned expert in the field of AI and specifically in LLM topics, shares his extensive knowledge on the use of Large Language Models. From prompt engineering to fine-tuning foundational models, David covers all aspects. His current project at Mediform aims to develop a telephone bot for medical practices that can communicate with patients in natural language thanks to modern AI. This innovative application represents a significant step forward and demonstrates the potential of LLM in practice.

Development and testing

At the heart of David’s work is a solidly developed testing process that focuses on acceptance testing. Through careful analysis of real dialogs and iterative improvements, a high-quality model is created. David talks about the challenges and approaches to developing LLMs, including dealing with non-determinism and the black box nature of this technology. By using specialized tools such as Eloifa’s Language Model Evaluation Harness, the team is able to effectively measure and verify the quality of their models.

Acceptance Test Driven LLM Development

Acceptance Test Driven LLM Development is not just a method, but a philosophy. It interweaves agile methods with machine learning, enabling rapid iteration cycles with direct end-user involvement. David explains the process in detail and shows how this approach has made it possible to develop a robust and effective system for medical practices. This approach ensures that the end product meets the exact requirements while remaining flexible for future customization.

The role of CPMAI in the development cycle

Another key element of David’s strategy is the use of CPMAI (Cognitive Process Management for AI), a modern process framework that combines agility with machine learning. This method supports the team at every stage of the development cycle - from understanding the business need to deploying the model. This structured approach allows problems to be identified and resolved quickly, enabling continuous improvement of the system.

The future of AI development

The discussion not only highlights the complexity behind the development of Large Language Models, but also the enormous potential of this technology. Innovative approaches such as Acceptance Test Driven LLM Development are opening the door to a new era of AI development - an era in which quality assurance and agile methods go hand in hand. This episode sheds light on the exciting future of AI technology and its many possible applications.