Abstracts

AI-based systems are changing not only how software behaves, but also how we define when testing is "enough".

In traditional systems, testing completeness is often guided by coverage metrics, the number of test cases, or passing results. However, when dealing with AI-driven behaviour, these indicators quickly become unreliable. The same input may produce different outputs, tests may pass without validating real risks, and teams often struggle to justify when testing can stop or a system can be considered ready for release.

This session presents a practical, exploration-driven approach to this challenge. Starting from hands-on experimentation with AI-based features, it becomes clear that traditional validation strategies are insufficient to build real confidence in non-deterministic systems.

Through structured exploration, several recurring problems emerge:

• inconsistent outputs across similar inputs
• difficulty defining expected results
• outputs that appear correct but fail to address real risks
• a false sense of confidence created by passing tests

To address this, the session introduces a practical decision framework designed to help answer a critical question: when is testing sufficient?

Instead of relying on pass/fail criteria, this approach focuses on evaluating confidence across key aspects such as system consistency, variation in behaviour, risk relevance, and level of understanding. This allows teams to identify gaps and make more informed decisions about whether further testing is needed or a release can be approved.

The session demonstrates how this model can be applied in practice using realistic AI scenarios, such as conversational systems and API-driven interactions. Attendees will see how variations are explored, how outputs are evaluated without fixed expectations, and how confidence is built step by step.

Participants will leave with a structured and practical approach to testing AI systems, enabling them to better evaluate behaviour, identify risks, and make more reliable decisions in non-deterministic environments.