Abstracts

AI systems work differently from regular software. A model that seems perfect in testing can behave very differently with real users and data. Many teams still rely on one-time tests, which can make them feel overly confident and lead to unexpected problems later.

This session will focus on moving from one-time AI tests to a steady, practical way of evaluating AI as part of the ongoing product process. By using model checks, product testing, small, high-quality datasets, and observing how things work in real life, teams can build real trust in AI systems instead of just guessing when issues arise.

Through examples and real-life situations, this talk will show what works well, what doesn't, and why ongoing evaluation is more important than trying to achieve perfect scores. Attendees will learn how to create simple evaluation plans, pick useful metrics, avoid common mistakes, and keep track of how AI behaves as it changes in the real world.

Key Takeaways:

• Why we need to evaluate AI systems
• The difference between testing models and testing your real product
• How to create your first evaluation dataset in a few hours
• Three simple ways to measure AI quality
• How to keep an eye on your AI system after it goes live
• Common mistakes people make