11:40 - 12:25 Uhr


Prof. Bruno Legeard
University of Franche-Comté / Smartesting

From Usage Analysis to Automated Regression Testing supported by Machine Learning: Experience Report

This presentation focuses on the automation of regression testing based on AI -assisted usage analysis in operation. We will describe the concepts, the process from acquiring anonymous usage data, and the visualization of usage traces to the generation and execution of automated test scripts. The focus is on functional regression testing of web applications in a DevOps and continuous testing context. Two full-scale experiments of this “Usage -driven test automation process” are analyzed. Each focuses on a different objective:

  1. SEPHORA: end-to-end regression tests on the eCommerce site
  2. GENERALI France: regression tests of usage scenarios on the partner APIs of their platform. We performed these experiments in close interaction with the QA teams. The results will be made public at the JFTL conference in Paris on June 14 (presentations by François VLERICK - SEPHORA, and Mahdi GARSALLAH & Mathieu MEYERE - GENERALI France) - see cftl.fr website to access to the presentation (but in French). Our presentation is divided into four parts:

1- Motivation for usage-driven test automation - the problem addressed: the relevance of automated regression tests to real-life usage and the maintenance costs of automated functional tests are significant challenges for Agile teams. This complicates the transformation to DevOps and continuous testing.

2- The process from usage analysis to test automation:

  • collecting anonymous usage data in compliance with the GPDR,
  • measuring and visualizing actual test coverage by comparing usage traces and test execution traces,
  • completing test coverage by selecting usage traces to be covered, and
  • generating & running automated tests on GUI or API.

3- Machine Learning (ML) techniques we experimented with usage and testing data: clustering traces to analyze the traces, anomaly detection on usage traces with unsupervised ML, and prioritization of test execution by ML models from past execution data.

4- Feedback from the field - SEPHORA and GENERALI France: what was their motivation and the context of implementing the approach? What was the obstacle in the data acquisition process, and what was the gain for tests' relevance? What was the gain for test automation? We conclude with a summary of how regression test generation from usage is a step towards autonomous software testing capabilities (for regression testing).

Style of the presentation
The target audience is Testers and Test Managers. The presentation will be concrete and factual. We will not detail the Machine Learning models used but rather the practical issues of acquiring and using anonymous usage data for regression test automation. The focus is on the response to the new challenges of the Agile transformation, with the acceleration of the release pace and the need to implement a Continuous Integration - Continuous Testing - Continuous Deployment approach.

Prof. Bruno Legeard, University of Franche-Comté / Smartesting

Bruno is a Professor of Software Engineering at the University of Franche-Comté (France), cofounder and scientific advisor of Smartesting, and an active member of the CFTL (ISTQB French Testing Board). He has 20 years of experience in software testing as a practitioner (consultant), researcher (Model-Based Testing; AI for testing), and trainer. Within the ISTQB, he has led the working groups for the Model-Based Testing and Acceptance Testing syllabi and contributed to an initial version of the AI Testing syllabus.
His current focus is on applying Machine Learning techniques for test automation, regression test generation, and test case prioritization. He is an experienced international speaker on topics related to Model-Based Testing in Agile, and AI for software testing, each time aiming to be concrete and aligned with the practical concerns of testers.