Reinventing Quality

ERROR: Content Element with uid "6796" and type "dce_dceuid16" has no rendering definition!

Test Coverage is one of the base problems in software testing. Whatever in black box testing or white box testing, achieving 100% coverage of code lines or requirements cannot guarantee that there is no bug left. Also it is not possible to measure the maximum possibility of fail by such test coverage metrics. Thus in test management, the confidence to the test result often is weak.

This report discusses a solution to this problem based on assessing test results by mathematical "Probability Theory".

In "Probability Theory", there is the "Law of Large Numbers". One instance of this law is "Hoeffding Inequality", which is the base of machine learning. This law can also be the base of software testing.

"Hoeffding Inequality" answers the relationship between sample amount and the "precise level" of the sample result. Thus the total number of test cases (equal to sampling amount) can be linked with "how precise" the test result is.

With "Hoeffding Inequality", we can

  1. assess, before testing, how many test case shall be planned to get precise enough test result
  2. assess, after testing, measure the risks remain in product in mathematical wa
  3. Make decisions based on reasonable data

The approach is to use "Confidence Interval" and "Confidence Level" to replace "Test Coverage". The "Confidence Interval" and "Confidence Level" are calculated by "Hoeffding Inequality" with sampling amounts.

Finally, some real examples are given to explain the utilization of "Hoeffding Inequality" in software test management.

Geng Chen, Sony

16 years experience in software development and testing
3 years in GSM Base System Development (Alcatel)
5 years in 3G mobile platform testing (NEC)
8 years in Customer Electronic Device Testing (TV, Walkman, Digital Camera and so on)  (Sony)

CSTQB Expert Group Member
ISO 29119 Chinese Mirror Group Vice Chairman
ISO 29119-6 Author