Functional verification quality – Poor regression phase

Poor quality in regression phase is like spoiling the climax of the functional verification. If the right things have been done in planning phase and development phase, good news is it is easy recover back from quality lapses in the regression phase.

If the planning phase and development phase also had poor execution regression phase can become hell. Effects of poor planning phase and poor development phase impacts are not mixed. Focus here is effects of poor regression phase on functional verification quality.  

Regression phase is primarily targeted towards enabling new features using the all the variants of tests without affecting the features that are already verified. Objective is to keep increasing the pass rate, catch the issues and meet the coverage goals by minimizing the backtracking of the regressions. Check-in regression and full regression are two important components of the regressions.

Poorly exercised test suites

Poor quality in regression phase leads to primarily poorly exercised test suites an unmet coverage goals coupled with increased time spent in the debugs of the failures.

Poor quality of regression have following problems:

• Tests not exercised in all the variants leading to hidden bugs
• Constrained random tests seeded inadequately leading to hidden bugs
• Code coverage and functional coverage goals will not be met but a review could fix it too
• Lapse in the process of the failure triaging leading to longer regression closure cycles
• Planned tests not enabled in the regressions due tests slipping through the cracks between the development phase and regression phase
• Redundant tests running in regression causing longer duration of the regressions and longer regression run times leading delayed reporting of the genuine issues
• False failures reported due to regression management infrastructure issues
• False pass tests due to regression management infrastructure issues leading escaped bugs
• Tests with the temporarily and accidentally disabled checks continuing in the regression leading to escaped bugs
• Inadequate check-in regression compromising full regression stability   
• Ambiguous failure debug assignment process leading overload or under loading of verification engineers

Nature of bugs

Type of bugs due to poor quality in regression phase:

• Bugs in features in different configurations and in combinations with the other features
• Corner case bugs getting discovered when the seeding is varied or missed configurations are enabled
• Bugs in already verified features due disabled checks or missed tests