Role of Regressions in Verification

Imagine a floor of the engineers working on design and verification of an interesting DUT. Designers are improving DUT’s strength everyday by adding new features. Verification team is busy catching up with the DUT’s features in test bench. Tests are written for the DUT’s features ready for verification. First milestone is approaching. Required features are ready in DUT, Test bench, BFM and in tests. Tests are executed for features required for milestone. Debug at war pace around for few days. Feature has passed all tests.

Parents of DUT and test bench are both happy. Little celebrations. Preparations start for next milestone.

Next question is.. 

What to do with passing tests?

If the Newton’s first law of inertia had stayed true in verification by keeping a passing test passing all the time, we did not have to do anything for passing tests. Unfortunately that’s not true.

Really, no kidding please. Are you saying, Newton’s first law is not holding up?

Yes. Passing test will not stay in passing state.

Why does passing test fail?

Although test itself is not changing, new seeds, changes to test bench code or DUT code, which is not even directly related to the test can lead to test failures. There is a complex web of dependency between test, test bench and DUT. Sometimes it may not be directly evident.

Thus a test, which is not changed, can still fail due to changes in other parts of the code. This means there is no guarantee that tests once passing will not stay in passing state. There needs to be effort and process to keep passing tests passing. Yes it may sound bit funny to engineers new to verification. But it’s true. Test in passing state will not stay passing state unless everything is frozen forever. This will not happen unless a company is wrapping up its business.

What can we do to keep passing test in passing?

Note that it’s impossible to keep passing test in passing state all the time during development. A passing test will fail several times throughout development cycles to chip tape out. A failure by itself is not bad. As long it is catching issues without significantly affecting forward progress.

What we do not want is sudden rise in failures, which forces to stop the development till they are fixed. This is bad for schedule as it prevents the forward progress.

Okay. I get that. But..

What can be done to increase the chances of keeping passing test passing?

Yes that’s a right question. We can only strive to improve chances of keeping it in passing state.

Every new feature being introduced will has potential to cause failure to already passing tests.  This happens, due to incomplete verification of already verified feature. It could be a missed configuration, missed stimulus or missing check for already verified feature. Random seeds or new checks added, new seeds could be leading to discovery of these failures. Another major cause is cross feature interaction. Interaction of the existing features with the new features can discover cases leading to failures in already passing tests.

Also it’s the humans who are developing the code and not the robots. Human mistakes are bound to happen. It’s not possible to completely eliminate it.

Regressions are the process to contain and minimize the impact of new changes on already passing tests. Regression in simple words is running all passing tests before every change is committed to development branch. This would ensure no bad changes can come in and passing test remains passing. This is not practically feasible as the number of tests grows. It will not be possible to run all the tests before committing every change to development branch.

This is not possible because of the time and resource limitations. If the entire regression can be completed within 30 minutes may be one run entire regressions before every check-in. But that’s hardly the case. Entire regressions can range from overnight to few days for any reasonably useful DUT. This delay is not acceptable for qualifying every change. If done can seriously slow down the development.

One common strategy used is to create a subset of full regression called check-in regression to be run before every code commit to development branch. This is designed to minimize the spikes in the failures of regression. Full regressions are run at a predetermined frequency separately at much lesser frequency than check-in regression.