Full Regressions in Verification

Full regressions are ideally are required for every change. But it’s not practically possible in many real life use cases. Role of regression is to keep all passing tests in passing state they have to be run periodically and check their status.  If some of the tests found failing, fix them and get back to passing state. This process is called regression. Typically combination of full regression and check-in regressions are used to maintain the health of overall regressions.

Before a test is declared as passing it should be exercised in all the applicable configurations and test benches. If there are multiple test benches care should be taken to exercise in all the different test bench areas. After all these qualifications if the test is still passing it becomes part of the periodic regressions. Basically regression is a process of protecting what has been already built.

Full regression is run with regress list. A regress list contains the series of commands that can be run in a batch mode to run all the qualified tests. It will contain both the directed and constrained random tests. Unless it’s strictly directed test, it should be seeded appropriately. A strictly directed test has fixed configurations and fixed stimulus sequence. In a constrained random environment it’s rather hard to write a strictly directed test. Directed tests of the constrained random environment randomize the configurations but keep the stimulus sequence fixed. Based on the state space coverage a test is providing the test should be seeded. Seeding constrained random tests helps extracts full value from it.

Typically full regressions are run with the seeds. Number of seeds across tests can be scaled according to value of test, license and compute resources availability.  

Who should run the full regression?

Typically single owner in verification team owns full regressions. The results published by the regression owner should be used as official number for tracking status. Regression phase project management is really challenging. Key decision on exceptions to rules laid have to be made to meet the critical milestones.

Considering the sheer size of the full regressions it’s not feasible for everyone to run full regressions.Only exception for this rule is for the major code changes or close to critical milestones, even the developers doing the code changes can run the full regressions to gain additional confidence in their changes.

Full regressions result reference should be from single source to maintain consistency and validity. Like a legal document, it should be stamped and signed by full regression owner.

 

How periodically full regressions should be run?

Given the compute capacity, total licenses available, total number of tests in regression will set the limit on minimum duration required to run the complete regression. This is a real constraint. Let’s call it as physics limit. Everything else should be worked around it to meet the requirements.

Next important factor in deciding frequency should be driven by the rate of the new developments. If the new code changes are happening once every month then it’s sufficient to run it once every month. This may happen at the later phase of the project close to tape out. During the peak of development, rate of code change will be far faster with several code changes coming every hour. When the rate code change is far faster than full regression time then it becomes challenging to decide the frequency of full regressions.

During development phase, attempt should be made to run as frequently as possible guided by the full regression physics limit. What happens when the full regressions are delayed? Check-in regression will attempt to hold the fort but it’s on best effort basis. It’s possible that some check-in can cause the damage to full regression results. Earlier this damage is detected better are chances that it can be contained and recovered. A very simple solution might be to just back out the culprit changes. But as the time passes and more code changes are committed to development branch the impact of damage increases. It becomes complicated even to back out the changes.

Now when the physical limit for full regression is very large to be useful then full regressions can be broken into multiple levels. Group the tests based on the importance of the configurations and features. These subsets should be divided till the time limits reach a useful point. There on these subsets can be run at acceptable frequency. Important levels be should be run more frequently. It’s like choosing to save most valuable possessions in case of emergencies.

When to enable coverage in full regressions?

Answer depends on the phase of the project. At early development phases this is not required. Coverage of any form is not seriously analyzed during early phases of development. So early phase full regressions coverage need not be enabled. Enabling coverage hits on the run time performance. Enable it when it’s planned to be looked in to. Generating coverage reports that are not analyzed is waste of time and resources.

At the later stage when development matures coverage should be enabled. The run time hit it takes has to be taken into consideration during planning regression duration. Frequency of enabling coverage can reduced compared to frequency of full regression based on the granularity of the coverage tracking.

A simple rule to figure out when to start enabling coverage is, unless bugs rate starts showing the downward trend coverage analysis is an unaffordable luxury. When the bug rate comes under control, coverage convergence process can be started. That’s when it makes sense to enable the coverage to find holes and tune stimulus to uncover bugs in those areas. Bottom line, do not enable unless there is bandwidth available to use the coverage data generated.

What do with the failures of full regression?

Debug. Considering the total failures process has to be defined to make debug productive.

The sheer size of failure numbers can become scary at times. Point to keep in mind is it’s not about total failure numbers but the debug effort involved which is more important. Debug effort involved is not directly proportional to total failing numbers in many cases. This is because the unique failures within total failures may be lesser. Debug effort is required for debugging the unique failures and not all the failures.

It’s important to group the failures with same failure signature. Among this group debug the failure that is showing earliest in simulation time. Picking the one showing earliest makes it faster to simulate the failing scenario with the debug verbosity for logging and waveform dumps.

Unique failures have to be assigned out. Till they are debugged either they can be excluded from further regression or excluded from analysis for failure debug assignment.  Failures debugged should be classified as design, test bench and bus functional model issues. Bugs should be tracked through a bug tracking system. The failing tests should be re-enabled in the regression after the fixes are committed in development branch.

Extra points for making list of failures expected to pass before starting next full regression. This gives an idea of what to expect. It also gives window to pull in any additional fixes to make the best of the next full regression.