Traceable verification plan

Traceable verification plan is the one where following are all stitched together.

Test plan <-> Tests <-> Test variants <-> Regression status 

There is accountability for every test yet to be written, passing and failing test in the test plan. A test added to test plan is guaranteed to be executed. A verification plan that can be trusted. A traceable verification plan can be trusted. Its anchor for the functional verification quality.

Right not the concept is called out in the context of test plan which is the one of the three plans of verification plans where traceability matters most. Ideally it should be implemented to all three plans.

Why we need traceable verification plan?

Three phase of functional verification are: Planning phase -> Development phase <-> Regression phase. Requirements specifications are not really written for the functional verification.

During the planning phase verification plan is created. Verification plans creation is a process of transforming requirement specifications for verification. Verification plan has to be realized using the testbench. Testbench architecture is architected using the verification plan. Testbench is implemented using the high level verification languages and verification methodologies. Verification plan is executed through the tests and regressions.

This shows there are series of the transformations taking place. Each of these transformations has potential for loss of intent. Each loss is a potential bug introduced. It’s practically impossible to make these transformation a zero loss process. But measures should be put in place to minimize the loss.

One of the key measures is ensuring that loss of intent is minimized between the planning phase, development phase and regression phase. This can be accomplished by closely tying the test plan <-> Tests <-> Test variants in form of regression lists together.

What happens when verification plan is not traceable?

The stitching of Test plan <-> Tests <-> Test variants <-> Regression status, is the key to get confidence in verification plan. If it’s not done then even when the excellent quality verification plan created initially will die soon.

Even when it’s kept updated it won’t be able to give confidence unless it’s maintained like accounting balance sheet. Every single rupee spent or earned has to match and every expense is accountable. It has to be part of the everyday action that is taking place and balanced at the close of the day like bank’s account keeping mechanism. This is where verification transforms from art to accounting.

How traceable verification plan improves quality of verification?

Verification plan created initially, might of the best quality but unless it’s executed completely it’s not much of use. There are always chances that items can fall through the cracks between the planning phase, development phase and regression phase. It’s critical that there is traceability from the regression results to verification plan to make sure nothing falls through the cracks.

Changing specification be it the standard specifications or the design specifications pose biggest challenge to quality of verification. A large number of the design errors found are attributed to the incorrect or incomplete specifications and changing specifications according to studies conducted by the 2014 Wilson Research group. Incorrect or incomplete specification from the verification plan point of view needs questioning and tracking the ambiguities. Execution cannot stop till all the ambiguities are clarified.

Test bench should be architected to be additionally programmable to accommodate possible resolutions to ambiguities, which can be different from the assumptions made initially, to keep the execution going.

Changing specifications require repetition of the planning phase. As part of the repeated planning phase the verification plan and testbench architecture will have to be updated. Verification plan updates are required for sure the testbench architecture updates will depend on the extent of the specification changes and programmability of the testbench in the area of change.