Functional verification quality – Poor planning phase

Poor quality in planning phase leads to poor verification plans and test bench architecture. This is a seed. This seed grows into tree as the project progresses and yields bitter fruits later. Poor planning phase has big impact on the functional verification quality.

Bulk of the planning activities about 60-80% completes during the planning phase of the project and remaining is completed during the development phase. This provides opportunity to recover from some of the mistakes of the planning phase during development phase.

If there are quality problems found with the planning phase deliverables during development phase, it’s best to fix them at that point rather than ignoring or putting it off to later. It’s well known fact that cost of fixing the mistakes increases exponentially as the project progresses.

Lets look at what are the manifestations and effects of  poor quality verification plans and test bench architecture.

Poor quality Verification plan

Verification plan is backbone of the verification project and it stitches the planning phase, development phase and regression phase. Verification plan consists of the three plans : test plan, checks plan and coverage plan.

During the initial part of the project, especially for new designs as there are no references for the quality of the verification plan. This can lead to very high likelihood of poor quality verification plan. Especially if it’s hurried to execution prematurely. As stated it may not really become 100 % complete but major areas should not be missed. Width should covered but depth can be missing in certain areas. This can be filled up in development phase but when major features are missed, it leads to poor testbench architecture.

Poor quality verification plan results in:

• Missed features and configurations resulting in holes in functional verification manifesting as
  • Test plan with
    • Missed features
    • Missed test scenarios under specific configuration
    • Multi feature interactions
  • Coverage plan with
    • Missed parameters coverage
    • Transition coverage or scenario coverage
    • Concurrent events combination
    • Pipeline coverage
  • Checks plan with missing
    • Checks within the interface
    • Checks across interfaces
    • Temporal checks
• Verification strategy
  • Undefined
  • Defined ambiguously
  • Defined incompletely
  • Defined in a non executable form
• Verification plan is written at very high level leading to misinterpretation or partial execution by verification engineers

Poor quality testbench architecture

Are your verification engineers scared to touch test bench code today? Be assured that its architecture was broken long ago. Missing out the fundamental thought of stimulus generation and checks while architecting test bench results in testbench that do everything but verification.

Another trap is just building laundry list of the verification blocks based on the standard verification methodologies. This is very dangerous. It may a give illusion of having it done right but it’s often very high level generic architecture that fails during execution. 

Mere structural division is not sufficient, the attention needs to be paid to functional division, data flow and control event handling. Testbenches should not be architected in pieces. At least 60-70 % complete test bench architecture should be put in place to start with. If it’s not done testbench architecture will not be able to gracefully accommodate the changes coming in later. This is why 60-80 % complete verification plan is important. Without a good verification plan a good test bench cannot be architected.

Last but not least over designed test benches with the promise of reusability never realized also contributes to poor quality. Bloated test benches with unrealized promise of reusability are maintenance burden.

Poor quality testbench architecture results in:

• Inability to create certain type of scenarios
• Inability to concurrently stimulate and check for parallel interfaces
• Inability to create the asynchronous events injection
• Missing hooks for scoreboard and functional coverage at right abstraction levels leading to additional effort in their integration
• Structured with the mix of data and control leading to confusion and issues with concurrent data & control event execution
• Lacking the controllability – Every randomization should be controllable
• Lacking the observability – No logging architecture defined
• Mixing of the functionality in the components – lacking the right functional division
• Multiple test benches without clean functionality division between them leading to redundant test benches, features falling between cracks
• Tests implementing the testbench functionality leading to redundant code
• Falling in repeated pattern instead of being pseudo random by design
• Highly design dependent checkers, monitors and scoreboards getting highly tied and changes to design leading false failures
• Lack of the scalability leading to challenges in verifying the multi-instance designs
• No provision for customization even around the areas where there was ambiguity while architecting

Nature of Bugs 

Type of bugs due to poor quality in planning phase will be of the type:

• Basic bugs – These get discovered because the features or configurations itself has been missed either in verification plan or test bench architecture
• Testbench bugs – Lots of the test bench bugs discovered due to poor test bench architecture. This should be measured by tracking number of test bench bugs per RTL bugs discovered