Successful Verification Team Composition

Verification team composition is one of the important aspect for successful closure of verification projects. Verification methodology has focused extensively on the engineering problem alone. This has distanced verification methodology from the reality.  Thus at times we see the verification environments using all aspects of methodology but yet failing to deliver the desired results.

Its time, the best-known practices of verification project management also got into the verification methodology. This will make verification methodology a holistic approach to help achieve functional verification quality. One of the key aspects of the verification project management is formation of verification teams.

It’s no secret that successful verification requires team of experts and engineers to come together. In order for this to happen effectively, at bare minimum various roles and guidelines for responsibilities of those roles have to be laid out.

Here is an attempt towards the key roles required to be played in the successful completion of verification project. Note that based on the complexity of project and availability, each role can be played by different individuals or single person could be playing multiple roles. Bottom line is all these roles have to be played for efficient execution and successful closure of verification.

I am capturing five key roles:

1. Specification expert
2. Verification expert
3. Verification manager
4. Verification engineers
5. Automation expert

 

Specification expert

Requirements for verification could be industry standard specifications or custom specifications for an innovative proprietary designs or mix of both.

Along with the requirements specifications, there are specifications of the design implementation. This is user point of view of design usage in terms of possible modes of operation, various interfaces of the design, programmable registers, clocks and reset requirements etc.

“Specification expert” needs to have thorough understanding of all these specifications. He plays a key role in the planning phase of the verification.

He is responsible for:

• Maintaining the latest specifications
• Keeping track of supported multiple revisions of specifications, if applicable. This includes the changes to design implementation specification as well as it evolves during the course of implementation
• If multiple specifications are implemented which needs to work together, keep track of which version of specifications are compatible with each other
• Should be able to answer any questions regarding specifications. If he cannot answer himself take it up with right forum and get the answers
• Should be able to spot the ambiguities in the specifications. If the answers to some of these ambiguities cannot be given immediately make sure to provide the reasonable direction to move ahead. Track those and take it up with the right forum and get the answers.
• Should have good understanding for why’s and motivations behind major features of the specification. This is key to answering some of the non explicit aspects, resolving ambiguities and prioritizing
• Should be participating and contributing to standard bodies for the specification creation. Also keep track of ongoing useful discussions and important roadmaps
• Should have good understanding of the use cases for design as well
• He is also one of the key contributor to verification plan by doing verificaiton plan reviews

Verification expert

Today’s verification world offers array of tools and techniques to achieve the results. To quote few of them are, portable stimulus, directed verification, constrained random verification, assertion driven formal verification, accelerated simulation and emulation technologies.

A “Verification expert” is the one who understands the fundamentals of the verification well. He is aware, experienced, understands the verification requirements and technologies available.

He also needs to have good understanding of the requirements specification as well as micro-architecture of the implementation.

He is responsible for:

• Selection of the right combination of the verification technologies for the project and Suggest the newer verification technologies for evaluation and adoption
• Should deliver,  Verification plan made up of execution ready Test plan, Coverage plan and Checks/Assertion plan. These may not be fully complete in terms of all leaf level cases. But the sections and major features need to be covered by it. Some of the missing cases can only be determined only during the course of the execution
• Should deliver, architectures of various test bench environments and components. These should not be just laundry lists suggested by the verification methodology. Rather should be able to call out the abstract APIs, data structures to be used, threads and their pseudo algorithms
• Sets up the sign off criteria for the verification closure
• He is also a technical lead who would provide the guidance to team on various verification components implementation
• Breaks up the implementation into various tasks. Works with the verification manager for the task assignments. Should make recommendation and request for right engineers for implementation
• He is also responsible in ensuring the quality of the verification done, maintainability of the code and writing down any technical documentations

Verification manager

Rise in complexity of projects has also lead to larger verification team sizes. Larger teams have resulted in project executions challenged with contradictory demands on execution.

“Verification manager” is single point contact responsible for delivering quality results. Clarity leads to results. Hence his primarily responsible for ensuring clarity among chaos of execution. Standard management skills are also important but I am not going to describe those over here. This is more specific from verification project management point of view to manage planning phase, development phase and regression phase of verification project.

When these verification project management principles are not applied for various reasons out of control, it results in poor quality in planning phase, poor quality in development phase and poor quality in regression phase. The overall poor quality eventually requires a functional verification quality overhaul.

He is responsible for:

• He has good understanding of the business requirements of the project in hand. He will have to balance between budgets, business requirements, engineering resource optimization, technical and resources constraints. He is responsible in periodic evaluations to determine the right priority of the features execution
• Clarity is life. Ambiguity is death. Clarity drives best results. He needs to ensure the entire team remains clear on the results to be achieved and goals to be met. Teams have to march as single entity towards aligned goal
• Creates and drives the creation of the tasks lists. Detailed tasks lists are as important as verification plan itself. The detailed tasks lists improves, the schedule forecast accuracy. Use of right task management systems is highly undervalued in many projects
• Although development tasks are relatively easy to plan but the tasks such as regression cleanup and certain debugs can be unbound in nature.  Its needs to be smartly tracked and closed. Ideas to manage this in another post.
• Creates the weekly or monthly plans aligning to overall project milestones and allocates the tasks from the task tracking to engineers for implementation
• Makes the critical calls about when to pause and resume development to handle the dynamic situations such as codes releases and code freeze
• Sets up and drives the process definition and deployment for productive and efficient project execution
• Defines the different metrics to be used during the course of life cycle of the project. Uses the right metrics at right points to achieve the desired results
• Periodically publishes the verification status. Clearly puts the status in right perspective to upper management and to various engineering teams to enable them to act upon. Goes beyond just numbers to make sense of those numbers for driving the further execution
• Ensures that the tasks allocation to engineers are grouped so that it helps engineers execute optimally rather than doing random allocation to save his own time and effort. Works to opportunistically align individual engineers goals to project goals to create win-win situations
• Dynamically creates and merges various sub teams for execution efficiency
• Smaller teams with the focused area of execution with a theme, execute better than chaotic individual random assignments spread out. Chaotic random assignments can ease the leader or manager’s job of task assignments. Simplest random task assignment algorithm is, as and when task comes in just assign to engineer who has some bandwidth at that point. Don’t blame it on agile. Agile is not random task assignments. Random task assignments is a morale killer. It should be strictly avoided wherever possible.
• However there are going to be certain crunch times where this type of assignments may be necessary but making it a default practice is a reflection of poor project planning
• Raising issue(/bug) counts requires attention from verification manager to ensure it stays within the limits. Any cross team prioritization needs to be handled by him
• Selects the right projects for its for adoption of new verification technologies and spread learning’s of one project to other projects

Verification engineers

“Verification engineers” are primary workhorses who translate the plans to results. Verification teams, which are larger than 6 members should be strictly organized in more smaller sub teams.

Verification engineers will participate broadly in three major areas of activities:

1. Test bench – Stimulus generators, monitors, Scoreboards, Bus functional models, checkers, Functional coverage
2. Tests – writing the tests on reusable environment, executing tests in different parameters and configurations. Debugging the failures between the test bench vs design under test (DUT)
3. Coverage and regression management: Running different regressions, failure bucketing, failure assignments and tracking them to closure. Generating various coverage metrics. Publishing and looping in the appropriate team members for getting inputs for the convergence.

He is responsible for:

• Verification engineer should have clarity on the fundamentals of the verification concepts and test benches
• Verification engineer should be expert or familiar with the HDLs, high level verification languages and verification methodologies used in the verification environment
• Verification engineers should be intimately familiar with the simulators, version control, operating system, code collaborations tools, task tracking systems, issue(/bug) tracking systems and any other internal custom tools being used
• Verification engineers exposure to basics of software engineering especially data structures can go a long way in helping build good verification environments
• Verification engineers are expected to question the verification plan and test bench architecture to ensure they have gained the clarity. Figure out what is the best implementation to achieve verification objective
• Verification engineers should follow the practice of raising the issues(bugs) through the issue tracking system religiously. They are responsible for follow up of the issues raised till its closed
• Verification engineers will face challenge of getting closure on the issues filed. They should do periodic publishing of open issues, prioritizing and using either team meetings of quick one to one phone calls to drive it to closure
• Verification engineer will own, running various type of regressions periodically either through automation or semi manual way
• Verification engineer will keep updating the regressions status and various coverage metrics status. This is key to convergence and closure of verification activity
• Last phase where the verification where the coverage convergence needs to achieved, verification engineers should work closely with the design engineers wherever desired. Verification engineer needs to do the right home work to ensure such interactions are meaningful and effective
• Verification engineers should help recreate any of the failures not seen in the verification environment but showing up in emulation or silicon bring up. These are opportunities to question the verification plan and test benches.
• Verification engineers are responsible for periodic updates to the verification plan with the various new cases discovered. All major categories needs to be identified for the verification lead but each sub-section or test cases it self will have holes. These holes will be discovered as a part of execution. It’s extremely important to look at the bigger picture and see if it’s just specific hole or is there a missing theme to holes. Appropriately the verification plan should be updated  
• Verification engineer’s familiarity with the various automation and scripting languages goes in long way improving the verification productivity. In fact every verification team needs at least one automation expert

Automation expert

Verification is not a work that is one-size fits all type of activity. Each of the verification projects has their own unique challenges. This requires one to continuously evolve and adapt to keep the efficiency high. Due to this uniqueness and dynamic nature, off the shelf tools may not meet all your requirements. Every verification teams forced to do something specific to meet their own requirements.

“Automation expert” is your own in-house EDA/productivity tool vendor.

Automation engineer on board should be part time automation engineer. He should also be part of the verification activity to understand the problems better.

Automation problems, proven to have great potential after prototype can be handed off to special software teams dedicated for building them in scalable and right ways. Prototyping aspect is best handled by on board automation engineer with the verification team who understands the requirements well.

He is responsible for:

• Evaluating and choosing the right tools and tool versions to be used during the course of the project
• Efficient utilization of the compute grid resources and tool licenses
• Efficient compile, simulation and regression flows
• Regression failure triaging and failure tracking infrastructures
• Test plan, coverage plan and assertion plan management flows
• Debugging flows
• Code generation automations for repeated boilerplate codes. RAL(Register abstraction layer) for instance many companies have their own flow. From single source of register information RTL code, RAL model for the test benches and coverage information is generated
• Keeps track of the new features introduced by the various EDA tool vendors. Attempts, irons out issues and helps team to adopt them
• Of-course goes without saying he should be well versed with scripting such as Perl, Python and Shell scripting. More than scripting it’s about being able to figure out the ideas and translate them quickly to usable prototypes

At times, it may be very easy to staff all these folks. Key to success is ensuring each of them works together towards achieving the project objectives and does not get isolated in his own universe.