Code coverage is composed of line coverage, expression coverage, toggle coverage, FSM coverage and conditional coverage. Code coverage is one of the oldest and reliable forms of coverage metric. It’s a working horse of verification engineers. Code coverage is one of the key metrics used in closure of verification.

There are two reasons for its popularity. First it’s automatically generated. And second it’s comprehensive.

Code coverage is automatically generated from the standard simulator tools. It just requires enabling few additional options during compile of the code and during run of test cases. A complete and comprehensive coverage report without any effort from user, that’s what, makes it so attractive.

While all this true, there are sides of code coverage that are not so attractive. There are some pain points. In this blog we will look in to three such pain points.

Pain point #1: Code coverage is not useful in early stages of verification cycles

Code coverage to become useful requires certain level of maturity to RTL code. That’s how it ends up being looked at the later in the verification cycle. What are the downsides of it?

While the comprehensive nature of the code coverage is its one of the advantage but it’s also a deterrent. Due to its comprehensive nature, code coverage requires good bit of time and effort to analyze and identify the actionable items for the verification to drive it to closure.

Sometimes a key coverage items not being covered can remain hidden till very end due to excessive information. Spotting these important holes early and prioritizing them can enable catching bugs where it matters. But spotting them is as difficult as proverbial saying of finding needle in hay.

Also in the early phases of the project the design code itself is not complete. So there may be little value being derived from code coverage. Code coverage cannot be cleanly divided into parts and organized to meet the requirements of intermediate milestones.

That’s how code coverage ends up being generally looked at towards the later point of the project. Since it’s indicating the coverage of the RTL design, it also requires good understanding of the design code to understand it and to use it effectively.

Pain point #2: Code coverage convergence dependency on designer

Verification engineers will have limited visibility and awareness about the RTL code. This makes it difficult for them to drive the code coverage closure all by themselves.

It’s also complicated by the fact that, a high level feature can be implemented in multiple ways. There may have been tradeoffs made for the synthesis or timing reasons, which may not directly evident in the context of feature requirements. Since verification engineers are more conversant with the feature rather than the implementation details, this can make the design understanding challenging. Verification engineers although have a broader idea of design but when it gets in to details it can become daunting for them. So they generally will be dependent on designers when it comes to analysis of the code coverage.

Especially while closing the last few remaining percentages of code coverage. They often require help from the designer to make sense of it. They need help translate the code coverage holes in to verifiable action items or exclusions.

This dependency comes at a point when RTL designer’s hands are full. We have already seen code coverage analysis and convergence starts during the later part of the project. This is a point where, RTL designers time is also requested by other tasks such as, timing closure, DFT, layout issues, CDC and general final checklist items etc. It’s challenging for designer to provide full attention to verification engineers to help close the code coverage holes.

Pain point#3: Coverage holes in code coverage

Code coverage will not understand the application intent of the design. For code coverage there is no difference between the network packet processors or next generation machine learning processor. As long as both are written in the same hardware description language, it treats them same. Of course that’s where the functional coverage makes its entry.

Leaving aside lack of application awareness for code there are few more inherent limitations which leads to holes in code coverage.

• Concurrency of operations
  • If there is N-input arbiter, how many inputs of arbiter were active at any given point of time cannot be covered by code coverage. It can tell you if branch of arbiter is covered but it will not tell you which other requests were active at the same time
  • If there N-instances of the design and you wanted to check if the all instances have asserted interrupts together, it can not be covered by the code coverage directly unless there is some expression or condition based on them
  • If there are multiple FSMs in the design when FSM1 was in state1 what states did FSM2 pass through cannot be addressed through the code coverage
• Sequences or order of certain events taking place
  • Whether resetn1 -> reseten2 were de-asserted as part of your reset sequencing cannot be found through code coverage
  • Whether FIFO full followed by incoming write request cannot be checked by the code coverage
• Sorry, not everything is made equal: Not all signals are same, not all expressions are same, not all conditions are same not all states of FSM are same. There is ranking among signals, expressions, conditions or states.
  • Not all interface or internal signals are same in their importance. An interrupt signal might be far more important than signal that is part of data bus
  • A functional reset might be more important to verification engineer than test mode related signals which are handled by DFT team
  • Not all expressions are made equal. Expressions controlling the key features are more important than expressions that are part of some fancy feature that is just going early in to this version of chip but only going to be enabled in later revision of the chip
  • Within expression certain combination might be more important than other. As all conditions are treated equal it might become difficult to spot the one of importance
  • Not all states of state machines are equally important. Some test mode state or compliance state may not be as important as error recovery state. Error recovery state will be entered and exited more frequent. Its entry and exit takes place during real operation in the field while test mode state is limited in its use and entered under very controlled lab environment or test setup
  • Not only that all states equally important, certain arcs from same state are more important than others. For certain state the duration for which design stays in it might also be of important.
  • For certain states how many times entry and exit occurred might be of great importance
  • For certain key signals not only its toggled but how many times it has toggled might be of importance
  • For certain signals how many times it has pulsed, irrespective of its duration might be of importance

Pain point#4: Code coverage exclusions are not portable

• Code coverage due to its comprehensiveness ends up showing some holes that are not relevant
• Now in order to enable sign off these non-relevant holes will have to be waived off. Line numbers in code, conditions are waived by directly pointing to them in the code. This ends up being highly non-portable. Code is living entity and it keeps changing and that makes maintaining this waivers a nightmare
• This is psychological block. In the next chip when the same code base with the enhancements and fixes is used, engineers will feel less motivated to make it converge as many holes that were waived off are showing up again

 

To summarize there are 4 pain points of code coverage:

• Pain point #1: Code coverage is not useful in early stages of verification cycles. Certain level of RTL maturity is required to make code coverage effective. This puts off the convergence to later part of the project
• Pain point #2: Due to lack of completed understanding of RTL code the verification engineers are dependent on RTL designers for final closure and exclusions
• Pain point#3: Code coverage although being comprehensive, lack application understanding, concurrency, sequences of operations and sufficiency of stimulus
• Pain point#4: Code coverage exclusions are very closely tied to RTL code. This makes them highly non-portable as code keeps changing.

Let’s continue the discussion further to see how functional coverage is helping relieve some of pain points of the code coverage.