Test plan writing process

When there is no test plan writing process is defined, test plan writing can be lot of fun for first 60 % and then it can soon turn out to be activity filled with anxiety about missing scenarios. To help you overcome this anxiety 8 step test plan writing process is defined below.

Remember primary goal of verification is to ensure:

• All features and feature combinations are covered in all configurations
• All checks for the illegal and legal behavior expectations are put in place

Wear verification goggles, you’ll need it.

While writing test plan do not worry about the schedule and resources to execute it. Focus on the completeness. Think theoretically and execute practically.

In short, think about all unique scenarios possible first. Subsequently think about all the permutations and combinations of unique scenarios possible. Apply to it all possible concurrency and configurations. Really, that’s it. Now let’s look at what are some of challenges and how to overcome them.

Test plan writers especially the ones using coverage driven constrained random approach will fall in the trap of functional coverage plan and skip enumerating the possible scenarios in test plan. Verification engineers fall in the trap of magically functional coverage plan will take care of everything. Coverage plan has its own place but it cannot replace the test plan. Both are not same, they are different. Coverage plan is goal and test plan is the way to achieve it.

Constrained random stimulus might exercise it. Functional coverage may even indicate the holes. But not enumerating the scenarios in test plan can result in surprises later. Also the test bench architecture can get affected due to the lack of visibility into possible scenario space. Use the test plan to make everything as explicit as possible. You will thank yourself for doing that later.

The surprises can show up as missed verification areas or large test bench infrastructures development tasks later. It isn’t worth escaping trouble of writing detailed test plan by hiding behind the coverage plan.

 

Test plan is transformation of requirement specifications to verifiable requirement specifications keeping the design in mind. If you want to know more about how to look at design from verification point of view check: design for verification engineer.

Following 8 steps can also be thought of as 8 dimensions or variables for thinking about the test plan scenarios enumeration process. Idea is to introduce one new variable or dimension in each step.

In the following eight steps process when it’s referred to list all the unique scenarios in either data or control interface consider doing following:

• Both positive and negative scenarios
• DUT implementation of the specification
  • Stages in the pipelines
  • State machines and state transitions
  • Buffer depths, FIFO/Counter underflow and overflow
  • Arbitration logic
  • Low power logic implementations
  • Bandwidth and Latency considerations
• Use case driven scenarios possible

Without any further delay, let’s take the plunge into 8 dimensional test plan universe.

8 steps test plan writing process

Step#1: For every “data interface” all “unique scenarios”

Through data interfaces design receives the data to be processed. There can be multiple data interfaces in the design. Multiple data interfaces can be of multiple instances of same type or of different types.

Enumerate the different possible unique scenarios on each of these unique data interfaces independently.

Step#2: For every “data interface” all “combinations” of unique scenarios

Figure out the permutation and combinations of the unique scenarios relevant for each of the unique data interfaces. These will form the data interface sequences.

Step#3: Across “data interfaces” possible “concurrency” of scenarios

Figure out how many of these data interfaces can operate concurrently at any given time. Idea of this step is to identify various combinations of the concurrencies and any of their relationships possible.

Exercise different combinations of concurrency of scenarios identified across allowed data interfaces.

Step#4: For every “control interface” all “unique scenarios”

Design receives the commands and status requests through control interface. There can be multiple control interfaces in the design. Multiple control interfaces can be of multiple instances of same type or of different types.

Enumerate the different possible unique scenarios on each of these unique control interfaces independently.

Step#5: For every “control interface” all “combinations” scenarios

Figure out the permutation and combinations of the unique scenarios relevant for each of the unique control interfaces. These will form the control interface sequences.

Step#6: Across “control interfaces” possible “concurrency” of scenarios

Figure out how many of these control interfaces can operate concurrently. Idea of this step is to identify various combinations of the concurrencies and their relationships to be exercised.

Exercise different combinations of scenarios identified concurrently across allowed control interfaces.

Step#7: Across “data and control interface” possible “concurrency”

In the previous steps the data and control interfaces were operating separately. Within both the data and control interfaces were exercised concurrently with all the possible combination of scenarios.

In this step we identify the possible interaction between the data and control interfaces. Identify the level of concurrency allowed between the data and control interfaces.

Exercise the various combinations of interactions between the data and control interfaces concurrently.

Step#8: Across “Configurations & modes” concurrent “data & control”

So far we have focused on exercising all the data interfaces and control interfaces concurrently in all possible combinations. We did not bother about the various configurations or modes of the design’s operations.

In this step we exercise the all of the above in the different configurations or modes of the design’s operation.

One last question..

Let’s assume for scenarios identified in each of these steps were mapped to independent tests. Let’s call tests of steps#1 to #7 as “simple tests” and tests of step#8 as “super tests” for differentiation. One question that pops out is, when we have super tests do we still need simple tests?

Answer is debatable. My answer is yes. One can choose not to run all of the simple tests every time during the full regression but it’s important to maintain the all tests. Simple tests can also be helpful primarily in two ways:

• Super test since it contains so many scenarios it may not be able to create certain repeated pattern scenarios quickly. Theoretically it will get to it but not quickly. For example lets say a memory controller design, for every memory write leaks 1 byte of memory. In super test number of memory writes might be limited compared to simple test of step#2 which does only combinations of memory writes. So simple test of step#2 doing multiple memory writes will catch this issue faster
• It’s easy to debug the failures in simple tests rather than in the super tests. If we don’t have simple test all failures will show up in super test making every debug harder and more time consuming