February 23, 2012 -- A streamlined verification planning processes is required to meet today's quality and productivity expectations. The Metric Driven Verification (MDV) flow introduced by Cadence provides many enablers to achieve these goals. The MDV flow starts by creating an executable verification plan, namely the vPlan, using a tool feature called Enterprise Planner which is then read into the Cadence Incisive Enterprise Manager tool, along with the coverage results. The term "executable" refers to the idea that Enterprise Manager shows the vPlan with the coverage results and coverage percentages projected onto it.

Verification planning with Enterprise Planner is the most critical phase of the MDV flow because the verification goals are set here. This also aids in estimating the schedule of the project. Later in the verification cycle the results can be automatically annotated back to the vPlan and are used as the sign-off criteria for the design.

This article gives the guidelines in creating a verification plan using the vPlan. It explains a complete verification flow starting from verification strategy planning to closure. It also gives the case study of a legacy IP verification project, where the verification plan was previously done using an extensive documentation suite using Word documents, Mind Maps and Excel spread sheets, and shows how the MDV-based flow was used to improve the overall quality and productivity of functional verification.

A step-by-step approach in verification planning

Spec study

A well documented design specification is the starting point of your verification. The goal of your verification is to ensure that the design conforms to the functional specification. It is also important to make sure that the design's functional intent is completely captured in the specification and to clearly specify the limitations and assumptions made.

While reading the specification, it is important to record any mistakes, incomplete description and lack of clarity in different portions of the spec. You can capture these findings in a spreadsheet format and publish to the design team. The spreadsheet should have columns like 1) Page/section number, 2) Description of the problem and, 3) Status. This will be a running document throughout the verification project cycle and is used at the final stage to sign off the design.

Checklists

Every verification project should leverage on the lessons learned from the past. It is also important to have a common document that explains the guidelines to be followed.

Maintain these findings as a checklist and every project should use it as a criteria for final sign-off. Filling the verification checklist should start at the spec study phase itself.

Capturing the design features in vPlan

The vPlan is a graphical representation of your design's functional intent. Cadence provides the Enterprise Planner feature in order to create the vPlan. You can install the tool on Windows or Linux. Enterprise Planner lets users embed Microsoft Word-based images while in Windows as well as support Open Office diagrams while in Linux.

Your design can be meaningfully organized into features and sub-features. Before this, you are assumed to have read the design spec and have understood the designs functionality at least to a moderate level. You can then break the design to its major functional features.

Figure 1 gives an example of the vPlan functional hierarchy.

Figure 1. A vPlan functional hierarchy.

The design is basically an OCP-based RAM wrapper design. The design has the following major features:

OCP-based CPU bus OCP-based MMR bus MMR registers RAM auto-initialization Parity in RAM

The vPlan should have a section that gives the high-level description of the testbench architecture. This explains the overall testbench strategy, the key stimulus generators and monitors used. The vPlan allows drawing diagrams using Microsoft Word drawing tool boxes or Open Office Images. This helps in giving a detailed description of the testbench architecture with diagrams.

The structure of vPlan may or may not reflect the spec. It should, however, reflect the design's functional features in a hierarchical way. Also at this stage you are not interested in figuring out the scenarios and check points. The team should concentrate only in capturing the functional features of the design.

It's important to describe a key capability in Enterprise Planner called Spec Annotation which lets users highlight the areas in the functional specification and associate them with features in the verification plan. This capability offers several benefits. For one, users can see clearly what areas have already been covered. The other benefit is that when there are spec changes, Enterprise Planner will notify the user of the changes and let them react to those changes and modify the vPlan accordingly.

Begin by creating functional sections in the vPlan that correspond to heading sections of the functional spec. This makes sure that none of the spec sections are omitted in your plan. Spec annotation makes this process much easier. Read through the spec and identify other features. Each identified feature will be represented as a section in the vPlan, even though it may not have a heading in the spec. It's possible that one section in the vPlan which represents a feature is associated with several different areas in the spec. Spec annotation makes it easy to connect these different areas in the Spec to the single feature in the vPlan.

The effectiveness of a vPlan is decided mostly by its readability. The same feature section in the vPlan could appear in multiple places just for the sake of readability. Hence, redundancy should not be considered as an issue. An example for this is MMR register features. There should be a MMR Configuration section under which all the MMRs should be sub-feature sections. The same MMR sub-feature might appear in a functional feature where the MMR fields are used.

The vPlan has the capability to reference particular sections from another plan. Bus interfaces such as OCP, AHB and AXI can be represented with their own vPlans and provided along with the respective verification components such as Specman eVCs (e verification components).

These vPlans are assumed to take care of the protocol aspects of the bus. The user should add a design-specific features branch for the bus protocol in addition to the standard vPlan branch. For example, if the bus interface is an OCP, there should be a branch that checks the OCP profile for the design.

It is recommended to have a project-wide template vPlan used for all the IP-level activities. This gives a common look and feel for all the IP-level vPlans. Also, the reusable verification components that are used should have their own vPlans.

Planning verification strategy

Each and every verification problem can be unique. Choosing the right strategy for the problem is critical. The strategy should be reviewed and approved. A small presentation on the verification strategy followed by the review will help in reducing a lot of rework in planning and coding.

The presentation shall cover the main feature classes of the design and the type of verification technique chosen and the rationale behind it. The team makes critical decisions like whether to use a reference model or an assertion, or to use a full-fledged data scoreboard or use a few directed test cases. The importance of the feature targeted, the effort required and the available skill set are the trade-offs involved.

Planning checks

Once you have the features captured, the verification strategy also is decided upon. Now the detailed design of checkers is required. Enterprise Planner's planning section has three portions — the description, implementation details, and the extended details.

In the description section, you can add the items to check. Now in the implementation tab, add the "How to check it" aspect. It is not necessary to add a very elaborate description of the checks. A pseudo-code description is appropriate in most cases.

Under each feature section you should add the required check branches. Check branches target specific aspects or scenarios of the feature to be checked. Map the specific lines that explain the aspects on the spec to the branches. This will make sure that you have not omitted any section in the spec.

The functional spec can be huge and a line-by-line granularity of spec annotation mapping to the vPlan is not practical. This is also not essential since most of the time the spec writer uses several sentences to explain the aspects of the features. Your task is to identify the aspects and the sentences of spec that are related to the verification, and map those sentences to the check branch.

The key point is to correctly identify the aspect from the possibly vast description in the spec, and then choose the correct check mechanism to verify the same.

Figure 2 gives an example for the mapping of spec annotation checks to vPlan spec sections.

Figure 2. Mapping of spec annotation checks to vPlan spec sections.

The Memory Auto initialization section in spec is mapped to the RAM Auto initialization section of the vPlan. After carefully reading the section description in spec, four different aspects are identified:

ram_init_value_chk MEM_AUTO_INIT_DONE_pulse Init_time Init_negate_check

This means that specific checks or assertions are planned for these aspects. Figure 3 gives the plan section for ram_init_value_chk. The description is concise; it mentions what to check. The implementation details explain how the verification will catch any issues in RAM value. It is not mandatory that each aspect needs a specific checker. In some cases one scoreboard checker can cover many aspects. In those cases, the check branches can mention that the common checker will cover these aspects also.

Figure 3. Plan section for ram_init_value_chk.

Some of the aspects are not explicitly mentioned in the spec. The team should think through the possible faults that can happen and add additional checks for these faults.

A possible pitfall is that the design-verification (DV) team is biased to the spec writer's view of the design. It is important that the DV team understand the functional intent of the spec, have enough discussions and use past lessons learned to capture the functionalities of the design. It is possible that the team identifies some features which were not explicitly mentioned in the spec. The team can add vPlan branches to these features also.

Planning functional coverage

MDV is most useful in a coverage driven verification methodology. Functional-coverage definition is the goal setting phase for the verification task. It defines what scenarios of the design will be tested. The coverage definition can be classified in to two groups.

Micro-level functional-coverage definitions - Under each and every feature section in the vPlan, the corresponding functional-coverage requirement is defined using cover branches. This makes sure that each and every minute detail of the design aspects is covered. Macro-level functional-coverage definition - Issues can arise when different sub-blocks in the design interact. This requires a macro-level functional definition. The practical way of doing the same is to define the functional coverage for blocks/ features at a micro-level-coverage definition, and then add cross coverage of parameters across blocks/features.

The DV architecture should be such that maximum automation is possible in stimulus generation and checking. A sequence-based stimulus such as described in the UVM-e based methodology is recommended. The testbench should be self-checking.

The description in coverage branch should be the reflection of actual functional-coverage code that will be written. In the coverage description use an "item: bin" format. The implementation section can contain details such as how the coverage is collected (i.e., from monitor or from reference model) and when it is collected (i.e., the event that collects the coverage).

Each coverage branch should be mapped with the corresponding lines in the spec also. Extreme care should be taken that the coverage definition actually covers all aspects of functional behavior.

Review of the plan

The vPlan can be quite large, hence a single review section is impractical. It is recommended to have four different review sections:

Features captured review - The goal is to make sure that all features are added in the vPlan. There needs to be an agreement on the structure of the vPlan. The spec headings to vPlan section mapping can also be reviewed. Checkers review - The checks that are required for each feature and sub-feature. Follow the methodology related to checking. Coverage - Review the functional coverage definition. Sign-off review - 100% coverage is desired, but some bins can be excluded as not possible and this needs to be agreed upon and sign-off. Also any changes from the initial plan can be highlighted in this discussion.

Change in functional specification

The vPlan supports spec changes. The update spec option in Enterprise Planner allows reading a newer version of the spec. The spec manager portion will show the differences from the previous spec to the new one. However, the new sections and paragraphs that are added in the spec will not be shown explicitly. It is recommended that the user do a quick review of the new spec to see if any sections in the spec are un-mapped. (Mapped sections will be highlighted by a black outline).

vPlan in the testbench-coding phase

During the testbench-implementation phase, the implementation sections can be updated with notes. It is not required to give file names directory structures. The vPlan elements are mapped to the actual testbench components through Enterprise Planner, saving lots of manual entries and time.

A case study

This section explains a case study of the use of MDV in an IP-level verification flow. Even before adopting MDV, the team had the tools to handle quality and productivity requirements. However, MDV helped to simplify the effort involved in verification planning.

Figure 4 explains the verification flow before adopting MDV.

Figure 4. Verification flow before adopting MDV.

The activity is started from spec study phase, deciding on verification strategy, defining the functional coverage using the Free Mind tool, creating a verification plan and traceability matrix, and testbench-implementation plan.

The documentation requirement was huge. Also there are other pain areas:

Spec traceability - The team used a traceability matrix. This was a daunting manual task. Also, there were limitations in the level of granularity that can be provided in the matrix. Correlation - The verification activity is defined in four different documents. It was difficult to correlate between different sections in these documents. The review process was tiresome. Schedule estimation - The team used Excel spreadsheets for schedule estimation, but it was not granular enough to estimate accurately. Tracking verification progress - The consolidated coverage numbers alone were not enough to track the progress of verification. It was difficult to prioritize the work on important features because the traditional flow would not give feature-wise coverage statistics

Figure 5 explains the verification flow after adopting MDV.

Figure 5. Verification flow with MDV.

The number of deliverables was reduced from 4 (i.e. Free Mind Map, traceability matrix, verification plan and TB implementation plan) to 1. The vPlan became the single reference document for verification planning. It solved the spec traceability and correlation issues. The MDV flow provided coverage statistics per feature which gave a more realistic assessment of the progress and it was easy to prioritize activities as and when required. The vPlan was detail oriented, which made schedule estimation much easier since the activities were already broken into fine-grain tasks.

In summary

Synchronizing the multiple documents needed to create a meaningful verification plan is often painful and tedious, especially when requirements change or the functional specification changes. Our experience in maintaining and updating these multiple documents to keep them current was one of the biggest challenges in making sure that what we were trying to test was reflective of the most current design intents.

Any change in any of the documents meant that we needed to go back and check all the other documents as well to make sure we were interpreting the requirements and specifications correctly. In addition, because we did not have an executable verification plan previously, there was the possibility that even the verification plan itself could not be verified to match the actual testbench implementation.

Our experience in using the Enterprise Planner feature within Incisive Enterprise Manager allowed us to do synchronization of the four major documents needed for verification. Keeping all four verification documents synchronized together from one spot proved to be the only way to manage document churn and avoid wasted time tracking down the most current information.

The Metric Driven Verification flow with Enterprise Planner helps to improve the quality of verification planning and allows an automated closed-loop correlation and DV progress from spec feature to actual implementation of the coverage, checker, and stimulus. This in turn increased the productivity and helped in attaining zero bug slip-through. Because of this, verification teams were encouraged to upgrade their legacy flows to MDV.

By Henry Nguyen and Jentil Jose.

Henry Nguyen is IP Development Manager at Texas Instruments, Inc. and Jentil Jose is an IP Design Verification Engineer at Wipro Technologies.