Open Source VHDL Verification Methodology/Описание пакета CoveragePkg/en

1.9 About STANDARD revision 2.3

Revision 2.3 adds the function GetBin. GetBin is an accessor function that returns a bin in the form of a record. It is only intended for debugging. In particular, the return value of this function may change as the internal data types evolve.

Revision 2.2 adds AtLeast and Weights to the coverage database. The AtLeast value allows individual bins to have a specific coverage goal. A conjunction of the AtLeast and Weight (depending on the WeightMode) are used to weight the random selection of coverage holes. These features are at the heart of intelligent coverage.

The STANDARD revision of this package requires VHDL-2008. It will work with a VHDL- 2002 compliant simulator by uncommenting the VHDL-2008 compatibility packages.

CoveragePkg 2.X represents the coverage database in a protected type rather than signals as was done in the versions 1.X. The signal based coverage methodology has been moved to the package, CoverageSigPkg. Rather than use CoverageSigPkg, it is recommended that you migrate to the

2 A First Example: Manually Tracking Transfer Sizes

Functional coverage can be captured with any code. CoveragePkg and language syntax are solely intended to simplify this effort. As a first example, lets just write the code.

In a packet based transfer (such as across an ethernet port), most interesting things happen when the transfer size is at or near either the minimum or maximum sized transfers. It is important that a number of medium sized transfers occur, but we do not need to see as many of them. For this example, lets assume that we are interested in tracking transfers that are either the following size or range: 1, 2, 3, 4 to 127, 128 to 252, 253, 254, or 255. The sizes we look for are specified by our test plan.

We also must decide when to capture (aka sample) the coverage. In the following code, we will use a rising edge of clock where the flag TransactionDone is 1.

signal Bin : integer_vector(1 to 8) ;
. . .
process
begin
  wait until rising_edge(Clk) and TransactionDone = '1' ;
  case to_integer(unsigned(ActualData)) is
    when 1 => Bin(1) <= Bin(1) + 1 ;
    when 2 => Bin(2) <= Bin(2) + 1 ;
    when 3 => Bin(3) <= Bin(3) + 1 ;
    when 4 to 127 => Bin(4) <= Bin(4) + 1 ;
    when 128 to 252 => Bin(5) <= Bin(5) + 1 ;
    when 253 => Bin(6) <= Bin(6) + 1 ;
    when 254 => Bin(7) <= Bin(7) + 1 ;
    when 255 => Bin(8) <= Bin(8) + 1 ;
    when others =>
  end case ;
end process ;

Any coverage can be written this way. However, this is too much work and too specific to the problem at hand. We could make a small improvement to this by capturing the code in a procedure. This would help with local reuse, but there are still no built-in operations (such as reporting or save database).

Basic Point Coverage (aka Item Coverage) with CoveragePkg

Point coverage examines values within a single object, such as we did in the example in the previous section. The basic steps done when collecting functional coverage are declare the coverage object, model (describe) the coverage, accumulate the coverage, interact with the coverage data structure, and report the coverage. In this section we will explore how to do this with CoveragePkg.

Item or point coverage (single object) is a collection of one or more bins, such as the 8 bins from the previous example: 1, 2, 3, 4 to 127, 128 to 252, 253, 254, and 255. Internal to CoveragePkg, each bin is represented by a minimum and maximum value (effectively a range). Bins that have only one value, such as 1 are represented by the pair 1, 1 (meaning 1 to 1). Internally, the minimum and maximum values are stored in a record with other bin information.

The data structure for coverage is hidden within the protected type, CovPType. The first step to modeling a coverage item is to create a shared variable to hold the data structure, such as CovBin1 shown below.

architecture Test1 of tb is
  shared variable CovBin1 : CovPType ;
begin

The next step is to call the function GenBin to create the bins and then call the method AddBins to add the bins to the protected type. The version of GenBin shown below has three parameters: min value, max value, and number of bins. The call, GenBin(1,3,3), breaks the range 1 to 3 into the 3 separate bins with ranges 1 to 1, 2 to 2, 3 to 3.

TestProc : process
begin
  -- min, max, #bins
  CovBin1.AddBins(GenBin(1, 3, 3)); -- bins 1 to 1, 2 to 2, 3 to 3
  . . .

Additional calls to AddBins will append additional bins to the data structure. As a result, the call, GenBin(4, 252, 2), adds two bins with the ranges 4 to 127 and 128 to 252 respectively.

CovBin1.AddBins(GenBin( 4, 252, 2)) ; -- bins 4 to 127 and 128 to 252

Since creating one bin per value in the range is common, there is also a version of GenBin that has two parameters: min value and max value which creates one bin per value. As a result, the call GenBin(253, 255) adds three bins with the ranges 253 to 253, 254 to 254, and 255 to 255.

CovBin1.AddBins(GenBin(253, 255)) ; -- bins 253, 254, 255

The next step is to call the method ICover to accumulate coverage. This methodology supports either clock based sampling (shown below) or transaction based sampling (by calling ICover after a transaction that returns a value - shown in later examples).

  -- Accumulating coverage using clock based sampling
  loop
    wait until rising_edge(Clk) and nReset = '1' ;
    CovBin1.ICover(to_integer(unsigned(RxData_slv))) ;
  end loop ;
end process ;

When we are done with our test, we want to print out a report on the coverage. One way to decide we are done is to call the method IsCovered. IsCovered returns a true when all count bins in the coverage data structure has reached its goal. Just like ICover, IsCovered is called at a sample point (either an event or a transaction). A call to the method WriteBin prints the coverage results to OUTPUT (generally the transcript window when running interactively). The following code prints the coverage results after complete coverage is reached.

ReportCov : process
begin
  wait until rising_edge(Clk) and Bin1.IsCovered ;
  CovBin1.WriteBin ;
  wait ;
end process ;

Putting the entire example together, we end up with the following. Note that when we are working with coverage, we primarily work with integer values. All of the inputs to GenBin and ICover are integers, WriteBin reports results in terms of integers. Each of the methods have additional overloading that we will explore later.

architecture Test1 of tb is shared variable CovBin1 : CovPType ; -- Coverage Object begin TestProc : process begin -- Model the coverage CovBin1.AddBins(GenBin(1, 3 )); -- bins 1, 2, 3 CovBin1.AddBins(GenBin( 4, 252, 2)) ; -- bins 4 to 127 and 128 to 252 CovBin1.AddBins(GenBin(253, 255 )) ; -- 253, 254, 255 -- Accumulating coverage using clock based sampling loop wait until rising_edge(Clk) and nReset = '1' ; CovBin1.ICover(to_integer(unsigned(RxData_slv))) ; end loop ; end process ;   ReportCov : process begin wait until rising_edge(Clk) and Bin1.IsCovered ; CovBin1.WriteBin ; wait ; end process ;

Cross Coverage

Cross coverage examines the relationships between different objects, such as an ALU has done all of its supported operations with every different input pair of registers. The hardware we are working with is as shown below.

For this problem, the only supported operation is addition. The cross coverage needs to see every register used with every other register. This results in the following matrix of desired coverage points.

When collecting cross coverage, we follow the same steps used to create point coverage: declare, model, accumulate, interact, and report. These steps are shown in the code below. The first step is to declare the coverage object, ACov as a shared variable. Next we model the coverage using AddCross. For cross coverage, AddCross replaces AddBins. More on this shortly. Next we loop generating one transaction until we have complete coverage. The first step in the loop is to check IsCovered to see if we are done. Next we use RandInt to randomize the register addresses and get a value that is in the range of 0 to 7. There is more on RandInt in the RandomPkg documentation (available at http://www.SynthWorks.com/downloads). These register addresses are used by the procedure DoAluOp to generate transactions. After the transaction completes, we accumulate the coverage using ICover. After the loop completes, write out the coverage results and end the testbench (using the procedure, EndStatus).

architecture Test2 of tb is
shared variable ACov : CovPType ; -- Declare Cov Object
begin
TestProc : process
variable RV : RandomPType ;
variable RegIn1, RegIn2 : integer ;
begin
ACov.AddCross( GenBin(0,7), GenBin(0,7) ); -- Model
while not ACov.IsCovered loop -- Interact
-- Randomize register addresses -- see RandomPkg documentation
RegIn1 := RV.RandInt(0, 7) ;
RegIn2 := RV.RandInt(0, 7) ;
DoAluOp(TRec, RegIn1, RegIn2) ; -- Do a transaction
ACov.ICover( (RegIn1, RegIn2) ) ; -- Accumulate
end loop ;
ACov.WriteBin ; -- Report
EndStatus(. . . ) ;
end process ;

Collecting cross coverage is almost identical to collecting point coverage, except that the model and accumulate steps have changed. AddCross creates the cross product of the point bins (created by GenBin) on its inputs. Each call to GenBin(0,7) creates the 8 bins: 0, 1, 2, 3, 4, 5, 6, 7. As a result then, AddCross creates 64 bins with the pairs: (0,0), (0,1), (0,2), (0,3), (0,4), (0,5), (0,6), (0,7), (1,0), (1,1), (1,2), (1,3), … , (7,0), (7,1), (7,2), (7,3), (7,4), (7,5), (7,6), (7,7).

In the accumulate step, we still call ICover, however, with cross coverage, ICover has an integer_vector input with one value for each point in the cross cover. As a result, the call requires a second set of parentheses to denote an integer_vector aggregate with RegIn1 and RegIn2 as its elements.

While this model only does a cross product of two items, AddCross supports crossing of up to 20 items.

The Problem with Constrained Random

Earlier it was noted that a constrained random testbench takes O(N * Log N) to generate N unique test cases. The "log N" can add 10X to 100X to simulation run times. Running the previous ALU testbench, we get the following coverage matrix when the code completes. It takes 315 ≈ 64 * log(64) randomizations to generate this. By changing the seed value, the exact number of randomizations may increase or decrease (but this would be a silly way to try to reduce the number of iterations a test runs).

SRC2
SRC1
6 6 9 1 4 6 6 5
3 4 3 6 9 5 5 4
4 1 5 3 2 3 4 6
5 5 6 3 3 4 4 6
4 5 5 10 9 10 7 7
4 6 3 6 3 5 3 8
3 6 3 4 7 1 4 6
7 3 4 6 6 5 4 5

Intelligent Coverage

Intelligent coverage is what differentiates CoveragePkg from other coverage methodologies (syntax based or other). The process is simple. We add coverage goals and/or weights to the data structure and randomly select holes in the coverage to be the next item generated. Using this, we can generate each item exactly the number of times specified in its goal. The code is shown below.

architecture Test3 of tb is
shared variable ACov : CovPType ; -- Declare Cov Object
begin
TestProc : process
variable RegIn1, RegIn2 : integer ;
begin
-- AtLeast, Bin1, Bin2
ACov.AddCross( 2, GenBin(0,7,8), GenBin(0,7,8) ); -- Model
while not ACov.IsCovered loop -- Interact
-- Randomize register addresses -- see RandomPkg documentation
(RegIn1, RegIn2) := ACov.RandCovPoint ;
DoAluOp(TRec, RegIn1, RegIn2) ; -- Do a transaction
ACov.ICover( (RegIn1, RegIn2) ) ; -- Accumulate
end loop ;
ACov.WriteBin ; -- Report
EndStatus(. . . ) ;
end process ;

In this approach we add coverage goals to the modeling steps (AddCross, AddBins, and/or GenBin) and use RandCovPoint to randomize. In the code above, the AtLeast parameter of AddCross is 2. This means that the bin is considered a coverage hole until it has been seen twice. The method RandCovPoint randomly selects a coverage bin that is not covered and then randomly selects a value that is within that coverage bin. RandCovPoint always returns an integer_vector value.

Incremental Coverage Construction

Not all coverage can be modeled as the cross product of a set of points. Instead it can exclude either points, rows of points, or columns of points. While AddCross and AddBins allows all information to be entered in one call, they also allow incremental addition to the coverage model. Rather thinking about what to exclude, with CoveragePkg we think about what we want in the model and incrementally construct the model bin by bin if necessary. Hence, creating a high fidelity cross coverage model is relatively straight forward.

A high fidelity cross coverage model is important. Without one, reaching 100% coverage can be impossible. In addition, the Intelligent Coverage methodology will not work without it. Incremental construction of the coverage also facilitates giving different bins different weights.

As an example, consider the ALU, except disallow a register to be used with itself. In addition give each row a different coverage goal, as shown in the table below. By default, the coverage goal is also used as the weight. The weight determines the relative frequency of occurrence an item during randomization. While this is not a real example, it does demonstrate the CoveragePkg coverage modeling capability.

Coverage Goal Bin1 Bin2
1 0 1, 2, 3, 4, 5, 6, 7
2 1 0, 2, 3, 4, 5, 6, 7
3 2 0, 1, 3, 4, 5, 6, 7
4 3 0, 1, 2, 4, 5, 6, 7
5 4 0, 1, 2, 3, 5, 6, 7
6 5 0, 1, 2, 3, 4, 6, 7
7 6 0, 1, 2, 3, 4, 5, 7
8 7 0, 1, 2, 3, 4, 5, 6

To model the above coverage, we can do a separate call for each different coverage goal.

architecture Test4 of tb is
shared variable ACov : CovPType ; -- Declare Cov Object
begin
TestProc : process
variable RegIn1, RegIn2 : integer ;
begin
-- Capture coverage model
ACov.AddCross( 1, GenBin (0), GenBin(1,7)) ;
ACov.AddCross( 2, GenBin (1), GenBin(0) & GenBin(2,7)) ;
ACov.AddCross( 3, GenBin (2), GenBin(0,1) & GenBin(3,7)) ;
ACov.AddCross( 4, GenBin (3), GenBin(0,2) & GenBin(4,7)) ;
ACov.AddCross( 5, GenBin (4), GenBin(0,3) & GenBin(5,7)) ;
ACov.AddCross( 6, GenBin (5), GenBin(0,4) & GenBin(6,7)) ;
ACov.AddCross( 7, GenBin (6), GenBin(0,5) & GenBin(7)) ;
ACov.AddCross( 8, GenBin (7), GenBin(0,6) ) ;
while not ACov.IsCovered loop -- Interact
-- Randomize register addresses -- see RandomPkg documentation
(RegIn1, RegIn2) := ACov.RandCovPoint ;
DoAluOp(TRec, RegIn1, RegIn2) ; -- Do a transaction
ACov.ICover( (RegIn1, RegIn2) ) ; -- Accumulate
end loop ;
ACov.WriteBin ; -- Report
EndStatus(. . . ) ;
end process ;

GenBin, IllegalBin, and IgnoreBin

GenBin, IllegalBin, and IgnoreBin are used to create bins of type CovBinType. Using the functions replaces the need to know the details of the types. CovBinType is an array of the record type, CovBinBaseType. These are shown below.

type CovBinBaseType is record
. . .
end record ;
type CovBinType is array (natural range <>) of CovBinBaseType ;

GenBin

The following are five variations of GenBin. The ones with AtLeast and Weight parameters are mainly intended to for use with constants.

function GenBin(Min, Max, NumBin : integer ) return CovBinType ;
function GenBin(Min, Max : integer) return CovBinType ;
function GenBin(A : integer) return CovBinType ;
-- Only intended for constants
function GenBin(AtLeast, Weight, Min, Max, NumBin : in integer)
return CovBinType ;
function GenBin(AtLeast, Min, Max, NumBin : integer ) return CovBinType ;

The version of GenBin shown below has three parameters: min value, max value, and number of bins. The call, GenBin(1, 3, 3), breaks the range 1 to 3 into the 3 separate

bins with ranges1 to 1, 2 to 2, 3 to 3.
-- min, max, #bins
CovBin1.AddBins(GenBin(1, 3, 3)); -- bins 1 to 1, 2 to 2, 3 to 3

If there are less values (between max and min) than bins, then only "max - min + 1" bins will be created. As a result, the call GenBin(1,3,20), will still create the three bins: 1 to 1, 2 to 2 and 3 to 3.

CovBin2.AddBins( GenBin(1, 3, 20) ) ; -- bins 1 to 1, 2 to 2, and 3 to 3

If there are more values (between max and min) than bins and the range does not divide evenly among bins, then each bin with have on average (max - min + 1)/bins. Later bins will have one more value than earlier bins. The exact formula used is (number of values remaining)/(number of bins remaining). As a result, the call GenBin(1, 14, 4) creates four bins with ranges 1 to 3, 4 to 6, 7 to 10, and 11 to 14. CovBin2.AddBins( GenBin(1, 14, 20) ) ; -- 1 to 3, 4 to 6, 7 to 10, 11 to 14 Since creating one bin per value in the range is common, there is also a version of GenBin that has two parameters: min value and max value which creates one bin per value. As a result, the first call to AddBins/GenBin can be shortened to the following.

 -- min, max
 CovBin1.AddBins(GenBin(1, 3)); -- bins 1 to 1, 2 to 2, and 3 to 3

GenBin can also be called with one parameter, the one value that is contained in the bin. Hence the call, GenBin(5) creates a single bin with the range 5 to 5. The following two calls are equivalent.

CovBin3.AddBins( GenBin(5) ) ;
CovBin3.AddBins( GenBin(5,5,1) ) ; -- equivalent call

8.2 Illegal and Ignore Bins

When creating bins, at times we need to mark bins as illegal and flag errors or as ignored actions and not to count them.

The functions IllegalBin and IgnoreBin are used to create illegal and ignore bins. One version of IllegalBin and IgnoreBin has three parameters: min value, max value, and number of bins (just like GenBin).

-- min, max, NumBin
IllegalBin( 1, 9, 3) -- creates 3 illegal bins: 1-3, 4-6, 7-9
IllegalBin( 1, 9, 1) -- creates one illegal bin with range 1-9
IgnoreBin ( 1, 3, 3) -- creates 3 ignore bins: 1, 2, 3

There are also two parameter versions of IgnoreBin and IllegalBin that creates a single bin. Some examples of this are illustrated below. While this is different from the action of the two parameter GenBin calls, it matches the common behavior of creating illegal and ignore bins.

-- min, max
IllegalBin( 1, 9) -- creates one illegal bin with range 1-9
IgnoreBin ( 1, 3) -- creates one ignore bin with range 1-3
There are also one parameter versions of IgnoreBin and IllegalBin that creates a single
bin with a single value. Some examples of this are illustrated below.
-- AVal
IllegalBin( 5 ) -- creates one illegal bin with range 5-5
IgnoreBin ( 7 ) -- creates one ignore bin with range 7-7

8.3 Predefined Bins

The following are predefined bins.

constant ALL_BIN : CovBinType := GenBin(integer'left, integer'right, 1) ;
constant ALL_COUNT : CovBinType := GenBin(integer'left, integer'right, 1) ;
constant ALL_ILLEGAL : CovBinType := IllegalBin(integer'left, integer'right, 1) ;
constant ALL_IGNORE : CovBinType := IgnoreBin(integer'left, integer'right, 1) ;
constant ZERO_BIN : CovBinType := GenBin(0) ;
constant ONE_BIN : CovBinType := GenBin(1) ;

8.4 Combining Bins

Since GenBin, IllegalBin, and IgnoreBin all return CovBinType, their results can be concatenated together. As a result, the following calls to GenBin creates the bins: 1 to 1, 2 to 2, 3 to 3, 2 to 127, 128 to 252, 253 to 253, 254 to 254, and 255 to 255.

CovBin1.AddBins(GenBin(0, 2) & GenBin(3, 252, 2) & GenBin(253, 255));

Calls to GenBin, IllegalBin, and IgnoreBin can also be combined. As a result the following creates the four separate legal bins (1, 2, 5, and 6), a single ignore bin (3 to 4), and everything else falls into an illegal bin.

CovBin2.AddBins( GenBin(1,2) & IgnoreBin(3,4) & GenBin(5,6) & ALL_ILLEGAL ) ;

9 Data Structure Construction

The methods AddBins and AddCross are used to create the coverage data structure.

9.1 AddBins

The method AddBins is used to add point coverage bins to the coverage data structure. Each time it is called new bins are appended after any existing bins. AddBins has additional parameters to allow specification of coverage goal (AtLeast) and randomization weight (Weight). By using separate calls to AddBins, each bin can have a different coverage goal and/or randomization weight.

procedure AddBins (CovBin : CovBinType) ;
procedure AddBins (AtLeast : integer ; CovBin : CovBinType) ;
procedure AddBins (AtLeast, Weight : integer ; CovBin : CovBinType) ;

9.2 AddCross

The method AddCross is used to add cross coverage bins to the coverage data structure. Each time it is called new bins are appended after any existing bins. AddCross has additional parameters to allow specification of coverage goal (AtLeast) and randomization weight (Weight). By using separate calls to AddCross, each bin can have a different coverage goal and/or randomization weight.

procedure AddCross(
Bin1, Bin2 : CovBinType ;
Bin3, Bin4, Bin5, Bin6, Bin7, Bin8, Bin9, Bin10, Bin11, Bin12, Bin13,
Bin14, Bin15, Bin16, Bin17, Bin18, Bin19, Bin20 : CovBinType := NULL_BIN
) ;
procedure AddCross(
AtLeast : integer ;
Bin1, Bin2 : CovBinType ;
Bin3, Bin4, Bin5, Bin6, Bin7, Bin8, Bin9, Bin10, Bin11, Bin12, Bin13,
Bin14, Bin15, Bin16, Bin17, Bin18, Bin19, Bin20 : CovBinType := NULL_BIN
) ;
procedure AddCross(
AtLeast : integer ;
Weight : integer ;
Bin1, Bin2 : CovBinType ;
Bin3, Bin4, Bin5, Bin6, Bin7, Bin8, Bin9, Bin10, Bin11, Bin12, Bin13,
Bin14, Bin15, Bin16, Bin17, Bin18, Bin19, Bin20 : CovBinType := NULL_BIN
) ;

9.3 Multiple Matches within the Coverage Data Structure

When coverage has more than one bin, bins are processed in order. By default, if bins overlap, only the first matching bin is considered.

This behavior is controlled by the CountMode variable. This is an experimental feature and may be removed from future versions if it impacts run time. The default value of the variable is COUNT_FIRST. Setting the count mode to COUNT_ALL, as shown below allows all matching bins to be counted.

type CountModeType is (COUNT_FIRST, COUNT_ALL) ;
CovBin4.SetCountMode(COUNT_ALL) ; -- Count all matching bins
CovBin4.SetCountMode(COUNT_FIRST) ; -- default. Only count first matching bin

Looking forward to revision 3.0 of the package, when CountMode is COUNT_FIRST, bins that are contained in earlier bins will be removed. Repeated count bins will be merged. This action is likely to subsume the need for COUNT_ALL and works better in an environment with Intelligent Coverage and a mixture of count, ignore, and illegal bins. If you have a needed use model for COUNT_ALL, make sure to contact the package author. Also make sure to set it before adding items to the coverage model.

9.4 Controlling Reporting for Illegal Bins

By default, illegal bins both count and flag an error. This behavior is controlled by the IllegalMode variable. The default value of the variable is ILLEGAL_ON. Setting the count mode to ILLEGAL_OFF, as shown below suppresses printing of messages on an error.

type IllegalModeType is (ILLEGAL_ON, ILLEGAL_OFF) ;
CovBin4.SetIllegalMode(ILLEGAL_OFF) ; -- Illegal printing off
CovBin4.SetIllegalMode(ILLEGAL_ON) ; -- Default: Illegal printing on

9.5 SetBinSize

SetBinSize predeclares the number of bins to be created in the coverage data structure. Use this for small bins to save space or for large bins to suppress the resize and copy that occurs when the bins resize.

procedure SetBinSize (NewNumBins : integer) ;

10 Accumulating Coverage

The method ICover is used to accumulate coverage. For point coverage, ICover accepts an integer value. For cross coverage, ICover accepts an integer_vector. The procedure interfaces are shown below. Since the coverage accumulation is written procedurally, ICover will support either clock based sampling or transaction based sampling (examples of both shown previously).

procedure ICover( CovPoint : in integer ) ;
procedure ICover( CovPoint : in integer_vector ) ;

Since the inputs must be either type integer or integer_vector, conversions must be used. To convert from std_logic_vector to integer, numeric_std_unsigned and numeric_std provide the following conversions.

CovBin3.ICover( to_integer(RxData_slv) ) ; -- using numeric_std_unsigned (2008)
CovBin3.ICover( to_integer(unsigned(RxData_slv)) ) ; -- using numeric_std

To convert either std_logic or boolean to integer, CoveragePkg provides overloading for to_integer.

CovBin3.ICover( to_integer(Empty) ) ; -- std_logic
CovBin3.ICover( to_integer(Empty = '1') ) ; -- boolean

To convert either std_logic_vector or boolean_vector to integer_vector (bitwise), CoveragePkg provides to_integer_vector functions.

CrossBin.ICover( to_integer_vector(CtrlReg_slv) ) ; -- std_logic_vector
CrossBin.ICover( to_integer_vector((Empty='1')&(Rdy='1')) ) ; -- boolean_vector

Since the language does not do introspection of aggregate values when determining the type of an expression, the boolean vector expression needs to be constructed using concatenation (as shown above) rather than aggregates (as shown below).

--! CrossBin.ICover( to_integer_vector( ((Empty='1'),(Rdy='1')) )); -- ambiguous

11 Reporting Coverage

The procedures WriteBin and WriteCovHoles are used to report coverage. The following overloading for these methods are supported. String typed parameters are used since the language does not allow file typed parameters with protected type methods.

procedure WriteBin ;
procedure WriteBin (FileName : string; OpenKind : File_Open_Kind := APPEND_MODE) ;
procedure WriteCovHoles ( PercentCov : real := 100.0 ) ;
procedure WriteCovHoles ( FileName : string; PercentCov : real := 100.0 ;
OpenKind : File_Open_Kind := APPEND_MODE ) ;

The procedure method WriteBin prints out the coverage results with one bin printed per line. There are two versions. The first has no arguments and prints to OUTPUT. This is shown below. Note bins marked as either ignore are not printed by WriteBin and bins marked as illegal are only printed if they have a non-zero count.

ReportCov : process
begin
wait until rising_edge(Clk) and Bin1.IsCovered ;
CovBin1.WriteBin ;
wait ;
end process ;

The other version accepts two arguments. The first argument, FileName specifies the file name as a string. The second argument specifies the OpenKind argument (to file_open) and accepts either WRITE_MODE or APPEND_MODE. The OpenKind argument is initialized to APPEND_MODE.

-- FileName, OpenKind
CovBin1.WriteBin ("Test1.txt", WRITE_MODE);

The procedure method WriteCovHoles prints out coverage results that are below the PercentCov parameter. Note bins marked as either illegal or ignore are not printed by WriteCovHoles. PercentCov is initialized to 100.0 and it is common to leave it off.

CovBin1.WriteCovHoles ;

Another version of WriteCovHoles specifies FileName, PercentCov, and OpenKind in a similar fashion to WriteBin. The OpenKind argument is initialized to APPEND_MODE. This is shown below.

-- FileName, PercentCov OpenKind
CovBin1.WriteCovHoles("Test1.txt", 100.0, APPEND_MODE);

The methods SetName and SetItemName are used to print a first and second line heading for WriteBin and WriteCovHoles. SetName is intended to reference the name or intent of the coverage bin. SetItemName is intended to print column headings for the coverage bins. Each of these also uses its string parameter to initialize the internal randomization seed.

CovBin1.SetName("DMA") ; -- Group name - prints first CovBin1.SetItemName("Stat, WordCnt") ; -- Item Names

12 Coverage Goals, Weights, and Randomization

Coverage goals, weights, and randomization are the core of the Intelligent Coverage methodology.

12.1 Coverage Goals and Weights

A coverage goal specifies how many times a value must land in a bin before the bin is considered covered. A randomization weight determines the relative number of times a bin will be selected in randomization. Any of the following can be specified to be used as the randomization weight: coverage goal, weight, or remaining coverage. Each bin can have a different coverage goal and weight.

Selection of the weight mode is done using SetWeightMode. The following table lists the current set of supported modes and how the randomization weight is calculated.

Mode Weight
At_Least AtLeast
Weight Weight
Remain Scale*AtLeast - Count
Scale is used to adjust the weight when the desired percent coverage
is above or below 100%

The interface for procedure SetWeightMode is shown below. Note that the Scale parameter to SetWeightMode and modes REMAIN_AT_LEAST and REMAIN_WEIGHT are experimental and may not be in the next revision.

type WeightModeType is (AT_LEAST, WEIGHT, REMAIN, REMAIN_AT_LEAST,
REMAIN_WEIGHT);
procedure SetWeightMode (A : WeightModeType; Scale : real := 1.5) ;

12.2 Adding Coverage Goals and Weights to the Coverage Model

All of the bin construction methods and functions support an extra parameter that allows specification of AtLeast (coverage goal) and Weight (a potential choice for randomization weight). Primarily these are specified in AddBins and AddCover as shown previously.

We will see later that GenBin, IllegalBin, IgnoreBin, and GenCross also support specification of AtLeast and Weight, but this is primarily intended to allow their specification in a constant.

12.3 Basic Randomization

Randomization is handled by either RandCovPoint and RandCovHole. RandCovPoint returns a randomly selected value within the randomly selected bin and is type integer_vector. RandCovHole returns the randomly selected bin and is type RangeArrayType. Only bins that have not reached the specified percentage of their coverage goal (100.0 by default) are considered for randomization. The type RangeArrayType and the function definitions are shown below. Note it is recommended to use RandCovPoint if possible as RangeArrayType may change.

type RangeType is record
min, max : integer ;
end record ;
type RangeArrayType is array (integer range <>) of RangeType;
impure function RandCovHole (PercentCov : real := 100.0) return RangeArrayType ;
impure function RandCovPoint (PercentCov : real := 100.0) return integer_vector ;

12.4 Randomization, Illegal, and Ignore Bins

RandCovPoint and RandCovHole will never select a bin marked as illegal or ignore. However, if count bin overlaps with an illegal or ignore bin then the illegal or ignore value may be generated by randomization. Currently the method around this is to carefully construct your bins so this will not happen.

12.5 Randomization Thresholds

Each call to RandCovPoint and RandCovHole allows the specification of PercentCov. For most tests the value is 100.0. RandCovPoint and RandCovHole randomly select any bin that has less than PercentCov of coverage. Setting PercentCov below 100.0 allows certain thresholds within the coverage bins to be met. Ideally this threshold would be some value that is within a certain percentage of the current minimum coverage in the coverage model and would increase as the minimum coverage increases. By setting PercentCov to a value less than the current minimum coverage value, to say 0.0, then automatic thresholding is used. The automatic threshold is set to be the current minimum coverage plus the internal

CovThresholdPercent variable value. The intenternal CovThresholdPercent variable is set by the procedure SetCovThreshold. When using automatic thresholding, it is recommended to set PercentCov to 0.0.

procedure SetCovThreshold (Percent : real) ;

Automatic thresholding is useful when trying to get a balanced solution across all bins when the bins have a coverage goal that is greater than 1. Automatic thresholding extends the notion of cyclic randomization to work across a cross coverage set of values where each value is to occur more than once.

12.6 Setting the Seeds

The internal seed for randomization can be initialized using the procedure methods, InitSeed [string] and InitSeed [integer]. Their overloading and an example call is shown below.

procedure InitSeed (S : string ) ;
procedure InitSeed (I : integer ) ;
. . .
CovBin1.InitSeed( CovBin1'path_name ) ; -- string
procedure SetSeed (RandomSeedIn : RandomSeedType ) ;
impure function GetSeed return RandomSeedType ;

In addition, the procedure methods SetName and SetItemName (used with reporting coverage) also call InitSeed with their parameter. Their overloading and an example call is shown below. As a result, for most tests, usage of SetName or SetItemName is sufficient to ensure that each coverage model has a unique randomization seed.

procedure SetName (NameIn : String) ;
procedure SetItemName (ItemNameIn : String) ;
. . .
CovBin1.SetName("DMA: Stat, WordCnt") ; -- also calls InitSeed

The methods GetSeed and SetSeed are intended for saving and restoring the seeds. In this case the seed value is of type RandomSeedType, which is defined in RandomBasePkg. RandomBasePkg also defines procedures for reading and writing RandomSeedType values.

13 Interacting with the Coverage Data Structure

In addition to randomization, the following methods are provided for interaction with the coverage data structure.

impure function IsCovered (PercentCov : real := 100.0) return boolean ;
impure function CovBinErrCnt return integer ;
impure function GetMinCov return real ;
impure function GetMaxCov return real ;
impure function CountCovHoles (PercentCov : real := 100.0) return integer ;
impure function GetCovHole(ReqHoleNum : integer := 1 ; PercentCov : real := 100.0)
return RangeArrayType ;

The function method, IsCovered, returns a true when all count bins in the coverage data structure has reached its goal. Just like ICover, IsCovered is called either at a sampling point of either the clock or a transaction.

The function method CovBinErrCnt sums up the count in each of the error bins and returns the resulting value. Generally CovBinErrCnt is called at the end of a testbench for coverage models that have bins marked as illegal.

TestErrCount := CovBin1.CovBinErrCnt + (Other_Error_Sources) ;

The function method GetMinCov returns the smallest number in any count field cover bins. The function method GetMaxCov returns the largest number in any count field of the cover bins. Both functions are called as follows.

MinCov := CovBin1.GetMinCov ;
MaxCov := CovBin1.GetMaxCov ;

The function method CountCovHoles returns the number of holes that are below the PercentCov parameter value (generally allowed to default to 100.0).

-- PercentCov
NumHoles := CovBin1.CountCovHoles( 100.0 ) ;

GetCovHole gets the ReqHoleNum bin with a coverage value less than the PercentCov value. The following call to GetCovHole gets the 5th bin that has less than 100% coverage. Note that ReqHoleNum must be between 1 and CountCovHoles. The value returned by GetCovHole is of type RangeArrayType (same type returned by RandCovHole).

-- ReqHoleNum, PercentCov
TestData := CovBin1.GetCovHole( 5, 100.0 ) ;

14 Coverage Database Operations

Coverage can be accumulated by writing out coverage results from one test and then reading the results back in before the next test. The methods used for this are shown below. String typed parameters are used since the language does not allow file typed parameters with protected type methods.

procedure ReadCovDb (FileName : in string) ;
procedure WriteCovDb (FileName : in string; OpenKind : File_Open_Kind :=
APPEND_MODE ) ;

The procedure method WriteCovDb saves coverage results into a file. It has two arguments. The first argument, FileName specifies the file name as a string. The second argument specifies the OpenKind argument (to file_open) and accepts either WRITE_MODE or APPEND_MODE. The OpenKind argument is initialized to APPEND_MODE. An example is shown below.

-- FileName, OpenKind
CovBin1.WriteCovDb( "CovDb.txt", WRITE_MODE ) ;

The procedure method ReadCovDb reads the values from a file. It has a single string typed FileName parameter. An example is shown below.

-- FileName
CovBin1.ReadCovDb( "CovDb.txt" );

The procedure method SetCovZero sets all the coverage counts in a coverage bin to zero. This allows the counts to be set to zero after reading in a coverage database. A simple call to it is shown below.

CovBin1.SetCovZero ; -- set all counts to 0

The procedure method Deallocate deallocates the entire database structure.

CovBin1.Deallocate ;

15 Creating Bin Constants

Constants are used for two purposes. The first is to create a short hand name for a point bin (normal constant stuff) and then use that name later in composing the coverage model. The second is to create the entire coverage model in the constant to facilitate reuse of the model.

15.1 Point Bin constants

In a previous model, we constructed a cross coverage model using the following call to AddCross.

ACov.AddCross( GenBin(0,7), GenBin(0,7) ); -- Model

On step of refinement is to create a point bin constant for the register addresses, such as REG_ADDR shown below. The type of REG_ADDR is CovBinType. Since constants can extract their range based on the object assigned to them, it is easiest to leave CovBinType unconstrained.

constant REG_ADDR : CovBinType := GenBin(0, 7) ;

Once creating the constant, it can be used in for further composition, such as shown below. Just like normal constants, this increases both the readability and maintainability of the code.

ACov.AddCross(REG_ADDR, REG_ADDR); -- Model

Since each element in a point bin may require different coverage goals or weights, additional overloading of GenBin were added. These are shown below.

function GenBin(AtLeast, Weight, Min, Max, NumBin : integer ) return CovBinType ;
function GenBin(AtLeast, Min, Max, NumBin : integer ) return CovBinType ;

As demonstrated earlier, point bins can be composed using concatenation. The following example creates two bins: 0 to 31 with coverage goal of 5, and 32 to 63 with coverage goal of 10.

constant A_BIN : CovBinType := GenBin(5, 0, 31, 1) & GenBin(10, 32, 63, 1) ;

15.2 Writing an Cross Coverage Model as a Constant

To capture a cross coverage model in a constant requires some additional types and functions. The following methodology is based the language prior to VHDL-2008 and requires a separate type definition for each size of cross coverage model. Currently up to a cross product of 9 separate items are supported by the following type. In VHDL- 2008 where composites are allowed to have unconstrained elements, this will be reduced to a single type (and cross products of greater than 9 can be easily supported).

type CovMatrix2Type is array (natural range <>) of CovMatrix2BaseType;
type CovMatrix3Type is array (natural range <>) of CovMatrix3BaseType;
type CovMatrix4Type is array (natural range <>) of CovMatrix4BaseType;
type CovMatrix5Type is array (natural range <>) of CovMatrix5BaseType;
type CovMatrix6Type is array (natural range <>) of CovMatrix6BaseType;
type CovMatrix7Type is array (natural range <>) of CovMatrix7BaseType;
type CovMatrix8Type is array (natural range <>) of CovMatrix8BaseType;
type CovMatrix9Type is array (natural range <>) of CovMatrix9BaseType;

The function GenCross is used to generate these cross products. We need a separate overloaded function for each of these types. The interface that generates CovMatrix2Type and CovMatrix9Type are shown below.

function GenCross( -- cross 2 point bins - see method AddCross
constant AtLeast : integer ;
constant Weight : integer ;
constant Bin1, Bin2 : in CovBinType
) return CovMatrix2Type ;
function GenCross(AtLeast : integer ; Bin1, Bin2 : CovBinType)
return CovMatrix2Type ;
function GenCross(Bin1, Bin2 : CovBinType) return CovMatrix2Type ;
function GenCross( -- cross 9 point bins - intended only for constants
constant AtLeast : integer ;
constant Weight : integer ;
constant Bin1, Bin2, Bin3, Bin4, Bin5, Bin6, Bin7, Bin8, Bin9 : in CovBinType
) return CovMatrix9Type ;
function GenCross(
AtLeast : integer ;
Bin1, Bin2, Bin3, Bin4, Bin5, Bin6, Bin7, Bin8, Bin9 : CovBinType
) return CovMatrix9Type ;
function GenCross(
Bin1, Bin2, Bin3, Bin4, Bin5, Bin6, Bin7, Bin8, Bin9 : CovBinType
) return CovMatrix9Type ;

Now we can write our constant for our simple ALU coverage model.

constant ALU_COV_MODEL : CovMatrix2Type := GenCross(REG_ADDR, REG_ADDR);

When we want to add this to our coverage data structure, we need methods that handle types CovMatrix2Type through CovMatrix9Type. This is handled by the overloaded versions of AddBins shown below.

procedure AddBins (CovBin : CovMatrix2Type) ;
procedure AddBins (CovBin : CovMatrix3Type) ;
procedure AddBins (CovBin : CovMatrix4Type) ;
procedure AddBins (CovBin : CovMatrix5Type) ;
procedure AddBins (CovBin : CovMatrix6Type) ;
procedure AddBins (CovBin : CovMatrix7Type) ;
procedure AddBins (CovBin : CovMatrix8Type) ;
procedure AddBins (CovBin : CovMatrix9Type) ;

To create the coverage data structure for the simple ALU coverage model, call AddBins as shown below.

ACov.AddBins( ALU_COV_MODEL ); -- Model

GenCross also allows specification of weights. In a similar manner to AddCross, we can build up our coverage model incrementally using constants and concatenation. This is shown in the following example.

architecture Test4 of tb is
shared variable ACov : CovPType ; -- Declare Cov Object
constant ALU_BIN_CONST : CovMatrix2Type :=
GenCross(1, GenBin (0), GenBin(1,7)) &
GenCross(2, GenBin (1), GenBin(0) & GenBin(2,7)) &
GenCross(3, GenBin (2), GenBin(0,1) & GenBin(3,7)) &
GenCross(4, GenBin (3), GenBin(0,2) & GenBin(4,7)) &
GenCross(5, GenBin (4), GenBin(0,3) & GenBin(5,7)) &
GenCross(6, GenBin (5), GenBin(0,4) & GenBin(6,7)) &
GenCross(7, GenBin (6), GenBin(0,5) & GenBin(7)) &
GenCross(8, GenBin (7), GenBin(0,6) ) ;
begin
TestProc : process
variable RegIn1, RegIn2 : integer ;
begin
-- Capture coverage model
ACov.AddBins( ALU_BIN_CONST ) ;
while not ACov.IsCovered loop -- Interact
-- Randomize register addresses -- see RandomPkg documentation
(RegIn1, RegIn2) := ACov.RandCovPoint ;
DoAluOp(TRec, RegIn1, RegIn2) ; -- Do a transaction
ACov.ICover( (RegIn1, RegIn2) ) ; -- Accumulate
end loop ;
ACov.WriteBin ; -- Report
EndStatus(. . . ) ;
end process ;

16 Reuse of Coverage

There are a couple of ways to reuse a coverage model. If the intent is to reuse and accumulate coverage across tests, then the only way to accomplish this is to use WriteCovDb and ReadCovDb. If the intent is to just reuse the coverage model itself, then either a constant or a subprogram can be used. The calls to ICover generally are simple enough that we do not try to abstract them.

17 Compiling

We compile all of our packages into a library named SynthWorks. Currently CoveragePkg does not use any VHDL-2008 features, so there is only one version of the package. Your programs will need to reference CoveragePkg.

library SynthWorks ;
use SynthWorks.CoveragePkg.all ;

18 CoveragePkg vs. Language Syntax

The basic level of item coverage (aka: point coverage) that can be captured with CoveragePkg is similar to when can be captured with IEEE 1647, 'e'. CoveragePkg and 'e' allow a item bin to consist of either a single value or a single range. SystemVerilog extends this to allow a value, a range, or a collection of values and ranges. While this additional capability of SystemVerilog is interesting, it did not seem to offer any compelling advantage that would justify the additional complexity required to specify it to the coverage model.

For cross coverage, both SystemVerilog and 'e' focus on first capturing item coverage and then doing a cross of the items. There is some capability to modify the bins contents within the cross, but at best it is awkward. On the other hand, CoveragePkg allows one to directly capture cross coverage, bin by bin and incrementally if necessary. Helper functions are provided to simplify the process. This means for simple things, such as making sure every register pair of an ALU is used, the coverage is captured in a very concise syntax, however, when more complex things need to be done, such as modeling the coverage for a CPU, the cross coverage can be captured on a line by line basis.

As a result, with CoveragePkg it is easier to capture high fidelity coverage within a single coverage object. A high fidelity coverage model in a single coverage object is required to do Intelligent Coverage.

19 Deprecated Methods

In the original design of the coverage feedback and randomization functions, there was no coverage goal or weight. Instead, each bin has a weight of 1 and the coverage goal is determined by the AtLeast parameter in the function calls. These functions are shown below. In this implementation, all bins had the same coverage goal. Usage of the AtLeast parameter has been subsumed by the real valued PercentCov parameter. In addition, each bin now has the capability to have a different coverage goal and weight. With different coverage goal values, PercentCov has replaced the AtLeast parameter. The functionality of the AtLeast parameter has been subsumed by has been subsumed by the PercentCov parameter. A PercentCov parameter of 200.0 is equivalent to an AtLeast parameter of 2.

impure function GetMinCov return integer ;
impure function GetMaxCov return integer ;
impure function CountCovHoles ( AtLeast : integer ) return integer ;
impure function IsCovered ( AtLeast : integer ) return boolean ;
impure function GetCovHole ( ReqHoleNum : integer := 1 ; AtLeast : integer )
return RangeArrayType ;
impure function RandCovHole ( AtLeast : in integer ) return RangeArrayType ;
impure function RandCovPoint (AtLeast : in integer ) return integer_vector ;
procedure WriteCovHoles ( AtLeast : in integer ) ;
procedure WriteCovHoles ( FileName : string; AtLeast : in integer ; OpenKind :
File_Open_Kind := APPEND_MODE ) ;

20 Future Work

CoveragePkg.vhd is a work in progress and will be updated from time to time. Some of the plans for the next revision are:

• Add a compress method that removes normal bins that are entirely contained in a previously defined ignore or illegal bin.
• Add a global coverage settings protected type.
  • Add capability for global on/off for coverage collection.
  • Set defaults for CountMode, IllegalMode, WeightMode, CovThresholdPercent in the global coverage model.
• Remove OrderCount (was for development purposes only).
• Consider overloading AddBins to subsume AddCross — that way AddBins is the only method needed to create the coverage data structure.

21 Other Packages

In addition to the CoveragePkg, we also are freely distributing our randomization packages (RandomPkg, RandomBasePkg, SortListPkg_int). See http://www.SynthWorks.com/downloads. Over time we will also be releasing other packages and hope to convince simulation vendors to distribute our libraries with their tools.

22 About the Author

Jim Lewis, the founder of SynthWorks, has twenty-six years of design, teaching, and problem solving experience. In addition to working as a Principal Trainer for SynthWorks, Mr Lewis has done ASIC and FPGA design, custom model development, and consulting. Mr Lewis is an active member of the VHDL standards effort and is the current IEEE VHDL Study Group chair. I am passionate about the use of VHDL for verification. If you find bugs with any of SynthWorks' packages or would like to request enhancements, you can reach me at jim@synthworks.com.

23 References

[1] Jim Lewis, VHDL Testbenches and Verification, student manual for SynthWorks' class.

[2] Andrew Piziali, Functional Verification Coverage Measurement and Analysis, Kluwer Academic Publishers 2004, ISBN 1-4020-8025-5

[3] IEEE Standard for System Verilog, 2005, IEEE, ISBN 0-7381-4811-3

[4] IEEE 1647, Standard for the Functional Verification Language 'e', 2006

[5] A Fitch, D Smith, Functional Coverage - without SystemVerilog!, DVCON 2010

24 When Code Coverage Fails

While code coverage is generally a useful metric, there are some cases where it does not accomplish what we want.

To help understand the issue, consider the following process. If SelA, SelB, and SelC all are 1 when Clk rises, then all of the lines of code execute and the code coverage is 100%. However, only the assignment, "Y <= A" has an observable impact on the output. The assignments, "Y <= C" and Y <= B" are not observable, and hence, are not validated.

PrioritySel : process (Clk)
begin
  if rising_edge(Clk) then
    Y <= "00000000" ;
    if (SelC = '1') then
      Y <= C ;
    end if ;
    if (SelB = '1') then
      Y <= B ;
    end if ;
    if (SelA = '1') then
      Y <= A ;
    end if ;
  end if ;
end process ;

In combinational logic, this issue only becomes worse. If we change the above process as shown below, then it runs due to any change on its inputs. It still has the issues shown above. In addition, the process now runs and accumulates coverage based on any signal change. Signals may change multiple times during a given clock period due to differences in delays - either delta cycle delays (in RTL) or real propagation delays (in gate simulations or from external models).

PrioritySel : process (SelA, SelB, SelC, A, B, C)
begin
  Y <= "00000000" ;
  if (SelC = '1') then
    Y <= C ;
  end if ;
  if (SelB = '1') then
    Y <= B ;
  end if ;
  if (SelA = '1') then
    Y <= A ;
  end if ;
end process ;

Since functional coverage depends on observing conditions in design, it may cover all of the gaps. There are also additional tools that address this issue with code coverage.