/usr/share/covered/doc/html/chapter.rank.html is in covered-doc 0.7.10-1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 | <html><head><meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"><title>Chapter 12. The rank Command</title><link rel="stylesheet" href="covered.css" type="text/css"><meta name="generator" content="DocBook XSL Stylesheets V1.71.1"><link rel="start" href="index.html" title="Covered User's Guide - 0.7.9"><link rel="up" href="part.command.line.usage.html" title="Part III. Command-line Usage"><link rel="prev" href="chapter.report.html" title="Chapter 11. The report Command"><link rel="next" href="chapter.exclude.html" title="Chapter 13. The exclude Command"><center><img src="img/banner.jpg"></center><hr></head><body bgcolor="#dfeef8" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"><div class="navheader"><table width="100%" summary="Navigation header"><tr><th colspan="3" align="center">Chapter 12. The rank Command</th></tr><tr><td width="20%" align="left"><a accesskey="p" href="chapter.report.html"><img src="img/prev.gif" alt="Prev"></a> </td><th width="60%" align="center">Part III. Command-line Usage</th><td width="20%" align="right"> <a accesskey="n" href="chapter.exclude.html"><img src="img/next.gif" alt="Next"></a></td></tr></table><hr></div><div class="chapter" lang="en"><div class="titlepage"><div><div><h2 class="title"><a name="chapter.rank"></a>Chapter 12. The rank Command</h2></div></div></div><div class="toc"><p><b>Table of Contents</b></p><dl><dt><span class="sect1"><a href="chapter.rank.html#section.rank.options">12.1. Options</a></span></dt><dt><span class="sect1"><a href="chapter.rank.html#section.rank.inputting">12.2. Methods for specifying CDDs to rank</a></span></dt><dt><span class="sect1"><a href="chapter.rank.html#section.rank.weighting">12.3. Specifying metric weights</a></span></dt><dt><span class="sect1"><a href="chapter.rank.html#section.rank.output">12.4. Understanding the rank command output information</a></span></dt><dt><span class="sect1"><a href="chapter.rank.html#section.rank.algorithm">12.5. Description of ranking algorithm</a></span></dt></dl></div><p>
The rank command allows the user to specify a number of CDD files that have been created from the same DUT (presumably in
a regression run -- but not necessarily) and rank them in order of coverage importance. This information can be useful
in accomplishing the following tasks:
</p><p>
</p><div class="itemizedlist"><ul type="disc"><li><p>
Understand which CDD files (simulations) can be eliminated from regression due to their inability to add
useful coverage information that selected CDD files already hit.
</p></li><li><p>
Understand the best order to run simulations in full regression.
</p></li><li><p>
It can help in creating a quick (confidence) regression which is meant to hit a good deal of the overall coverage
points in a short period of time to verify that recent changes don't massively break a previously working DUT.
This can be accomplished by selecting the first N simulation CDD files in the ranked list. The selection can
easily be done by either using the simulation order, accumulated hit, hit percent or timestep column values as
a means of selection.
</p></li></ul></div><p>
</p><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="section.rank.options"></a>12.1. Options</h2></div></div></div><p>
</p><div class="table"><a name="table.rank.options"></a><p class="title"><b>Table 12.1. Options to rank Command</b></p><div class="table-contents"><table summary="Options to rank Command" border="1"><colgroup><col><col></colgroup><thead><tr><th>
Option
</th><th>
Description
</th></tr></thead><tbody><tr><td>
-depth <span class="emphasis"><em>number</em></span>
</td><td>
Specifies the minimum number of CDD files to hit each coverage point. The value of
<span class="emphasis"><em>number</em></span> should be a value of 1 or more. Default is 1.
</td></tr><tr><td>
-d <span class="emphasis"><em>directory</em></span>
</td><td>
Directory to search for CDD files to include. This option is used in conjunction with the -ext option
which specifies the file extension to use for determining which files in the directory are CDD files.
</td></tr><tr><td>
-ext <span class="emphasis"><em>extension</em></span>
</td><td>
Used in conjunction with the -d option. If no -ext options are specified on the command-line, the
default value of '.cdd' is used. Note that a period (.) should be specified.
</td></tr><tr><td>
-f <span class="emphasis"><em>filename</em></span>
</td><td>
Name of file containing additional arguments to parse.
</td></tr><tr><td>
-h
</td><td>
Displays help information for the rank command.
</td></tr><tr><td>
-names-only
</td><td>
If specified, outputs only the needed CDD filenames that need to be run in the order they need to be run.
If this option is not set, a report-style output is provided with additional information.
</td></tr><tr><td>
-o <span class="emphasis"><em>filename</em></span>
</td><td>
Name of file to output ranking information to. Default is stdout.
</td></tr><tr><td>
-require-cdd <span class="emphasis"><em>filename</em></span>
</td><td>
Name of CDD file to use as a required CDD file for ranking, regardless of whether it is necessary to achieve
full coverage or not.
</td></tr><tr><td>
-require-list <span class="emphasis"><em>filename</em></span>
</td><td>
Name of file containing list of CDD files which are required to be in the list of ranked CDDs to be run,
regardless of whether they are necessary to achieve full coverage or not.
</td></tr><tr><td>
-v
</td><td>
Outputs verbose information during the run of the rank command. Useful for selection process understanding
and command performance.
</td></tr><tr><td>
-weight-assert <span class="emphasis"><em>number</em></span>
</td><td>
Specifies a relative weighting for assertion coverage used to rank non-unique coverage points.
</td></tr><tr><td>
-weight-comb <span class="emphasis"><em>number</em></span>
</td><td>
Specifies a relative weighting for combinational logic coverage used to rank non-unique coverage points.
</td></tr><tr><td>
-weight-fsm <span class="emphasis"><em>number</em></span>
</td><td>
Specifies a relative weighting for FSM state/state transition coverage used to rank non-unique coverage
points.
</td></tr><tr><td>
-weight-line <span class="emphasis"><em>number</em></span>
</td><td>
Specifies a relative weighting for line coverage used to rank non-unique coverage points.
</td></tr><tr><td>
-weight-memory <span class="emphasis"><em>number</em></span>
</td><td>
Specifies a relative weighting for memory coverage used to rank non-unique coverage points.
</td></tr><tr><td>
-weight-toggle <span class="emphasis"><em>number</em></span>
</td><td>
Specifies a relative weighting for toggle coverage used to rank non-unique coverage points.
</td></tr></tbody></table></div></div><p><br class="table-break">
</p></div><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="section.rank.inputting"></a>12.2. Methods for specifying CDDs to rank</h2></div></div></div><p>
The rank command provides several mechanisms for specifying CDD files to rank. Feel free to use any combination
of methods for providing this information to Covered as it makes sense to do in your environment. The following is
the list of input methods:
</p><p>
</p><div class="itemizedlist"><ul type="disc"><li><p>
Specify each CDD file after the rank command options have been specified. This method is best when there are
a relatively small number of CDD files to rank as the length of the command-line in a shell is typically
limited to a certain number of characters.
</p></li><li><p>
Create a file containing the list of CDD files to rank and pass this file to the rank command's
<span class="bold"><strong>-f</strong></span> option. Using this method, one can selectively specify a large number
of CDD files. You may specify as many "-f <span class="emphasis"><em>filename</em></span>" options as is necessary.
</p></li><li><p>
Use shell commands to create a list of CDD files and pass them to Covered's standard input via command-line
piping and the <span class="bold"><strong>-f -</strong></span> command-line option (the - means use standard input).
The following line is an example of this usage:
</p><p>
</p><pre class="programlisting">
ls foo*.cdd | covered rank -f -
</pre><p>
</p><p>
You may specify only one "-f -" option to the rank command.
</p></li><li><p>
Use the <span class="bold"><strong>-d <span class="emphasis"><em>directory</em></span></strong></span> and
<span class="bold"><strong>-ext <span class="emphasis"><em>extension</em></span></strong></span> options to tell
Covered to retrieve all files found in the directory specified by <span class="emphasis"><em>directory</em></span> that contain
the file extension of <span class="emphasis"><em>extension</em></span> (if the <span class="bold"><strong>-ext</strong></span> option
is not specified, the default extension of ".cdd" is used). The following is an example of this usage:
</p><p>
</p><pre class="programlisting">
covered rank -d /home/bubba/cdd_dir -ext .mycdd
</pre><p>
</p><p>
You may specify as many -d options on the command-line as is necessary; however, only one -ext option is allowed.
</p></li><li><p>
Specify a single CDD file that MUST be included in the final ranking list via the
<span class="bold"><strong>-require-cdd</strong></span> option. You can specify as many -require-cdd options to the rank
command as necessary.
</p></li><li><p>
Create a file containing a list of CDDs that MUST be included in the final ranking list where each CDD filename
is separated by a newline or whitespace. Pass this file to the <span class="bold"><strong>-require-list</strong></span>
option. You can specify as many -require-list options to the rank command as necessary.
</p></li></ul></div><p>
</p></div><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="section.rank.weighting"></a>12.3. Specifying metric weights</h2></div></div></div><p>
The rank command has several options that allow the user to weight any and all of the coverage metrics that Covered
supplies. Weights are used to ascribe the importance of its metric in ranking the value of a CDD file.
</p><p>
Weight values are non-negative numbers whose values are relative to each other. A value of 0 effectively disables
the associated weight from consideration in ranking. All other values are considered relative to each other. In
other words, if the line weight is set to a value of 5 while the toggle weight is set to 1, line coverage is
considered 5 times more important than toggle coverage in ranking.
</p><p>
By default, each metric is given an equal weight of 1. Adjust each of the metrics as necessary to potentially
modify which CDD files are chosen and what order those CDD files are given in the ranked list of diagnostics to run.
</p></div><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="section.rank.output"></a>12.4. Understanding the rank command output information</h2></div></div></div><p>
After the rank command has been executed, the output will either be displayed to standard output (if no
<span class="bold"><strong>-o</strong></span> option was specified) or it will be output to a file. If the
<span class="bold"><strong>-names_only</strong></span> option was specified, the output will be a list of CDD filenames (one
per line) which are output in the order in which they should be run to achieve full coverage in the shortest amount
of time. If the -names_only option was not specified, the output will be in the form of a table (similar to reports
generated with the report command) which describes not only the order that CDD files should be run but also
various statistical data and the names of CDD files which are considered redundant in terms of coverage information.
The following example shows an example of this report output.
</p><p>
</p><div class="example"><a name="example.rank.noreq"></a><p class="title"><b>Example 12.1. Rank Output With No Required Files</b></p><div class="example-contents"><pre class="programlisting">
::::::::::::::::::::::::::::::::::::::::::::::::::::
:: ::
:: Covered -- Simulation Ranked Run Order ::
:: ::
::::::::::::::::::::::::::::::::::::::::::::::::::::
* Reduced 3 CDD files down to 2 needed to maintain coverage ( 33% reduction, 1.5x improvement)
* Reduced 9 timesteps down to 6 needed to maintain coverage ( 33% reduction, 1.5x improvement)
-----------+-------------------------------------------+----------------------------------------------------------
| ACCUMULATIVE | CDD
Simulation |-------------------------------------------+----------------------------------------------------------
Order | Hit / Total % Timesteps | R Name Hit / Total % Timesteps
-----------+-------------------------------------------+----------------------------------------------------------
1 9 17 53% 3 rank1c.cdd 9 17 53% 3
2 13 17 76% 6 rank1b.cdd 8 17 47% 3
--------------------------------------- The following CDD files add no additional coverage ----------------------------------------------
3 13 17 76% 9 rank1a.cdd 8 17 47% 3
</pre></div></div><p><br class="example-break">
</p><p>
After the header information, there is a summary of the ranking reduction (if one occurred) specified in both number
of files and number of simulation timesteps. <a href="chapter.rank.html#example.rank.req" title="Example 12.2. Rank Output With Required Files">Example 12.2, “Rank Output With Required Files”</a> shows what this information looks
like when no file reduction has occurred.
</p><p>
After the summary information, there is the rank output table which displays both the individual CDD information and
the accumulative coverage information. Each row of the table contains one CDD file where the first row contains
the first CDD that should be run in regression for maximum coverage impact. Each successive row contains CDD files
that add less and less coverage to the overall coverage gained. If there were CDD files that were found to not
add any coverage information to the above CDD files, a line is drawn and these CDD files displayed below that line.
These CDD files can be excluded from regression because they don't add any useful coverage information over-and-above
the CDD files that are selected to be run in regression.
</p><p>
The first major column is the simulation order. This value increments by one for each successive row. If you would
like to create a shorter running regression that is limited to a certain number of simulations, this row information
can be useful for figuring out which simulations to include in these type of regressions.
</p><p>
The second major column is the accumulative coverage statistics which include the following:
</p><p>
</p><div class="itemizedlist"><ul type="disc"><li><p><span class="bold"><strong>Hit</strong></span></p><p>
The first row contains the number of coverage points that the first CDD file hit. All successive rows add
the unique coverage points hit by the CDD file listed on that line to the above hit total.
</p><p>
For example, in <a href="chapter.rank.html#example.rank.noreq" title="Example 12.1. Rank Output With No Required Files">Example 12.1, “Rank Output With No Required Files”</a> "rank1c.cdd" contains 9 hit coverage points (accumulative
hit count = 9). The second highest ranked file, "rank1b.cdd", contains 8 hit coverage points but only 4 of
those coverage points are different than the coverage points hit by "rank1c.cdd"; therefore, the accumulative
coverage hit count is 13. The third coverage file, "rank1c.cdd", contains 8 coverage points but none of them
add to the coverage points hit by the first two coverage files; therefore, it is considered unnecessary for
regression.
</p></li><li><p><span class="bold"><strong>Total</strong></span></p><p>
The "accumulative" total represents the total number of coverage points in each CDD file. Since this number
must be the same for each CDD file, this value will be the same for each row.
</p></li><li><p><span class="bold"><strong>%</strong></span></p><p>
This column represents the hit percentage and is a simple calculation of <code class="code">hit / total</code> for each
row.
</p></li><li><p><span class="bold"><strong>Timesteps</strong></span></p><p>
This column represents the number of accumulated simulation timesteps. This value represents the number of
timesteps that need to transpire to gain the amount of accumulated coverage. One of the goals of the
ranking algorithm is to gain the most amount of coverage in the least amount of simulation time.
</p></li></ul></div><p>
</p><p>
The third major column is the individual coverage information for the CDD file listed in the row which includes
the following information:
</p><p>
</p><div class="itemizedlist"><ul type="disc"><li><p><span class="bold"><strong>R</strong></span></p><p>
This column represents whether the current CDD file was a required CDD file by the user (a file listed
via the -require-list or -require-cdd options). If the file is a required CDD file, an asterisk (*)
is listed in this column. If the file was not required, nothing will be listed in this column.
<a href="chapter.rank.html#example.rank.req" title="Example 12.2. Rank Output With Required Files">Example 12.2, “Rank Output With Required Files”</a> provides an example of what this output looks like.
</p></li><li><p><span class="bold"><strong>Name</strong></span></p><p>
Filename of the CDD file that was selected for this row.
</p></li><li><p><span class="bold"><strong>Hit</strong></span></p><p>
This column represents the number of coverage points hit by the CDD file.
</p></li><li><p><span class="bold"><strong>Total</strong></span></p><p>
This column represents the total number of coverage points within the CDD file that could be hit.
</p></li><li><p><span class="bold"><strong>%</strong></span></p><p>
This column represents the hit percentage (a calculation of <code class="code">hit / total</code>.
</p></li><li><p><span class="bold"><strong>Timesteps</strong></span></p><p>
This column represents the number of simulation timesteps that the CDD file took to run during simulation.
This value does not necessarily represent the ending simulation time.
</p></li></ul></div><p>
</p><p>
</p><div class="example"><a name="example.rank.req"></a><p class="title"><b>Example 12.2. Rank Output With Required Files</b></p><div class="example-contents"><pre class="programlisting">
::::::::::::::::::::::::::::::::::::::::::::::::::::
:: ::
:: Covered -- Simulation Ranked Run Order ::
:: ::
::::::::::::::::::::::::::::::::::::::::::::::::::::
No reduction occurred
-----------+-------------------------------------------+------------------------------------------------------------
| ACCUMULATIVE | CDD
Simulation |-------------------------------------------+------------------------------------------------------------
Order | Hit / Total % Timesteps | R Name Hit / Total % Timesteps
-----------+-------------------------------------------+------------------------------------------------------------
1 9 17 53% 3 rank1.1c.cdd 9 17 53% 3
2 13 17 76% 6 * rank1.1b.cdd 8 17 47% 3
3 13 17 76% 9 * rank1.1a.cdd 8 17 47% 3
</pre></div></div><p><br class="example-break">
</p></div><div class="sect1" lang="en"><div class="titlepage"><div><div><h2 class="title" style="clear: both"><a name="section.rank.algorithm"></a>12.5. Description of ranking algorithm</h2></div></div></div><p>
This section describes the overall algorithm that is used to rank the specified CDD files. The algorithm is being
presented in this document for the purposes of providing an explanation for how diagnostics are selected for
ranking and how they are ranked. Knowing this information is not necessary for using the rank command or
understanding its results.
</p><p>
The algorithm makes use of three different types of structures. The first structure is a bit-compressed structure
where each bit corresponds to one coverage point in the associated CDD file. The purpose of this structure is to
store a large amount of information as concisely and immediately available as possible. In addition to coverage
point information, each structure also contains the following information:
</p><p>
</p><div class="itemizedlist"><ul type="disc"><li><p>
Name of CDD file that the structure is associated with (for reporting purposes only).
</p></li><li><p>
The number of timesteps that occurred during the CDD simulation. This is used for calculating the average
number of coverage points hit during a single clock period -- used for ranking selection.
</p></li><li><p>
The total number of coverage points hit by the CDD. This is used for calculating the average number of
coverage points hit during a single clock period -- used for ranking selection.
</p></li><li><p>
The total number of coverage points that were hit by this CDD that were not hit by any other CDD listed in
the rank command-line. This information is used to figure out which CDDs are absolutely necessary to be
included in the ranked list.
</p></li><li><p>
The current score associated with the CDD file. This value is updated for each selection iteration.
</p></li></ul></div><p>
</p><p>
Each of these structures are maintained in an array that is iterated on during the ranking process.
</p><p>
The second structure is an array of 16-bit values where each value corresponds to a single coverage point. Its
value is a sum of the number of unranked CDD files that have hit that coverage point. Each value is decremented
by one when a CDD file is moved from the unranked list to the ranked list and that CDD file hits the given
coverage point. If a coverage point value is a value of 1 and a CDD file has its corresponding coverage bit set,
we know that the CDD is the only one that hits the coverage point.
</p><p>
The third structure is similar to the second and works similarly, except that it stores the number of CDD files in
the ranked list that hit a particular coverage point.
</p><p>
The specified CDD files are ranked in a series of steps, some of which are reiterated for each CDD that is
selected to be ranked. Here is the basic algorithm (in pseudo-code):
</p><p>
</p><pre class="programlisting">
calculate the size of the compressed coverage point structure
foreach CDD in CDD_list {
convert CDD to compressed coverage point structure
}
foreach CDD in CDD_list {
if this CDD contains uniquely hit coverage points {
move the CDD file into the ranked list
}
}
foreach CDD in unranked list {
foreach CDD in unranked list {
calculate score (((total_num_cov_points_hit / num_timesteps) * 100) * metric_weight_value)
if any unranked elements are less than the -depth option value and are hit by this CDD {
set unique_found to true
}
if our score > highest score and a "unique" coverage point was found {
set highest_score to our score
}
}
move the CDD file indexed by highest_score from the unranked list to the ranked list
}
resort ranked list from most unique to least unique (if there is a tie, select the CDD that hit the most coverage
points in the least amount of time)
</pre><p>
</p></div></div><div class="navfooter"><hr><table width="100%" summary="Navigation footer"><tr><td width="40%" align="left"><a accesskey="p" href="chapter.report.html"><img src="img/prev.gif" alt="Prev"></a> </td><td width="20%" align="center"><a accesskey="u" href="part.command.line.usage.html"><img src="img/up.gif" alt="Up"></a></td><td width="40%" align="right"> <a accesskey="n" href="chapter.exclude.html"><img src="img/next.gif" alt="Next"></a></td></tr><tr><td width="40%" align="left" valign="top">Chapter 11. The report Command </td><td width="20%" align="center"><a accesskey="h" href="index.html"><img src="img/home.gif" alt="Home"></a></td><td width="40%" align="right" valign="top"> Chapter 13. The exclude Command</td></tr></table></div></body></html>
|