/usr/share/doc/collectl/html/Data-verbose.html is in collectl 3.6.9-1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 | <html>
<head>
<link rel=stylesheet href="style.css" type="text/css">
<title>Verbose Data</title>
</head>
<body>
<center><h1>Verbose Data</h1></center>
<p>
Data is reported in this form when either --verbose is used OR if there is at least one
type of data requested that doesn't have a brief form such as any detail data or
ionodes, processes or slabs. Specifying some of the lustre output options with --lustopts
such as B, D and M will also force verbose format.
<h3>Buddy (Memory Fragmentation) Data, <i>collectl -sb</i></h3>
<p>
<div class=terminal-wide14><pre>
# MEMORY FRAGMENTATION SUMMARY (4K pages)
# 1Pg 2Pgs 4Pgs 8Pgs 16Pgs 32Pgs 64Pgs 128Pgs 256Pgs 512Pgs 1024Pgs
</pre></div>
This table shows the total number of memory fragments by pagesize in increasing powers of 2 for
all the memory types.
<p>
<p><h3>CPU, <i>collectl -sc</i></h3>
<div class=terminal-wide14>
<pre>
# CPU SUMMARY (INTR, CTXSW & PROC /sec)
# USER NICE SYS WAIT IRQ SOFT STEAL IDLE INTR CTXSW PROC RUNQ RUN AVG1 AVG5 AVG15 RunT BlkT
</pre>
</div>
These are the percentage of time the system in running is one of the modes, noting that
these are averaged across all CPUs. While User and Sys modes are self-eplanitory, the others
may not be:
<p>
<table>
<tr valign=top><td><b>User</b></td>
<td>Time spent in User mode, not including time spend in "nice" mode.</td></tr>
<tr valign=top><td><b>Nice</b></td>
<td>Time spent in Nice mode, that is lower priority as adjusted by
the nice command and have the "N" status flag set when examined with "ps".</td></tr>
<tr valign=top><td><b>Sys</b></td>
<td>This is time spent in "pure" system time.</td></tr>
<tr valign=top><td><b>Wait</b></td>
<td>Also known as "iowait", this is the time the CPU was idle during an
outstanding disk I/O request. This is not considered to be part of the total or
system times reported in brief mode.</td></tr>
<tr valign=top><td><b>Irq</b></td>
<td>Time spent processing interrupts and also considered to be part of
the summary system time reported in "brief" mode.</td></tr>
<tr valign=top><td><b>Soft</b></td>
<td>Time spent processing soft interrupts and also considered to be part
of the summary system time reported in "brief" mode.</td></tr>
<tr valign=top><td><b>Steal</b></td>
<td>Time spend in involuntary wait state while the hypervisor was servicing
another virtual processor.</td></tr>
</table>
<p>
This next set of fields apply to processes
<p>
<table>
<tr><td><b>Proc</b></td><td>Process creations/sec.</li></td></tr>
<tr><td><b>Runq</b></td><td>Number of processes in the run queue.</li></td></tr>
<tr><td><b>Run</b></td><td>Number of processes in the run state.</li></td></tr>
<tr><td><b>Avg1, Avg5, Avg15</b></td><td>Load average over the last 1,5 and 15 minutes.</td></tr>
<tr><td><b>RunT</b></td><td>Total number of process in the run state, not counting collectl itself</td></tr>
<tr><td><b>BlkT</b></td><td>Total number of process blocked, waiting on I/O</td></tr>
</table>
<p><h3>Disks, <i>collectl -sd</i></h3>
<p>
If you specify filtering with <i>--dskfilt</i>, the disks that match the pattern(s)
will either be included or excluded from the the summary data. However, the data will
<i>still</i> be collected so if recorded to a file can later be viewed.
<div class=terminal-wide14>
<pre>
# DISK SUMMARY (/sec)
#KBRead RMerged Reads SizeKB KBWrit WMerged Writes SizeKB
</div></pre>
<table>
<tr><td><b>KBRead</b></td><td>KB read/sec</td></tr>
<tr valign=top><td><b>RMerged</b></td><td>Read requests merged per second when being dequeued.</td></tr>
<tr><td><b>Reads</b></td><td>Number of reads/sec</td></tr>
<tr><td><b>SizeKB</b></td><td>Average read size in KB</td></tr>
<tr><td><b>KBWrite<b></td><td>KB written/sec</td></tr>
<tr valign=top><td><b>WMerged</b></td>
<td>Write requests merged per second when being dequeued.</tr></tr>
<tr><td><b>Writes</b></td><td>Number of writes/sec</td></tr>
<tr><td><b>SizeKB</b></td><td>Average write size in KB</td></tr>
</table>
<p><h3>Inodes/Filesystem, <i>collectl -si</i></h3>
<div class=terminal-wide14>
<pre>
# INODE SUMMARY
# Dentries File Handles Inodes
# Number Unused Alloc % Max Number
40585 39442 576 0.17 38348
</pre></div>
<table>
<tr><td colspan=2>DCache</td></tr>
<tr><td><b>Dentries Number</b></td><td>Number of entries in directory cache</td</tr>
<tr><td><b>Dentried Unused</b></td><td>Number of unused entries in directory cache</td</tr>
<tr><td><b>Handles Alloc</b></td><td>Number of allocated file handles</td></tr>
<tr><td><b>handles % Max</b></td><td>Percentage of maximum available file handles</td></tr>
<tr><td><b>Inodes Number</b></td><td>Number of inodes in use</td></tr>
</table>
<p><i>NOTE - as of this writing I'm baffled by the dentry unused field. No matter how
many files and/or directories I create, this number goes up! Sholdn't it go down?</i>
<p><h3>Infiniband, <i>collectl -sx</i></h3>
<div class=terminal-wide14>
<pre>
# INFINIBAND SUMMARY (/sec)
# KBIn PktIn SizeIn KBOut PktOut SizeOut Errors
</pre></div>
<table>
<tr><td><b>KBIn</b></td><td>KB received/sec.</td></tr>
<tr><td><b>PktIn</b></td><td>Packets received/sec.</td></tr>
<tr><td><b>SizeIn</b></td><td>Average incoming packet size in KB</td></tr>
<tr><td><b>KBOut</b></td><td>KB transmitted/sec.</td></tr>
<tr><td><b>PktOut</b></td><td>Packets transmitted/sec.</td></tr>
<tr><td><b>SizeOut</b></td><td>Average outgoing packet size in KB</td></tr>
<tr valign=top><td><b>Errs</b></td><td>Count of current errors. Since these
are typically infrequent, it is felt that reporting them as a rate would result
in either not seeing them OR round-off hiding their values.</td></tr>
</table>
<p><h3>Lustre</h3>
<p><b>Lustre Client</b>, <i>collectl -sl</i>
<p>There are several formats here controlled by the --lustopts switch. There is
also detail data for these available as well. Specifying -sL results in
data broken out by the file system and --lustopts O further breaks it out by OST.
Also note the average read/write sizes are only reported when --lustopts is <i>not</i> specified.
<div class=terminal-wide14>
<pre>
# LUSTRE CLIENT SUMMARY
# KBRead Reads SizeKB KBWrite Writes SizeKB
</pre></div>
<table>
<tr><td><b>KBRead</b></td><td>KB/sec delivered to the client.</td></tr>
<tr><td valign=top><b>Reads</b></td><td>Reads/sec delivered to the client,
not necessarily from the lustre storage servers.</td></tr>
<tr><td><b>SizeKB</b></td><td>Average read size in KB</td></tr>
<tr><td><b>KBWrite</b></td><td>KB Writes/sec delievered to the storage servers.</td></tr>
<tr><td><b>Writes</b></td><td>Writes/sec delievered to the storage servers.</td></tr>
<tr><td><b>SizeKB</b></td><td>Average write size in KB</td></tr>
</table>
<div class=terminal-wide14>
<pre>
# LUSTRE CLIENT SUMMARY: METADATA
# KBRead Reads KBWrite Writes Open Close GAttr SAttr Seek Fsynk DrtHit DrtMis
</pre></div>
<table>
<tr><td><b>KBRead</b></td><td>KB/sec delivered to the client.</td></tr>
<tr><td valign=top><b>Reads</b></td><td>Reads/sec delivered to the client,
not necessarily from the lustre storage servers.</td></tr>
<tr><td><b>KBWrite</b></td><td>KB Writes/sec delievered to the storage servers.</td></tr>
<tr><td><b>Writes</b></td><td>Writes/sec delievered to the storage servers.</td></tr>
<tr><td><b>Open</b></td><td>File opens/sec</td></tr>
<tr><td><b>Close</b></td><td>File closes/sec</td></tr>
<tr><td><b>GAttr</b></td><td>getattrs/sec</td></tr>
<tr><td><b>Seek</b></td><td>seeks/sec</td></tr>
<tr><td><b>Fsync</b></td><td>fsyncs/sec</td></tr>
<tr><td><b>DrtHit</b></td><td>dirty hits/sec</td></tr>
<tr><td><b>DrtMis</b></td><td>dirty misses/sec</td></tr>
</table>
<div class=terminal-wide14>
<pre>
# LUSTRE CLIENT SUMMARY: READAHEAD
# KBRead Reads KBWrite Writes Pend Hits Misses NotCon MisWin FalGrb LckFal Discrd ZFile ZerWin RA2Eof HitMax Wrong
</pre></div>
<table>
<tr><td><b>KBRead</b></td><td>KB/sec delivered to the client.</td></tr>
<tr><td valign=top><b>Reads</b></td><td>Reads/sec delivered to the client,
not necessarily from the lustre storage servers.</td></tr>
<tr><td><b>KBWrite</b></td><td>KB Writes/sec delievered to the storage servers.</td></tr>
<tr><td><b>Writes</b></td><td>Writes/sec delievered to the storage servers.</td></tr>
<tr><td><b>Pend</b></td><td>Pending issued pages</td></tr>
<tr><td><b>Hits</b></td><td>prefetch cache hits</td></tr>
<tr><td><b>Misses</b></td><td>prefetch cache misses</td></tr>
<tr><td><b>NotCon</b></td><td>The current pages read that were not consecutive with the previous ones./td></tr>
<tr><td><b>MisWin</b></td><td>Miss inside window. The pages that were expected to be in the
prefetch cache but weren't. They were probably
reclaimed due to memory pressure</td></tr>
<tr><td><b>LckFal</b></td><td>Failed grab_cache_pages. Tried to prefetch page but it was locked.</td></tr>
<tr><td><b>Discrd</b></td><td>Read but discarded. Prefetched pages (but not read by applicatin)
have been discarded either becuase of memory pressure or lock
revocation.</td></tr>
<tr><td><b>ZFile</b></td><td>Zero length file.</td></tr>
<tr><td><b>ZerWin</b></td><td>Zero size window.</td></tr>
<tr><td><b>RA2Eof</b></td><td>Read ahead to end of file</td></tr>
<tr><td><b>HitMax</b></td><td>Hit maximum readahead issue. The read-ahead window has grown to the
maximum specified by <i>max_read_ahead_mb</i></td></tr>
</table>
<div class=terminal-wide14>
<pre>
# LUSTRE CLIENT SUMMARY: RPC-BUFFERS (pages)
#RdK Rds 1K 2K ... WrtK Wrts 1K 2K ...
</pre></div>
This display shows the size of rpc buffer distribution buckets in K-pages. You can find the
page size for you system in the header (<i>collectl --showheader</i>).
<p>
<table>
<tr><td valign=top><b>RdK</b></td><td>KBs read/sec</td></tr>
<tr><td valign=top><b>Rds</b></td><td>Reads/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of pages of of this size read</td></tr>
<tr><td valign=top><b>WrtK</b></td><td>KBs written/sec</td></tr>
<tr><td valign=top><b>Wrts</b></td><td>Writes/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of pages of of this size written</td></tr>
</table>
<p><b>Lustre Meta-Data Server</b>, <i>collectl -sl</i>
<p>As of Lustre 1.6.5, the data reported for the MDS had changed, breaking out the <i>Reint</i>
data into 5 individual buckets which are the last 5 fields described below. For earlier
versions those 5 fields will be replaced by a single one named <i>Reint</i>.
<div class=terminal-wide14>
<pre>
# LUSTRE MDS SUMMARY
#Getattr GttrLck StatFS Sync Gxattr Sxattr Connect Disconn Create Link Setattr Rename Unlink
</pre></div>
<table>
<tr><td valign=top><b>Getattr</b></td><td>Number of getattr calls, for example <i>lfs osts</i>.
Note that this counter is <i>not</i> incremented as the result of <i>ls</i> - see <i>Gxattr</i></td></tr>
<tr><td><b>GttrLck</b></td><td>These are getattrs that also return a lock on the file</td></tr>
<tr><td valign=top><b>StatFS</b></td><td>Number of stat calls, for example <i>df</i> or <i>lfs df</i>.
Note that lustre caches data for up to a second so many calls within a second may only show
up as a single <i>statfs</i></td></tr>
<tr><td><b>Sync</b></td><td>Number of sync calls</td></tr>
<tr><td valign=top><b>Gxattr</b></td><td>Extended attribute get operations, for example <i>getfattr</i>,
<i>getfacl</i> or even <i>ls.</i> Note that the MDS must have been mounted with <i>-o acl</i>
for this counter to be enabled.</td></tr>
<tr><td><b>Sxattr</b></td><td>Extended attribute set operations, for example <i>setfattr</i> or <i>setfacl</i></td></tr>
<tr><td><b>Connect</b></td><td>Client mount operations</td></tr>
<tr><td><b>Disconn</b></td><td>Client umount operations</td></tr>
<tr><td><b>Create</b></td><td>Count of mknod and mkdir operations, also used by NFS servers internally when creating files</td></tr>
<tr><td><b>Link</b></td><td>Hard and symbolic links, for example <i>ln</i></td></tr>
<tr><td><b>Setattr</b></td><td>All operations that modify inode attributes including chmod, chown, touch, etc</td></tr>
<tr><td><b>Rename</b></td><td>File and directory renames, for example <i>mv</i></td></tr>
<tr><td><b>Unlink</b></td><td>File/directory removals, for example <i>rm</i> or <i>rmdir</i></td></tr>
</table>
<p>
The following display is very similar the the RPC buffers in that the sizes of different size
I/O requests are reported. In this case there are requests sent to the disk driver.
Note that this report is only available for HP's SFS.
<div class=terminal-wide14>
<pre>
# LUSTRE DISK BLOCK LEVEL SUMMARY
#Rds RdK 0.5K 1K ... Wrts WrtK 0.5K 1K ...
</pre></div>
<table>
<tr><td valign=top><b>Rds</b></td><td>Reads/sec</td></tr>
<tr><td valign=top><b>RdK</b></td><td>KBs read/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of blocks of of this size read</td></tr>
<tr><td valign=top><b>Wrts</b></td><td>Writes/sec</td></tr>
<tr><td valign=top><b>WrtK</b></td><td>KBs written/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of blocks of of this size written</td></tr>
</table>
<p><b>Lustre Object Storage Server</b>, <i>collectl -sl</i>
<div class=terminal-wide14>
<pre>
# LUSTRE OST SUMMARY
# KBRead Reads SizeKB KBWrite Writes SizeKB
</pre></div>
<table>
<tr><td><b>KBRead</b></td><td>KB/sec read</td></tr>
<tr><td><b>Reads</b></td><td>Reads/sec</td></tr>
<tr><td><b>SizeKB</b></td><td>Average read size in KB</td></tr>
<tr><td><b>KBWrite</b></td><td>KB/sec written</td></tr>
<tr><td><b>Writes</b></td><td>Writes/sec</li>
<tr><td><b>SizeKB</b></td><td>Average write size in KB</td></tr>
</table>
<p><b>Lustre Object Storage Server</b>, <i>collectl -sl --lustopts B</i>
<p>
As with client data, when you only get read/write average sizes when
--lustopt is <i>not</i> specified.
<div class=terminal-wide14>
<pre>
# LUSTRE OST SUMMARY
#<--------reads-----------|----writes-----------------
#RdK Rds 1K 2K ... WrtK Wrts 1K 2K ....
</pre></div>
<table>
<tr><td valign=top><b>RdK</b></td><td>KBs read/sec</td></tr>
<tr><td valign=top><b>Rds</b></td><td>Reads/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of pages of of this size read</td></tr>
<tr><td valign=top><b>WrtK</b></td><td>KBs written/sec</td></tr>
<tr><td valign=top><b>Wrts</b></td><td>Writes/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of pages of of this size written</td></tr>
</table>
<p><b>Lustre Object Storage Server</b>, <i>collectl -sl --lustopts D</i>
<div class=terminal-wide14>
<pre>
# LUSTRE DISK BLOCK LEVEL SUMMARY
#RdK Rds 0.5K 1K ... WrtK Wrts 0.5K 1K ...
</pre></div>
<table>
<tr><td valign=top><b>RdK</b></td><td>KBs read/sec</td></tr>
<tr><td valign=top><b>Rds</b></td><td>Reads/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of blocks of of this size read</td></tr>
<tr><td valign=top><b>WrtK</b></td><td>KBs written/sec</td></tr>
<tr><td valign=top><b>Wrts</b></td><td>Writes/sec</td></tr>
<tr><td valign=top><b>nK</b></td><td>Number of blocks of of this size written</td></tr>
</table>
<p><h3>Memory, <i>collectl -sm</i></h3>
<div class=terminal-wide14>
<pre>
# MEMORY SUMMARY
#<-------------------------------Physical Memory-------------------------------------><-----------Swap------------><-------Paging------>
# Total Used Free Buff Cached Slab Mapped Anon Commit Locked Inact Total Used Free In Out Fault MajFt In Out
</pre></div>
<table>
<tr><td><b>Total</b></td>
<td>Total physical memory</td></tr>
<tr valign=top><td><b>Used</b></td>
<td>Used physical memory. This does not include memory used by the kernel itself.</td></tr>
<tr valign=top><td><b>Free</b></td>
<td>Unallocated memory</td></tr>
<tr valign=top><td><b>Buff</b></td>
<td>Memory used for system buffers</td></tr>
<tr valign=top><td><b>Cached</b></td>
<td>Memory used for caching data beween the kernel and disk, noting direct I/O does not use the cache</td></tr>
<tr valign=top><td><b>Slab</b></td>
<td>Memory used for slabs, see <i>collectl -sY</i></td></tr>
<tr valign=top><td><b>Mapped</b></td>
<td>Memory mapped by processes</td></tr>
<tr valign=top><td><b>Anon</b></td>
<td>Anonymous memory. <i>NOTE - this </i>is included<i> with mapped memory in brief format</i></td></tr>
<tr valign=top><td><b>Commit</b></td>
<td>Accorting to RedHat: <i>"An estimate of how much RAM you would need to make a 99.99% guarantee
that there never is OOM (out of memory) for this workload."</i></td></tr>
<tr valign=top><td><b>Locked</b></td>
<td>Locked Memory</td>
<tr valign=top><td><b>Inactive</b></td>
<td>Inactive pages. On ealier kernels this number is the sum of the clean, dirty
and laundry pages.</td></tr>
<tr><td><b>Swap Total</b></td>
<td>Total Swap</td></tr>
<tr><td><b>Swap Used</b></td>
<td>Used Swap</td></tr>
<tr><td><b>Swap Free</b></td>
<td>Free Swap</td></tr>
<tr><td><b>Swap In</b></td>
<td>Pages swapped in/sec</td></tr>
<tr><td><b>Swap Out</b></td>
<td>Pages swapped out/sec</td></tr>
<tr><td><b>Fault</b></td>
<td>Page faults/sec resolved by not going to disk</td></tr>
<tr><td><b>MajFt</b></td>
<td>These page faults are resolved by going to disk</td></tr>
<tr><td><b>Paging In</b></td>
<td>Total number of pages read by block devices</td></tr>
<tr><td><b>Paging Out</b></td>
<td>Total number of pages written by block devices</td></tr>
</table>
<p>
<center><b><i>Notes</i></b></center>
If you include <i>--memopts R</i>, memory and swap values wil be displayed as changes/sec between intervals
rather than absolute values in addition to page fault information, which is already displayed as rates.
This switch will also honor -on in that the values will not be normalized to a rate but rather displayed
as changes in size per interval.
<p>
If you include <i>--memopts</i> with P or V, collectl will only display Physical or Virtual memory.
The default is PV and will display both.
<p><h3>Memory, <i>collectl -sm --memopts ps</i></h3>
<p>
The p and s options allow you to display data about page and/or steal and scan information. If you want this data combined with
the standard physical or virtual data you must explicitly request them as well. The columns show how the memory is allocated
for the respective sections.
<div class=terminal-wide14>
<pre>
# MEMORY SUMMARY
#<---Other---|-------Page Alloc------|------Page Refill-----><------Page Steal-------|-------Scan KSwap------|------Scan Direct----->
# Free Activ Dma Dma32 Norm Move Dma Dma32 Norm Move Dma Dma32 Norm Move Dma Dma32 Norm Move Dma Dma32 Norm Move
14M 136K 2 69 13M 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
</pre></div>
<p><h3>Network, <i>collectl -sn</i></h3>
<p>
The entries for error counts in the following are actually the total of several
types of errors. To get individual error counts, you must either include <i>--netopts e</i>
or report details on individual interfaces in plot format by specifying -P.
Transmission errors are categorized by errors, dropped, fifo, collisions and carrier.
Receive errors are broken out for errors, dropped, fifo and framing errors.
<p>
If you specify filtering with <i>--netfilt</i>, the names that match the pattern(s)
will either be included or excluded from the the summary data. However, the data will
<i>still</i> be collected so if recorded to a file can later be viewed.
<div class=terminal-wide14>
<pre>
# NETWORK SUMMARY (/sec)
# KBIn PktIn SizeIn MultI CmpI ErrsI KBOut PktOut SizeO CmpO ErrsO
</pre></div>
<table>
<tr><td><b>KBIn</b></td>
<td>Incoming KB/sec</td></tr>
<tr><td><b>PktIn</b></td>
<td>Incoming packets/sec</td></tr>
<tr><td><b>SizeI</b></td>
<td>Average incoming packet size in bytes</td></tr>
<tr><td><b>MultI</b></td>
<td>Incoming multicast packets/sec</td></tr>
<tr><td><b>CmpI</b></td>
<td>Incoming compressed packets/sec</td></tr>
<tr valign=top><td><b>ErrsI</b></td>
<td>Total incoming errors/sec. This is an aggregation of incoming error counters. To see explicit
error counters use <i>--netopts e</i></td></tr>
<tr><td><b>KBOut</b></td>
<td>Outgoing KB/sec</td></tr>
<tr><td><b>PktOut</b></td>
<td>Outgoing packets/sec</td></tr>
<tr><td><b>SizeO</b></td>
<td>Average outgoing packet size in bytes</td></tr>
<tr><td><b>CmpO</b></td>
<td>Outgoing compressed packets/sec</td></tr>
<tr valign=top><td><b>ErrsO</b></td>
<td>Total outgoing errors/sec. This is an aggregation of outgoing error counters. To see explicit
error counters use <i>--netopts e</i> </td></tr>
</table>
<p><h3>Network, <i>collectl -sn --netopts e</i></h3>
<p>
This alternative format, which is displayed when you specify <i>--netopts e</i> enumerates
the individual error types. You cannot see both output formats at the same time.
<div class=terminal-wide14>
<pre>
# NETWORK ERRORS SUMMARY (/sec)
# ErrIn DropIn FifoIn FrameIn ErrOut DropOut FifoOut CollOut CarrOut
</pre></div>
<table>
<tr><td><b>ErrIn</b></td>
<td>Receive errors/sec detected by the device driver</td></tr>
<tr><td><b>DropIn</b></td>
<td>Receive packets dropped/sec</td></tr>
<tr><td><b>FifoIn</b></td>
<td>Receive packet FIFO buffer errors/sec</td></tr>
<tr><td><b>FrameIn</b></td>
<td>Receive packet framing errors/sec</td></tr>
<tr><td><b>ErrOut</b></td>
<td>Transmit errors/sec detected by the device driver</td></tr>
<tr><td><b>DropOut</b></td>
<td>Transmit packets dropped/sec</td></tr>
<tr><td><b>FifoOut</b></td>
<td>Transmit packet FIFO buffer errors/sec</td></tr>
<tr><td><b>CollOut</b></td>
<td>Transmit collisions/sec detected on the interface</td></tr>
<tr><td><b>CarrOut</b></td>
<td>Transmit packet carrier loss errors detected/sec</td></tr>
</table>
<p><h3>NFS, <i>collectl -sf</i></h3>
<p>
As of version 3.2.1, by default collectl collects and reports on all versions of nfs data,
both clients and servers. One can limit the types of data reported with <i>--nfsfilt</i>
and if only server or client data has been selected, only that type of data will be
reported as shown in the 2 forms below. When both server and client data are being
reported they will be displayed side by side. As with brief format, if filters have been
selected they will be displayed in the header.
<div class=terminal-wide14>
<pre>
# NFS SUMMARY (/sec)
#<---------------------------server--------------------------->
# Reads Writes Meta Comm UDP TCP TCPConn BadAuth BadClnt
</pre></div>
<table>
<tr><td><b>Reads</b></td><td>Total reads/sec</td></tr>
<tr><td><b>Writes</b></td><td>Total writes/sec</td></tr>
<tr><td><b>Meta</b></td><td>Total nfs meta data calls/sec, where meta data is
considered to be any of: lookup, access, getattr, setattr, readdir and readdirplus,
noting that not all types of nfs version report all as V3 clients/servers do.</td></tr>
<tr><td><b>Comm</b></td><td>Total commits/sec</td></tr>
<tr><td><b>UDP</b></td><td>Number of UDP packets/sec</td></tr>
<tr><td><b>TCP</b></td><td>Number of TCP packets/sec</td></tr>
<tr><td><b>TCPConn</b></td><td>Number of TCP connections/sec</td></tr>
<tr><td><b>BadAuth</b></td><td>Number of authentication failures/sec</td></tr>
<tr><td><b>BadClnt</b></td><td>Number of unknown clients/sec</td></tr>
</table>
<div class=terminal-wide14>
<pre>
# NFS SUMMARY (/sec)
#<----------------client---------------->
# Reads Writes Meta Comm Retrans Authref
</pre></div>
<table>
<tr><td><b>Reads</b></td><td>Total reads/sec</td></tr>
<tr><td><b>Writes</b></td><td>Total writes/sec</td></tr>
<tr><td><b>Meta</b></td><td>Total nfs meta data calls/sec, where meta data is
considered to be any of: lookup, access, getattr, setattr, readdir and readdirplus,
noting that not all types of nfs version report all as V3 clients/servers do.</td></tr>
<tr><td><b>Comm</b></td><td>Total commits/sec</td></tr>
<tr><td><b>Retrans</b></td><td>Number of retransmissions/sec</td></tr>
<tr><td><b>Authref</b></td><td>Number of authrefreshes/sec</td></tr>
</table
<p><h3>NFS, <i>collectl -sf -nfsopts C</i></h3>
<p>The data reported for clients is slightly different, specifically the
retrans and authref fields.
<div class=terminal-wide14>
<pre>
# NFS CLIENT (/sec)
#<----------RPC---------><---NFS V3--->
#CALLS RETRANS AUTHREF READ WRITE
</pre></div>
<table>
<tr><td><b>Calls</b></td><td>Number of RPC calls/sec</td></tr>
<tr><td><b>Retrans</b></td><td>Retransmitted calls</td></tr>
<tr><td><b>Authref</b></td><td>Authentication failed</td></tr>
<tr><td><b>Read</b></td><td>Number of reads/sec</td></tr>
<tr><td><b>Write</b></td><td>Number of writes/sec</td></tr>
</table>
<p><h3>Slabs, <i>collectl -sy</i></h3>
<p>As of the 2.6.22 kernel, there is a new slab allocator, called SLUB, and since
there is not a 1:1 mapping between what it reports and the older slab allocator,
the format of this listing will depend on which allocator is being used. The following
format is for the older allocator.
<div class=terminal-wide14>
<pre>
# SLAB SUMMARY
#<------------Objects------------><--------Slab Allocation-------><--Caches--->
# InUse Bytes Alloc Bytes InUse Bytes Total Bytes InUse Total
</pre></div>
<table>
<tr><td colspan=2>Objects</td></tr>
<tr valign=top><td><b>InUse</b></td>
<td>Total number of objects that are currently in use.</td></tr>
<tr valign=top><td><b>Bytes</b></td>
<td>Total size of all the objects in use.</td></tr>
<tr valign=top><td><b>Alloc</b></td>
<td>Total number of objects that have been allocated but not necessarily in use.</td></tr>
<tr valign=top><td><b>Bytes</b></td>
<td>Total size of all the allocated objects whether in use or not.</td></tr>
<tr><td colspan=2>Slab Allocation</td></tr>
<tr valign=top><td><b>InUse</b></td>
<td>Number of slabs that have at least one active object in them.</td></tr>
<tr valign=top><td><b>Bytes</b></td>
<td>Total size of all the slabs.</td></tr>
<tr valign=top><td><b>Total</b></td>
<td>Total number of slabs that have been allocated whether in use or not.</td></tr>
<tr valign=top><td><b> Bytes</b></td>
<td>Total size of all the slabs that have been allocted whether in use or not.</td></tr>
<tr><td colspan=2>Caches</td></tr>
<tr valign=top><td><b>InUse</b></td>
<td>Not all caches are actully in use. This included only those with non-zero
counts.</td></tr>
<tr valign=top><td><b>Total</b></td>
<td>This is the count of all caches, whether currently in use or not.</td></tr>
</table>
<p>This is format for the new <i>slub</i> allocator
<div class=terminal-wide14>
<pre>
# SLAB SUMMARY
#<---Objects---><-Slabs-><-----memory----->
# In Use Avail Number Used Total
</pre></div>
One should note that this report summarizes those slabs being monitored. In general
this represents all slabs, but if filering is being used these numbers will only
apply to those slabs that have matched the filter.
<p>
<table>
<tr><td colspan=2>Objects</td></tr>
<tr><td><b>InUse</b></td>
<td>The total number of objects that have been allocated to processes.</td></tr>
<tr valign=top><td><b>Avail</b></td>
<td>The total number of objects that are available in the currently allocated slabs.
This includes those that have already been allocated toprocesses.</td></tr>
<tr><td colspan=2>Slabs</td></tr>
<tr valign=top><td><b>Number</b></td>
<td>This is the number of individual slabs that have been allocated and
taking physical memory.</td></tr>
<tr><td colspan=2>Memory</td></tr>
<tr valign=top><td><b>Used</b></td>
<td>Used memory corresponds to those objects that have been allocated to
processes.</td></tr>
<tr valign=top><td><b>Total</b></td>
<td>Total physical memory allocated to processes. When there is no filtering
in effect, this number will be equal to the Slabs field reported by -sm.</td></tr>
</table>
<p><h3>Sockets, <i>collectl -ss</i></h3>
<div class=terminal-wide14>
<pre>
# SOCKET STATISTICS
# <-------------Tcp-------------> Udp Raw <---Frag-->
#Used Inuse Orphan Tw Alloc Mem Inuse Inuse Inuse Mem
</pre></div>
<table>
<tr><td valign=top><b>Used</b></td><td>Total number if socket allocated which can include additional types such as domain.</td></tr>
<tr><td colspan=2>Tcp</td></tr>
<tr><td><b>Inuse</td><td>Number of TCP connections in use</td></tr>
<tr><td><b>Orphan</td><td>Number of TCP orphaned connections</td></tr>
<tr><td><b>Tw</td><td>Number of connections in <i>TIME_WAIT</i></td></tr>
<tr><td><b>Alloc</td><td>TCP sockets allocated</td></tr>
<tr><td><b>Mem</td><td></td></tr>
<tr><td colspan=2>Udp</td></tr>
<tr><td><b>Inuse</td><td>Number of UCP connections in use</td></tr>
<tr><td colspan=2>Raw</td></tr>
<tr><td><b>Inuse</td><td>Number of RAW connections in use</td></tr>
<tr><td colspan=2>Frag</td></tr>
<tr><td><b>Inuse</td><td></td></tr>
<tr><td><b>Mem</td><td></td></tr>
</table>
<p><h3>TCP, <i>collectl -st</i></h3>
These are the counters one sees when running the command <i>netstat -s</i>, whose output is very verbose. Since this
format is an attemt to compress those field names to 6 characters or less, sometime something gets lost in the translation.
As described in the <a href=Data-brief.html>brief</a> data formats, the actual TCP data displayed is based on the value of
<i>--tcpfilt</i> and like brief data, everything is displayed on a single line which can be quite wide, even more reason to
use this switch, espcially since the default format is over 200 columns wide! The following definitions are based the
value of that filter:
<p><b>--tcpfilt i</b>
<div class=terminal-wide14>
<pre>
# TCP SUMMARY (/sec)# TCP STACK SUMMARY (/sec)
#<----------------------------------IpPkts----------------------------------->
# Receiv Delivr Forwrd DiscdI InvAdd Sent DiscrO ReasRq ReasOK FragOK FragCr
</pre></div>
<table>
<tr><td><b>Receiv</b><td></td><td>- total packets received/sec</td></tr>
<tr><td><b>Delivr</b><td></td><td>- incoming packets delivered/sec</td></tr>
<tr><td><b>Forwrd</b><td></td><td>- packets forwarded</td></tr>
<tr><td><b>DiscdI</b><td></td><td>- discarded incoming packets</td></tr>
<tr><td><b>InvAdd</b><td></td><td>- packets received with invalid addresses</td></tr>
<tr><td><b>Sent</b><td></td><td> - requests sent out/sec</td></tr>
<tr><td><b>DiscrO</b><td></td><td>- discarded outbound requests</td></tr>
<tr><td><b>ReasRq</b><td></td><td>- reassembled requests</td></tr>
<tr><td><b>ReasOK</b><td></td><td>- reassembled OK</td></tr>
<tr><td><b>FragOK</b><td></td><td>- fragments received OK</td></tr>
<tr><td><b>FragCr</b><td></td><td>- fragments created</td></tr>
</table>
<p><b>--tcpfilt t</b>
<div class=terminal-wide14>
<pre>
# TCP SUMMARY (/sec)# TCP STACK SUMMARY (/sec)
#<---------------------------------Tcp--------------------------------->
# ActOpn PasOpn Failed ResetR Estab SegIn SegOut SegRtn SegBad SegRes
</pre></div>
<table>
<tr><td><b>ActOpn</b></td><td>- active connections opened/sec</td></tr>
<tr><td><b>PasOpn</b></td><td>- passive connection opened/sec</td></tr>
<tr><td><b>Failed</b></td><td>- failed connection attempts</td></tr>
<tr><td><b>ResetR</b></td><td>- connection resets received</td></tr>
<tr><td><b>Estab</b></td><td> - connections established</td></tr>
<tr><td><b>SegIn</b></td><td> - segments received/sec</td></tr>
<tr><td><b>SegOut</b></td><td>- segments sent out/sec</td></tr>
<tr><td><b>SegRtn</b></td><td>- segments retransmitted</td></tr>
<tr><td><b>SegBad</b></td><td>- bad segments received</td></tr>
<tr><td><b>SegRes</b></td><td>- resets sent</td></tr>
</table>
<p><b>--tcpfilt u</b>
<div class=terminal-wide14>
<pre>
# TCP SUMMARY (/sec)# TCP STACK SUMMARY (/sec)
#<------------Udp----------->
# InDgm OutDgm NoPort Errors
</pre></div>
<table>
<tr><td><b>InDgm</b></td><td>- packets received/sec</tr>
<tr><td><b>OutDgm</b></td><td>- packets sent/sec</tr>
<tr><td><b>NoPort</b></td><td>- packets received to unknown port</tr>
<tr><td><b>Errors</b></td><td>- packet receive errors</tr>
</table>
<p><b>--tcpfilt c</b>
<div class=terminal-wide14>
<pre>
# TCP SUMMARY (/sec)# TCP STACK SUMMARY (/sec)
#<----------------------------Icmp--------------------------->
# Recvd FailI UnreI EchoI ReplI Trans FailO UnreO EchoO ReplO
</pre></div>
<table>
<tr><td><b>Recvd</b></td><td>- ICMP messages received</td></tr>
<tr><td><b>FailI</b></td><td>- incoming ICMP messages failed</td></tr>
<tr><td><b>UnreI</b></td><td>- input destination unreachable</td></tr>
<tr><td><b>EchoI</b></td><td>- input echo requests</td></tr>
<tr><td><b>ReplI</b></td><td>- input echo reploes</td></tr>
<tr><td><b>Trans</b></td><td>- ICMP messages sent</td></tr>
<tr><td><b>FailO</b></td><td>- outbound ICMP messages failed</td></tr>
<tr><td><b>UnreO</b></td-><td>- output destination unreachable</td></tr>
<tr><td><b>EchoO</b></td><td> - output echo requests</td></tr>
<tr><td><b>ReplO</b></td><td> - output echo replies</td></tr>
</table>
<p><b>--tcpfilt T</b>
<div class=terminal-wide14>
<pre>
# TCP SUMMARY (/sec)# TCP STACK SUMMARY (/sec)
#<------------------------------------------TcpExt----------------------------------------->
# FasTim Reject DelAck QikAck PktQue PreQuB HdPdct AkNoPy PreAck DsAcks RUData REClos SackS
</pre></div>
<table>
<tr><td><b>FasTim</b></td><td>- TCP sockets finished time wait in fast timer</td></tr>
<tr><td><b>Reject</b></td><td>- packet rejects in established connections because of timestamp</td></tr>
<tr><td><b>DelAck</b></td><td>- delayed ACKs sent</td></tr>
<tr><td><b>QikAck</b></td><td>- times quick ACK mode activated </td></tr>
<tr><td><b>PktQue</b></td><td>- packets directly queued to recvmsg prequeue</td></tr>
<tr><td><b>PreQuB</b></td><td>- bytes directly received in process context from prequeue</td></tr>
<tr><td><b>HdPdct</b></td><td>- packet headers predicted</td></tr>
<tr><td><b>AkNoPy</b></td><td>- acknowledgements for received packets not containing data </td></tr>
<tr><td><b>PreAck</b></td><td>- predicted acknowledgements</td></tr>
<tr><td><b>DsAcks</b></td><td>- DSACKS sent for old packets</td></tr>
<tr><td><b>RUData</b></td><td>- connections reset to do unexpected data</td></tr>
<tr><td><b>REClos</b></td><td>- connections reset due to early close</td></tr>
<tr><td><b>SackS</b></td><td>- SackShiftFallback</td></tr>
</table>
<table width=100%><tr><td align=right><i>updated November 9, 2011</i></td></tr></colgroup></table>
</body>
</html>
|