/usr/share/collectl/FAQ-colplot.html

<title>Colplot Frequently Asked Questions</title>

<body>

<h1>Colplot Frequently Asked Questions</h1>
<h2>General Questions</h2>
<ul>
  <li><a href="#gen1">What is colplot?</a></li>
  <li><a href="#gen2">What kind of systems will it run on?</a></li>
  <li><a href="#gen3">Are there any prerequisites?</a></li>
  <li><a href="#gen4">What are plot <i>types</i>?</a></li>
</ul>
<h2>Running colplot</h2>
<ul>
  <li><a href="#run1">How do I get started?</a></li>
  <li><a href="#run2">How does the CLI work?</a></li>
  <li><a href="#run3">How do I generate plot files?</a></li>
</ul>
<h2>Creating and Using Custom Plots</h2>
<ul>
  <li><a href="#cust1">How do I create custom plots?</a></li>
  <li><a href="#cust3">I built my own custom plot but it's not working correctly.</a></li>
  <li><a href="#cust4">How can I learn more about customizing plots?</a></li>
</ul>
<h2>Operational Problems</h2>
<ul>
  <li><a href="#ops1">I'm not getting all the plots I requested.</a></li>
  <li><a href="#ops2">Why aren't plots updating more frequently in live mode?</a></li>
  <li><a href="#ops3">How can I override the default directory?</a></li>
  <li><a href="#ops4">Disk MB and Disk I/O lines on the <i>diskboth</i> plot obscure each other</a></li>
  <li><a href="#ops5">I want to generate detail plots but forgot to specify detail switches when
  <li><a href="#ops6">Why am I getting a message that there was a conflit in datafile and plot sub-options?</a></li>
  <li><a href="#ops7">I'm trying to generate detail plots and there are zillions of them.</a></li>
  <li><a href="#ops8">Colplot runs without errors but also without plots</a></li>
  <li><a href="#ops9">What do messages about no X11 or png support mean?</a></li>
  <li><a href="#ops10">Why do I get blank CPU plots when requestion logarithmic Y-Axis?</a></li>
  <li><a href="#ops11">I'm getting the error: <i>can't find column name '[CPU]Irq%' for plot cpu</i></a></li>
</ul

<h1>General Questions</h1>
<a name="gen1"></a><h2>What is colplot?</h2>
<p>
Colplot is a plotting tool designed to work with <i>plot files</i>
generated by collectl.  It is <i>not</i> a general purpose plotting
package.

<a name="gen2"></a><h2>What kind of systems will it run on?</h2>
<p>
The main focus for development and testing for colplot has been linux
systems running a web server and/or an X server.  It has also been
tested on PCs and seems to work fine there are well.

<a name="gen3"></a><h2>Are there any prerequisites?</h2>
To run as a web-based system, a web server is required.  It has been
tested on linux using apache and IIS on windows though there's no reason
why it shouldn't work with other web servers.  It also requires gnuplot
to generate the plots and also ghostscript if you wish to generate pdf
files.

<a name="gen4"></a><h2>What are plot <i>types</i>?</h2>
<p>
There are 4 types of plots colplot is capable of displaying.  Rather than
increasing the number of different plots by 4-fold, it was decided to select a
type for each one that produced the most favorable look for each under most
conditions.
<p>
As it turns out, some plots look best when displayed one way and  can be very
difficult to see when displayed another.  Perhaps the best example is network
data.  When communications is in one direction, the use of lines help give one
a better feel for the load since the plot is filled with a single color.
However, when a system is transmitting and receiving at very close speeds,
the second line drawn usually overwrites the first making the first invisible.
One solution is a <i>point</i> plot which has no lines and can greatly help
show what is really happening.  At the same time, if one chooses to <i>zoom in</i>
by choosing a narrower time frame, there may be too few points to get a good
feel for the data.
<p>
When one looks at memory utilization, one may want to see how much of each type
is being used and so a line plot show just that.  Unlike network data, different
types of memory virtually never change at the same rate in the same direction
and so data occlusion is rarely a problem.  
One can easily read values off the left scale and get useful information.  
However, other times one might be interested in total memory 
used, making this format more difficult to interpret.  A <i>stacked</i>
plot literally stacks data points on top of each other so each data point
actually represents a total of all those below it, the topmost therefore
showing <i>total memory used</i>.
<p>
There are many other examples where one format might be more useful than the
other so the real solution here it whenever you find yourself having a hard
time understanding what you're seeing, consider a different type.

<h1>Running colplot</h1>
<a name="run1"></a><h2>How do I get started?</h2>
<p>
To get started, you first need to make sure <i>colplot</i> has been installed
correctly.  If you instaled colplot on a linux system running an apache web
server via the <i>collectl-utils</i> rpm, all you should need to do is restart
apache and you're ready to begin plotting.  If not, see the <i>INSTALL</i> 
document that comes with the kit.
<p>
Next you need one or more plottable files. Typically plot
files are generated by playing back a collectl raw or raw.gz file using a 
command like this: "collectl -p <i>filespec</i> -P -f 
<i>destination-directory</i> -ocz".  Alternatively you can use the 
<i>genplotfiles</i> utility, which can be pretty handy for generating 
plots files from a set of raw files, especially when used as a cron job.
See <i>man collectl</i> for more help and options when running collectl
directly or <i>man genplotfiles</i> for the use of that utility.
<p>
There are also some sample plot files included in <i>/usr/share/collectl/plotfiles</i> 
directory which you can easily set by clicking on the <i><b>ChangeDir></b></i>
button at the top of the display if you've installed colplot on a Linux machine.
To keep the size down, these files only contain a small set of data and you 
will need to <i>zoom in</i> by picking the appropriate <i>from</i> and 
<i>thru</i> times to get a good look at the quality of plots colplot is
capable of producing
<p>
To actually generate a plot, make sure colplot is pointing to the directory in
which the plottable files are stored.  This can be identified by the second line
of the web page and furthermore.  Colplot won't let you change 
to a directory that doesn't contain at least one plottable file in it.  
Now you simply select one or more plots and hit <i>return</i> or click on 
the <i>Generate Plot</i> button.
<p>
<a name="run2"></a><h2>How does the CLI work?</h2>
<p>
The first step is to make sure your configuration supports it.  This also assume
you have all the necessary supporting modules such as <i>gnuplot</i> and
<i>ghostscript</i> if you want to generate files.  If you're running on a Linux
machine, you also need to make sure you're running X to display the graphics.
If you're running on a PC connected to a Linux machine through a remote terminal
session not only does that Linux machine need to be running X you also need to
set the display back to your PC on which you have an X Server running.  If
colplot is installed on a PC, you're ready to begin.
<p>
The first thing to do is to make sure colplot is installed correctly and to do
this you can just run the command <i>colplot -help</i> to make sure it's
there.	If you haven't installed it via the rpm the soft-link <i>colplot</i>
which points to the executable <i>colplot.pl</i> will not be there and you may
need to invoke it in a system-specific manner.
At the very top of the display it will show its defaults including the
directory it's looking in for plot files.
<p>
To generate your first plot simply type <i>colplot.pl -plot cpu</i> and you
should see a CPU plot for every file in your default directory.  To see what
other plot names are available to you type <i>colplot.pl -showplots</i>.  The
first column is the name of the plot, the second the <i>collectl</i> subsystem
it corresponds to and the third column is a description of that plot.

See <i>colplot -help</i> for more details.

<p><a name="run3"></a><h2>How do I generate plot files?</h2>
There is a utility that ships with colplot in the <i>examples</i> directory
called <i>genplotfiles.pl</i> which can greatly facilitate the generation of 
plotfiles every day on a Linux based system by simply putting it in your crontab
and specifying a single parameter to tell it which directory to put the plot
files in.  Naturally that system would need to be running <i>collectl</i> to 
generate the data.  If the data is on another system you can always automate its
movement to the plotting system using a utility such as <i>rsync</i> and then
pointing <i>genplotfiles.pl</i> to the directory containing those <i>raw</i>
files.
<P>
For more information on <i>genplotfiles.pl</i> type <i>man genplotfiles</i>.
<p>

<h1>Creating and Using Custom Plots</h1>
<a name="cust1"></a><h2>How do I create custom plots?</h2>
If you examine a plottable collectl file such as one with the extension 
<i>.tab</i>, you'll see a number of uniquely named 
column headers.  It is these names that drive the plotting process.
The fastest way to write a custom plot definition is to look at 
<i>colplotlib.defs</i> to see how the existing definitions are built as entries
in several main sections which are introduced with <>s.  These entries start 
with a name and are followed by a list of elements surrounded by braces.  
<p>
At the most basic level, you create your plot definition in the 
<i>allplots</i> section by giving the entry a name, title and list the names
of the columns you wish to plot.  If you want to limit the
range of the y-axis, you can set the <i>mask</i> to 1 and include values
for ymin/ymax, noting the default for ymin is 0.  You should also include a
descriptive text string for you plot in the <i><descplots></i> section so it
will show up when to request a list of availble plots when running colplot.
<p>
At this point you are essentially done, but you can further customize your
plot definitions at the individual field level.  In some cases you may want to
use a name other than the column header, apply specific ymin/ymax values
(which can result in 2 or more y-axes, though colplot limits you to 2) 
or even divide the plottable values by a 
constant for the purpose of converting to some other units such as MBs.
<p>
Finally, if you want to create your own macro simply give it a name and list
the corresponding plot names in the <i>macros</i> section as either a space or
comma separate list.  Since there are no optional fields involved you do not
put these entries in braces.

<a name="cust3"></a><h2>I built my own custom plot but it's not working correctly.</h2>
<p>
This is almost certainly related to conflicting data formats in the
data file and the plot definition in the <i>.defs</i> file or the 
column names to plot are just wrong.
Carefully example the header of the data file and make sure
the column names and data ranges are consistent.  If you're still not sure
what's going on, consider trying the <i>--showparams</i> switch in CLI mode.

<a name="cust4"></a><h2>How can I learn more about customizing plots?</h2>
<p>
If you installed colplot on a Linux system with RPM, these definitions can be 
found in <i>/opt/hp/collectl/lib/colplotlib.defs</i>.  You can pick 
up a number of ideas just
from examining the parameters used to define the standard plots.  
It is also possible that some questions you have about colplot may have been 
answered in the collectl man pages, FAQs or documentation on sourceforge
so be sure to check there too.

<h1>Operational Problems</h1>
<a name="ops1"></a><h2>I'm not getting all the plots I requested</h2>
<p>
One often has a variety of files for which data has been collected for 
different subsystems.  These may be from different physical computer systems 
or simply collected on 
different days with different settings for <i>collectl</i>.  It has been
found to be more convenient to silently ignore files for which there is
no data to plot.  If you think a plot should have been generated, inspect
the header of the data file in question and you will usually find it 
doesn't contain the data you thought it did.  Alternatively you may have
simply requested the wrong plot.  Another typical scenario is requesting
detail plots for which there are no detail files.

<a name="ops2"></a><h2>Why aren't plots updating more frequently in live mode?</h2>
Perhaps collectl's I/O buffers aren't being flushed frequenly enough.  See
collectl's <i>-F</i> switch.

<a name="ops3"></a><h2>How can I override the default directory?</h2>
<p>
To make the change permanent set <i>PlotDir=</i> in colplot.conf or append 
<i>?dir=dirname</i> to the URL when invoking it.

<a name="ops4"></a><h2>Disk MB and Disk I/O lines on the <i>diskboth</i> plot obscure each other</h2>

Some devices will occasionally generate data of the same values, and therefore 
overwrite each other.  This is not just a disk problem.  When this happens you 
have a couple of choices:
<ul>
  <li>Choose separate plots for each category, in this case disk and diskio</li>
  <li>Consider writing your own custom definition - it really isn't that hard 
      especially if you start with an existing definition and hack it up.  
      Now you can choose your own plotting dimensions and change the scale 
      for the MB and I/O so they don't overlap as much.
</ul>

<a name="ops5"></a><h2>I want to generate detail plots but forgot to specify detail switches when
I collected the data.</h2>
<p>
As it turns out, when you request summary data to be collected, collectl actually
has to generate data at the detail level so that it can summarize it.
So, for example, if you collect data using <i>collectl -sdn ...</i> you can
generate your plot files using <i>collectl -sdnDN ... </i> and get both summary
as well as detail data for disks and networks.
<p>
NOTE - if you try to playback data that you haven't recorded you will get 0s.

<a name="ops6"></a><h2>Why am I getting a message that there was 
a conflit in datafile and plot sub-options?</h2>
<p>
This message is related to lustre data, which uses the -O switch in <i>collectl</i>
to specify additional data such as rpc-buffers, metadata, readahead or disk
block I/O counts.  If one requests a plot of one of these and that data
doesn't exist in the file, this message will be produced.  This is an exception
to the earlier comment about silently ignoring  plots because these are
rarer and so harder to diagnose without the message when they occur.

<a name="ops7"></a><h2>I'm trying to generate detail plots and 
there are zillions of them.</h2>
<p>
As they say, <i>be careful what you ask for</i> because colplot will happily try
to do what you ask it to.  Most of the time this happens with lustre
rpc-buffers or disk block I/O statistics which occur at the OST level and
can therefore be a lot of them, especially when there can be multiple
plots generated for a single request (such as with the lusblkR macro).
This can also happen on systems with many disks, cpus or even networks.
<p>
If you really are interested in detail data consider reducing the number
of devices by filters or customer plot defintions that specify let information.

<a name="ops8"></a><h2>Colplot runs without errors but also without plots</h2>
<p>
This could happen if you are running a vesion of gnuplot that doesn't have the
appropriate graphics packages linked in.  In order to run from a brower you need
<i>png</i> support and from a terminal <i>X11</i> support.  The easiest way to
tell is run gnuplot from a terminal window and it should say something like 
<i>Terminal type set to 'X11'</i>.  If not it doesn't support <i>X</i> graphics
and will not work from the command line.  Now while in gnuplot enter the command
<i>set terminal png</i> and gnuplot should respond with 
<i>Terminal type set to 'png'</i> and if not it won't support web-based graphics.
See the next section for even more details.

<a name="ops9"></a><h2>What do messages about no X11 or png support mean?</h2>
<p>
As discussed in the previous section, colplot requires png or X11 support to
run from a browser or command line respectively.  As of version 4.1.0, if you try 
to run colplot from an environment that doesn't support that capability it will not 
run and you will get the associted error message.
Additionally, the -version information displayed has been extended to explicitly state
the types of graphics supported by gnuplot as addtional confirmation.
<p>
To proceed you will need to install a version of gnuplot that has been built to
support these types of graphics OR build it yourself!  If building it yourself note
that you will need to install the <i>png-devel</i> package to support png graphics
which in turn also requires <i>gd</i> and <i>/gd-devel</i>.  X11 graphics will 
require the installation of <i>xorg-x11-devel</i> as well as supporting packages
<i>freetype2-devel</i> and <i>fontconfig-devel</i>.


<a name="ops10"></a><h2>Why do I get blank CPU plots when requestion logarithmic Y-Axis?</h2>
<p>
CPU plots are defined to have a Y-Axis range from 0 to 100 and by definition a logarithmic axis
must not begin at 0.  If you must do this try defining a custom plot that starts at 1.

<a name="ops11"></a><h2>I'm getting the error: <i>can't find column name '[CPU]Irq%' for plot cpu</i></h2>
<p>
Some of the CPU plots have been enhanced to add additional data fields that were not
captured with eariler versions of collectl.  Since these represent real CPU time they
should have been included earlier and weren't.  Now, cpu time for IRQs, Soft Interrupts
and Steal are all included as <i>SysAll</i> as well as <i>SysMore</i> in case you want
to be able to differentiate between pure system time and these other fields.  If you
want to see the breakout by these specific fields, you can choose <i>cpumore</i> in
the <i>plot by name</i> field.
<p>
If you are getting this error you have two choices - regenerated the plot files with a
newer version of collectl and thse new fields will be included.  Otherwise you will not
be able to select the standard cpu plot and instead will need to choose <i>cpuold</i>
by name and keep in mind this will not account for any cpu usage in these 3 unreported 
fields.

</body>
</html>
collectl-utils 4.8.1-2 / usr / share / collectl / FAQ-colplot.html
