/usr/src/blcr-0.8.5/doc/html/FAQ.html is in blcr-dkms 0.8.5-2.1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 508 509 510 511 512 513 514 515 516 517 518 519 520 521 522 523 524 525 526 527 528 529 530 531 532 533 534 535 536 537 538 539 540 541 542 543 544 545 546 547 548 549 550 551 552 553 554 555 556 557 558 559 560 561 562 563 564 565 566 567 568 569 570 571 572 573 574 575 576 577 578 579 580 581 582 583 584 585 586 587 588 589 590 591 592 593 594 595 596 597 598 599 600 601 602 603 604 605 606 607 608 609 610 611 612 613 614 615 616 617 618 619 620 621 622 623 624 625 626 627 628 629 630 631 632 633 634 635 636 637 638 639 640 641 642 643 644 645 646 647 648 649 650 651 652 653 654 655 656 657 658 659 660 661 662 663 664 665 666 667 668 669 670 671 672 673 674 675 676 677 678 679 680 681 682 683 684 685 686 687 688 689 690 691 692 693 694 695 696 697 698 699 700 701 702 703 704 705 706 707 708 709 710 711 712 713 714 715 716 717 718 719 720 721 722 723 724 725 726 727 728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790 791 792 793 794 795 796 797 798 799 800 801 802 803 804 805 806 807 808 809 810 811 812 813 814 815 816 817 818 819 820 821 822 823 824 825 826 827 828 829 830 831 832 833 834 | <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html>
<head>
<title>BLCR Frequently Asked Questions (for version 0.8.5)</title>
<meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1">
</head>
<body bgcolor="#ffffff">
<h1>BLCR Frequently Asked Questions (for version 0.8.5)</h1>
<h3>General Questions</h3>
<ul>
<li><a href="#whatisblcr">What is BLCR?</a>
<li><a href="#sigstop">How is checkpoint/restart different than SIGSTOP/SIGCONT?</a>
<li><a href="#condor">How is BLCR different than "user-level" checkpointing libraries like Condor, etc.?</a>
<li><a href="#platforms">What kinds of Linux systems does BLCR support?</a>
<li><a href="#distros">What Linux distributions does BLCR work with?</a>
<li><a href="#threads">Can BLCR checkpoint/restart multithreaded programs?</a>
<li><a href="#procs">Can BLCR checkpoint/restart multi-process applications?</a>
<li><a href="#limitations">Are there limits to the types of programs can BLCR checkpoint? </a>
<li><a href="#callbacks">What if I want to checkpoint a program that uses resources that BLCR can't checkpoint?</a>
<li><a href="#mpi">Does BLCR support checkpointing parallel/distributed applications?</a>
<li><a href="#rootbuild">Do I need root access in order to use BLCR?</a>
<li><a href="#batch">Is BLCR integrated with any batch systems? </a>
<li><a href="#porting">How hard is it to port BLCR to an architecture that isn't currently supported?</a>
</ul>
<h3>Build/Install Questions</h3>
<ul>
<li><a href="#patch">Does BLCR require a kernel patch?</a>
<li><a href="#buildreqs">What do I need in order to build and use BLCR?</a>
<li><a href="#unconfig">What if I my kernel sources are unconfigured?</a>
<li><a href="#modvesion">How do I get past the error "linux/modversions.h:
No such file or directory"?</a>
<li><a href="#systemmap"> What if I get an error about my <tt>System.map</tt> file?</a>
<li><a href="#kcompiler"> Why do I get "Invalid module format" when loading BLCR's kernel modules?</a>
</ul>
<h3>Usage Questions</h3>
<ul>
<li><a href="#samepid">Why do I get this error: "Restart failed:
Device or resource busy"?</a>
<li><a href="#sig31">Why does my application dies with "Real-time signal 31"
(or 32, etc.) when I try to checkpoint it?</a>
<li><a href="#prelink">Why can I restart jobs on the original machine they ran
on, but not a different node in my cluster?</a>
<li><a href="#enoent">Why do I get the error "Restart failed: No such file or directory"?</a>
<li><a href="#eperm">Why do I get the error "Restart failed: Permission denied"?</a>
<li><a href="#preload">Why do I get the error "ERROR: ld.so: object 'libcr_run.so.0' from LD_PRELOAD cannot be preloaded: ignored"?</a>
<li><a href="#nscd">Why do I get "vmadump: mmap failed: /var/db/nscd/[something]" in the system logs?</a>
<li><a href="#hsperfdata">Why do I get "vmadump: mmap failed: /tmp/hsperfdata_[user]/[pid]" in the system logs?</a>
<li><a href="#staticlink">Why can't I checkpoint my statically linked application?</a>
<li><a href="#auditing">Can I use Linux kernel auditing support with BLCR?</a>
</ul>
<h3>Additional Resources</h3>
<ul>
<li><a href="#website">Where can I download BLCR?</a>
<li><a href="#website">Where can I find more information?</a>
<li><a href="#mailaddr">Is there a mailing list for BLCR?</a>
<li><a href="#bugzilla">Where can I report BLCR bugs?</a>
</ul>
<hr>
<h3>General Questions</h3>
<h4><a name="whatisblcr">What is BLCR?</a></h4>
<blockquote>
BLCR (Berkeley Lab Checkpoint/Restart) allows programs running on Linux to
be "checkpointed" (written entirely to a file), and then later "restarted".
BLCR can be found
at <a href="http://ftg.lbl.gov/checkpoint">http://ftg.lbl.gov/checkpoint</a>.
</blockquote>
<h4><a name="sigstop">How is checkpoint/restart different than SIGSTOP/SIGCONT?</a></h4>
<blockquote>
<p> Putting a process to sleep (via the SIGSTOP signal) implies stopping its
execution. Taking a checkpoint writes a snapshot of a process to disk:
the process may either be allowed to continue running after the checkpoint is
complete, or you can kill the process to release all of its resources .
<p> With sleep, a process's resources are not all fully released (such as virtual
memory, network connections, process id, etc.). Checkpointing then killing a
process fully releases all system resources.
<p> Restarts from checkpoint files can be used across machine reboots, and/or even
on different machines than the one that the checkpoint was taken on. This is
not true for SIGCONT.
</blockquote>
<h4><a name="condor">How is BLCR different than "user-level" checkpointing libraries like Condor, etc.?</a></h4>
<blockquote>
BLCR performs checkpointing and restarting inside the linux kernel. While
this makes it less portable than solutions that use user-level libraries, it
also means that it has full access to all kernel resources, and can thus
restore resources (like process IDs) that user-level libraries cannot.
This also allows BLCR to checkpoint/restart groups of processes (such as
shell scripts and their subprocesses), together with the pipes that
connect them.
</blockquote>
<h4><a name="platforms">What kinds of Linux systems does BLCR support?</a></h4>
<blockquote>
BLCR runs on x86 and x86_64 (Opteron/EM64T) systems running Linux 2.6.x
and 3.x.y kernels.
With the 0.8.5 release, we believe the following to work:
<ul>
<li>2.6.0 through 3.7.1 on x86 and x86_64.
</ul>
<p>BLCR 0.7.0 added experimental support for PPC (32-bit), and 0.6.0 added experimental
support for PPC64 and ARM. These three architectures have been tested as follows:
<ul>
<li>2.6.14 and newer on PPC
<li>2.6.12 and newer on PPC64
<li>2.6.17 and newer on ARM
</ul>
We are interested to hear of your success or failure with these three experimental
architectures, especially on kernels older than those we have tested.
<p>Note that 0.6.x was the last release series to support 2.4.x kernels.
</blockquote>
<h4><a name="distros">What Linux distributions does BLCR work with?</a></h4>
<blockquote>
<p> BLCR uses a set of configure-based tests to determine which kernel features
are available, and so in principle, BLCR should work with any
distribution that uses a supported CPU/kernel combination
(see <a href="#platforms">above</a>).
<p> Historically, BLCR has been tested with kernels from numerous versions of SuSE,
RHEL (and clones such as CentOS and Scientific Linux), Fedora, Debian, Ubuntu,
and many vanilla Linux kernels (from kernel.org)
from 2.6.0 on up. We have not tested every single version of the kernel from every
vendor, nor is each BLCR release retested against all distributions tested in the
past. However, we believe that BLCR should work on most distributions using
kernels in the ranges given <a href="#platforms">above</a> (except where vendors
may have applied patches that bring in problematic changes from kernels outside
that range).
<p> If after reading this question and the one <a href="#platforms">above</a> you
believe your platform should be supported but cannot get BLCR to work, then please
consult our <a href="#bugzilla">bug database</a> for possible solutions and then
report the problem if you don't find it already reported. We can't try every
possible platform ourselves and count on user's bug reports to let us know when
our testing has missed something.
</blockquote>
<h4><a name="threads">Can BLCR checkpoint/restart multithreaded programs?</a></h4>
<blockquote>
Yes, BLCR can checkpoint both single- and multithreaded (pthreads) programs
linked with the NPTL implementation of pthreads.
<strong>Since with the 0.7.0 release, BLCR no longer provides full support
for the older LinuxThreads implementation of pthreads. Please contact us if
you are able to devote some effort to restoring/maintaining such support in
the future.</strong>
<p> BLCR has not been tested with other threading packages,
such as those used by some Java runtimes. We are interested in hearing of
both success and failure with other threading packages.
</blockquote>
<h4><a name="procs">Can BLCR checkpoint/restart multi-process applications?</a></h4>
<blockquote>
Yes, starting with version 0.5.0 BLCR is able to save and restore groups of
related processes together with the pipes that connect them. To do this, BLCR
must be given a single request that covers all the processes involved. Currently
there are three ways to specify a group request to BLCR:
<ul>
<li>A process "tree", consisting of a process and all its non-orphaned descendants.
(An orphaned process is one who's parent has exited, leaving it as a child
of the <tt>init</tt> process).
<li>A POSIX process group, consisting of all processes with a given PGID (including
orphaned processes, if any).
<li>A POSIX session, consisting of all processes with a given SID (including
orphaned processes, if any).
</ul>
<p>While BLCR can save and restore the pipes used for IPC among processes in these
groups, it is unable at this time to deal with most other IPC mechanisms
(see <a href="#limitations">next FAQ</a>).
</blockquote>
<h4><a name="limitations">Are there limits to the types of programs can BLCR checkpoint? </a></h4>
<blockquote>
Yes. BLCR does not support checkpointing certain process resources.
While the following list is not exhaustive, it lists the most significant
issues we are aware of.
<ul>
<li>BLCR will not checkpoint and/or restore open sockets
(TCP/IP, Unix domain, etc.). At restart time any sockets will appear
to have been closed.</li>
<li>BLCR will not checkpoint and/or restore open character or block devices
(e.g. serial ports or raw partitions). At restart time any devices will
appear to have been closed.</li>
<li>BLCR does not handle SysV IPC objects (man 5 ipc). Such
resources are silently ignored at checkpoint time and are not
restored.</li>
<li>If a checkpoint is taken of a process with any "zombie" children, then
these children will not be recreated at restart time. A "zombie" is
defined as a process that has exited, but who's exit status has not yet
been reaped by its parent (via <tt>wait()</tt> or a related function).
This means that a wait()-family call made after a restart will never return
a status for such a child.</li>
</ul>
<p>If needed, applications can arrange to save any information necessary to
recreate/reacquire such resources at restart time
(see <a href="#callbacks">next FAQ</a>).
</blockquote>
<h4><a name="callbacks">What if I want to checkpoint a program that uses resources that BLCR can't checkpoint?</a></h4>
<blockquote>
BLCR supports adding 'callbacks' to user-level code, which are called when a
checkpoint is about to be performed, and when it is restarted (or continues
on after the checkpoint). This is how MPI communication can be handled
(see <a href="#mpi">next FAQ</a>).
<p>Full documentation of the callback interface has not yet been written because some of
the interfaces are still subject to change. However, the comments in the file
<tt>libcr.h</tt> should provide enough to get started.
</blockquote>
<h4><a name="mpi">Does BLCR support checkpointing parallel/distributed applications?</a></h4>
<blockquote>
Not by itself. But by using checkpoint callbacks (see <a href="#callbacks">previous FAQ</a>).
many MPI implementations have made themselves
checkpointable by BLCR. You can checkpoint/restart an MPI application running
across an entire cluster of machines with BLCR, without any application code
modifications, if you use one of these MPI implementations (listed alphabetically):
<ul>
<li> Cray MPI for their current x86-64 systems
<li> Intel MPI 4.1 or later
<li> LAM/MPI 7.x or later
<li> MPICH-V 1.0.x
<li> MPICH2-1.3a2 or later, including MPICH v3.x
<li> MVAPICH2 0.9.8 or later
<li> Open MPI 1.3 or later
<li> SGI MPT since September 2010
</ul>
See the documentation of your specific MPI for usage instructions. In almost
all cases you will need to use a tool provided by the MPI implementation
to request a checkpoint or restart, rather then using BLCR's <tt>cr_checkpoint</tt>
and <tt>cr_restart</tt> utilities.
<p>
At this time we are not aware of other MPI implementations that are
working on BLCR support, but surprisingly our information is not always the latest.
If in doubt, check the support channels of your favorite MPI implementation
<p> Note that any questions about using these MPI implementations with BLCR is more likely
to receive a useful response if directed to the support channels of the specific
MPI implementation than to the checkpoint@lbl.gov list.
</blockquote>
<h4><a name="rootbuild">Do I need root access in order to use BLCR?</a></h4>
<blockquote>
Root access is needed to install the BLCR kernel modules. However, once
these are installed, any user can checkpoint and restart their own programs
without needing root permission.
</blockquote>
<h4><a name="batch">Is BLCR integrated with any batch systems? </a></h4>
<blockquote>
We are aware of the following, but we are not always informed of new efforts
to integrate with BLCR. For the most up-to-date information you should consult
the support channels of your favorite batch system.
<dl>
<dt> TORQUE version 2.3 and later</dt><dd>
Support for serial and parallel jobs, including periodic checkpoints and
<tt>qhold</tt>/<tt>qrls</tt>.
</dd>
<dt> SLURM version 2.0 and later</dt><dd>
Support for automatic (periodic) and manually requested checkpoints.
</dd>
<dt> SGE (aka Sun Grid Engine)</dt><dd>
Information on configuring SGE to use BLCR can be found
<a href="http://gridengine.sunsource.net/howto/APSTC-TB-2004-005.pdf">here</a>.
There is also a thread on the checkpoint@lbl.gov list about modifications
to those instructions. The thread begins with
<a href="http://mantis.lbl.gov/hypermail/checkpoint/0952.html">this</a> posting.
</dd>
<dt> LSF</dt><dd>
Information on configuring LSF to use BLCR can be found in
<a href="http://mantis.lbl.gov/hypermail/checkpoint/1211.html">this</a> posting
on the checkpoint@lbl.gov list.
</dd>
<dt> Condor</dt><dd>
Information on configuring Condor to use BLCR to checkpoint "Vanilla Universe"
jobs with the help of Parrot can be found
<a href="http://www.escience.cam.ac.uk/projects/camgrid/blcr.html">here</a>.
</dd>
</dl>
Work is ongoing by third
parties to integrate BLCR into other batch systems. If you are
interested in adding BLCR support to a job launcher/scheduler, please
contact us!
<p> Note that any questions about using these batch systems with BLCR is more likely
to receive a useful response if directed to the support channels of the specific
batch system than to the checkpoint@lbl.gov list.
</blockquote>
<h4><a name="porting">How hard is it to port BLCR to an architecture that isn't currently supported?</a></h4>
<blockquote>
<p> Most of the architecture-specific code in BLCR is confined to small set of
logic to save and restore the CPU-specific register set (vmadump) and some
gcc inline assembly for atomic operations and special system calls. The majority of
BLCR's code base is entirely processor-independent.
<p> If you are interested in
seeing BLCR run on other chips, and are able to devote programmer resources,
please contact us! The Alpha platform is likely to be
the easiest since vmadump already supports this architecture for
Linux 2.6 kernels.
</blockquote>
<hr>
<h3>Build/Install Questions</h3>
<h4><a name="patch">Does BLCR require a kernel patch?</a></h4>
<blockquote>
No. All of the kernel logic used by BLCR is implemented within kernel
modules. You can thus compile BLCR and load it into a running kernel (with
<tt>modprobe</tt> or <tt>insmod</tt>) without needing to recompile your kernel or reboot.
</blockquote>
<h4><a name="buildreqs">What do I need in order to build and use BLCR?</a></h4>
<blockquote>
A machine that is running a supported architecture (x86 and x86_64 are fully
supported and PPC/PPC64 and ARM are "Experimental") and
Linux kernel 2.6.x or 3.x.y.
<p> A set of <b>configured</b> kernel headers that matches the kernel you wish to build against.
By configured, we mean that <tt>include/linux/version.h</tt> and the files in
<tt>include/linux/modules/</tt> match the target kernel.
For many distributions a kernel-devel or linux-headers package is often
enough if using the vendor's kernel. For a custom kernel, the actual kernel build
directory is often required.
<p> The kernel's symbol table. Normally the file <tt>/boot/System.map</tt>, or
equivalent will serve this purpose.
</blockquote>
<h4><a name="unconfig">What if I my kernel sources are unconfigured?</a></h4>
<blockquote>
BLCR needs to be able to examine a linux kernel source tree that has been
configured, and this configuration must match the kernel that you will run
BLCR against.
<p>If you do not have a configured linux kernel source tree, you may be able
to create one fairly easily. Many distributions provide a 'config' file
that is all you need to easily produce a configured kernel source tree.
Good places to look for a config file include
<tt>/boot/config-2.6.5-1.358</tt> or <tt>/config-2.6.5-1.358</tt>. In
some distributions, the kernel is actually setup to include its
configuration in <tt>/proc/config.gz</tt> (or <tt>/proc/config.bz2</tt>).
If you can find any one of these files then we can proceed with the
following steps:
<ol>
<li>Make a copy of the unconfigured source for the linux kernel you are using, and copy in the file you located:
<pre>
$ cp -a /usr/src/kernel-source-2.6.5 /tmp/linux-2.6.5-1.358
$ cd /tmp/linux-2.6.5-1.358
$ cp [CONFIG_FILE] .config
</pre></li>
<li>Configure it using one of the following:
<ul>
<li>For kernels 2.6.6 and newer:
<pre>
$ make modules_prepare
</pre></li>
<li>For 2.6.x kernels prior to 2.6.6:
<pre>
$ make prepare-all scripts
</pre></li>
</ul>
<li>Once that is done, you should be able to configure BLCR using the newly
configured kernel source. You should continue to use the <tt>System.map</tt>
file from your running kernel. What you want is probably something like
<pre>
$ ./configure --with-system-map=/boot/System.map-2.6.5-1.358 --with-linux=/tmp/linux-2.6.5-1.358.
</pre></li>
</ol>
</blockquote>
<h4><a name="modvesion">How do I get past the error "linux/modversions.h: No such file or directory"?</a></h4>
<blockquote>
<p> Please try rebuilding blcr after commenting out the following six lines
near the top of the files <tt>vmadump/vmadump.c</tt> and <tt>blcr_imports/imports.c</tt>:
<pre>
#if defined(CONFIG_MODVERSIONS) && ! defined(MODVERSIONS)
#define MODVERSIONS
#endif
#if defined(MODVERSIONS)
#include <linux/modversions.h>
#endif
</pre>
Let us know if your compilation still doesn't work.
</blockquote>
<h4><a name="systemmap">What if I get an error about my <tt>System.map</tt> file?</a></h4>
<blockquote>
To build, BLCR needs to read the <tt>System.map</tt> file that corresponds to the
kernel you will use BLCR with. Generally, BLCR will find this file
"automagically" during <tt>./configure</tt>, but some distributions do not
provide it, and/or you may not keep yours in a standard place.
<p>If you know where the correct <tt>System.map</tt> file is, use
<pre>
$ ./configure --with-system-map=PATH_TO_YOUR_SYSTEM.MAP
</pre>
<p>If your <tt>System.map</tt> is absent, it may still be available as an optional RPM.
For instance you may be able to get it by installing (depending
on the release) either the <tt>kernel-source</tt> or <tt>kernel-devel</tt> RPMs
for the kernel you will use BLCR with.
<p>However, Fedora Core 2 and some of its derivatives are shipping a "stripped-down"
<tt>System.map</tt> file. If this is the case, BLCR will abort during configuration
with an error stating
that the <tt>System.map</tt> cannot be used. You must install an additional RPM which
contains a full <tt>System.map</tt> in order to build BLCR. In Fedora Core 2
the '<tt>kernel-debuginfo</tt>' RPM contains a full <tt>System.map</tt> file, which it
will install into the <tt>/usr/lib/debug/boot</tt> directory. BLCR's configure
script will search this directory, but just to be certain you may still wish to pass
'<tt>--with-system-map</tt>' to point configure at the correct <tt>System.map</tt> file.
<blockquote>
<strong>Important Note</strong>: If you need to install the
<tt>kernel-debuginfo</tt> RPM, <em> make sure the correct version
is installed</em>. Specifically, the 'arch' type must be the same. If your
kernel was built for the '<tt>i386</tt>' (or '<tt>i586</tt>', or
'<tt>i686</tt>'), the kernel-debuginfo RPM must have the same value. Thus,
for an <tt>i586</tt> kernel, install
'<tt>kernel-debuginfo-2.6.5-1.358.i586.rpm</tt>'. To determine which
kernel version you have, use
<pre>
$ rpm -q kernel --qf '%{version}-%{release}.%{arch}\n'
</pre>
To make sure that you have installed compatible <tt>kernel</tt> and
<tt>kernel-debuginfo</tt> RPMs, use
<pre>
$ rpm -q kernel kernel-debuginfo --qf '%{version}-%{release}.%{arch}\n'
</pre>
(replace '<tt>kernel</tt>' with '<tt>kernel-smp</tt>' if you are using an SMP
kernel). You should see the same string, repeated twice.
<p>If you try to use BLCR with the wrong <tt>System.map</tt>, BLCR will
build without complaints, but will probably detect the problem when the
<tt>blcr.o</tt> kernel module is loaded (it does this by comparing some
well-known exported kernel symbols' addresses to those provided by the
<tt>System.map</tt> file), and the module load will be aborted.
</blockquote>
</blockquote>
<h4><a name="kcompiler">Why do I get "Invalid module format" when loading BLCR's kernel modules?</a></h4>
<blockquote>
Kernels from 2.6.0 through 2.6.18 check at module load time that the same compiler version (major and minor
numbers) were used to build both the module and the kernel. This is the error you will see
if they don't match. When this happens, you will need to reconfigure and build BLCR with
the correct compiler. When a module fails to load due to a version mismatch, you should
be able to find a message in the system logs indicating the required compiler version:
<pre>
blcr_imports: version magic '2.6.17 SMP mod_unload PENTIUM4 gcc-3.4' should be '2.6.17 SMP mod_unload PENTIUM4 gcc-3.2'
</pre>
Alternatively, the following should find the "signature" in existing kernel modules:
<pre>
$ find /lib/modules/`uname -r` -name '*.ko' -print | head -1 | xargs strings | grep gcc
vermagic=2.6.17 SMP mod_unload PENTIUM4 gcc-3.2
</pre>
In this case a gcc-3.2.X version is required.
<p>Regardless of which method is used to find the correct version, you will need to
reconfigure BLCR to use the correct compiler. To do so, rerun <tt>configure</tt> with the
addition of "KCC=/path/to/the/correct/gcc" to the command line to set the compiler used
for building BLCR's kernel modules.
</blockquote>
<hr>
<h3>Usage Questions</h3>
<h4><a name="samepid">Why do I get this error: "Restart failed:
Device or resource busy"?</a></h4>
<blockquote>
This is because a resource needed into order to restart the process is already
in use. The most common problem is that another process already exists with the
same pid (process ID)--the operating system will not allow you to create two
programs with the same pid. Very frequently this is because a user is trying
to 'restart' a process from a checkpoint, when the original process they took
the checkpoint of is still running!
<p>If you are unlucky enough that some other, unrelated process has grabbed the
PID of your application, you must figure out some way to get rid of that
process. If you own the process, you can of course simply kill it (or
checkpoint it!). Otherwise, consider becoming root, or consulting your system
administrator. BLCR will not kill another process for you (this 'feature' would
raise certain security issues).
</blockquote>
<h4><a name="prelink">Why can I restart jobs on the original machine they ran
on, but not a different node in my cluster?</a></h4>
<blockquote>
You should be able to restart a BLCR-checkpointed job on a different node
(or set of nodes, for a parallel job), provided that all the nodes involved
provide the exact same libraries and other files that your executable needs.
By default BLCR does <em>not</em> save the contents of shared libraries that your program
uses, nor does it save the contents of any files your program has
<tt>open()</tt>'ed. But so long as all of these libraries and/or files
exist on another node, your program should restart fine.
<p>Note that libraries must be <em>exactly</em> the same for a restart to
work; if they are not the same size, for instance, restart will not work.
If you've installed the same version of the operating system to all of your
nodes (and you've updated them all the same way), you would think things would be fine.
However, some Linux distributions are using "prelinking", which is a
method for assigning fixed addresses for shared libraries to load into
executables. Prelinking is a feature which enables applications that use many
shared libraries to load faster. The fixed address used by the same library
on different nodes is often deliberately randomized (in order to defeat
buffer overflow attacks that could otherwise rely on standard libraries
being loaded at the same address on every machine with the same OS version).
Alas, if the prelinked addresses are different, you will not be able to
restart BLCR checkpoints on another node.
<p> The solution for this problem is to disable prelinking on both the
source and destination nodes of any process migration before starting any
process you may wish to migrate. For most cluster environments, that means
disabling it on all nodes before using BLCR for migration.
Prelinking is a systemwide setting, so you will need to be root.
On Fedora Core 2, at least, the fix is to edit
<tt>/etc/sysconfig/prelink</tt> and set '<tt>PRELINKING=no</tt>'. The
comments claim that this will cause prelinking to be undone automatically
the next night. We've never been patient, and instead "undo" prelinking
immediately by running (as root)
<pre> # /usr/sbin/prelink --undo --all</pre>
Automating this process for an entire cluster depends on your specific environment.
<p>BLCR 0.7.0 introduced the <tt>--save-*</tt> family
of options to <tt>cr_checkpoint</tt> to cause the executable and/or shared
libraries to be included in the context file. This may significantly increase
the size of the context files. Therefore we recommend this approach only
if you cannot ensure uniform library versions (w/o prelinking) across the
machines you wish to migrate among.
<p>Some versions of <tt>glibc</tt> use <tt>mmap()</tt> to load locale data.
CentOS 5 and Scientific Linux 5, for example, store the locale information
in the file <tt>/usr/lib/locale/locale-archive</tt>. This file is
generated when <tt>glibc</tt> (specifically, the <tt>glibc-common</tt> RPM)
is installed, and contains archived versions of all of the localization
databases. Since the <tt>locale-archive</tt> file is not included in
the RPM, but is generated at install time, the contents of this file
can differ between your compute nodes. This causes problems migrating
programs that use <tt>glibc</tt>'s locale support, <tt>bash</tt> being
the most notable. If you see problems migrating <tt>bash</tt> scripts
between nodes, you might have this problem.
<p>You can fix this problem in one of three ways. First, you can use the
<tt>--save-all</tt> option to <tt>cr_checkpoint</tt> to ensure that the
correct localization data is loaded at restart time. Second, you can
disable localization support completely by using <tt>export LANG=C</tt>
in your shell environment. Finally, we've been successful in copying
one version of the <tt>/usr/lib/locale/locale-archive</tt> across all of
your compute nodes. You'll need to update this file every time you
update your <tt>glibc-common</tt> RPM.
</blockquote>
<h4><a name="enoent">Why do I get the error "Restart failed: No such file or directory"?</a></h4>
<blockquote>
This error normally means that a file that was open at the time the checkpoint
was taken is no longer present (it is either completely gone, or perhaps just
not present at the same pathname as it was previously). You should examine your
system logs (such as <tt>/var/log/messages</tt> or dmesg) for an indication of the
file that caused the problem. You will probably find a message like one of the following:
<pre> vmadump: mmap failed: /tmp/hsperfdata_[user]/[pid]<br> Failed to open file '/tmp/foobar'<br></pre>
In the case of files in a directory of the form <tt>/tmp/hsperfdata_[user]</tt> see
<a href="#hsperfdata">the hsperfdata FAQ entry</a>. For other files, there are a
a couple of things you might try.
<p> If the file is a temporary created by your application, it is
possible that it has been removed when the application terminated. For instance, if
you checkpoint an application with the <tt>--term</tt> option to the <tt>cr_checkpoint</tt>
utility, then <tt>SIGTERM</tt> was sent to the application, causing it to cleanup before
terminating. If this is the case, then passing <tt>--kill</tt> will cause the uncatchable
signal <tt>SIGKILL</tt> to be sent, thus preventing any cleanups by the application. Of
course, if your application ran to completion and removed its temporary file at its
normal conclusion, then you are on your own as to how to recover the file.
<p> If you are trying to perform migration of a process from one machine to another, then
it is possible that the file exists, but not at the full pathname that was saved by BLCR.
This is especially true if network filesystem mountpoints differ between machines. You
may be able to work around such issues with symbolic links
among directories, but BLCR provides a <tt>--relocate</tt> option to <tt>cr_restart</tt>
that can easily deal with such situations.
<p>Future versions of BLCR will make it possible to capture the contents of all open
files, thus providing a mechanism to eliminate problems of this sort (at the cost of
increased context file size, of course).
</blockquote>
<h4><a name="eperm">Why do I get the error "Restart failed: permission denied"?</a></h4>
<blockquote>
There are multiple reasons why a restart would fail with this message, but the most
common is filesystem permissions. You should examine your system logs (such as
<tt>/var/log/messages</tt> or dmesg) for an indication of the file that caused the
problem. You will probably find a message like one of the following:
<pre> vmadump: mmap failed: /var/db/nscd/hosts<br> Failed to open file '/tmp/foobar'<br></pre>
You should verify that our user has permission to access the file. In the case
of files in the directory <tt>/var/db/nscd</tt> see <a href="#nscd">the NSCD FAQ entry</a>.
</blockquote>
<h4><a name="preload">
Why do I get the error "ERROR: ld.so: object 'libcr_run.so.0' from LD_PRELOAD cannot be preloaded: ignored"?
</a></h4>
<blockquote>
This error is almost always a symptom that BLCR's shared libraries
are not located for one of two reasons:
<ul>
<li> The libraries have been installed in a "standard" directory, but
the utility <tt>ldconfig</tt> has not been run as root to update the
loader cache.
<li> The libraries have been installed in a "non-standard" directory
and the directory has not been added to the LD_LIBRARY_PATH environment
variable.
</ul>
Please see, respectively, the sections "Updating ld.so.cache" and
"Configuring Users' environments" in the BLCR Admin Guide for
information on resolving these installation issues.
</blockquote>
<h4><a name="nscd">Why do I get "vmadump: mmap failed: /var/db/nscd/[something]" in the system logs?</a></h4>
<blockquote>
The files in the directory <tt>/var/db/nscd/</tt> are created by the Name Service
Cache Daemon (NSCD, for short) and are protected against normal file accesses.
When the NSCD is enabled, certain C library operations talk to the daemon which uses
filedescriptor passing to allow the application to <tt>mmap()</tt> these files. Unfortunately, BLCR
cannot replay the filedescriptor passing (there is no way to know that a given filedescriptor
was obtained in this way). This leaves BLCR to rely on normal filesystem permissions when
trying to reestablish the <tt>mmap()</tt>, which in this case finds insufficient permissions.
<p>
C library functions that may use NSCD include host database lookups (such as
<tt>gethostbyname()</tt>), user database lookups (such as <tt>getpwuid()</tt>) and
other database lookup functions. Since use of NSCD may speed these lookups relative
to network-based lookups such as NIS, LDAP or DNS, this is generally a good thing.
<p>
If you are experiencing the "mmap failed: /var/db/nscd/..." error, you have at
least the following three options:
<ul>
<li> You may choose to disable NSCD or remove it from you system entirely. If you
choose this option then consult the documentation for your specific Linux distribution
for instructions on disabling services or removing packages.</li>
<li> You may choose to keep NSCD enabled, but disable the <tt>mmap()</tt> mechanism.
This will still allow NSCD to cache lookups, but the interaction between the
application and NSCD will be slower. If the database lookups are primarily
resolved by network services such as NIS, LDAP and DNS then this is still a net
win when compared to disabling or removing NSCD entirely.<br>
To disable the <tt>mmap()</tt> mechanism add the following lines (or modify
similar ones) in the file <tt>/etc/nscd.conf</tt>:
<pre>
shared passwd no
shared group no
shared hosts no
</pre>
You will need to restart NSCD for the change to take effect (consult the
documentation for your specific Linux distribution for instructions on
restarting services).</li>
<li> Since 0.8.3, BLCR has the option to disable NSCD per-process when <tt>libcr</tt> is
initialized. This option has the benefit of avoiding this error in many cases, while
allowing the full benefit of NSCD for applications that run without <tt>libcr</tt>.
To enable this behavior set the environment variable <tt>LIBCR_DISABLE_NSCD</tt> to any
non-empty value.
Unlike the previous two options, this approach does not require one to have root permission.
However there is one weakness: if NSCD is used by a process prior to <tt>libcr</tt>'s
initialization, then one could still experience this problem. If this happens to
you then you will need to apply one of the other two options.</li>
</ul>
Our thanks to Guy Coates <tt><gmpc AT sanger DOT ac DOT uk></tt>
for information on the <tt>/etc/nscd.conf</tt> settings.<br>
Our thanks to Hongjia Cao <tt><hjcao AT nudt DOT edu DOT cn></tt> for information leading to the
implementation of <tt>LIBCR_DISABLE_NSCD</tt>.
</blockquote>
<h4><a name="hsperfdata">Why do I get "vmadump: mmap failed: /tmp/hsperfdata_[user]/[pid]" in the system logs?</a></h4>
<blockquote>
Directories of the form <tt>/tmp/hsperfdata_[user]/</tt> are created by Sun's Java
runtime environment (JRE), and the individual files in this directory are removed
when the application exits. This includes exits due to catchable fatal signals, so if a
checkpointed Java application is terminated by such a signal these files will be
unavailable for a subsequent restart. You have several options to deal with this issue:
<ul>
<li> If you are trying to perform migration of a process from one machine to another, then
you may try manually copying the file from one machine to another. While this is not a
safe practice in general, we have had reports of success from users who have tried this.</li>
<li> If you experiencing this problem when running Matlab but don't need the Java features,
you can avoid this issue entirely by passing <tt>--nojre</tt> when launching Matlab.</li>
<li> You may disable creation of hsperfdata files by passing <tt>-XX:-UsePerfData</tt>
when starting the Java application. Note, however, that certain performance monitoring
or debugging tools may require the hsperfdata files.</li>
<li> If the signal causing termination of the Java application is due to the <tt>--term</tt>
option to the <tt>cr_checkpoint</tt> utility, passing <tt>--kill</tt> instead will
cause the uncatchable signal <tt>SIGKILL</tt> to be sent, thus preventing any cleanups by
the JRE.</li>
<li> If the signal is due to the <tt>cr_signal</tt> value in the arguments to the
<tt>cr_request_checkpoint()</tt> function, then use of <tt>SIGKILL</tt> will prevent
cleanups, as above.</li>
</ul>
Our thanks to Guy Coates <tt><gmpc AT sanger DOT ac DOT uk></tt>
for information on the <tt>-XX:-UsePerfData</tt> JRE option.
</blockquote>
<h4><a name="sig31">Why does my application dies with "Real-time signal 31" (or 32, etc.) when I try to checkpoint it?</a></h4>
<blockquote>
This error was possible in older releases of BLCR when an application was not
checkpointable. This should not happen in release 0.5.0 or newer and should
be reported as a bug if seen.
</blockquote>
<h4><a name="staticlink">Why can't I checkpoint my statically linked application?</a></h4>
<blockquote>
If you can checkpoint and restart a dynamically linked application correctly, but
cannot do so with the same application linked statically, this FAQ entry is for you.
There are multiple reasons why BLCR may have problems with statically executables.
<ul>
<li>The <tt>cr_run</tt> utility only supports dynamic executables<br>
If you wish to checkpoint an unmodified executable, the typical recipe is
<pre>
$ cr_run my_app my_args
</pre>
However, the <tt>cr_run</tt> utility does its work using the "LD_PRELOAD"
environment variable to force loading of BLCR's support code into the
address space the applications. That mechanism is only functional for
dynamically linked executables. There is no magic we can perform today
that will resolve this (though in the future we'd like to replace our
use of LD_PRELOAD with a kernel-side mechanism). So, you'll need to
relink any statically linked executables to include BLCR support.
</li>
<li>Linking BLCR's libraries statically takes special care<br>
OK, we've told you why <tt>cr_run</tt> doesn't work and told you to
relink. You tried linking with <tt>-lcr_run</tt> and/or <tt>-lcr</tt>
and still can't get a checkpoint to work. What went wrong?<p>
Please reread the "Cautionary linker notes" section of the BLCR Users
Guide. You need a <tt>-u</tt> option in addition the the <tt>-l</tt>
or the static linking will simply ignore BLCR's library.
</li>
<li>BLCR doesn't support LinuxThreads<br>
Ok, what else could go wrong? You've followed the guidance given
in the "Cautionary linker notes" section of the BLCR Users Guide
when you linked your application. You even ran
<pre>
$ nm my_app | grep link_me
</pre>
to be sure the symbol you specified with <tt>-u</tt> is linked in.
However, you are seeing weird crashes of your application when you
try to checkpoint.<p>
The culprit might be LinuxThreads. Why? Because at the time this
FAQ entry is being written, there are many Linux distributions
that install the static libs for LinuxThreads in the default library
search path, and with the NPTL static libs elsewhere. The resolution
could be as simple as linking your application with <tt>-L/usr/lib/nptl</tt>
or <tt>-L/usr/lib64/nptl</tt>, perhaps by setting an "LDFLAGS" variable
(though it is possible that your distribution has picked some other location).<p>
While it is not strictly required due to binary compatibility between
LinuxThreads and NPTL, we'd recommend that you at least consider a
recompile with <tt>-I/usr/include/nptl</tt> in CFLAGS.<p>
Note, of course, that if BLCR's utilities are statically linked to
LinuxThreads, then they need to be rebuilt too. See the BLCR
Admin Guide for more information on that.
</li>
</ul>
</blockquote>
<h4><a name="auditing">Can I use Linux kernel auditing support with BLCR?</a></h4>
<blockquote>
We recommend that you avoid auditing when using BLCR. In particular, we've
seen odd restart failures (on return to user space) when migrating processes
between nodes where auditing is enabled (with <tt>/sbin/auditctl -e1</tt>) to nodes
where auditing is disabled. The migrating processes themselves were not being
audited, neither at checkpoint time nor during restart. In some cases, but not
all, these restart failures produce warning messages about leaked audit
contexts.
<p>More importantly, BLCR does not generate audit records
during checkpoint or restart. As far as the auditing code is concerned, all of
the BLCR kernel calls are described by <tt>ioctl()</tt> records. You don't see audit
records describing the creation of the context file, the <tt>mmap()</tt> established
during restart, or the file descriptors that are restored. Please contact the
mailing list if you need auditing support <em>during</em> checkpoint/restart calls.
<p>Prior to version 0.8.4, the <tt>linked_fifo</tt> test caused a kernel BUG when audited
during restart. If you encounter any other problems, please report a bug
to the mailing list.
</blockquote>
<hr>
<h3>Additional Resources</h3>
<h4><a name="website">Where can I download BLCR?<br>
Where can I find more information?</a></h4>
<blockquote>
To download the BLCR software, or for links to all the available information
about BLCR, please
visit <a href="http://ftg.lbl.gov/checkpoint">http://ftg.lbl.gov/checkpoint</a>.
</blockquote>
<h4><a name="mailaddr">Is there a mailing list for BLCR?</a></h4>
<blockquote>
There is a mailing list of BLCR developers and some of the users
at <a href="mailto:checkpoint@lbl.gov">checkpoint@lbl.gov</a> and
which is archived:
<ul>
<li><a href="http://hpcrdm.lbl.gov/pipermail/checkpoint/">Current messages</a></li>
<li><a href="http://mantis.lbl.gov/hypermail/checkpoint/">Messages prior to May 1, 2010</a></li>
</ul>
<p> This list is managed by Mailman. So, to subscribe (or unsubscribe)
visit <a href='https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/checkpoint'>
https://hpcrdm.lbl.gov/cgi-bin/mailman/listinfo/checkpoint</a>.
</blockquote>
<h4><a name="bugzilla">Where can I report BLCR bugs?</a></h4>
<blockquote>
If you think you've found a bug in BLCR, please <em>do</em> let us know about it.
There are many kernel-dependencies in BLCR and we could easily have missed testing on
a system like yours. We count on user's bug reports to help ensure wide testing
coverage.
<p> The BLCR bug database is managed by a Bugzilla, located
at <a href="http://mantis.lbl.gov/bugzilla">http://mantis.lbl.gov/bugzilla</a>.
<p> Before reporting a bug, you are encouraged to search the database to see if
a bug report exists for your problem. For some issues a solution can be
found in just a day. So, a patch to fix your problem may already be attached
to an existing bug report. BLCR is just one of a group of projects managed on
this server, so be sure to select product "BLCR" in your queries.
</blockquote>
</body>
</html>
|