/usr/share/doc/slurm-wlm-doc/html/bluegene.html

<!DOCTYPE html>
<html lang="en-US">
<head>
    <meta charset="UTF-8" />
	<meta name="viewport" content="width=device-width" />

	<title>Slurm Workload Manager</title>

	<link rel="shortcut icon" href="favicon.ico" />

	<link rel="stylesheet" type="text/css" href="fonts.css" />
	<link rel="stylesheet" type="text/css" href="reset.css" />
	<link rel="stylesheet" type="text/css" href="style.css" />
	<link rel="stylesheet" type="text/css" href="slurm.css" />

	<script src="jquery.min.js"></script>
</head>

<body>

<div class="container container--main">

	<header class="site-header" role="banner">

		<div class="site-masthead">
			<h1 class="site-masthead__title site-masthead__title--slurm">
				<a href="/" rel="home">
					<span class="slurm-logo">Slurm Workload Manager</span>
				</a>
			</h1>
			<div class="site-masthead__title">
				<a href="https://www.schedmd.com/" rel="home">
					<span class="site-logo">SchedMD</span>
				</a>
			</div>

			<button class="site-masthead__trigger menu-trigger" type="button" role="button" aria-label="Toggle Navigation"><span class="menu-trigger__lines"></span></button>
		</div>


		<nav class="site-nav">
			<h2 class="site-nav__title">Navigation</h2>

			<div class="slurm-title">
				<div class="slurm-logo"><a href="/">Slurm Workload Manager</a></div>
				<div class="slurm-title__version">Version 17.11</div>
			</div>

			<ul class="site-nav__menu site-menu menu" role="navigation">
				<li class="site-menu__item">
				        <div>About</div>
					<ul>
						<li><a href="overview.html">Overview</a></li>
						<li><a href="news.html">What's New</a></li>
						<li><a href="team.html">Slurm Team</a></li>
						<li><a href="meetings.html">Meetings</a></li>
						<li><a href="testimonials.html">Testimonials</a></li>
						<li><a href="disclaimer.html">Legal Notices</a></li>
					</ul>
				</li>
				<li class="site-menu__item">
					<div>Using</div>
					<ul>
						<li><a href="tutorials.html">Tutorials</a></li>
						<li><a href="documentation.html">Documentation</a></li>
						<li><a href="faq.html">FAQ</a></li>
						<li><a href="publications.html">Publications</a></li>
					</ul>
				</li>
				<li class="site-menu__item">
					<div>Installing</div>
					<ul>
						<li><a href="download.html">Download</a></li>
						<li><a href="quickstart_admin.html">Installation Guide</a></li>
					</ul>
				</li>
				<li class="site-menu__item">
					<div>Getting Help</div>
					<ul>
						<li><a href="https://www.schedmd.com/services.php">Support</a></li>
						<li><a href="mail.html">Mailing Lists</a></li>
						<li><a href="https://www.schedmd.com/services.php">Training</a></li>
						<li><a href="troubleshoot.html">Troubleshooting</a></li>
					</ul>
				</li>
			</ul>

		</nav>

	</header>

	<div class="content" role="main">
		<section class="slurm-search">
			<div class="container" id="cse">
				<gcse:search></gcse:search>
			</div>
		</section>

		<div class="section">
			<div class="container">


<h1>BlueGene/Q User and Administrator Guide</h1>

<p><b>Support for BlueGene/Q systems is deprecated as of 17.11, and will be
removed in a future release.</b></p>

<p><b>Beginning with the 17.02 release only BlueGene/Q systems are supported
by Slurm. Support for the BlueGene/L and BlueGene/P systems has been removed.
</b></p>

<h2>Overview</h2>

<p>This document describes the unique features of Slurm on the
<a href="http://www.research.ibm.com/bluegene/">IBM BlueGene/Q</a> systems.
You should be familiar with Slurm's mode of operation on Linux clusters
before studying the relatively few differences in BlueGene operation
described in this document.</p>

<p>BlueGene systems have several unique features making for a few
differences in how Slurm operates there.
BlueGene systems consists of one or more <i>base partitions</i> or
<i>midplanes</i> connected in a four-dimensional - AXYZ - torus.
Each <i>midplane</i> typically includes 512 <i>c-nodes</i> or compute
nodes each containing two or more cores;
one core is typically designed primarily for managing communications while the
other cores are used primarily for computations.
Each <i>c-node</i> can execute only one process and thus are unable to execute
both the user's application plu and Slurm's <i>slurmd</i> daemon.
Thus the <i>slurmd</i> daemon(s) executes on one or more of the BlueGene
<i>Front End Nodes</i>.
The <i>slurmd</i> daemons provide (almost) all of the normal Slurm services
for every <i>midplane</i> on the system. </p>

<p>Internally Slurm treats each <i>midplane</i> as one node with
a processor count equal to the number of cores on the midplane, which
keeps the number of entities being managed by Slurm more
reasonable.</p>

<p>All BlueGene systems can sub-allocate a <i>midplane</i>
into smaller blocks, this allows more than one user job to execute on
each <i>midplane</i>.</p>
<p>In the case of BlueGene/Q systems, more than one user job can also
execute in each block (see AllowSubBlockAllocation option in 'man
bluegene.conf').</p>

<p>To effectively utilize this environment, Slurm tools present the user with
the view that each <i>c-node</i> is a separate node, so allocation requests
and status information use <i>c-node</i> counts.
Since the <i>c-node</i> count can be very large, the suffix "k" can be used
to represent multiples of 1024 or "m" for multiples of 1,048,576 (1024 x 1024).
For example, "2k" is equivalent to "2048".</p>

<p>If you are running a system that is smaller than 1 midplane (a
nodecard/nodeboard or such) you can configure your system up like
this in the bluegene.conf file.  Below is an example for a BlueGene/Q system:</p>
<pre>
# Excerpt from bluegene.conf file for BlueGene/Q system
...
BasePartitionNodeCnt=512
NodeCardNodeCnt=32
SubMidplaneSystem=YES
LayoutMode=STATIC
MPs=0000 type=small 32cnblocks=16
...
</pre>
<p>This will create a small block on each nodeboard on the system.  If your
system is different than this, adjust appropriately.  The idea is Slurm
will create the smallest block possible on every possible hardware
location.  The system will then check for missing hardware and remove
blocks that are invalid.  This will get around the problem if you
have, for instance, the 4th nodeboard populated instead of the 1st.
</p>

<h2>User Tools</h2>

<p>The normal set of Slurm user tools: <i>sbatch</i>, <i>scancel</i>,
<i>sinfo</i>, <i>squeue</i>, and <i>scontrol</i> provide all of the expected
services except support for job steps, which is detailed later.
The sstat command is not supported on BlueGene systems.</p>

<p>Four job submission options are available exclusively on BlueGene/Q systems:</p>
<table>
<tr VALIGN=TOP><td><i>--geometry</i></td><td>Specify job size in each dimension,
    (i.e. 1x4x4 = 16 nodes)</td></tr>
<tr VALIGN=TOP><td><i>--no-rotate</i></td><td>Disable rotation of geometry, by default
    1x4x4 could be rotated to be 4x1x4)</td>
<tr VALIGN=TOP><td><i>--conn-type</i></td><td>Specify interconnect
    type between midplanes, mesh or torus. On BlueGene/Q systems you can
    specify a different conn-type for each dimension, TTMT would
    give you Torus in all dimensions except the Y dimension, where
    it would be Mesh.</td></tr>
<tr VALIGN=TOP><td><i>--mloader-image</i></td><td>Specify
    alternative mloader image for bluegene block. Default if not set.</td></tr>
</table>

<p>The <i>--nodes</i> option with a minimum and (optionally) maximum node count
continues to be available.
Note that this is a c-node count.</p>

<h3>Task Launch on BlueGene/Q only</h3>

<p>Use Slurm's <i>srun</i> command to launch tasks (<i>srun</i> uses
an api interface into IBM's <i>runjob</i> command).
Slurm job step information, including accounting, functions as
expected.  Totalview and other debuggers will also work with srun.
If Slurm is installed and configured correctly IBM's runjob will not
work.</p>
<p>The <i>srun --launcher-opts</i> option is designed to convey
options to <i>runjob</i> that are not available using <i>srun</i>
alone.  Node selection options are conveyed automatically
by <i>srun</i> and cannot be overridden using this option.  Two
leading dashes are required when listing the <i>runjob</i>
options, e.g., <i>srun --launcher-opts='--mapping TEDCBA'</i>.  See
the <i>runjob</i> man page for the list of available options.</p>

<h3><a name="naming">Naming Conventions</a></h3>
<p>The naming of midplanes includes a numeric suffix representing the its
coordinates with a zero origin. The suffix contains three digits on BlueGene/L
and BlueGene/P systems, while four digits are required for the BlueGene/Q
systems. For example, "bgp012" represents the midplane whose coordinate
is at X=0, Y=1 and Z=2.
Slurm uses an abbreviated format for describing midplanes in which the
end-points of the block enclosed are in square-brackets and separated by an "x".
For example, "bgp[620x731]" is used to represent the eight midplanes
enclosed in a block with end-points and bgp620 and bgp731 (bgp620, bgp621,
bgp630, bgp631, bgp720, bgp721, bgp730 and bgp731).</p>

<p><b>IMPORTANT:</b> Slurm can support up to 36 elements in each
BlueGene dimension by supporting "A-Z" as valid numbers. Slurm requires the
prefix to be lower case and any letters in the suffix must always be upper
case. This schema must be used in both the <i>slurm.conf</i> and bluegene.conf
configuration files when specifying midplane/node names (the prefix is
optional). This schema should also be used to specify midplanes or locations
in configure mode of <i>smap</i>:
<br>
valid: bgl[000xC44], bgl000, bglZZZ
<br>
invalid: BGL[000xC44], BglC00, bglb00, Bglzzz
</p>

<p><b>IMPORTANT:</b> Slurm requires that all systems start with 0 in each
dimension.  So if you have a BlueGene/Q system and only want Slurm to
run on a portion of it you need to define the entire system and mark
midplanes down in the slurm.conf file or with scontrol/sview.
<br>
In example of this with a BGQ system of [0000x2333] but only can
use [2000x2333] you could define it in your slurm.conf like this.
<pre>
...
NodeName=bgq[0000x1333] state=down
NodeName=bgq[2000x2333] state=unknown
...
</pre>
This would mark your nodes not managed as down and only create blocks
on the portion of the machine you want to use.
</p>

<p>In a system configured with <i>small blocks</i> (any block less
than a full midplane), there will be divisions in the midplane
notation. On BlueGene/L and BlueGene/P systems, the midplane name may
be followed by a square bracket enclosing ID numbers of the IO nodes associated
with the block. For example, if there are 64 psets in a BlueGene/L
configuration, "bgl012[0-15]" represents the first quarter or first 16 IO nodes
of a midplane.  In BlueGene/L this would be 128 c-node block.  To represent
the first nodecard in the second quarter or IO nodes 16-19, the notation would
be "bgl012[16-19]", or a 32 c-node block. On BlueGene/Q systems, the specific
c-nodes would be identified in square brackets using their five digit
coordinates. For example "bgq0123[00000x11111]" would represent the 32 c-nodes
in midplane "bgq0123" having coordinates (within that midplane) from zero to
one in each of the five dimensions.</p>

<p>Two topology-aware graphical user interfaces are provided: <i>smap</i> and
<i>sview</i> (<i>sview</i> provides more viewing and configuring options).
See each command's man page for details.
A sample of <i>smap</i> output is provided below showing the location of five jobs.
Note the format of the list of midplanes allocated to each job.
Also note that idle (unassigned) midplanes are indicated by a period.
Down and drained midplanes (those not available for use) are
indicated by a number sign (bg703 in the display below).
The legend is for illustrative purposes only.
The origin (zero in every dimension) is shown at the rear left corner of the bottom plane.
Each set of four consecutive lines represents a plane in the Y dimension.
Values in the X dimension increase to the right.
Values in the Z dimension increase down and toward the left.</p>

<pre>
   a a a a b b d d    ID JOBID PARTITION BG_BLOCK USER   NAME ST TIME NODES BP_LIST
  a a a a b b d d     a  12345 batch     RMP0     joseph tst1 R  43:12  32k bg[000x333]
 a a a a b b c c      b  12346 debug     RMP1     chris  sim3 R  12:34   8k bg[420x533]
a a a a b b c c       c  12350 debug     RMP2     danny  job3 R   0:12   4k bg[622x733]
		      d  12356 debug     RMP3     dan    colu R  18:05   8k bg[600x731]
   a a a a b b d d    e  12378 debug     RMP4     joseph asx4 R   0:34   2k bg[612x713]
  a a a a b b d d
 a a a a b b c c
a a a a b b c c

   a a a a . . d d
  a a a a . . d d
 a a a a . . e e              Y
a a a a . . e e               |
			      |
   a a a a . . d d            0----X
  a a a a . . d d            /
 a a a a . . . .            /
a a a a . . . #            Z
</pre>

<p>If the block is in a READY state, the job will begin execution almost
immediately.
Otherwise the execution of the job will not actually begin until the
block is in a READY state, which can require booting the block and
a delay of minutes to do so. During this time a job will be in the
CONFIGURING state.
You can identify the block associated with your job using the command
<i>smap -Dj -c</i> and the state of the block with the command
<i>smap -Db -c</i>.
The time to boot a block is related to its size, but should range from
from a few minutes to about 15 minutes for a block containing 128
midplanes (on a BlueGene/L system).
Only after the block is READY will your job's output file be created
and the script execution begin.
If the block boot fails, Slurm will attempt to reboot several times (3)
before draining the associated midplanes and aborting/requeueing the job.</p>

<p>The job will continue to be in a RUNNING state until the bgjob has
completed and the block ownership is changed.
The time for completing a bgjob has frequently been on the order of
five minutes.
In summary, your job may appear in Slurm as RUNNING for 15 minutes
before the script actually begins to 5 minutes after it completes.
These delays are the result of the BlueGene infrastructure issues and are
not due to anything in Slurm.  These times have improved considerably on the
more recent BlueGene/P and BlueGene/Q systems.</p>

<p>When using <i>smap</i> in default output mode you can scroll through
the different windows using the arrow keys.
The <b>up</b> and <b>down</b> arrow keys scroll
the window containing the grid, and the <b>left</b> and <b>right</b> arrow
keys scroll the window containing the text information.</p>

<p class="footer"><a href="#top">top</a></p>

<h2>System Administration for BlueGene/Q only</h2>

<p><b>IMPORTANT:</b> The SlurmUser defined in the slurm.conf must be
added to the bgadmin group.  This allows the slurmctld to access
information from the system and manipulate blocks.
</p>

<p>In order to make <i>srun</i> operate correctly with the underlying system
and to ensure security for new MPI jobs, it is necessary to enable the
Slurm plugin for the IBM runjob_mux.  This is
done by altering the bg.properties file. In the [runjob.mux]
section of the bg.properties file change the plugin option to
<i>$prefix/lib/slurm/runjob_plugin.so</i> and also set the plugin_flags
option to <i>0x0109</i> (RTLD_LAZY | RTLD_GLOBAL | RTLD_DEEPBIND)
which allows the forwarding of symbols to shared objects like Slurm
uses for plugins.</p>
<pre>
[runjob.mux]
...
plugin = /usr/lib64/slurm/runjob_plugin.so
    # Path to the plugin used for communicating with a
    # job scheduler. This value can be updated by the
    # runjob_mux_refresh_config command on the
    # Login Node where a runjob_mux process runs.
...
plugin_flags = 0x0109 # RTLD_LAZY | RTLD_GLOBAL | RTLD_DEEPBIND
</pre>

<p>You also need to set things up so the runjob_mux is ran by the
  SlurmUser.  This can be done by editing 2 files.</p>
<p>Back in your bg.properties file alter the [master.user] section.
<pre>
[master.user]
...
runjob_mux=<b>slurm</b>
</pre>
<p>Then in /etc/init.d/bgagent add SlurmUser to the --users line.</p>
<pre>
OPTIONS="--users bgqadmin,bgws,bgqsysdb,<b>slurm</b>"
</pre>
<p>After these settings are set flush the runjob_server and (re)start
  each runjob_mux running on your system.</p>
<pre>
> /bgsys/drivers/ppcfloor/sbin/master_stop binaries
stopped
> sudo /etc/init.d/bgagent restart
Shutting down bgagentd:                                    [  OK  ]
Starting bgagentd:
Startup of bgagentd completed:                             [  OK  ]
> /bgsys/drivers/ppcfloor/sbin/master_start binaries
> /bgsys/drivers/ppcfloor/sbin/bgmaster_server_refresh_config
success!
> /bgsys/drivers/ppcfloor/sbin/master_start runjob_mux
started runjob_mux
> ps aux | grep runjob_mux
<b>slurm</b>       25461  0.0  0.3 518528 48064 ?        Sl   13:00   0:00 runjob_mux
</pre>

<p>When a new version of Slurm is installed it is a wise idea to "refresh" the
runjob_mux with the new plugin.  This can be done in one of two ways.
<ul>
<li>Stopping and restarting the runjob_mux.  While this
  option works every time jobs running under the runjob_mux will not
  survive so plan your updates accordingly.
<pre>
> /bgsys/drivers/ppcfloor/sbin/master_stop runjob_mux
stopped runjob_mux
> /bgsys/drivers/ppcfloor/sbin/master_start runjob_mux
started runjob_mux
</pre>
</li>
<li><b>WARNING! You need at least IBM driver V1R1M1 efix 008 or this
method will not work.  Previous versions would load the old
plugin (presumably still in memory) other than the new one.
Slurm will print its version when the plugin is loaded for validation.</b><br>
This method allows for no job loss using the
IBM <i>runjob_mux_refresh_config</i> command.  This should
reload the plugin and all should be good afterwards.  After doing this
you may see some warning/error messages about the current running jobs
when finishing not being known.  This is expected and can usually be
ignored.</li>
</ul>

<b>Notes about sub-block allocations:</b>
<p>
There is a current limitation for sub-block jobs and how the system
(used for I/O) and user (used for MPI) torus class routes are configured. The
network device hardware has cutoff registers to prevent packets from flowing
outside of the sub-block. Unfortunately, when the sub-block has a size 3,
the job can attempt to send user packets outside of its sub-block. This causes
it to be terminated by signal 36.  To prevent this from happening Slurm does
not allow a sub-block to be used with any dimension of 3.
</p><p>
In the current IBM API it does not allow wrapping inside a midplane.
Meaning you can not create a sub-block of 2 with nodes in the 0 and 3 position.
Slurm will support this in the future when the underlying system allows it.
</p>

<p class="footer"><a href="#top">top</a></p>

<h2>System Administration for all BlueGene Systems</h2>

<p>The <i>slurmctld</i> daemon should execute on the system's service node.
If an optional backup daemon is used, it must be in some location where
it is capable of executing Bridge APIs.
The <i>slurmd</i> daemons executes the user scripts and there must be
at least one front end node configured for this purpose. Multiple
front end nodes may be configured for <i>slurmd</i> use to improve
performance and fault tolerance.  Each <i>slurmd</i> can execute
jobs for every midplane and the work will be distributed among
the <i>slurmd</i> daemons to balance the workload.  You can use
the <i>scontrol</i> command to drain individual compute nodes as
desired and return them to service.</p>

<p>The <i>slurm.conf</i> (configuration) file needs to have the value of
<i>InactiveLimit</i> set to zero or not specified (it defaults to a value of zero).
This is because we don't want to purge jobs prematurely if there are no job steps.
The value of <i>SelectType</i> must be set to "select/bluegene" (which
happens automatically) in order to have
node selection performed using a system aware of the system's topography
and interfaces.
The value of <i>TopologyPlugin</i> must be set to "topology/none" (which
happens automatically) since topology information is managed by the
select/bluegene plugin.
The value of <i>Prolog</i> should be set to the full pathname of a program that
will delay execution until the job's block is ready for
use by the user running the job. It is recommended that you construct a script
that serves this function and calls the supplied program <i>sbin/slurm_prolog</i>.
The value of <i>Epilog</i> should be set to the full pathname of a program that
will wait until the job's block has relinquished the resources acquired
by the job and is no longer usable by this job. It is recommended that
you construct a script that serves this function and calls the
supplied program <i>sbin/slurm_epilog</i>.
The prolog and epilog programs are used to ensure proper synchronization
between the <i>slurmctld</i> daemon, the user job, and MMCS.
A multitude of other functions may also be placed into the prolog and
epilog as desired (e.g. enabling/disabling user logins, purging file systems,
etc.).  Sample prolog and epilog scripts follow. </p>

<pre>
#!/bin/bash
# Sample BlueGene Prolog script
#
# Wait for block to be ready for this job's use
/usr/sbin/slurm_prolog
</pre>

<pre>
#!/bin/bash
# Sample BlueGene Epilog script
#
# Cancel job to start the termination process for this job
# and release the block
/usr/bin/scancel $SLURM_JOB_ID
#
# Wait for block to be released from this job's use
/usr/sbin/slurm_epilog
</pre>

<p>Since jobs with different geometries or other characteristics might not
interfere with each other, scheduling is somewhat different on a BlueGene
system than typical clusters.</p>

<p>Starting in 2.4.3 SchedType=sched/backfill works in all modes and
for all job sizes.  Before this release there were issues backfilling
jobs smaller than a midplane.  It is encouraged to upgrade to at least
2.4.3 for better backfill behavior.</p>

<p>Slurm does support different partitions with an assortment of
different scheduling parameters.
For example, Slurm can have defined a partition for full system jobs that
is enabled to execute jobs only at certain times; while a default partition
could be configured to execute jobs at other times.
Jobs could still be queued in a partition that is configured in a DOWN
state and scheduled to execute when changed to an UP state.
Midplanes can also be moved between Slurm partitions either by changing
the <i>slurm.conf</i> file and restarting the <i>slurmctld</i> daemon or by using
the <i>scontrol</i> reconfig command. </p>

<p>Slurm node and partition descriptions should make use of the
<a href="#naming">naming</a> conventions described above. For example,
"NodeName=bg[000x733]"
is used in <i>slurm.conf</i> to define a BlueGene/L system with 128 midplanes
in an 8 by 4 by 4 matrix. The node name prefix of "bg" defined by
NodeName can be anything you want, but needs to be consistent
throughout the <i>slurm.conf</i> file. No computer is actually
expected to a hostname of "bg000" and no attempt will be made to route
message traffic to this address. Starting in version 2.4, Slurm can determine
how many Sockets, CoresPerSocket, and ThreadsPerCore are available on each
midplane, so no configuration is needed to determine how many cores
are on each midplane.</p>

<p>Front end nodes used for executing the <i>slurmd</i> daemons must also be defined
in the <i>slurm.conf</i> file.
It is recommended that at least two front end nodes be dedicated to use by
the <i>slurmd</i> daemons for fault tolerance.
For example:
"FrontendName=frontend[00-03] State=UNKNOWN"
is used to define four front end nodes for running <i>slurmd</i> daemons.</p>

<pre>
# Portion of slurm.conf for BlueGene system
InactiveLimit=0
SelectType=select/bluegene
Prolog=/usr/sbin/prolog
Epilog=/usr/sbin/epilog
#
FrontendName=frontend[00-01] State=UNKNOWN
NodeName=bg[000x733] State=UNKNOWN
</pre>

<p>It is best to minimize other work on the front end nodes executing <i>slurmd</i>
so as to maximize its performance and minimize other risk factors.</p>

<a name="bluegene-conf"><h2>bluegene.conf File Creation</h2></a>
<p>In addition to the normal <i>slurm.conf</i> file, a new
<i>bluegene.conf</i> configuration file is required with information pertinent
to the system.
Put <i>bluegene.conf</i> into the Slurm configuration directory with
<i>slurm.conf</i>.
A sample file is installed in <i>bluegene.conf.example</i>.
If a System administrators chooses against dynamic partitioning for
some reason they should use the <i>smap</i> tool to build appropriate
configuration file for static/overlap partitioning.
Note that <i>smap -Dc</i> can be run without the Slurm daemons
active to establish the initial configuration.
Note when using static partitioning the blocks defined
using <i>smap</i> may not overlap (except for the full-system block,
which is implicitly created).
See the <i>smap</i> man page for more information.</p>

<p>There are 3 different modes which the system administrator can define
BlueGene partitions (or blocks) available to execute jobs: static,
overlap, and dynamic.
Jobs must then execute in one of the created blocks.
(<b>NOTE:</b> blocks are unrelated to Slurm partitions.)</p>

<p>The default mode of partitioning is <i>static</i>.
In this mode, the system administrator must explicitly define each
of the blocks in the <i>bluegene.conf</i> file.
Each of these blocks are explicitly configured with either a
mesh or torus interconnect.
They must also not overlap, except for the implicitly defined full-system
block.
Note that blocks are not rebooted between jobs in the mode
except when going to/from full-system jobs.
Eliminating block booting can significantly improve system
utilization (eliminating boot time) and reliability.</p>

<p>The second mode is <i>overlap</i> partitioning.
Overlap partitioning is very similar to static partitioning in that
each blocks must be explicitly defined in the <i>bluegene.conf</i>
file, but these partitions can overlap each other.
In this mode <b>it is highly recommended that none of the blocks
have any passthroughs in the X-dimension associated to them</b>.
Usually this is only an issue on larger BlueGene systems.
<b>It is advisable to use this mode with extreme caution.</b>
Make sure you know what you doing to assure the blocks will
boot without dependency on the state of any midplane
not included the block.</p>

<p>In the two previous modes you must ensure that the midplanes
defined in <i>bluegene.conf</i> are consistent with
those defined in <i>slurm.conf</i>.
Note the <i>bluegene.conf</i> file contains only the numeric
coordinates of midplanes while <i>slurm.conf</i> contains
the name prefix in addition to the numeric coordinates.</p>

<p>The final mode is <i>dynamic</i> partitioning.
While dynamic partitioning was developed primarily for smaller
BlueGene systems, it is commonly used on larger systems.
A warning about dynamic partitioning is it may introduce fragmentation
of resources. Dynamic partitioning is very capable,
easy to set up, and is the default for many systems including LLNL's
Sequoia.  With the advent of sub-block allocations (see
AllowSubBlockAllocation option in 'man bluegene.conf') fragmentation has
become less of a concern.
Blocks need not be assigned in the <i>bluegene.conf</i> file
for this mode.</p>

<p>Blocks can be freed or set in an error state using the <i>scontrol</i>,
command (i.e. "<i>scontrol update BlockName=RMP0 state=error</i>").
This will terminate any job on the block and set the state of the block to ERROR
making it so no job will run on the block.  To set it back to a usable
state, you can resume the block with the <i>scontrol</i> option state=resume
(i.e. "<i>scontrol update BlockName=RMP0 state=resume</i>").  This is useful
if you temporarily put the block in an error state and the block is
really booted and ready to start jobs.  You can also put the block
in free state using the state=free option.  Valid states are Error, Free,
Recreate, Remove, Resume.

<p>Alternatively, if only part of a midplane needs to be put
into an error state which isn't already in a block of the size you
need, you can set a collection of IO nodes into an error state using
<i>scontrol</i> (i.e. "<i>scontrol update submpname=bg000[0-3]
  state=error</i>").
<b>NOTE:</b> Even on BGQ where node names are given in bg0000[00000] format
this option takes an ionode name bg0000[0].

This will end any job on the nodes listed, create a block there, and set
the state of the block to ERROR making it so no job will run on the
block.  Then resume the block when it is ready to be used again (i.e.
"<i>scontrol update BlockName=RMP0 state=resume</i>"). This is
helpful to allow other jobs to run on the unaffected nodes in
the midplane.</p>

<p>One of these modes must be defined in the <i>bluegene.conf</i> file
with the option <i>LayoutMode=MODE</i> (where MODE=STATIC, DYNAMIC or OVERLAP).</p>

<p>The number of c-nodes in a midplane and in a node card must
be defined.
This is done using the keywords <i>MidplaneNodeCnt=NODE_COUNT</i>
and <i>NodeCardNodeCnt=NODE_COUNT</i> respectively in the <i>bluegene.conf</i>
file (i.e. <i>MidplaneNodeCnt=512</i> and <i>NodeCardNodeCnt=32</i>).</p>

<p>Note that the <i>IONodesPerMP</i> value defined in
<i>bluegene.conf</i> represents how many ionodes are on each midplane.
Slurm does not support heterogeneous ionode configurations so if your
environment is like this place the smallest number here.  For most BlueGene/L
systems this value is either 8 (for IO poor systems) or 64 (for IO rich
systems). For BlueGene/Q systems 4 to 16 are most common.</p>

<p>The <i>Images</i> file specifications identify which images are used when
booting a block and the valid images are different for each BlueGene system
type (e.g. L, P and Q). Their values can change during job allocation based on
input from the user.
If you change the block layout, then <i>slurmctld</i> and <i>slurmd</i> should
both be cold-started (without preserving any state information,
"/etc/init.d/slurm startclean").</p>

<p>If you wish to modify the <i>IONodesPerMP</i> value after blocks
have already been created, either modify the blocks manually or destroy them
and le. Slurm recreate them.
Note that in addition to the blocks defined in <i>bluegene.conf</i>, an
additional block is created containing all resources defined
all of the other defined blocks.
Make use of the Slurm partition mechanism to control access to these
blocks.
A sample <i>bluegene.conf</i> file is shown below.</p>
<pre>
###############################################################################
# Global specifications for a BlueGene/L system
#
# BlrtsImage:           BlrtsImage used for creation of all blocks.
# LinuxImage:           LinuxImage used for creation of all blocks.
# MloaderImage:         MloaderImage used for creation of all blocks.
# RamDiskImage:         RamDiskImage used for creation of all blocks.
#
# You may add extra images which a user can specify from the srun
# command line (see man srun).  When adding these images you may also add
# a Groups= at the end of the image path to specify which groups can
# use the image.
#
# AltBlrtsImage:           Alternative BlrtsImage(s).
# AltLinuxImage:           Alternative LinuxImage(s).
# AltMloaderImage:         Alternative MloaderImage(s).
# AltRamDiskImage:         Alternative RamDiskImage(s).
#
# LayoutMode:           Mode in which Slurm will create blocks:
#                       STATIC:  Use defined non-overlapping blocks
#                       OVERLAP: Use defined blocks, which may overlap
#                       DYNAMIC: Create blocks as needed for each job
# MidplaneNodeCnt: Number of c-nodes per midplane
# NodeCardNodeCnt:      Number of c-nodes per node card.
# IONodesPerMP:         Number of ionodes per midplane, needed to
#                       determine smallest creatable block..
#
# BridgeAPILogFile:  Pathname of file in which to write the
#                    Bridge API logs.
# BridgeAPIVerbose:  How verbose the BG Bridge API logs should be
#                    0: Log only error and warning messages
#                    1: Log level 0 and information messages
#                    2: Log level 1 and basic debug messages
#                    3: Log level 2 and more debug message
#                    4: Log all messages
# DenyPassthrough:   Prevents use of passthrough ports in specific
#                    dimensions, X, Y, and/or Z, plus ALL
#
# NOTE: The bgl_serial value is set at configuration time using the
#       "--with-bgl-serial=" option. Its default value is "BGL".
###############################################################################
# These are the default images with are used if the user doesn't specify
# which image they want
BlrtsImage=/bgl/BlueLight/ppcfloor/bglsys/bin/rts_hw.rts
LinuxImage=/bgl/BlueLight/ppcfloor/bglsys/bin/zImage.elf
MloaderImage=/bgl/BlueLight/ppcfloor/bglsys/bin/mmcs-mloader.rts
RamDiskImage=/bgl/BlueLight/ppcfloor/bglsys/bin/ramdisk.elf

#Only group jette can use these images
AltBlrtsImage=/bgl/BlueLight/ppcfloor/bglsys/bin/rts_hw2.rts Groups=jette
AltLinuxImage=/bgl/BlueLight/ppcfloor/bglsys/bin/zImage2.elf Groups=jette
AltMloaderImage=/bgl/BlueLight/ppcfloor/bglsys/bin/mmcs-mloader2.rts Groups=jette
AltRamDiskImage=/bgl/BlueLight/ppcfloor/bglsys/bin/ramdisk2.elf Groups=jette

# Since no groups are specified here any user can use them
AltBlrtsImage=/bgl/BlueLight/ppcfloor/bglsys/bin/rts_hw3.rts
AltLinuxImage=/bgl/BlueLight/ppcfloor/bglsys/bin/zImage3.elf
AltMloaderImage=/bgl/BlueLight/ppcfloor/bglsys/bin/mmcs-mloader3.rts
AltRamDiskImage=/bgl/BlueLight/ppcfloor/bglsys/bin/ramdisk3.elf

# Another option for images would be a "You can use anything you like image" *
# This allows the user to use any image entered with no security checking
AltBlrtsImage=* Groups=da,adamb
AltLinuxImage=* Groups=da,adamb
AltMloaderImage=* Groups=da,adamb
AltRamDiskImage=*  Groups=da,adamb

LayoutMode=STATIC
MidplaneNodeCnt=512
NodeCardNodeCnt=32
IONodesPerMP=64	# An I/O rich environment
BridgeAPILogFile=/var/log/slurm/bridgeapi.log
BridgeAPIVerbose=0

#DenyPassthrough=X,Y,Z

###############################################################################
# Define the static/overlap partitions (blocks)
#
# BPs: The midplanes (midplanes) in the block using XYZ coordinates
# Type:  Connection type "MESH" or "TORUS" or "SMALL", default is "TORUS"
#        Type SMALL will divide a midplane into multiple blocks
#        based off options NodeCards and Quarters to determine type of
#        small blocks.
#
# IMPORTANT NOTES:
# * Ordering is very important for laying out switch wires.  Please create
#   blocks with smap, and once done don't move the order of blocks
#   created.
# * A block is implicitly created containing all resources on the system
# * Blocks must not overlap (except for implicitly created block)
#   This will be the case when smap is used to create a configuration file
# * All midplanes defined here must also be defined in the slurm.conf file
# * Define only the numeric coordinates of the blocks here. The prefix
#   will be based upon the name defined in slurm.conf
###############################################################################
# LEAVE NEXT LINE AS A COMMENT, Full-system block, implicitly created
# BPs=[000x001] Type=TORUS       # 1x1x2 = 2 midplanes
###############################################################################
# volume = 1x1x1 = 1
BPs=[000x000] Type=TORUS                            # 1x1x1 =  1 midplane
BPs=[001x001] Type=SMALL 32CNBlocks=4 128CNBlocks=3 # 1x1x1 = 4-Nodecard sized
						    # c-node blocks 3-Base
						    # Partition Quarter sized
						    # c-node blocks
</pre>

<p>The above <i>bluegene.conf</i> file defines multiple blocks to be
created in a single midplane (see the "SMALL" option).
Using this mechanism, up to 32 independent jobs each consisting of
32 c-nodes can be executed
simultaneously on a one-rack BlueGene system.
If defining blocks of <i>Type=SMALL</i>, the Slurm partition
containing them as defined in <i>slurm.conf</i> must have the
parameter <i>OverSubscribe=force</i> to enable scheduling of multiple
jobs on what Slurm considers a single node.
Slurm partitions that do not contain blocks of <i>Type=SMALL</i>
may have the parameter <i>OverSubscribe=no</i> for a slight improvement in
scheduler performance.
As in all Slurm configuration files, parameters and values
are case insensitive.</p>

<p>The valid image names on a BlueGene/P system are <i>CnloadImage</i>,
<i>MloaderImage</i>, and <i>IoloadImage</i>. The only image name on BlueGene/Q
systems is <i>MloaderImage</i>. Alternate images may be specified as described
above for all BlueGene system types.</p>

<p>When <i>slurmctld</i> is initially started on an idle system, the blocks
already defined in MMCS are read using the Bridge APIs.
If these blocks do not correspond to those defined in the <i>bluegene.conf</i>
file, the old blocks with a prefix of "RMP" are destroyed and new ones
created.
When a job is scheduled, the appropriate block is identified,
its user set, and it is booted.
Subsequent jobs use this same block without rebooting by changing
the associated user field.
The only time blocks should be freed and rebooted, in normal operation,
is when going to or from full-system
jobs (two or more blocks sharing midplanes can not be in a
ready state at the same time).
When this logic became available at LLNL, approximately 85 percent of
block boots were eliminated and the overhead of job startup went
from about 24% to about 6% of total job time.
Note that blocks will remain in a ready (booted) state when
the Slurm daemons are stopped.
This permits Slurm daemon restarts without loss of running jobs
or rebooting of blocks.  </p>

<p>Be aware tha. Slurm will issue multiple block boot requests as
needed (e.g. when the boot fails).
If the block boot requests repeatedly fail (>3 times), Slurm will configure
the failing block to an ERROR state so as to avoid continuing
repeated reboots and the likely failure of user jobs.
A system administrator should address the problem before returning
the block to service with scontrol.</p>

<p>If the <i>slurmctld</i> daemon is cold-started (<i>/etc/init.d/slurm startclean</i>
or <i>slurmctld -c</i>) it is recommended that the <i>slurmd</i> daemon(s) be
cold-started at the same time.
Failure to do so may result in errors being reported by both <i>slurmd</i>
and <i>slurmctld</i> due to blocks that previously existed being deleted.</p>

<h4>Resource Reservations</h4>

<p>Slurm's advance reservation mechanism can accept a node count specification
as input rather than identification of specific nodes/midplanes.
Cnodes can be reserved instead of full midplanes, so asking for 32 nodes will
result in a reservation consisting of 32 cnodes. Also you can request specific
cnodes in your reservation as well (e.g. <i>scontrol create reservation
nodes=bgq0000[00000x11111]</i> will result in the first nodecard of
midplane bgq0000 being reserved).  Multiple block sizes can also be
specified and a reservation will be made that includes those block sizes
(e.g. <i>scontrol create reservation nodecnt=4k,2k ...</i>). In earlier
versions of Slurm, the nodes/midplanes selected for a reservation when
specifying a node count might not be suitable for creating block(s) of the
desired size(s).</p>

<h4>Debugging</h4>

<p>All of the testing and debugging guidance provided in
<a href="quickstart_admin.html"> Quick Start Administrator Guide</a>
apply to BlueGene systems.
One can start the <i>slurmctld</i> and <i>slurmd</i> daemons in the foreground
with extensive debugging to establish basic functionality.
Once running in production, the configured <i>SlurmctldLog</i> and
<i>SlurmdLog</i> files will provide historical system information.
On BlueGene systems, there is also a <i>BridgeAPILogFile</i> defined
in <i>bluegene.conf</i> which can be configured to contain detailed
information about every Bridge API call issued.</p>

<p>Note that <i>slurmctld</i> log messages of the sort
<i>Nodes bg[000x133] not responding</i> are indicative of the <i>slurmd</i>
daemon serving as a front-end to those midplanes is not responding (on
non-BlueGene systems, the <i>slurmd</i> actually does run on the compute
nodes, so the message is more meaningful there). </p>

<p>Note that you can emulate a BlueGene/Q system on stand-alone Linux
system.
Run <i>configure</i> with the <i>--enable-bgq-emulation</i> option.
This will define "HAVE_BG", "HAVE_BGQ", and "HAVE_FRONT_END" in the
config.h file.
Then execute <i>make</i> normally.
These variables will build the code as if it were running
on an actual BlueGene computer, but avoid making calls to the
Bridge library (that is controlled by the variable "HAVE_BG_FILES",
which is left undefined). You can use this to test configurations,
scheduling logic, etc. </p>

<p class="footer"><a href="#top">top</a></p>

<p style="text-align:center;">Last modified 7 November 2017</p>

			</div> <!-- END .container -->
		</div> <!-- END .section -->
	</div> <!-- END .content -->
</div> <!-- END .main -->

<footer class="site-footer" role="contentinfo">
	<nav class="footer-nav section">
		<div class="container">
			<p><a href="disclaimer.html" target="_blank" class="privacy">Legal Notices</a></p>
		</div>
	</nav>
</footer>

</body>
</html>
slurm-wlm-doc 17.11.2-1build1 / usr / share / doc / slurm-wlm-doc / html / bluegene.html
