/usr/share/doc/ganeti/html/design-hroller.html is in ganeti-doc 2.16.0~rc2-1build1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>HRoller tool — Ganeti 2.16.0~rc2 documentation</title>
<link rel="stylesheet" href="_static/style.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '2.16.0~rc2',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="HSqueeze tool" href="design-hsqueeze.html" />
<link rel="prev" title="GlusterFS Ganeti support" href="design-glusterfs-ganeti-support.html" />
</head>
<body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-hsqueeze.html" title="HSqueeze tool"
accesskey="N">next</a></li>
<li class="right" >
<a href="design-glusterfs-ganeti-support.html" title="GlusterFS Ganeti support"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="hroller-tool">
<h1><a class="toc-backref" href="#id1">HRoller tool</a><a class="headerlink" href="#hroller-tool" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Created:</th><td class="field-body">2013-Feb-15</td>
</tr>
<tr class="field-even field"><th class="field-name">Status:</th><td class="field-body">Implemented</td>
</tr>
<tr class="field-odd field"><th class="field-name">Ganeti-Version:</th><td class="field-body">2.8.0, 2.9.0</td>
</tr>
</tbody>
</table>
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#hroller-tool" id="id1">HRoller tool</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings" id="id2">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes" id="id3">Proposed changes</a><ul>
<li><a class="reference internal" href="#new-options" id="id4">New options</a></li>
<li><a class="reference internal" href="#calculating-rolling-maintenances" id="id5">Calculating rolling maintenances</a><ul>
<li><a class="reference internal" href="#down-instances" id="id6">Down instances</a></li>
<li><a class="reference internal" href="#drbd" id="id7">DRBD</a></li>
<li><a class="reference internal" href="#non-drbd" id="id8">Non-DRBD</a></li>
<li><a class="reference internal" href="#full-evacuation" id="id9">Full-Evacuation</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#future-work" id="id10">Future work</a></li>
</ul>
</li>
</ul>
</div>
<p>This is a design document detailing the cluster maintenance scheduler,
HRoller.</p>
<div class="section" id="current-state-and-shortcomings">
<h2><a class="toc-backref" href="#id2">Current state and shortcomings</a><a class="headerlink" href="#current-state-and-shortcomings" title="Permalink to this headline">¶</a></h2>
<p>To enable automating cluster-wide reboots a new htool, called HRoller,
was added to Ganeti starting from version 2.7. This tool helps
parallelizing cluster offline maintenances by calculating which nodes
are not both primary and secondary for a DRBD instance, and thus can be
rebooted at the same time, when all instances are down.</p>
<p>The way this is done is documented in the <em class="manpage">hroller(1)</em> manpage.</p>
<p>We would now like to perform online maintenance on the cluster by
rebooting nodes after evacuating their primary instances (rolling
reboots).</p>
</div>
<div class="section" id="proposed-changes">
<h2><a class="toc-backref" href="#id3">Proposed changes</a><a class="headerlink" href="#proposed-changes" title="Permalink to this headline">¶</a></h2>
<div class="section" id="new-options">
<h3><a class="toc-backref" href="#id4">New options</a><a class="headerlink" href="#new-options" title="Permalink to this headline">¶</a></h3>
<ul class="simple">
<li>HRoller should be able to operate on single nodegroups (-G flag) or
select its target node through some other mean (eg. via a tag, or a
regexp). (Note that individual node selection is already possible via
the -O flag, that makes hroller ignore a node altogether).</li>
<li>HRoller should handle non redundant instances: currently these are
ignored but there should be a way to select its behavior between “it’s
ok to reboot a node when a non-redundant instance is on it” or “skip
nodes with non-redundant instances”. This will only be selectable
globally, and not per instance.</li>
<li>Hroller will make sure to keep any instance which is up in its current
state, via live migrations, unless explicitly overridden. The
algorithm that will be used calculate the rolling reboot with live
migrations is described below, and any override on considering the
instance status will only be possible on the whole run, and not
per-instance.</li>
</ul>
</div>
<div class="section" id="calculating-rolling-maintenances">
<h3><a class="toc-backref" href="#id5">Calculating rolling maintenances</a><a class="headerlink" href="#calculating-rolling-maintenances" title="Permalink to this headline">¶</a></h3>
<p>In order to perform rolling maintenance we need to migrate instances off
the nodes before a reboot. How this can be done depends on the
instance’s disk template and status:</p>
<div class="section" id="down-instances">
<h4><a class="toc-backref" href="#id6">Down instances</a><a class="headerlink" href="#down-instances" title="Permalink to this headline">¶</a></h4>
<p>If an instance was shutdown when the maintenance started it will be
considered for avoiding contemporary reboot of its primary and secondary
nodes, but will <em>not</em> be considered as a target for the node evacuation.
This allows avoiding needlessly moving its primary around, since it
won’t suffer a downtime anyway.</p>
<p>Note that a node with non-redundant instances will only ever be
considered good for rolling-reboot if these are down (or the checking of
status is overridden) <em>and</em> an explicit option to allow it is set.</p>
</div>
<div class="section" id="drbd">
<h4><a class="toc-backref" href="#id7">DRBD</a><a class="headerlink" href="#drbd" title="Permalink to this headline">¶</a></h4>
<p>Each node must migrate all instances off to their secondaries, and then
can either be rebooted, or the secondaries can be evacuated as well.</p>
<p>Since currently doing a <code class="docutils literal"><span class="pre">replace-disks</span></code> on DRBD breaks redundancy,
it’s not any safer than temporarily rebooting a node with secondaries on
them (citation needed). As such we’ll implement for now just the
“migrate+reboot” mode, and focus later on replace-disks as well.</p>
<p>In order to do that we can use the following algorithm:</p>
<ol class="arabic simple">
<li>Compute node sets that don’t contain both the primary and the
secondary of any instance, and also don’t contain the primary
nodes of two instances that have the same node as secondary. These
can be obtained by computing a coloring of the graph with nodes
as vertexes and an edge between two nodes, if either condition
prevents simultaneous maintenance. (This is the current algorithm of
<em class="manpage">hroller(1)</em> with the extension that the graph to be colored
has additional edges between the primary nodes of two instances sharing
their secondary node.)</li>
<li>It is then possible to migrate in parallel all nodes in a set
created at step 1, and then reboot/perform maintenance on them, and
migrate back their original primaries, which allows the computation
above to be reused for each following set without N+1 failures
being triggered, if none were present before. See below about the
actual execution of the maintenance.</li>
</ol>
</div>
<div class="section" id="non-drbd">
<h4><a class="toc-backref" href="#id8">Non-DRBD</a><a class="headerlink" href="#non-drbd" title="Permalink to this headline">¶</a></h4>
<p>All non-DRBD disk templates that can be migrated have no “secondary”
concept. As such instances can be migrated to any node (in the same
nodegroup). In order to do the job we can either:</p>
<ul class="simple">
<li>Perform migrations on one node at a time, perform the maintenance on
that node, and proceed (the node will then be targeted again to host
instances automatically, as hail chooses targets for the instances
between all nodes in a group. Nodes in different nodegroups can be
handled in parallel.</li>
<li>Perform migrations on one node at a time, but without waiting for the
first node to come back before proceeding. This allows us to continue,
restricting the cluster, until no more capacity in the nodegroup is
available, and then having to wait for some nodes to come back so that
capacity is available again for the last few nodes.</li>
<li>Pre-Calculate sets of nodes that can be migrated together (probably
with a greedy algorithm) and parallelize between them, with the
migrate-back approach discussed for DRBD to perform the calculation
only once.</li>
</ul>
<p>Note that for non-DRBD disks that still use local storage (eg. RBD and
plain) redundancy might break anyway, and nothing except the first
algorithm might be safe. This perhaps would be a good reason to consider
managing better RBD pools, if those are implemented on top of nodes
storage, rather than on dedicated storage machines.</p>
</div>
<div class="section" id="full-evacuation">
<h4><a class="toc-backref" href="#id9">Full-Evacuation</a><a class="headerlink" href="#full-evacuation" title="Permalink to this headline">¶</a></h4>
<p>If full evacuation of the nodes to be rebooted is desired, a simple
migration is not enough for the DRBD instances. To keep the number of
disk operations small, we restrict moves to <code class="docutils literal"><span class="pre">migrate,</span> <span class="pre">replace-secondary</span></code>.
That is, after migrating instances out of the nodes to be rebooted,
replacement secondaries are searched for, for all instances that have
their then secondary on one of the rebooted nodes. This is done by a
greedy algorithm, refining the initial reboot partition, if necessary.</p>
</div>
</div>
</div>
<div class="section" id="future-work">
<h2><a class="toc-backref" href="#id10">Future work</a><a class="headerlink" href="#future-work" title="Permalink to this headline">¶</a></h2>
<p>Hroller should become able to execute rolling maintenances, rather than
just calculate them. For this to succeed properly one of the following
must happen:</p>
<ul class="simple">
<li>HRoller handles rolling maintenances that happen at the same time as
unrelated cluster jobs, and thus recalculates the maintenance at each
step</li>
<li>HRoller can selectively drain the cluster so it’s sure that only the
rolling maintenance can be going on</li>
</ul>
<p>DRBD nodes’ <code class="docutils literal"><span class="pre">replace-disks</span></code>’ functionality should be implemented. Note
that when we will support a DRBD version that allows multi-secondary
this can be done safely, without losing replication at any time, by
adding a temporary secondary and only when the sync is finished dropping
the previous one.</p>
<p>Non-redundant (plain or file) instances should have a way to be moved
off as well via plain storage live migration or <code class="docutils literal"><span class="pre">gnt-instance</span> <span class="pre">move</span></code>
(which requires downtime).</p>
<p>If/when RBD pools can be managed inside Ganeti, care can be taken so
that the pool is evacuated as well from a node before it’s put into
maintenance. This is equivalent to evacuating DRBD secondaries.</p>
<p>Master failovers during the maintenance should be performed by hroller.
This requires RPC/RAPI support for master failover. Hroller should also
be modified to better support running on the master itself and
continuing on the new master.</p>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">HRoller tool</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes">Proposed changes</a><ul>
<li><a class="reference internal" href="#new-options">New options</a></li>
<li><a class="reference internal" href="#calculating-rolling-maintenances">Calculating rolling maintenances</a><ul>
<li><a class="reference internal" href="#down-instances">Down instances</a></li>
<li><a class="reference internal" href="#drbd">DRBD</a></li>
<li><a class="reference internal" href="#non-drbd">Non-DRBD</a></li>
<li><a class="reference internal" href="#full-evacuation">Full-Evacuation</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#future-work">Future work</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="design-glusterfs-ganeti-support.html"
title="previous chapter">GlusterFS Ganeti support</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="design-hsqueeze.html"
title="next chapter">HSqueeze tool</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/design-hroller.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-hsqueeze.html" title="HSqueeze tool"
>next</a></li>
<li class="right" >
<a href="design-glusterfs-ganeti-support.html" title="GlusterFS Ganeti support"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2018, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Google Inc..
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
</div>
</body>
</html>
|