/usr/share/doc/ganeti/html/design-kvmd.html is in ganeti-doc 2.16.0~rc2-1build1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 | <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<title>KVM daemon — Ganeti 2.16.0~rc2 documentation</title>
<link rel="stylesheet" href="_static/style.css" type="text/css" />
<link rel="stylesheet" href="_static/pygments.css" type="text/css" />
<script type="text/javascript">
var DOCUMENTATION_OPTIONS = {
URL_ROOT: './',
VERSION: '2.16.0~rc2',
COLLAPSE_INDEX: false,
FILE_SUFFIX: '.html',
HAS_SOURCE: true,
SOURCELINK_SUFFIX: '.txt'
};
</script>
<script type="text/javascript" src="_static/jquery.js"></script>
<script type="text/javascript" src="_static/underscore.js"></script>
<script type="text/javascript" src="_static/doctools.js"></script>
<link rel="search" title="Search" href="search.html" />
<link rel="next" title="Improving location awareness of Ganeti" href="design-location.html" />
<link rel="prev" title="Detection of user-initiated shutdown from inside an instance" href="design-internal-shutdown.html" />
</head>
<body>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-location.html" title="Improving location awareness of Ganeti"
accesskey="N">next</a></li>
<li class="right" >
<a href="design-internal-shutdown.html" title="Detection of user-initiated shutdown from inside an instance"
accesskey="P">previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="document">
<div class="documentwrapper">
<div class="bodywrapper">
<div class="body" role="main">
<div class="section" id="kvm-daemon">
<h1>KVM daemon<a class="headerlink" href="#kvm-daemon" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Created:</th><td class="field-body">2014-Jan-09</td>
</tr>
<tr class="field-even field"><th class="field-name">Status:</th><td class="field-body">Implemented</td>
</tr>
<tr class="field-odd field"><th class="field-name">Ganeti-Version:</th><td class="field-body">2.11.0</td>
</tr>
</tbody>
</table>
<div class="toctree-wrapper compound">
</div>
<p>This design document describes the KVM daemon, which is responsible for
determining whether a given KVM instance was shutdown by an
administrator or a user.</p>
<div class="section" id="current-state-and-shortcomings">
<h2>Current state and shortcomings<a class="headerlink" href="#current-state-and-shortcomings" title="Permalink to this headline">¶</a></h2>
<p>This design document describes the KVM daemon which addresses the KVM
side of the user-initiated shutdown problem introduced in
<a class="reference internal" href="design-internal-shutdown.html"><span class="doc">Detection of user-initiated shutdown from inside an instance</span></a>. We are also interested in keeping this
functionality optional. That is, an administrator does not necessarily
have to run the KVM daemon if either he is running Xen or even, if he
is running KVM, he is not interested in instance shutdown detection.
This requirement is important because it means the KVM daemon should
be a modular component in the overall Ganeti design, i.e., it should
be easy to enable and disable it.</p>
</div>
<div class="section" id="proposed-changes">
<h2>Proposed changes<a class="headerlink" href="#proposed-changes" title="Permalink to this headline">¶</a></h2>
<p>The instance shutdown feature for KVM requires listening on events from
the Qemu Machine Protocol (QMP) Unix socket, which is created together
with a KVM instance. A QMP socket typically looks like
<code class="docutils literal"><span class="pre">/var/run/ganeti/kvm-hypervisor/ctrl/<instance>.qmp</span></code> and implements
the QMP protocol. This is a bidirectional protocol that allows Ganeti
to send commands, such as, system powerdown, as well as, receive events,
such as, the powerdown and shutdown events.</p>
<p>Listening in on these events allows Ganeti to determine whether a given
KVM instance was shutdown by an administrator, either through
<code class="docutils literal"><span class="pre">gnt-instance</span> <span class="pre">stop|remove</span> <span class="pre"><instance></span></code> or <code class="docutils literal"><span class="pre">kill</span> <span class="pre">-KILL</span>
<span class="pre"><instance-pid></span></code>, or by a user, through <code class="docutils literal"><span class="pre">poweroff</span></code> from inside the
instance. Upon an administrator powerdown, the QMP protocol sends two
events, namely, a powerdown event and a shutdown event, whereas upon a
user shutdown only the shutdown event is sent. This is enough to
distinguish between an administrator and a user shutdown. However,
there is one limitation, which is, <code class="docutils literal"><span class="pre">kill</span> <span class="pre">-TERM</span> <span class="pre"><instance-pid></span></code>. Even
though this is an action performed by the administrator, it will be
considered a user shutdown by the approach described in this document.</p>
<p>Several design strategies were considered. Most of these strategies
consisted of spawning some process listening on the QMP socket when a
KVM instance is created. However, having a listener process per KVM
instance is not scalable. Therefore, a different strategy is proposed,
namely, having a single process, called the KVM daemon, listening on the
QMP sockets of all KVM instances within a node. That also means there
is an instance of the KVM daemon on each node.</p>
<p>In order to implement the KVM daemon, two problems need to be addressed,
namely, how the KVM daemon knows when to open a connection to a given
QMP socket and how the KVM daemon communicates with Ganeti whether a
given instance was shutdown by an administrator or a user.</p>
<div class="section" id="qmp-connections-management">
<h3>QMP connections management<a class="headerlink" href="#qmp-connections-management" title="Permalink to this headline">¶</a></h3>
<p>As mentioned before, the QMP sockets reside in the KVM control
directory, which is usually located under
<code class="docutils literal"><span class="pre">/var/run/ganeti/kvm-hypervisor/ctrl/</span></code>. When a KVM instance is
created, a new QMP socket for this instance is also created in this
directory.</p>
<p>In order to simplify the design of the KVM daemon, instead of having
Ganeti communicate to this daemon through a pipe or socket the creation
of a new KVM instance, and thus a new QMP socket, this daemon will
monitor the KVM control directory using <code class="docutils literal"><span class="pre">inotify</span></code>. As a result, the
daemon is not only able to deal with KVM instances being created and
removed, but also capable of overcoming other problematic situations
concerning the filesystem, such as, the case when the KVM control
directory does not exist because, for example, Ganeti was not yet
started, or the KVM control directory was removed, for example, as a
result of a Ganeti reinstallation.</p>
</div>
<div class="section" id="shutdown-detection">
<h3>Shutdown detection<a class="headerlink" href="#shutdown-detection" title="Permalink to this headline">¶</a></h3>
<p>As mentioned before, the KVM daemon is responsible for opening a
connection to the QMP socket of a given instance and listening in on the
shutdown and powerdown events, which allow the KVM daemon to determine
whether the instance stopped because of an administrator or user
shutdown. Once the instance is stopped, the KVM daemon needs to
communicate to Ganeti whether the user was responsible for shutting down
the instance.</p>
<p>In order to achieve this, the KVM daemon writes an empty file, called
the shutdown file, in the KVM control directory with a name similar to
the QMP socket file but with the extension <code class="docutils literal"><span class="pre">.qmp</span></code> replaced with
<code class="docutils literal"><span class="pre">.shutdown</span></code>. The presence of this file indicates that the shutdown
was initiated by a user, whereas the absence of this file indicates that
the shutdown was caused by an administrator. This strategy also handles
crashes and signals, such as, <code class="docutils literal"><span class="pre">SIGKILL</span></code>, to be handled correctly,
given that in these cases the KVM daemon never receives the powerdown
and shutdown events and, therefore, never creates the shutdown file.</p>
</div>
<div class="section" id="kvm-daemon-launch">
<h3>KVM daemon launch<a class="headerlink" href="#kvm-daemon-launch" title="Permalink to this headline">¶</a></h3>
<p>With the above issues addressed, a question remains as to when the KVM
daemon should be started. The KVM daemon is different from other Ganeti
daemons, which start together with the Ganeti service, because the KVM
daemon is optional, given that it is specific to KVM and should not be
run on installations containing only Xen, and, even in a KVM
installation, the user might still choose not to enable it. And finally
because the KVM daemon is not really necessary until the first KVM
instance is started. For these reasons, the KVM daemon is started from
within Ganeti when a KVM instance is started. And the job process
spawned by the node daemon is responsible for starting the KVM daemon.</p>
<p>Given the current design of Ganeti, in which the node daemon spawns a
job process to handle the creation of the instance, when launching the
KVM daemon it is necessary to first check whether an instance of this
daemon is already running and, if this is not the case, then the KVM
daemon can be safely started.</p>
</div>
</div>
<div class="section" id="design-alternatives">
<h2>Design alternatives<a class="headerlink" href="#design-alternatives" title="Permalink to this headline">¶</a></h2>
<p>At first, it might seem natural to include the instance shutdown
detection for KVM in the node daemon. After all, the node daemon is
already responsible for managing instances, for example, starting and
stopping an instance. Nevertheless, the node daemon is more complicated
than it might seem at first.</p>
<p>The node daemon is composed of the main loop, which runs in the main
thread and is responsible for receiving requests and spawning jobs for
handling these requests, and the jobs, which are independent processes
spawned for executing the actual tasks, such as, creating an instance.</p>
<p>Including instance shutdown detection in the node daemon is not viable
because adding it to the main loop would cause KVM specific code to
taint the generality of the node daemon. In order to add it to the job
processes, it would be possible to spawn either a foreground or a
background process. However, these options are also not viable because
they would lead to the situation described before where there would be a
monitoring process per instance, which is not scalable. Moreover, the
foreground process has an additional disadvantage: it would require
modifications the node daemon in order not to expect a terminating job,
which is the current node daemon design.</p>
<p>There is another design issue to have in mind. We could reconsider the
place where to write the data that tell Ganeti whether an instance was
shutdown by an administrator or the user. Instead of using the KVM
shutdown files presented above, in which the presence of the file
indicates a user shutdown and its absence an administrator shutdown, we
could store a value in the KVM runtime state file, which is where the
relevant KVM state information is. The advantage of this approach is
that it would keep the KVM related information in one place, thus making
it easier to manage. However, it would lead to a more complex
implementation and, in the context of the general transition in Ganeti
from Python to Haskell, a simpler implementation is preferred.</p>
<p>Finally, it should be noted that the KVM runtime state file benefits
from automatic migration. That is, when an instance is migrated so is
the KVM state file. However, the instance shutdown detection for KVM
does not require this feature and, in fact, migrating the instance
shutdown state would be incorrect.</p>
</div>
<div class="section" id="further-considerations">
<h2>Further considerations<a class="headerlink" href="#further-considerations" title="Permalink to this headline">¶</a></h2>
<p>There are potential race conditions between Ganeti and the KVM daemon,
however, in practice they seem unlikely. For example, the KVM daemon
needs to add and remove watches to the parent directories of the KVM
control directory until this directory is finally created. It is
possible that Ganeti creates this directory and a KVM instance before
the KVM daemon has a chance to add a watch to the KVM control directory,
thus causing this daemon to miss the <code class="docutils literal"><span class="pre">inotify</span></code> creation event for the
QMP socket.</p>
<p>There are other problems which arise from the limitations of
<code class="docutils literal"><span class="pre">inotify</span></code>. For example, if the KVM daemon is started after the first
Ganeti instance has been created, then the <code class="docutils literal"><span class="pre">inotify</span></code> will not produce
any event for the creation of the QMP socket. This can happen, for
example, if the KVM daemon needs to be restarted or upgraded. As a
result, it might be necessary to have an additional mechanism that runs
at KVM daemon startup or at regular intervals to ensure that the current
KVM internal state is consistent with the actual contents of the KVM
control directory.</p>
<p>Another race condition occurs when Ganeti shuts down a KVM instance
using force. Ganeti uses <code class="docutils literal"><span class="pre">TERM</span></code> signals to stop KVM instances when
force is specified or ACPI is not enabled. However, as mentioned
before, <code class="docutils literal"><span class="pre">TERM</span></code> signals are interpreted by the KVM daemon as a user
shutdown. As a result, the KVM daemon creates a shutdown file which
then must be removed by Ganeti. The race condition occurs because the
KVM daemon might create the shutdown file after the hypervisor code that
tries to remove this file has already run. In practice, the race
condition seems unlikely because Ganeti stops the KVM instance in a
retry loop, which allows Ganeti to stop the instance and cleanup its
runtime information.</p>
<p>It is possible to determine if a process, in this particular case the
KVM process, was terminated by a <code class="docutils literal"><span class="pre">TERM</span></code> signal, using the <a class="reference external" href="https://web.archive.org/web/20121025062848/http://netsplit.com/2011/02/09/the-proc-connector-and-socket-filters/">proc
connector and socket filters</a>.
The proc connector is a socket connected between a userspace process and
the kernel through the netlink protocol and can be used to receive
notifications of process events, and the socket filters is a mechanism
for subscribing only to events that are relevant. There are several
<a class="reference external" href="http://lwn.net/Articles/157150/">process events</a> which can be
subscribed to, however, in this case, we are interested only in the exit
event, which carries information about the exit signal.</p>
</div>
</div>
</div>
</div>
</div>
<div class="sphinxsidebar" role="navigation" aria-label="main navigation">
<div class="sphinxsidebarwrapper">
<h3><a href="index.html">Table Of Contents</a></h3>
<ul>
<li><a class="reference internal" href="#">KVM daemon</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes">Proposed changes</a><ul>
<li><a class="reference internal" href="#qmp-connections-management">QMP connections management</a></li>
<li><a class="reference internal" href="#shutdown-detection">Shutdown detection</a></li>
<li><a class="reference internal" href="#kvm-daemon-launch">KVM daemon launch</a></li>
</ul>
</li>
<li><a class="reference internal" href="#design-alternatives">Design alternatives</a></li>
<li><a class="reference internal" href="#further-considerations">Further considerations</a></li>
</ul>
</li>
</ul>
<h4>Previous topic</h4>
<p class="topless"><a href="design-internal-shutdown.html"
title="previous chapter">Detection of user-initiated shutdown from inside an instance</a></p>
<h4>Next topic</h4>
<p class="topless"><a href="design-location.html"
title="next chapter">Improving location awareness of Ganeti</a></p>
<div role="note" aria-label="source link">
<h3>This Page</h3>
<ul class="this-page-menu">
<li><a href="_sources/design-kvmd.rst.txt"
rel="nofollow">Show Source</a></li>
</ul>
</div>
<div id="searchbox" style="display: none" role="search">
<h3>Quick search</h3>
<form class="search" action="search.html" method="get">
<div><input type="text" name="q" /></div>
<div><input type="submit" value="Go" /></div>
<input type="hidden" name="check_keywords" value="yes" />
<input type="hidden" name="area" value="default" />
</form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
</div>
</div>
<div class="clearer"></div>
</div>
<div class="related" role="navigation" aria-label="related navigation">
<h3>Navigation</h3>
<ul>
<li class="right" style="margin-right: 10px">
<a href="design-location.html" title="Improving location awareness of Ganeti"
>next</a></li>
<li class="right" >
<a href="design-internal-shutdown.html" title="Detection of user-initiated shutdown from inside an instance"
>previous</a> |</li>
<li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> »</li>
</ul>
</div>
<div class="footer" role="contentinfo">
© Copyright 2018, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Google Inc..
Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
</div>
</body>
</html>
|