/usr/share/doc/ganeti/html/design-daemons.html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Ganeti daemons refactoring &#8212; Ganeti 2.16.0~rc2 documentation</title>
    <link rel="stylesheet" href="_static/style.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    './',
        VERSION:     '2.16.0~rc2',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true,
        SOURCELINK_SUFFIX: '.txt'
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Conversion between disk templates" href="design-disk-conversion.html" />
    <link rel="prev" title="Design for adding UUID and name to devices" href="design-device-uuid-name.html" /> 
  </head>
  <body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="design-disk-conversion.html" title="Conversion between disk templates"
             accesskey="N">next</a></li>
        <li class="right" >
          <a href="design-device-uuid-name.html" title="Design for adding UUID and name to devices"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> &#187;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="ganeti-daemons-refactoring">
<h1><a class="toc-backref" href="#id1">Ganeti daemons refactoring</a><a class="headerlink" href="#ganeti-daemons-refactoring" title="Permalink to this headline">¶</a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Created:</th><td class="field-body">2013-Sep-27</td>
</tr>
<tr class="field-even field"><th class="field-name">Status:</th><td class="field-body">Implemented</td>
</tr>
<tr class="field-odd field"><th class="field-name">Ganeti-Version:</th><td class="field-body">2.12.0</td>
</tr>
</tbody>
</table>
<div class="contents topic" id="contents">
<p class="topic-title first">Contents</p>
<ul class="simple">
<li><a class="reference internal" href="#ganeti-daemons-refactoring" id="id1">Ganeti daemons refactoring</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings" id="id2">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes" id="id3">Proposed changes</a></li>
<li><a class="reference internal" href="#implementation" id="id4">Implementation</a></li>
<li><a class="reference internal" href="#further-considerations" id="id5">Further considerations</a></li>
</ul>
</li>
</ul>
</div>
<p>This is a design document detailing the plan for refactoring the internal
structure of Ganeti, and particularly the set of daemons it is divided into.</p>
<div class="section" id="current-state-and-shortcomings">
<h2><a class="toc-backref" href="#id2">Current state and shortcomings</a><a class="headerlink" href="#current-state-and-shortcomings" title="Permalink to this headline">¶</a></h2>
<p>Ganeti is comprised of a growing number of daemons, each dealing with part of
the tasks the cluster has to face, and communicating with the other daemons
using a variety of protocols.</p>
<p>Specifically, as of Ganeti 2.8, the situation is as follows:</p>
<dl class="docutils">
<dt><code class="docutils literal"><span class="pre">Master</span> <span class="pre">daemon</span> <span class="pre">(MasterD)</span></code></dt>
<dd><p class="first">It is responsible for managing the entire cluster, and it’s written in Python.
It is executed on a single node (the master node). It receives the commands
given by the cluster administrator (through the remote API daemon or the
command line tools) over the LUXI protocol.  The master daemon is responsible
for creating and managing the jobs that will execute such commands, and for
managing the locks that ensure the cluster will not incur in race conditions.</p>
<p>Each job is managed by a separate Python thread, that interacts with the node
daemons via RPC calls.</p>
<p class="last">The master daemon is also responsible for managing the configuration of the
cluster, changing it when required by some job. It is also responsible for
copying the configuration to the other master candidates after updating it.</p>
</dd>
<dt><code class="docutils literal"><span class="pre">RAPI</span> <span class="pre">daemon</span> <span class="pre">(RapiD)</span></code></dt>
<dd>It is written in Python and runs on the master node only. It waits for
requests issued remotely through the remote API protocol. Then, it forwards
them, using the LUXI protocol, to the master daemon (if they are commands) or
to the query daemon if they are queries about the configuration (including
live status) of the cluster.</dd>
<dt><code class="docutils literal"><span class="pre">Node</span> <span class="pre">daemon</span> <span class="pre">(NodeD)</span></code></dt>
<dd>It is written in Python. It runs on all the nodes. It is responsible for
receiving the master requests over RPC and execute them, using the appropriate
backend (hypervisors, DRBD, LVM, etc.). It also receives requests over RPC for
the execution of queries gathering live data on behalf of the query daemon.</dd>
<dt><code class="docutils literal"><span class="pre">Configuration</span> <span class="pre">daemon</span> <span class="pre">(ConfD)</span></code></dt>
<dd>It is written in Haskell. It runs on all the master candidates. Since the
configuration is replicated only on the master node, this daemon exists in
order to provide information about the configuration to nodes needing them.
The requests are done through ConfD’s own protocol, HMAC signed,
implemented over UDP, and meant to be used by parallely querying all the
master candidates (or a subset thereof) and getting the most up to date
answer. This is meant as a way to provide a robust service even in case master
is temporarily unavailable.</dd>
<dt><code class="docutils literal"><span class="pre">Query</span> <span class="pre">daemon</span> <span class="pre">(QueryD)</span></code></dt>
<dd>It is written in Haskell. It runs on all the master candidates. It replies
to Luxi queries about the current status of the system, including live data it
obtains by querying the node daemons through RPCs.</dd>
<dt><code class="docutils literal"><span class="pre">Monitoring</span> <span class="pre">daemon</span> <span class="pre">(MonD)</span></code></dt>
<dd>It is written in Haskell. It runs on all nodes, including the ones that are
not vm-capable. It is meant to provide information on the status of the
system. Such information is related only to the specific node the daemon is
running on, and it is provided as JSON encoded data over HTTP, to be easily
readable by external tools.
The monitoring daemon communicates with ConfD to get information about the
configuration of the cluster. The choice of communicating with ConfD instead
of MasterD allows it to obtain configuration information even when the cluster
is heavily degraded (e.g.: when master and some, but not all, of the master
candidates are unreachable).</dd>
</dl>
<p>The current structure of the Ganeti daemons is inefficient because there are
many different protocols involved, and each daemon needs to be able to use
multiple ones, and has to deal with doing different things, thus making
sometimes unclear which daemon is responsible for performing a specific task.</p>
<p>Also, with the current configuration, jobs are managed by the master daemon
using python threads. This makes terminating a job after it has started a
difficult operation, and it is the main reason why this is not possible yet.</p>
<p>The master daemon currently has too many different tasks, that could be handled
better if split among different daemons.</p>
</div>
<div class="section" id="proposed-changes">
<h2><a class="toc-backref" href="#id3">Proposed changes</a><a class="headerlink" href="#proposed-changes" title="Permalink to this headline">¶</a></h2>
<p>In order to improve on the current situation, a new daemon subdivision is
proposed, and presented hereafter.</p>
<img src="_images/graphviz-e3c4c711c84d984cd5287fb6327323cd62fc910e.png" alt="digraph &quot;new-daemons-structure&quot; {
{rank=same; RConfD LuxiD;}
{rank=same; Jobs rconfigdata;}
node [shape=box]
RapiD [label=&quot;RapiD [M]&quot;]
LuxiD [label=&quot;LuxiD [M]&quot;]
WConfD [label=&quot;WConfD [M]&quot;]
Jobs [label=&quot;Jobs [M]&quot;]
RConfD [label=&quot;RConfD [MC]&quot;]
MonD [label=&quot;MonD [All]&quot;]
NodeD [label=&quot;NodeD [All]&quot;]
Clients [label=&quot;gnt-*\nclients [M]&quot;]
p1 [shape=none, label=&quot;&quot;]
p2 [shape=none, label=&quot;&quot;]
p3 [shape=none, label=&quot;&quot;]
p4 [shape=none, label=&quot;&quot;]
configdata [shape=none, label=&quot;config.data&quot;]
rconfigdata [shape=none, label=&quot;config.data\n[MC copy]&quot;]
locksdata [shape=none, label=&quot;locks.data&quot;]

RapiD -&gt; LuxiD [label=&quot;LUXI&quot;]
LuxiD -&gt; WConfD [label=&quot;WConfD\nproto&quot;]
LuxiD -&gt; Jobs [label=&quot;fork/exec&quot;]
Jobs -&gt; WConfD [label=&quot;WConfD\nproto&quot;]
Jobs -&gt; NodeD [label=&quot;RPC&quot;]
LuxiD -&gt; NodeD [label=&quot;RPC&quot;]
rconfigdata -&gt; RConfD
configdata -&gt; rconfigdata [label=&quot;sync via\nNodeD RPC&quot;]
WConfD -&gt; NodeD [label=&quot;RPC&quot;]
WConfD -&gt; configdata
WConfD -&gt; locksdata
MonD -&gt; RConfD [label=&quot;RConfD\nproto&quot;]
Clients -&gt; LuxiD [label=&quot;LUXI&quot;]
p1 -&gt; MonD [label=&quot;MonD proto&quot;]
p2 -&gt; RapiD [label=&quot;RAPI&quot;]
p3 -&gt; RConfD [label=&quot;RConfD\nproto&quot;]
p4 -&gt; Clients [label=&quot;CLI&quot;]
}" />
<dl class="docutils">
<dt><code class="docutils literal"><span class="pre">LUXI</span> <span class="pre">daemon</span> <span class="pre">(LuxiD)</span></code></dt>
<dd>It will be written in Haskell. It will run on the master node and it will be
the only LUXI server, replying to all the LUXI queries. These includes both
the queries about the live configuration of the cluster, previously served by
QueryD, and the commands actually changing the status of the cluster by
submitting jobs. Therefore, this daemon will also be the one responsible with
managing the job queue. When a job needs to be executed, the LuxiD will spawn
a separate process tasked with the execution of that specific job, thus making
it easier to terminate the job itself, if needed.  When a job requires locks,
LuxiD will request them from WConfD.
In order to keep availability of the cluster in case of failure of the master
node, LuxiD will replicate the job queue to the other master candidates, by
RPCs to the NodeD running there (the choice of RPCs for this task might be
reviewed at a second time, after implementing this design).</dd>
<dt><code class="docutils literal"><span class="pre">Configuration</span> <span class="pre">management</span> <span class="pre">daemon</span> <span class="pre">(WConfD)</span></code></dt>
<dd>It will run on the master node and it will be responsible for the management
of the authoritative copy of the cluster configuration (that is, it will be
the daemon actually modifying the <code class="docutils literal"><span class="pre">config.data</span></code> file). All the requests of
configuration changes will have to pass through this daemon, and will be
performed using a LUXI-like protocol (“WConfD proto” in the graph. The exact
protocol will be defined in the separate design document that will detail the
WConfD separation).  Having a single point of configuration management will
also allow Ganeti to get rid of possible race conditions due to concurrent
modifications of the configuration.  When the configuration is updated, it
will have to push the received changes to the other master candidates, via
RPCs, so that RConfD daemons and (in case of a failure on the master node)
the WConfD daemon on the new master can access an up-to-date version of it
(the choice of RPCs for this task might be reviewed at a second time). This
daemon will also be the one responsible for managing the locks, granting them
to the jobs requesting them, and taking care of freeing them up if the jobs
holding them crash or are terminated before releasing them.  In order to do
this, each job, after being spawned by LuxiD, will open a local unix socket
that will be used to communicate with it, and will be destroyed when the job
terminates.  LuxiD will be able to check, after a timeout, whether the job is
still running by connecting here, and to ask WConfD to forcefully remove the
locks if the socket is closed.
Also, WConfD should hold a serialized list of the locks and their owners in a
file (<code class="docutils literal"><span class="pre">locks.data</span></code>), so that it can keep track of their status in case it
crashes and needs to be restarted (by asking LuxiD which of them are still
running).
Interaction with this daemon will be performed using Unix sockets.</dd>
<dt><code class="docutils literal"><span class="pre">Configuration</span> <span class="pre">query</span> <span class="pre">daemon</span> <span class="pre">(RConfD)</span></code></dt>
<dd>It is written in Haskell, and it corresponds to the old ConfD. It will run on
all the master candidates and it will serve information about the static
configuration of the cluster (the one contained in <code class="docutils literal"><span class="pre">config.data</span></code>). The
provided information will be highly available (as in: a response will be
available as long as a stable-enough connection between the client and at
least one working master candidate is available) and its freshness will be
best effort (the most recent reply from any of the master candidates will be
returned, but it might still be older than the one available through WConfD).
The information will be served through the ConfD protocol.</dd>
<dt><code class="docutils literal"><span class="pre">Rapi</span> <span class="pre">daemon</span> <span class="pre">(RapiD)</span></code></dt>
<dd>It remains basically unchanged, with the only difference that all of its LUXI
query are directed towards LuxiD instead of being split between MasterD and
QueryD.</dd>
<dt><code class="docutils literal"><span class="pre">Monitoring</span> <span class="pre">daemon</span> <span class="pre">(MonD)</span></code></dt>
<dd>It remains unaffected by the changes in this design document. It will just get
some of the data it needs from RConfD instead of the old ConfD, but the
interfaces of the two are identical.</dd>
<dt><code class="docutils literal"><span class="pre">Node</span> <span class="pre">daemon</span> <span class="pre">(NodeD)</span></code></dt>
<dd>It remains unaffected by the changes proposed in the design document. The only
difference being that it will receive its RPCs from LuxiD (for job queue
replication), from WConfD (for configuration replication) and for the
processes executing single jobs (for all the operations to be performed by
nodes) instead of receiving them just from MasterD.</dd>
</dl>
<p>This restructuring will allow us to reorganize and improve the codebase,
introducing cleaner interfaces and giving well defined and more restricted tasks
to each daemon.</p>
<p>Furthermore, having more well-defined interfaces will allow us to have easier
upgrade procedures, and to work towards the possibility of upgrading single
components of a cluster one at a time, without the need for immediately
upgrading the entire cluster in a single step.</p>
</div>
<div class="section" id="implementation">
<h2><a class="toc-backref" href="#id4">Implementation</a><a class="headerlink" href="#implementation" title="Permalink to this headline">¶</a></h2>
<p>While performing this refactoring, we aim to increase the amount of
Haskell code, thus benefiting from the additional type safety provided by its
wide compile-time checks. In particular, all the job queue management and the
configuration management daemon will be written in Haskell, taking over the role
currently fulfilled by Python code executed as part of MasterD.</p>
<p>The changes describe by this design document are quite extensive, therefore they
will not be implemented all at the same time, but through a sequence of steps,
leaving the codebase in a consistent and usable state.</p>
<ol class="arabic simple">
<li>Rename QueryD to LuxiD.
A part of LuxiD, the one replying to configuration
queries including live information about the system, already exists in the
form of QueryD. This is being renamed to LuxiD, and will form the first part
of the new daemon. NB: this is happening starting from Ganeti 2.8. At the
beginning, only the already existing queries will be replied to by LuxiD.
More queries will be implemented in the next versions.</li>
<li>Let LuxiD be the interface for the queries and MasterD be their executor.
Currently, MasterD is the only responsible for receiving and executing LUXI
queries, and for managing the jobs they create.
Receiving the queries and managing the job queue will be extracted from
MasterD into LuxiD.
Actually executing jobs will still be done by MasterD, that contains all the
logic for doing that and for properly managing locks and the configuration.
At this stage, scheduling will simply consist in starting jobs until a fixed
maximum number of simultaneously running jobs is reached.</li>
<li>Extract WConfD from MasterD.
The logic for managing the configuration file is factored out to the
dedicated WConfD daemon. All configuration changes, currently executed
directly by MasterD, will be changed to be IPC requests sent to the new
daemon.</li>
<li>Extract locking management from MasterD.
The logic for managing and granting locks is extracted to WConfD as well.
Locks will not be taken directly anymore, but asked via IPC to WConfD.
This step can be executed on its own or at the same time as the previous one.</li>
<li>Jobs are executed as processes.
The logic for running jobs is rewritten so that each job can be managed by an
independent process. LuxiD will spawn a new (Python) process for every single
job. The RPCs will remain unchanged, and the LU code will stay as is as much
as possible.
MasterD will cease to exist as a daemon on its own at this point, but not
before.</li>
<li>Improve job scheduling algorithm.
The simple algorithm for scheduling jobs will be replaced by a more
intelligent one. Also, the implementation of <a class="reference internal" href="design-optables.html"><span class="doc">Filtering of jobs for the Ganeti job queue</span></a> can be
started.</li>
</ol>
<div class="section" id="job-death-detection">
<h3>Job death detection<a class="headerlink" href="#job-death-detection" title="Permalink to this headline">¶</a></h3>
<p><strong>Requirements:</strong></p>
<ul class="simple">
<li>It must be possible to reliably detect a death of a process even under
uncommon conditions such as very heavy system load.</li>
<li>A daemon must be able to detect a death of a process even if the
daemon is restarted while the process is running.</li>
<li>The solution must not rely on being able to communicate with
a process.</li>
<li>The solution must work for the current situation where multiple jobs
run in a single process.</li>
<li>It must be POSIX compliant.</li>
</ul>
<p>These conditions rule out simple solutions like checking a process ID
(because the process might be eventually replaced by another process
with the same ID) or keeping an open connection to a process.</p>
<p><strong>Solution:</strong> As a job process is spawned, before attempting to
communicate with any other process, it will create a designated empty
lock file, open it, acquire an <em>exclusive</em> lock on it, and keep it open.
When connecting to a daemon, the job process will provide it with the
path of the file. If the process dies unexpectedly, the operating system
kernel automatically cleans up the lock.</p>
<p>Therefore, daemons can check if a process is dead by trying to acquire
a <em>shared</em> lock on the lock file in a non-blocking mode:</p>
<ul class="simple">
<li>If the locking operation succeeds, it means that the exclusive lock is
missing, therefore the process has died, but the lock
file hasn’t been cleaned up yet. The daemon should release the lock
immediately. Optionally, the daemon may delete the lock file.</li>
<li>If the file is missing, the process has died and the lock file has
been cleaned up.</li>
<li>If the locking operation fails due to a lock conflict, it means
the process is alive.</li>
</ul>
<p>Using shared locks for querying lock files ensures that the detection
works correctly even if multiple daemons query a file at the same time.</p>
<p>A job should close and remove its lock file when completely finishes.
The WConfD daemon will be responsible for removing stale lock files of
jobs that didn’t remove its lock files themselves.</p>
<p><strong>Statelessness of the protocol:</strong> To keep our protocols stateless,
the job id and the path the to lock file are sent as part of every
request that deals with resources, in particular the Ganeti Locks.
All resources are owned by the pair (job id, lock file). In this way,
several jobs can live in the same process (as it will be in the
transition period), but owner death detection still only depends on the
owner of the resource. In particular, no additional lookup table is
needed to obtain the lock file for a given owner.</p>
<p><strong>Considered alternatives:</strong> An alternative to creating a separate lock
file would be to lock the job status file. However, file locks are kept
only as long as the file is open. Therefore any operation followed by
closing the file would cause the process to release the lock. In
particular, with jobs as threads, the master daemon wouldn’t be able to
keep locks and operate on job files at the same time.</p>
</div>
<div class="section" id="job-execution">
<h3>Job execution<a class="headerlink" href="#job-execution" title="Permalink to this headline">¶</a></h3>
<p>As the Luxi daemon will be responsible for executing jobs, it needs to
start jobs in such a way that it can properly detect if the job dies
under any circumstances (such as Luxi daemon being restarted in the
process).</p>
<p>The name of the lock file will be stored in the corresponding job file
so that anybody is able to check the status of the process corresponding
to a job.</p>
<p>The proposed procedure:</p>
<ol class="arabic simple">
<li>The Luxi daemon saves the name of its own lock file into the job file.</li>
<li>The Luxi daemon forks, creating a bi-directional pipe with the child
process.</li>
<li>The child process creates and locks its own, proper lock file and
handles its name to the Luxi daemon through the pipe.</li>
<li>The Luxi daemon saves the name of the lock file into the job file and
confirms it to the child process.</li>
<li>Only then the child process can replace itself by the actual job
process.</li>
</ol>
<p>If the child process detects that the pipe is broken before receiving the
confirmation, it must terminate, not starting the actual job.
This way, the actual job is only started if it is ensured that its lock
file name is written to the job file.</p>
<p>If the Luxi daemon detects that the pipe is broken before successfully
sending the confirmation in step 4., it assumes that the job has failed.
If the pipe gets broken after sending the confirmation, no further
action is necessary. If the child doesn’t receive the confirmation,
it will die and its death will be detected by Luxid eventually.</p>
<p>If the Luxi daemon dies before completing the procedure, the job will
not be started. If the job file contained the daemon’s lock file name,
it will be detected as dead (because the daemon process died). If the
job file already contained its proper lock file, it will also be
detected as dead (because the child process won’t start the actual job
and die).</p>
</div>
<div class="section" id="wconfd-details">
<h3>WConfD details<a class="headerlink" href="#wconfd-details" title="Permalink to this headline">¶</a></h3>
<p>WConfD will communicate with its clients through a Unix domain socket for both
configuration management and locking. Clients can issue multiple RPC calls
through one socket. For each such a call the client sends a JSON request
document with a remote function name and data for its arguments. The server
replies with a JSON response document containing either the result of
signalling a failure.</p>
<p>Any state associated with client processes will be mirrored on persistent
storage and linked to the identity of processes so that the WConfD daemon will
be able to resume its operation at any point after a restart or a crash. WConfD
will track each client’s process start time along with its process ID to be
able detect if a process dies and it’s process ID is reused.  WConfD will clear
all locks and other state associated with a client if it detects it’s process
no longer exists.</p>
<div class="section" id="configuration-management">
<h4>Configuration management<a class="headerlink" href="#configuration-management" title="Permalink to this headline">¶</a></h4>
<p>The new configuration management protocol will be implemented in the following
steps:</p>
<dl class="docutils">
<dt>Step 1:</dt>
<dd><ol class="first arabic">
<li><p class="first">Implement the following functions in WConfD and export them through
RPC:</p>
<ul class="simple">
<li>Obtain a single internal lock, either in shared or
exclusive mode. This lock will substitute the current lock
<code class="docutils literal"><span class="pre">_config_lock</span></code> in config.py.</li>
<li>Release the lock.</li>
<li>Return the whole configuration data to a client.</li>
<li>Receive the whole configuration data from a client and replace the
current configuration with it. Distribute it to master candidates
and distribute the corresponding <em>ssconf</em>.</li>
</ul>
<p>WConfD must detect deaths of its clients (see <a class="reference internal" href="#job-death-detection">Job death
detection</a>) and release locks automatically.</p>
</li>
<li><p class="first">In config.py modify public methods that access configuration:</p>
<ul class="simple">
<li>Instead of acquiring a local lock, obtain a lock from WConfD
using the above functions</li>
<li>Fetch the current configuration from WConfD.</li>
<li>Use it to perform the method’s task.</li>
<li>If the configuration was modified, send it to WConfD at the end.</li>
<li>Release the lock to WConfD.</li>
</ul>
</li>
</ol>
<p>This will decouple the configuration management from the master daemon,
even though the specific configuration tasks will still performed by
individual jobs.</p>
<p class="last">After this step it’ll be possible access the configuration from separate
processes.</p>
</dd>
<dt>Step 2:</dt>
<dd><ol class="first last arabic simple">
<li>Reimplement all current methods of <code class="docutils literal"><span class="pre">ConfigWriter</span></code> for reading and
writing the configuration of a cluster in WConfD.</li>
<li>Expose each of those functions in WConfD as a separate RPC function.
This will allow easy future extensions or modifications.</li>
<li>Replace <code class="docutils literal"><span class="pre">ConfigWriter</span></code> with a stub (preferably automatically
generated from the Haskell code) that will contain the same methods
as the current <code class="docutils literal"><span class="pre">ConfigWriter</span></code> and delegate all calls to its
methods to WConfD.</li>
</ol>
</dd>
<dt>Step 3:</dt>
<dd>In a later step, the impact of the config lock will be reduced by moving
it more and more into an internal detail of WConfD. This forthcoming process
of <a class="reference internal" href="design-configlock.html"><span class="doc">Removal of the Config Lock Overhead</span></a> is described separately.</dd>
</dl>
</div>
<div class="section" id="locking">
<h4>Locking<a class="headerlink" href="#locking" title="Permalink to this headline">¶</a></h4>
<p>The new locking protocol will be implemented as follows:</p>
<p>Re-implement the current locking mechanism in WConfD and expose it for RPC
calls. All current locks will be mapped into a data structure that will
uniquely identify them (storing lock’s level together with it’s name).</p>
<p>WConfD will impose a linear order on locks. The order will be compatible
with the current ordering of lock levels so that existing code will work
without changes.</p>
<p>WConfD will keep the set of currently held locks for each client. The
protocol will allow the following operations on the set:</p>
<dl class="docutils">
<dt><em>Update:</em></dt>
<dd>Update the current set of locks according to a given list. The list contains
locks and their desired level (release / shared / exclusive). To prevent
deadlocks, WConfD will check that all newly requested locks (or already held
locks requested to be upgraded to <em>exclusive</em>) are greater in the sense of
the linear order than all currently held locks, and fail the operation if
not. Only the locks in the list will be updated, other locks already held
will be left intact. If the operation fails, the client’s lock set will be
left intact.</dd>
<dt><em>Opportunistic union:</em></dt>
<dd>Add as much as possible locks from a given set to the current set within a
given timeout. WConfD will again check the proper order of locks and
acquire only the ones that are allowed wrt. the current set.  Returns the
set of acquired locks, possibly empty. Immediate. Never fails. (It would also
be possible to extend the operation to try to wait until a given number of
locks is available, or a given timeout elapses.)</dd>
<dt><em>List:</em></dt>
<dd>List the current set of held locks. Immediate, never fails.</dd>
<dt><em>Intersection:</em></dt>
<dd>Retain only a given set of locks in the current one. This function is
provided for convenience, it’s redundant wrt. <em>list</em> and <em>update</em>. Immediate,
never fails.</dd>
<dt>Additional restrictions due to lock implications:</dt>
<dd><p class="first">Ganeti supports locks that act as if a lock on a whole group (like all nodes)
were held. To avoid dead locks caused by the additional blockage of those
group locks, we impose certain restrictions. Whenever <cite>A</cite> is a group lock and
<cite>B</cite> belongs to <cite>A</cite>, then the following holds.</p>
<ul class="last simple">
<li><cite>A</cite> is in lock order before <cite>B</cite>.</li>
<li>All locks that are in the lock order between <cite>A</cite> and <cite>B</cite> also belong to <cite>A</cite>.</li>
<li>It is considered a lock-order violation to ask for an exclusive lock on <cite>B</cite>
while holding a shared lock on <cite>A</cite>.</li>
</ul>
</dd>
</dl>
<p>After this step it’ll be possible to use locks from jobs as separate processes.</p>
<p>The above set of operations allows the clients to use various work-flows. In particular:</p>
<dl class="docutils">
<dt>Pessimistic strategy:</dt>
<dd>Lock all potentially relevant resources (for example all nodes), determine
which will be needed, and release all the others.</dd>
<dt>Optimistic strategy:</dt>
<dd>Determine what locks need to be acquired without holding any. Lock the
required set of locks. Determine the set of required locks again and check if
they are all held. If not, release everything and restart.</dd>
</dl>
<p>Future aims:</p>
<ul class="simple">
<li>Add more fine-grained locks to prevent unnecessary blocking of jobs. This
could include locks on parameters of entities or locks on their states (so that
a node remains online, but otherwise can change, etc.). In particular,
adding, moving and removing instances currently blocks the whole node.</li>
<li>Add checks that all modified configuration parameters belong to entities
the client has locked and log violations.</li>
<li>Make the above checks mandatory.</li>
<li>Automate optimistic locking and checking the locks in logical units.
For example, this could be accomplished by allowing some of the initial
phases of <cite>LogicalUnit</cite> (such as <cite>ExpandNames</cite> and <cite>DeclareLocks</cite>) to be run
repeatedly, checking if the set of locks requested the second time is
contained in the set acquired after the first pass.</li>
<li>Add the possibility for a job to reserve hardware resources such as disk
space or memory on nodes. Most likely as a new, special kind of instances
that would only block its resources and allow to be converted to a regular
instance. This would allow long-running jobs such as instance creation or
move to lock the corresponding nodes, acquire the resources and turn the
locks into shared ones, keeping an exclusive lock only on the instance.</li>
<li>Use more sophisticated algorithm for preventing deadlocks such as a
<a class="reference external" href="http://en.wikipedia.org/wiki/Wait-for_graph">wait-for graph</a>. This would allow less <em>union</em> failures and allow more
optimistic, scalable acquisition of locks.</li>
</ul>
</div>
</div>
</div>
<div class="section" id="further-considerations">
<h2><a class="toc-backref" href="#id5">Further considerations</a><a class="headerlink" href="#further-considerations" title="Permalink to this headline">¶</a></h2>
<p>There is a possibility that a job will finish performing its task while LuxiD
and/or WConfD will not be available.
In order to deal with this situation, each job will update its job file
in the queue. This is race free, as LuxiD will no longer touch the job file,
once the job is started; a corollary of this is that the job also has to
take care of replicating updates to the job file. LuxiD will watch job files for
changes to determine when a job was cleanly finished. To determine jobs
that died without having the chance of updating the job file, the <a class="reference internal" href="#job-death-detection">Job death
detection</a> mechanism will be used.</p>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h3><a href="index.html">Table Of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Ganeti daemons refactoring</a><ul>
<li><a class="reference internal" href="#current-state-and-shortcomings">Current state and shortcomings</a></li>
<li><a class="reference internal" href="#proposed-changes">Proposed changes</a></li>
<li><a class="reference internal" href="#implementation">Implementation</a><ul>
<li><a class="reference internal" href="#job-death-detection">Job death detection</a></li>
<li><a class="reference internal" href="#job-execution">Job execution</a></li>
<li><a class="reference internal" href="#wconfd-details">WConfD details</a><ul>
<li><a class="reference internal" href="#configuration-management">Configuration management</a></li>
<li><a class="reference internal" href="#locking">Locking</a></li>
</ul>
</li>
</ul>
</li>
<li><a class="reference internal" href="#further-considerations">Further considerations</a></li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="design-device-uuid-name.html"
                        title="previous chapter">Design for adding UUID and name to devices</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="design-disk-conversion.html"
                        title="next chapter">Conversion between disk templates</a></p>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="_sources/design-daemons.rst.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <div><input type="text" name="q" /></div>
      <div><input type="submit" value="Go" /></div>
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="design-disk-conversion.html" title="Conversion between disk templates"
             >next</a></li>
        <li class="right" >
          <a href="design-device-uuid-name.html" title="Design for adding UUID and name to devices"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> &#187;</li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &#169; Copyright 2018, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Google Inc..
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
    </div>
  </body>
</html>
ganeti-doc 2.16.0~rc2-1build1 / usr / share / doc / ganeti / html / design-daemons.html
