This file is indexed.

/usr/share/doc/ganeti/html/design-cpu-pinning.html is in ganeti-doc 2.16.0~rc2-1build1.

This file is owned by root:root, with mode 0o644.

The actual contents of the file can be viewed below.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml" lang="en">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    <title>Ganeti CPU Pinning &#8212; Ganeti 2.16.0~rc2 documentation</title>
    <link rel="stylesheet" href="_static/style.css" type="text/css" />
    <link rel="stylesheet" href="_static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    './',
        VERSION:     '2.16.0~rc2',
        COLLAPSE_INDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true,
        SOURCELINK_SUFFIX: '.txt'
      };
    </script>
    <script type="text/javascript" src="_static/jquery.js"></script>
    <script type="text/javascript" src="_static/underscore.js"></script>
    <script type="text/javascript" src="_static/doctools.js"></script>
    <link rel="search" title="Search" href="search.html" />
    <link rel="next" title="Allocation for Partitioned Ganeti" href="design-dedicated-allocation.html" />
    <link rel="prev" title="Taking relative CPU speed into account" href="design-cpu-speed.html" /> 
  </head>
  <body>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="design-dedicated-allocation.html" title="Allocation for Partitioned Ganeti"
             accesskey="N">next</a></li>
        <li class="right" >
          <a href="design-cpu-speed.html" title="Taking relative CPU speed into account"
             accesskey="P">previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> &#187;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body" role="main">
            
  <div class="section" id="ganeti-cpu-pinning">
<h1>Ganeti CPU Pinning<a class="headerlink" href="#ganeti-cpu-pinning" title="Permalink to this headline"></a></h1>
<table class="docutils field-list" frame="void" rules="none">
<col class="field-name" />
<col class="field-body" />
<tbody valign="top">
<tr class="field-odd field"><th class="field-name">Created:</th><td class="field-body">2011-May-28</td>
</tr>
<tr class="field-even field"><th class="field-name">Status:</th><td class="field-body">Implemented</td>
</tr>
<tr class="field-odd field"><th class="field-name">Ganeti-Version:</th><td class="field-body">2.6.0</td>
</tr>
</tbody>
</table>
<div class="section" id="objective">
<h2>Objective<a class="headerlink" href="#objective" title="Permalink to this headline"></a></h2>
<p>This document defines Ganeti’s support for CPU pinning (aka CPU
affinity).</p>
<p>CPU pinning enables mapping and unmapping entire virtual machines or a
specific virtual CPU (vCPU), to a physical CPU or a range of CPUs.</p>
<p>At this stage Pinning will be implemented for Xen and KVM.</p>
</div>
<div class="section" id="command-line">
<h2>Command Line<a class="headerlink" href="#command-line" title="Permalink to this headline"></a></h2>
<p>Suggested command line parameters for controlling CPU pinning are as
follows:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=&lt;</span><span class="n">cpu</span><span class="o">-</span><span class="n">pinning</span><span class="o">-</span><span class="n">info</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">instance</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>cpu-pinning-info can be any of the following:</p>
<ul>
<li><p class="first">One vCPU mapping, which can be the word “all” or a combination
of CPU numbers and ranges separated by comma. In this case, all
vCPUs will be mapped to the indicated list.</p>
</li>
<li><p class="first">A list of vCPU mappings, separated by a colon ‘:’. In this case
each vCPU is mapped to an entry in the list, and the size of the
list must match the number of vCPUs defined for the instance. This
is enforced when setting CPU pinning or when setting the number of
vCPUs using <code class="docutils literal"><span class="pre">-B</span> <span class="pre">vcpus=#</span></code>.</p>
<p>The mapping list is matched to consecutive virtual CPUs, so the first entry
would be the CPU pinning information for vCPU 0, the second entry
for vCPU 1, etc.</p>
</li>
</ul>
<p>The default setting for new instances is “all”, which maps the entire
instance to all CPUs, thus effectively turning off CPU pinning.</p>
<p>Here are some usage examples:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># Map vCPU 0 to physical CPU 1 and vCPU 1 to CPU 3 (assuming 2 vCPUs)</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=</span><span class="mi">1</span><span class="p">:</span><span class="mi">3</span> <span class="n">my</span><span class="o">-</span><span class="n">inst</span>

<span class="c1"># Pin vCPU 0 to CPUs 1 or 2, and vCPU 1 to any CPU</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=</span><span class="mi">1</span><span class="o">-</span><span class="mi">2</span><span class="p">:</span><span class="nb">all</span> <span class="n">my</span><span class="o">-</span><span class="n">inst</span>

<span class="c1"># Pin vCPU 0 to any CPU, vCPU 1 to CPUs 1, 3, 4 or 5, and CPU 2 to</span>
<span class="c1"># CPU 0</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=</span><span class="nb">all</span><span class="p">:</span><span class="mi">1</span>\\<span class="p">,</span><span class="mi">3</span><span class="o">-</span><span class="mi">5</span><span class="p">:</span><span class="mi">0</span> <span class="n">my</span><span class="o">-</span><span class="n">inst</span>

<span class="c1"># Pin entire VM to CPU 0</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=</span><span class="mi">0</span> <span class="n">my</span><span class="o">-</span><span class="n">inst</span>

<span class="c1"># Turn off CPU pinning (default setting)</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=</span><span class="nb">all</span> <span class="n">my</span><span class="o">-</span><span class="n">inst</span>
</pre></div>
</div>
<p>Assuming an instance has 3 vCPUs, the following commands will fail:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># not enough mappings</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=</span><span class="mi">0</span><span class="p">:</span><span class="mi">1</span> <span class="n">my</span><span class="o">-</span><span class="n">inst</span>

<span class="c1"># too many</span>
<span class="n">gnt</span><span class="o">-</span><span class="n">instance</span> <span class="n">modify</span> <span class="o">-</span><span class="n">H</span> <span class="n">cpu_mask</span><span class="o">=</span><span class="mi">2</span><span class="p">:</span><span class="mi">1</span><span class="p">:</span><span class="mi">1</span><span class="p">:</span><span class="nb">all</span> <span class="n">my</span><span class="o">-</span><span class="n">inst</span>
</pre></div>
</div>
</div>
<div class="section" id="validation">
<h2>Validation<a class="headerlink" href="#validation" title="Permalink to this headline"></a></h2>
<p>CPU pinning information is validated by making sure it matches the
number of vCPUs. This validation happens when changing either the
cpu_mask or vcpus parameters.
Changing either parameter in a way that conflicts with the other will
fail with a proper error message.
To make such a change, both parameters should be modified at the same
time. For example:
<code class="docutils literal"><span class="pre">gnt-instance</span> <span class="pre">modify</span> <span class="pre">-B</span> <span class="pre">vcpus=4</span> <span class="pre">-H</span> <span class="pre">cpu_mask=1:1:2-3:4\\,6</span> <span class="pre">my-inst</span></code></p>
<p>Besides validating CPU configuration, i.e. the number of vCPUs matches
the requested CPU pinning, Ganeti will also verify the number of
physical CPUs is enough to support the required configuration. For
example, trying to run a configuration of vcpus=2,cpu_mask=0:4 on
a node with 4 cores will fail (Note: CPU numbers are 0-based).</p>
<p>This validation should repeat every time an instance is started or
migrated live. See more details under Migration below.</p>
<p>Cluster verification should also test the compatibility of other nodes in
the cluster to required configuration and alert if a minimum requirement
is not met.</p>
</div>
<div class="section" id="failover">
<h2>Failover<a class="headerlink" href="#failover" title="Permalink to this headline"></a></h2>
<p>CPU pinning configuration can be transferred from node to node, unless
the number of physical CPUs is smaller than what the configuration calls
for.  It is suggested that unless this is the case, all transfers and
migrations will succeed.</p>
<p>In case the number of physical CPUs is smaller than the numbers
indicated by CPU pinning information, instance failover will fail.</p>
<p>In case of emergency, to force failover to ignore mismatching CPU
information, the following switch can be used:
<code class="docutils literal"><span class="pre">gnt-instance</span> <span class="pre">failover</span> <span class="pre">--fix-cpu-mismatch</span> <span class="pre">my-inst</span></code>.
This command will try to failover the instance with the current cpu mask,
but if that fails, it will change the mask to be “all”.</p>
</div>
<div class="section" id="migration">
<h2>Migration<a class="headerlink" href="#migration" title="Permalink to this headline"></a></h2>
<p>In case of live migration, and in addition to failover considerations,
it is required to remap CPU pinning after migration. This can be done in
realtime for instances for both Xen and KVM, and only depends on the
number of physical CPUs being sufficient to support the migrated
instance.</p>
</div>
<div class="section" id="data">
<h2>Data<a class="headerlink" href="#data" title="Permalink to this headline"></a></h2>
<p>Pinning information will be kept as a list of integers per vCPU.
To mark a mapping of any CPU, we will use (-1).
A single entry, no matter what the number of vCPUs is, will always mean
that all vCPUs have the same mapping.</p>
</div>
<div class="section" id="configuration-file">
<h2>Configuration file<a class="headerlink" href="#configuration-file" title="Permalink to this headline"></a></h2>
<p>The pinning information is kept for each instance’s hypervisor
params section of the configuration file as the original string.</p>
</div>
<div class="section" id="xen">
<h2>Xen<a class="headerlink" href="#xen" title="Permalink to this headline"></a></h2>
<p>There are 2 ways to control pinning in Xen, either via the command line
or through the configuration file.</p>
<p>The commands to make direct pinning changes are the following:</p>
<div class="highlight-default"><div class="highlight"><pre><span></span><span class="c1"># To pin a vCPU to a specific CPU</span>
<span class="n">xm</span> <span class="n">vcpu</span><span class="o">-</span><span class="n">pin</span> <span class="o">&lt;</span><span class="n">domain</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">vcpu</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">cpu</span><span class="o">&gt;</span>

<span class="c1"># To unpin a vCPU</span>
<span class="n">xm</span> <span class="n">vcpu</span><span class="o">-</span><span class="n">pin</span> <span class="o">&lt;</span><span class="n">domain</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">vcpu</span><span class="o">&gt;</span> <span class="nb">all</span>

<span class="c1"># To get the current pinning status</span>
<span class="n">xm</span> <span class="n">vcpu</span><span class="o">-</span><span class="nb">list</span> <span class="o">&lt;</span><span class="n">domain</span><span class="o">&gt;</span>
</pre></div>
</div>
<p>Since currently controlling Xen in Ganeti is done in the configuration
file, it is straight forward to use the same method for CPU pinning.
There are 2 different parameters that control Xen’s CPU pinning and
configuration:</p>
<dl class="docutils">
<dt>vcpus</dt>
<dd>controls the number of vCPUs</dd>
<dt>cpus</dt>
<dd>maps vCPUs to physical CPUs</dd>
</dl>
<p>When no pinning is required (pinning information is “all”), the
“cpus” entry is removed from the configuration file.</p>
<p>For all other cases, the configuration is “translated” to Xen, which
expects either <code class="docutils literal"><span class="pre">cpus</span> <span class="pre">=</span> <span class="pre">&quot;a&quot;</span></code> or <code class="docutils literal"><span class="pre">cpus</span> <span class="pre">=</span> <span class="pre">[</span> <span class="pre">&quot;a&quot;,</span> <span class="pre">&quot;b&quot;,</span> <span class="pre">&quot;c&quot;,</span> <span class="pre">...]</span></code>,
where each a, b or c are a physical CPU number, CPU range, or a
combination, and the number of entries (if a list is used) must match
the number of vCPUs, and are mapped in order.</p>
<p>For example, CPU pinning information of <code class="docutils literal"><span class="pre">1:2,4-7:0-1</span></code> is translated
to this entry in Xen’s configuration <code class="docutils literal"><span class="pre">cpus</span> <span class="pre">=</span> <span class="pre">[</span> <span class="pre">&quot;1&quot;,</span> <span class="pre">&quot;2,4-7&quot;,</span> <span class="pre">&quot;0-1&quot;</span> <span class="pre">]</span></code></p>
</div>
<div class="section" id="kvm">
<h2>KVM<a class="headerlink" href="#kvm" title="Permalink to this headline"></a></h2>
<p>Controlling pinning in KVM is a little more complicated as there is no
configuration to control pinning before instances are started.</p>
<p>The way to change or assign CPU pinning under KVM is to use <code class="docutils literal"><span class="pre">taskset</span></code> or
its underlying system call <code class="docutils literal"><span class="pre">sched_setaffinity</span></code>. Setting the affinity for
the VM process will change CPU pinning for the entire VM, and setting it
for specific vCPU threads will control specific vCPUs.</p>
<p>The sequence of commands to control pinning is this: start the instance
with the <code class="docutils literal"><span class="pre">-S</span></code> switch, so it halts before starting execution, get the
process ID or identify thread IDs of each vCPU by sending <code class="docutils literal"><span class="pre">info</span> <span class="pre">cpus</span></code>
to the monitor, map vCPUs as required by the cpu-pinning information,
and issue a <code class="docutils literal"><span class="pre">cont</span></code> command on the KVM monitor to allow the instance
to start execution.</p>
<p>For example, a sequence of commands to control CPU affinity under KVM
may be:</p>
<ul class="simple">
<li>Start KVM: <code class="docutils literal"><span class="pre">/usr/bin/kvm</span> <span class="pre"></span> <span class="pre">&lt;kvm-command-line-options&gt;</span> <span class="pre"></span> <span class="pre">-S</span></code></li>
<li>Use socat to connect to monitor</li>
<li>send <code class="docutils literal"><span class="pre">info</span> <span class="pre">cpus</span></code> to monitor to get thread/vCPU information</li>
<li>call <code class="docutils literal"><span class="pre">sched_setaffinity</span></code> for each thread with the CPU mask</li>
<li>send <code class="docutils literal"><span class="pre">cont</span></code> to KVM’s monitor</li>
</ul>
<p>A CPU mask is a hexadecimal bit mask where each bit represents one
physical CPU. See man page for <em class="manpage">sched_setaffinity(2)</em> for more
details.</p>
<p>For example, to run a specific thread-id on CPUs 1 or 3 the mask is
0x0000000A.</p>
<p>As of 2.12, the psutil python package
(<a class="reference external" href="https://github.com/giampaolo/psutil">https://github.com/giampaolo/psutil</a>) will be used to control process
and thread affinity. The affinity python package
(<a class="reference external" href="http://pypi.python.org/pypi/affinity">http://pypi.python.org/pypi/affinity</a>) was used before, but it was not
invoking the two underlying system calls appropriately, using a cast
instead of the CPU_SET macro, causing failures for masks referencing
more than 63 CPUs.</p>
</div>
<div class="section" id="alternative-design-options">
<h2>Alternative Design Options<a class="headerlink" href="#alternative-design-options" title="Permalink to this headline"></a></h2>
<ol class="arabic simple">
<li>There’s an option to ignore the limitations of the underlying
hypervisor and instead of requiring explicit pinning information
for <em>all</em> vCPUs, assume a mapping of “all” to vCPUs not mentioned.
This can lead to inadvertent missing information, but either way,
since using cpu-pinning options is probably not going to be
frequent, there’s no real advantage.</li>
</ol>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar" role="navigation" aria-label="main navigation">
        <div class="sphinxsidebarwrapper">
  <h3><a href="index.html">Table Of Contents</a></h3>
  <ul>
<li><a class="reference internal" href="#">Ganeti CPU Pinning</a><ul>
<li><a class="reference internal" href="#objective">Objective</a></li>
<li><a class="reference internal" href="#command-line">Command Line</a></li>
<li><a class="reference internal" href="#validation">Validation</a></li>
<li><a class="reference internal" href="#failover">Failover</a></li>
<li><a class="reference internal" href="#migration">Migration</a></li>
<li><a class="reference internal" href="#data">Data</a></li>
<li><a class="reference internal" href="#configuration-file">Configuration file</a></li>
<li><a class="reference internal" href="#xen">Xen</a></li>
<li><a class="reference internal" href="#kvm">KVM</a></li>
<li><a class="reference internal" href="#alternative-design-options">Alternative Design Options</a></li>
</ul>
</li>
</ul>

  <h4>Previous topic</h4>
  <p class="topless"><a href="design-cpu-speed.html"
                        title="previous chapter">Taking relative CPU speed into account</a></p>
  <h4>Next topic</h4>
  <p class="topless"><a href="design-dedicated-allocation.html"
                        title="next chapter">Allocation for Partitioned Ganeti</a></p>
  <div role="note" aria-label="source link">
    <h3>This Page</h3>
    <ul class="this-page-menu">
      <li><a href="_sources/design-cpu-pinning.rst.txt"
            rel="nofollow">Show Source</a></li>
    </ul>
   </div>
<div id="searchbox" style="display: none" role="search">
  <h3>Quick search</h3>
    <form class="search" action="search.html" method="get">
      <div><input type="text" name="q" /></div>
      <div><input type="submit" value="Go" /></div>
      <input type="hidden" name="check_keywords" value="yes" />
      <input type="hidden" name="area" value="default" />
    </form>
</div>
<script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related" role="navigation" aria-label="related navigation">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="design-dedicated-allocation.html" title="Allocation for Partitioned Ganeti"
             >next</a></li>
        <li class="right" >
          <a href="design-cpu-speed.html" title="Taking relative CPU speed into account"
             >previous</a> |</li>
        <li class="nav-item nav-item-0"><a href="index.html">Ganeti 2.16.0~rc2 documentation</a> &#187;</li> 
      </ul>
    </div>
    <div class="footer" role="contentinfo">
        &#169; Copyright 2018, 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015 Google Inc..
      Created using <a href="http://sphinx-doc.org/">Sphinx</a> 1.6.7.
    </div>
  </body>
</html>