/usr/share/doc/corosync/html/votequorum.5.html is in libvotequorum-dev 2.3.3-1ubuntu1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 | <!-- Creator : groff version 1.22.2 -->
<!-- CreationDate: Thu Mar 20 15:54:29 2014 -->
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<meta name="generator" content="groff -Thtml, see www.gnu.org">
<meta http-equiv="Content-Type" content="text/html; charset=US-ASCII">
<meta name="Content-Style" content="text/css">
<style type="text/css">
p { margin-top: 0; margin-bottom: 0; vertical-align: top }
pre { margin-top: 0; margin-bottom: 0; vertical-align: top }
table { margin-top: 0; margin-bottom: 0; vertical-align: top }
h1 { text-align: center }
</style>
<title>VOTEQUORUM</title>
</head>
<body>
<h1 align="center">VOTEQUORUM</h1>
<a href="#NAME">NAME</a><br>
<a href="#OVERVIEW">OVERVIEW</a><br>
<a href="#SPECIAL FEATURES">SPECIAL FEATURES</a><br>
<a href="#VARIOUS NOTES">VARIOUS NOTES</a><br>
<a href="#BUGS">BUGS</a><br>
<a href="#SEE ALSO">SEE ALSO</a><br>
<hr>
<h2>NAME
<a name="NAME"></a>
</h2>
<p style="margin-left:11%; margin-top: 1em">votequorum
− Votequorum Configuration Overview</p>
<h2>OVERVIEW
<a name="OVERVIEW"></a>
</h2>
<p style="margin-left:11%; margin-top: 1em">The votequorum
service is part of the corosync project. This service can be
optionally loaded into the nodes of a corosync cluster to
avoid split-brain situations. It does this by having a
number of votes assigned to each system in the cluster and
ensuring that only when a majority of the votes are present,
cluster operations are allowed to proceed. The service must
be loaded into all nodes or none. If it is loaded into a
subset of cluster nodes the results will be
unpredictable.</p>
<p style="margin-left:11%; margin-top: 1em">The following
corosync.conf extract will enable votequorum service within
corosync:</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">votequorum
reads its configuration from corosync.conf. Some values can
be changed at runtime, others are only read at corosync
startup. It is very important that those values are
consistent across all the nodes participating in the cluster
or votequorum behavior will be unpredictable.</p>
<p style="margin-left:11%; margin-top: 1em">votequorum
requires an expected_votes value to function, this can be
provided in two ways. The number of expected votes will be
automatically calculated when the nodelist { } section is
present in corosync.conf or expected_votes can be specified
in the quorum { } section. Lack of both will disable
votequorum. If both are present at the same time, the
quorum.expected_votes value will override the one calculated
from the nodelist.</p>
<p style="margin-left:11%; margin-top: 1em">Example (no
nodelist) of an 8 node cluster (each node has 1 vote):</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 8 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">Example (with
nodelist) of a 3 node cluster (each node has 1 vote):</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">nodelist { <br>
node { <br>
ring0_addr: 192.168.1.1 <br>
} <br>
node { <br>
ring0_addr: 192.168.1.2 <br>
} <br>
node { <br>
ring0_addr: 192.168.1.3 <br>
} <br>
}</p>
<h2>SPECIAL FEATURES
<a name="SPECIAL FEATURES"></a>
</h2>
<p style="margin-left:11%; margin-top: 1em"><b>two_node:
1</b></p>
<p style="margin-left:11%; margin-top: 1em">Enables two
node cluster operations (default: 0).</p>
<p style="margin-left:11%; margin-top: 1em">The "two
node cluster" is a use case that requires special
consideration. With a standard two node cluster, each node
with a single vote, there are 2 votes in the cluster. Using
the simple majority calculation (50% of the votes + 1) to
calculate quorum, the quorum would be 2. This means that the
both nodes would always have to be alive for the cluster to
be quorate and operate.</p>
<p style="margin-left:11%; margin-top: 1em">Enabling
two_node: 1, quorum is set artificially to 1.</p>
<p style="margin-left:11%; margin-top: 1em">Example
configuration 1:</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 2 <br>
two_node: 1 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">Example
configuration 2:</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
two_node: 1 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">nodelist { <br>
node { <br>
ring0_addr: 192.168.1.1 <br>
} <br>
node { <br>
ring0_addr: 192.168.1.2 <br>
} <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">NOTES: enabling
two_node: 1 automatically enables wait_for_all. It is still
possible to override wait_for_all by explicitly setting it
to 0. If more than 2 nodes join the cluster, the two_node
option is automatically disabled.</p>
<p style="margin-left:11%; margin-top: 1em"><b>wait_for_all:
1</b></p>
<p style="margin-left:11%; margin-top: 1em">Enables Wait
For All (WFA) feature (default: 0).</p>
<p style="margin-left:11%; margin-top: 1em">The general
behaviour of votequorum is to switch a cluster from
inquorate to quorate as soon as possible. For example, in an
8 node cluster, where every node has 1 vote, expected_votes
is set to 8 and quorum is (50% + 1) 5. As soon as 5 (or
more) nodes are visible to each other, the partition of 5
(or more) becomes quorate and can start operating.</p>
<p style="margin-left:11%; margin-top: 1em">When WFA is
enabled, the cluster will be quorate for the first time only
after all nodes have been visible at least once at the same
time.</p>
<p style="margin-left:11%; margin-top: 1em">This feature
has the advantage of avoiding some startup race conditions,
with the cost that all nodes need to be up at the same time
at least once before the cluster can operate.</p>
<p style="margin-left:11%; margin-top: 1em">A common
startup race condition based on the above example is that as
soon as 5 nodes become quorate, with the other 3 still
offline, the remaining 3 nodes will be fenced.</p>
<p style="margin-left:11%; margin-top: 1em">It is very
useful when combined with last_man_standing (see below).</p>
<p style="margin-left:11%; margin-top: 1em">Example
configuration:</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 8 <br>
wait_for_all: 1 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em"><b>last_man_standing:
1</b> / <b>last_man_standing_window: 10000</b></p>
<p style="margin-left:11%; margin-top: 1em">Enables Last
Man Standing (LMS) feature (default: 0). Tunable
last_man_standing_window (default: 10 seconds, expressed in
ms).</p>
<p style="margin-left:11%; margin-top: 1em">The general
behaviour of votequorum is to set expected_votes and quorum
at startup (unless modified by the user at runtime, see
below) and use those values during the whole lifetime of the
cluster.</p>
<p style="margin-left:11%; margin-top: 1em">Using for
example an 8 node cluster where each node has 1 vote,
expected_votes is set to 8 and quorum to 5. This condition
allows a total failure of 3 nodes. If a 4th node fails, the
cluster becomes inquorate and it will stop providing
services.</p>
<p style="margin-left:11%; margin-top: 1em">Enabling LMS
allows the cluster to dynamically recalculate expected_votes
and quorum under specific circumstances. It is essential to
enable WFA when using LMS in High Availability clusters.</p>
<p style="margin-left:11%; margin-top: 1em">Using the above
8 node cluster example, with LMS enabled the cluster can
retain quorum and continue operating by losing, in a cascade
fashion, up to 6 nodes with only 2 remaining active.</p>
<p style="margin-left:11%; margin-top: 1em">Example chain
of events: <br>
1) cluster is fully operational with 8 nodes. <br>
(expected_votes: 8 quorum: 5)</p>
<p style="margin-left:11%; margin-top: 1em">2) 3 nodes die,
cluster is quorate with 5 nodes.</p>
<p style="margin-left:11%; margin-top: 1em">3) after
last_man_standing_window timer expires, <br>
expected_votes and quorum are recalculated. <br>
(expected_votes: 5 quorum: 3)</p>
<p style="margin-left:11%; margin-top: 1em">4) at this
point, 2 more nodes can die and <br>
cluster will still be quorate with 3.</p>
<p style="margin-left:11%; margin-top: 1em">5) once again,
after last_man_standing_window <br>
timer expires expected_votes and quorum are <br>
recalculated. <br>
(expected_votes: 3 quorum: 2)</p>
<p style="margin-left:11%; margin-top: 1em">6) at this
point, 1 more node can die and <br>
cluster will still be quorate with 2.</p>
<p style="margin-left:11%; margin-top: 1em">7) one more
last_man_standing_window timer <br>
(expected_votes: 2 quorum: 2)</p>
<p style="margin-left:11%; margin-top: 1em">NOTES: In order
for the cluster to downgrade automatically from 2 nodes to a
1 node cluster, the auto_tie_breaker feature must also be
enabled (see below). If auto_tie_breaker is not enabled, and
one more failure occours, the remaining node will not be
quorate. LMS does not work with asymmetric voting schemes,
each node must vote 1.</p>
<p style="margin-left:11%; margin-top: 1em">Example
configuration 1:</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 8 <br>
last_man_standing: 1 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">Example
configuration 2 (increase timeout to 20 seconds):</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 8 <br>
last_man_standing: 1 <br>
last_man_standing_window: 20000 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em"><b>auto_tie_breaker:
1</b></p>
<p style="margin-left:11%; margin-top: 1em">Enables Auto
Tie Breaker (ATB) feature (default: 0).</p>
<p style="margin-left:11%; margin-top: 1em">The general
behaviour of votequorum allows a simultaneous node failure
up to 50% - 1 node, assuming each node has 1 vote.</p>
<p style="margin-left:11%; margin-top: 1em">When ATB is
enabled, the cluster can suffer up to 50% of the nodes
failing at the same time, in a deterministic fashion. The
cluster partition, or the set of nodes that are still in
contact with the node that has the lowest nodeid will remain
quorate. The other nodes will be inquorate.</p>
<p style="margin-left:11%; margin-top: 1em">Example
configuration 1:</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 8 <br>
auto_tie_breaker: 1 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em"><b>allow_downscale:
1</b></p>
<p style="margin-left:11%; margin-top: 1em">Enables allow
downscale (AD) feature (default: 0).</p>
<p style="margin-left:11%; margin-top: 1em">THIS FEATURE IS
INCOMPLETE AND CURRENTLY UNSUPPORTED.</p>
<p style="margin-left:11%; margin-top: 1em">The general
behaviour of votequorum is to never decrease expected votes
or quorum.</p>
<p style="margin-left:11%; margin-top: 1em">When AD is
enabled, both expected votes and quorum are recalculated
when a node leaves the cluster in a clean state (normal
corosync shutdown process) down to configured
expected_votes.</p>
<p style="margin-left:11%; margin-top: 1em">Example use
case:</p>
<p style="margin-left:11%; margin-top: 1em">1) N node
cluster (where N is any value higher than 3)</p>
<p style="margin-left:11%; margin-top: 1em">2)
expected_votes set to 3 in corosync.conf</p>
<p style="margin-left:11%; margin-top: 1em">3) only 3 nodes
are running</p>
<p style="margin-left:11%; margin-top: 1em">4) admin
requires to increase processing power and adds 10 nodes</p>
<p style="margin-left:11%; margin-top: 1em">5) internal
expected_votes is automatically set to 13</p>
<p style="margin-left:11%; margin-top: 1em">6) minimum
expected_votes is 3 (from configuration)</p>
<p style="margin-left:11%; margin-top: 1em">- up to this
point this is standard votequorum behavior -</p>
<p style="margin-left:11%; margin-top: 1em">7) once the
work is done, admin wants to remove nodes from the
cluster</p>
<p style="margin-left:11%; margin-top: 1em">8) using an
ordered shutdown the admin can reduce the cluster size <br>
automatically back to 3, but not below 3, where normal
quorum <br>
operation will work as usual.</p>
<p style="margin-left:11%; margin-top: 1em">Example
configuration:</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 3 <br>
allow_downscale: 1 <br>
} <br>
allow_downscale implicitly enabled EVT (see below).</p>
<p style="margin-left:11%; margin-top: 1em"><b>expected_votes_tracking:
1</b></p>
<p style="margin-left:11%; margin-top: 1em">Enables
Expected Votes Tracking (EVT) feature (default: 0).</p>
<p style="margin-left:11%; margin-top: 1em">Expected Votes
Tracking stores the highest-seen value of expected votes on
disk and uses that as the minimum value for expected votes
in the absence of any higher authority (eg a current quorate
cluster). This is useful for when a group of nodes becomes
detached from the main cluster and after a restart could
have enough votes to provide quorum, which can happen after
using allow_downscale.</p>
<p style="margin-left:11%; margin-top: 1em">Note that even
if the in-memory version of expected_votes is reduced, eg by
removing nodes or using corosync-quorumtool, the stored
value will still be the highest value seen - it never gets
reduced.</p>
<p style="margin-left:11%; margin-top: 1em">The value is
held in the file /var/lib/corosync/ev_tracking which can be
deleted if you really do need to reduce the expected votes
for any reason, like the node has been moved to a different
cluster.</p>
<h2>VARIOUS NOTES
<a name="VARIOUS NOTES"></a>
</h2>
<p style="margin-left:11%; margin-top: 1em">* WFA / LMS /
ATB / AD can be used combined together.</p>
<p style="margin-left:11%; margin-top: 1em">* In order to
change the default votes for a node there are two
options:</p>
<p style="margin-left:11%; margin-top: 1em">1)
nodelist:</p>
<p style="margin-left:11%; margin-top: 1em">nodelist { <br>
node { <br>
ring0_addr: 192.168.1.1 <br>
quorum_votes: 3 <br>
} <br>
.... <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">2) quorum
section (deprecated):</p>
<p style="margin-left:11%; margin-top: 1em">quorum { <br>
provider: corosync_votequorum <br>
expected_votes: 2 <br>
votes: 2 <br>
}</p>
<p style="margin-left:11%; margin-top: 1em">In the event
that both nodelist and quorum { votes: } are defined, the
value from the nodelist will be used.</p>
<p style="margin-left:11%; margin-top: 1em">* Only votes,
quorum_votes, expected_votes and two_node can be changed at
runtime. Everything else requires a cluster restart.</p>
<h2>BUGS
<a name="BUGS"></a>
</h2>
<p style="margin-left:11%; margin-top: 1em">No known bugs
at the time of writing. The authors are from outerspace.
Deal with it.</p>
<h2>SEE ALSO
<a name="SEE ALSO"></a>
</h2>
<p style="margin-left:11%; margin-top: 1em"><b>corosync</b>(8),
<b>corosync.conf</b>(5), <b>corosync-quorumtool</b>(8),
<b>votequorum_overview</b>(8)</p>
<hr>
</body>
</html>
|