/usr/share/doc/xapian-omega/termprefixes.html

<?xml version="1.0" encoding="utf-8" ?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.14: http://docutils.sourceforge.net/" />
<title>Term Prefixes</title>
<style type="text/css">

/*
:Author: David Goodger (goodger@python.org)
:Id: $Id: html4css1.css 7952 2016-07-26 18:15:59Z milde $
:Copyright: This stylesheet has been placed in the public domain.

Default cascading style sheet for the HTML output of Docutils.

See http://docutils.sf.net/docs/howto/html-stylesheets.html for how to
customize this style sheet.
*/

/* used to remove borders from tables and images */
.borderless, table.borderless td, table.borderless th {
  border: 0 }

table.borderless td, table.borderless th {
  /* Override padding for "table.docutils td" with "! important".
     The right padding separates the table cells. */
  padding: 0 0.5em 0 0 ! important }

.first {
  /* Override more specific margin styles with "! important". */
  margin-top: 0 ! important }

.last, .with-subtitle {
  margin-bottom: 0 ! important }

.hidden {
  display: none }

.subscript {
  vertical-align: sub;
  font-size: smaller }

.superscript {
  vertical-align: super;
  font-size: smaller }

a.toc-backref {
  text-decoration: none ;
  color: black }

blockquote.epigraph {
  margin: 2em 5em ; }

dl.docutils dd {
  margin-bottom: 0.5em }

object[type="image/svg+xml"], object[type="application/x-shockwave-flash"] {
  overflow: hidden;
}

/* Uncomment (and remove this text!) to get bold-faced definition list terms
dl.docutils dt {
  font-weight: bold }
*/

div.abstract {
  margin: 2em 5em }

div.abstract p.topic-title {
  font-weight: bold ;
  text-align: center }

div.admonition, div.attention, div.caution, div.danger, div.error,
div.hint, div.important, div.note, div.tip, div.warning {
  margin: 2em ;
  border: medium outset ;
  padding: 1em }

div.admonition p.admonition-title, div.hint p.admonition-title,
div.important p.admonition-title, div.note p.admonition-title,
div.tip p.admonition-title {
  font-weight: bold ;
  font-family: sans-serif }

div.attention p.admonition-title, div.caution p.admonition-title,
div.danger p.admonition-title, div.error p.admonition-title,
div.warning p.admonition-title, .code .error {
  color: red ;
  font-weight: bold ;
  font-family: sans-serif }

/* Uncomment (and remove this text!) to get reduced vertical space in
   compound paragraphs.
div.compound .compound-first, div.compound .compound-middle {
  margin-bottom: 0.5em }

div.compound .compound-last, div.compound .compound-middle {
  margin-top: 0.5em }
*/

div.dedication {
  margin: 2em 5em ;
  text-align: center ;
  font-style: italic }

div.dedication p.topic-title {
  font-weight: bold ;
  font-style: normal }

div.figure {
  margin-left: 2em ;
  margin-right: 2em }

div.footer, div.header {
  clear: both;
  font-size: smaller }

div.line-block {
  display: block ;
  margin-top: 1em ;
  margin-bottom: 1em }

div.line-block div.line-block {
  margin-top: 0 ;
  margin-bottom: 0 ;
  margin-left: 1.5em }

div.sidebar {
  margin: 0 0 0.5em 1em ;
  border: medium outset ;
  padding: 1em ;
  background-color: #ffffee ;
  width: 40% ;
  float: right ;
  clear: right }

div.sidebar p.rubric {
  font-family: sans-serif ;
  font-size: medium }

div.system-messages {
  margin: 5em }

div.system-messages h1 {
  color: red }

div.system-message {
  border: medium outset ;
  padding: 1em }

div.system-message p.system-message-title {
  color: red ;
  font-weight: bold }

div.topic {
  margin: 2em }

h1.section-subtitle, h2.section-subtitle, h3.section-subtitle,
h4.section-subtitle, h5.section-subtitle, h6.section-subtitle {
  margin-top: 0.4em }

h1.title {
  text-align: center }

h2.subtitle {
  text-align: center }

hr.docutils {
  width: 75% }

img.align-left, .figure.align-left, object.align-left, table.align-left {
  clear: left ;
  float: left ;
  margin-right: 1em }

img.align-right, .figure.align-right, object.align-right, table.align-right {
  clear: right ;
  float: right ;
  margin-left: 1em }

img.align-center, .figure.align-center, object.align-center {
  display: block;
  margin-left: auto;
  margin-right: auto;
}

table.align-center {
  margin-left: auto;
  margin-right: auto;
}

.align-left {
  text-align: left }

.align-center {
  clear: both ;
  text-align: center }

.align-right {
  text-align: right }

/* reset inner alignment in figures */
div.align-right {
  text-align: inherit }

/* div.align-center * { */
/*   text-align: left } */

.align-top    {
  vertical-align: top }

.align-middle {
  vertical-align: middle }

.align-bottom {
  vertical-align: bottom }

ol.simple, ul.simple {
  margin-bottom: 1em }

ol.arabic {
  list-style: decimal }

ol.loweralpha {
  list-style: lower-alpha }

ol.upperalpha {
  list-style: upper-alpha }

ol.lowerroman {
  list-style: lower-roman }

ol.upperroman {
  list-style: upper-roman }

p.attribution {
  text-align: right ;
  margin-left: 50% }

p.caption {
  font-style: italic }

p.credits {
  font-style: italic ;
  font-size: smaller }

p.label {
  white-space: nowrap }

p.rubric {
  font-weight: bold ;
  font-size: larger ;
  color: maroon ;
  text-align: center }

p.sidebar-title {
  font-family: sans-serif ;
  font-weight: bold ;
  font-size: larger }

p.sidebar-subtitle {
  font-family: sans-serif ;
  font-weight: bold }

p.topic-title {
  font-weight: bold }

pre.address {
  margin-bottom: 0 ;
  margin-top: 0 ;
  font: inherit }

pre.literal-block, pre.doctest-block, pre.math, pre.code {
  margin-left: 2em ;
  margin-right: 2em }

pre.code .ln { color: grey; } /* line numbers */
pre.code, code { background-color: #eeeeee }
pre.code .comment, code .comment { color: #5C6576 }
pre.code .keyword, code .keyword { color: #3B0D06; font-weight: bold }
pre.code .literal.string, code .literal.string { color: #0C5404 }
pre.code .name.builtin, code .name.builtin { color: #352B84 }
pre.code .deleted, code .deleted { background-color: #DEB0A1}
pre.code .inserted, code .inserted { background-color: #A3D289}

span.classifier {
  font-family: sans-serif ;
  font-style: oblique }

span.classifier-delimiter {
  font-family: sans-serif ;
  font-weight: bold }

span.interpreted {
  font-family: sans-serif }

span.option {
  white-space: nowrap }

span.pre {
  white-space: pre }

span.problematic {
  color: red }

span.section-subtitle {
  /* font-size relative to parent (h1..h6 element) */
  font-size: 80% }

table.citation {
  border-left: solid 1px gray;
  margin-left: 1px }

table.docinfo {
  margin: 2em 4em }

table.docutils {
  margin-top: 0.5em ;
  margin-bottom: 0.5em }

table.footnote {
  border-left: solid 1px black;
  margin-left: 1px }

table.docutils td, table.docutils th,
table.docinfo td, table.docinfo th {
  padding-left: 0.5em ;
  padding-right: 0.5em ;
  vertical-align: top }

table.docutils th.field-name, table.docinfo th.docinfo-name {
  font-weight: bold ;
  text-align: left ;
  white-space: nowrap ;
  padding-left: 0 }

/* "booktabs" style (no vertical lines) */
table.docutils.booktabs {
  border: 0px;
  border-top: 2px solid;
  border-bottom: 2px solid;
  border-collapse: collapse;
}
table.docutils.booktabs * {
  border: 0px;
}
table.docutils.booktabs th {
  border-bottom: thin solid;
  text-align: left;
}

h1 tt.docutils, h2 tt.docutils, h3 tt.docutils,
h4 tt.docutils, h5 tt.docutils, h6 tt.docutils {
  font-size: 100% }

ul.auto-toc {
  list-style-type: none }

</style>
</head>
<body>
<div class="document" id="term-prefixes">
<h1 class="title">Term Prefixes</h1>

<p>Xapian itself doesn't put any restrictions on the contents of a term, other
than that terms can't be empty, and there's an upper limit on the length
(which is backend dependent - chert and glass allow 245 bytes, except
that zero bytes count double in this length).</p>
<p>However, Omega and <tt class="docutils literal"><span class="pre">Xapian::QueryParser</span></tt> impose some rules to aid
interoperability and make it easier to write code that doesn't require
excessive configuring.  It's probably wise to follow these rules unless
you have a good reason not to.  Right now you might not intend to use Omega
or the QueryParser, not to combine a search with another database.  But if
you later find you do, it'll be much easier if you're using compatible
rules!</p>
<p>The basic idea is that terms won't begin with a capital letter (since they're
usually lower-cased and often stemmed), so any term which starts with a capital
letter is assumed to have a prefix.  For all letters apart from X, this is a
single character prefix and these have predefined standard meanings (or are
reserved for standard meanings but currently unallocated).</p>
<p>X starts a multi-capital letter user-defined prefix.  If you want a prefix for
something without a standard prefix, you create your own starting with an X
(e.g. XSHOESIZE).  The prefix ends with the first non-capital.  If the term
you're prefixing starts with a capital, add a &quot;:&quot; between prefix and term to
resolve ambiguity about where the prefix ends and the term begins.</p>
<p>Here's the current allocation list:</p>
<dl class="docutils">
<dt>A</dt>
<dd>Author</dd>
<dt>B</dt>
<dd>Topic (mnemonic: what the document is aBout)</dd>
<dt>D</dt>
<dd>Date (numeric format: YYYYMMDD or &quot;latest&quot; - e.g. D20050224 or Dlatest)</dd>
<dt>E</dt>
<dd>Extension (folded to lowercase - e.g. Ehtml, or E for no extension)</dd>
<dt>F</dt>
<dd>Filename</dd>
<dt>G</dt>
<dd>newsGroup (or similar entity - e.g. a web forum name)</dd>
<dt>H</dt>
<dd>Hostname</dd>
<dt>I</dt>
<dd>boolean filter term for &quot;can see&quot; permission (mnemonic: Include)</dd>
<dt>J</dt>
<dd>Site term (mnemonic: Jumping off point)</dd>
<dt>K</dt>
<dd>Keyword</dd>
<dt>L</dt>
<dd>ISO Language code</dd>
<dt>M</dt>
<dd>Month (numeric format: YYYYMM)</dd>
<dt>N</dt>
<dd>ISO couNtry code (or domaiN name)</dd>
<dt>O</dt>
<dd>Owner</dd>
<dt>P</dt>
<dd>Pathname</dd>
<dt>Q</dt>
<dd>uniQue id</dd>
<dt>R</dt>
<dd>Raw (i.e. unstemmed) term (unused by Xapian since 1.0.0)</dd>
<dt>S</dt>
<dd>Subject (or title)</dd>
<dt>T</dt>
<dd>mimeType</dd>
<dt>U</dt>
<dd>full URL of indexed document - if the resulting term would be &gt; 240
bytes, a hashing scheme is used to prevent overflowing
the Xapian term length limit (see omindex for how to do this).</dd>
<dt>V</dt>
<dd>boolean filter term for &quot;can't see&quot; permission (mnemonic: grep -v)</dd>
<dt>X</dt>
<dd>longer prefix for user-defined use</dd>
<dt>Y</dt>
<dd>year (four digits)</dd>
<dt>Z</dt>
<dd>stemmed term</dd>
</dl>
<p>Reserved but currently unallocated: CW</p>
<p>There are two main uses for prefixes - boolean filters and probabilistic
(i.e. free text) fields.</p>
<div class="section" id="boolean-filters">
<h1>Boolean Filters</h1>
<p>If the documents being indexed describe objects in a museum, you might
have a 'material' field, which records what each object is primarily made of.
So a sundial might be 'material=Stone', a letter might be 'material=paper',
etc.  There's no standard prefix for 'material', so you might allocate <tt class="docutils literal">XM</tt>.
If you lowercase the field contents, you can avoid having to add a colon to
seprated the prefix and content, so documents would be indexed by terms such as
<tt class="docutils literal">XMstone`</tt> or <tt class="docutils literal">XMpaper</tt>.</p>
<p>If you're indexing using scriptindex, and have a field in the input file
such as &quot;material=Stone&quot;, and then your index script would have a rule
such as:</p>
<pre class="literal-block">
material : lower boolean=XM
</pre>
<p>You can then restrict a search in Omega by passing a B parameter with one
of these as the value, e.g. <tt class="docutils literal">B=XMstone</tt></p>
<p>In your HTML search form, you can allow the user to select this using a set of
radio buttons:</p>
<pre class="literal-block">
Material:&lt;br&gt;
&lt;input type=&quot;radio&quot; name=&quot;B&quot; value=&quot;&quot;&gt; Any&lt;br&gt;
&lt;input type=&quot;radio&quot; name=&quot;B&quot; value=&quot;XMpaper&quot;&gt; Paper&lt;br&gt;
&lt;input type=&quot;radio&quot; name=&quot;B&quot; value=&quot;XMstone&quot;&gt; Stone&lt;br&gt;
</pre>
<p>If you want to have multiple sets of radio buttons for selecting different
boolean filters, you can make use of Omega's preprocessing of CGI parameter
names by calling them &quot;B 1&quot;, &quot;B 2&quot;, etc (names are truncated at the first
space - see <a class="reference external" href="cgiparams.html">cgiparams.html</a> for full details).</p>
<p>You can also use a select tag:</p>
<pre class="literal-block">
Material:
&lt;select name=&quot;B&quot;&gt;
&lt;option value=&quot;&quot;&gt;Any&lt;/option&gt;
&lt;option value=&quot;XMpaper&quot;&gt;Paper&lt;/option&gt;
&lt;option value=&quot;XMstone&quot;&gt;Stone&lt;/option&gt;
&lt;/select&gt;
</pre>
<p>Or if you want the user to be able to select more than one material to filter
by, you can use checkboxes instead of radio buttons:</p>
<pre class="literal-block">
Material:&lt;br&gt;
&lt;input type=&quot;checkbox&quot; name=&quot;B&quot; value=&quot;XMpaper&quot;&gt; Paper&lt;br&gt;
&lt;input type=&quot;checkbox&quot; name=&quot;B&quot; value=&quot;XMstone&quot;&gt; Stone&lt;br&gt;
</pre>
<p>Or a multiple select:</p>
<pre class="literal-block">
Material:
&lt;select multiple name=&quot;B&quot;&gt;
&lt;option value=&quot;XMpaper&quot;&gt;Paper&lt;/option&gt;
&lt;option value=&quot;XMstone&quot;&gt;Stone&lt;/option&gt;
&lt;/select&gt;
</pre>
<p>These will work in the natural way - if no materials are selected, then no
filtering by material will happen; if multiple materials are selected, then
items made of any of the materials will match (in details, groups of filter
terms with the same prefix are combined with <tt class="docutils literal">OP_OR</tt>; then these groups
are combined with <tt class="docutils literal">OP_AND</tt>).</p>
<p>Or perhaps the museum records multiple materials per object - e.g. a clock
might be made of brass, glass and wood.  This can be handled smoothly too - you
can specify multiple material fields to scriptindex:</p>
<pre class="literal-block">
material=brass
material=glass
material=wood
</pre>
<p>You may then want multiple filters on material to be mean &quot;find me objects
which contain <strong>all</strong> of these materials&quot; (rather than the default meaning
of &quot;find me objects which contain <strong>any</strong> of these materials&quot;) - to do this
you want to set <tt class="docutils literal">XM</tt> as a non-exclusive prefix, which you do like so (this
needs Omega 1.3.4 or later):</p>
<pre class="literal-block">
$setmap{nonexclusiveprefix,XM,true}
</pre>
<p>You can also allow the user to restrict a search with a boolean filter
specified in text query (e.g. <tt class="docutils literal">material:paper</tt> -&gt; <tt class="docutils literal">XMpaper</tt>) by adding this
to the start of your OmegaScript template:</p>
<pre class="literal-block">
$setmap{boolprefix,material,XM}
</pre>
<p>Multiple aliases are allowed:</p>
<pre class="literal-block">
$setmap{boolprefix,material,XM,madeof,XM}
</pre>
<p>This decoupling of internal and external names is also useful if you want
to offer search frontends in more than one language, as it allows the
prefixes the user sees to be translated.</p>
<p>If the user specified multiple filters in the query string, for example
<cite>material:wood material:paper</cite>, then these are combined using similar logic
to that used for filters specified by <tt class="docutils literal">B</tt> CGI parameters, with terms with the
same prefixed combined with <tt class="docutils literal">OP_OR</tt> by default, or <tt class="docutils literal">OP_AND</tt> specified by
<tt class="docutils literal"><span class="pre">$setmap{nonexclusiveprefix,...}</span></tt>.</p>
</div>
<div class="section" id="probabilistic-fields">
<h1>Probabilistic Fields</h1>
<p>Say you want to index the title of the document such that the user can
search within the title by specifying title:report (for example) in their
query.</p>
<p>Title has standard prefix S, so you'd generate terms as normal, but then
add an &quot;S&quot; prefix.  If you're using scriptindex, then you do this by
adding &quot;index=S&quot; to the scriptindex rule like so:</p>
<pre class="literal-block">
title : field=title index=S
</pre>
<p>You then need to tell Xapian::QueryParser that &quot;title:&quot; maps to an &quot;S&quot; prefix.
If you're using Omega, then you do so by adding this to your OmegaScript
template (at the start is best):</p>
<pre class="literal-block">
$setmap{prefix,title,S}
</pre>
<p>Or if you're writing your own search frontend, like this:</p>
<pre class="literal-block">
Xapian::QueryParser qp;
qp.add_prefix(&quot;subject&quot;, &quot;S&quot;);
// And similar lines for other probabilistic prefixes...
// And any other QueryParser configuration (e.g. stemmer, stopper).
Xapian::Query query = qp.parse_query(user_query_string);
</pre>
<p>You can add multiple aliases for a prefix (e.g. title and subject for S), and
the decoupling of &quot;UI prefix&quot; and &quot;term prefix&quot; means you can easily translate
the &quot;UI prefixes&quot; if you have frontends in different languages.</p>
<p>Note that if you want words from the subject to be found without a prefix, you
either need to generate unprefixed terms as well as the prefixed ones, or map
the empty prefix to both &quot;&quot; and &quot;S&quot; like so:</p>
<pre class="literal-block">
Xapian::QueryParser qp;
// Search both subject and body if no field is specified:
qp.add_prefix(&quot;&quot;, &quot;&quot;);
qp.add_prefix(&quot;&quot;, &quot;S&quot;);
// Search just the subject if 'subject:' is specified:
qp.add_prefix(&quot;subject&quot;, &quot;S&quot;);
Xapian::Query query = qp.parse_query(user_query_string);
</pre>
</div>
</div>
</body>
</html>
xapian-omega 1.4.5-1 / usr / share / doc / xapian-omega / termprefixes.html
