/usr/share/doc/python-apsw/html/_sources/vtable.txt is in python-apsw-doc 3.7.6.3-r1-1build1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 499 500 501 502 503 504 505 506 507 | .. Automatically generated by code2rst.py
code2rst.py src/vtable.c doc/vtable.rst
Edit src/vtable.c not this file!
.. currentmodule:: apsw
.. _virtualtables:
Virtual Tables
**************
`Virtual Tables <http://www.sqlite.org/cvstrac/wiki?p=VirtualTables>`_
are an experimental feature introduced in SQLite 3.3.7. They let a
developer provide an underlying table implementations, while still
presenting a normal SQL interface to the user. The person writing SQL
doesn't need to know or care that some of the tables come from
elsewhere.
Some examples of how you might use this:
* Translating to/from information stored in other formats (eg a csv/ini format file)
* Accessing the data remotely (eg you could make a table that backends into `Amazon's API <http://www.josephson.org/projects/pyamazon>`_)
* Dynamic information (eg currently running processes, files and directories, objects in your program)
* Information that needs reformatting (eg if you have complex rules about how to convert strings to/from Unicode
in the dataset)
* Information that isn't relationally correct (eg if you have data that has ended up with duplicate "unique" keys
with code that dynamically corrects it)
* There are other examples on the `SQLite page <http://www.sqlite.org/cvstrac/wiki?p=VirtualTables>`_
You need to have 3 types of object. A :class:`module <VTModule>`, a
:class:`virtual table <VTTable>` and a :class:`cursor
<VTCursor>`. These are documented below. You can also read the `SQLite
C method documentation
<http://www.sqlite.org/cvstrac/wiki?p=VirtualTableMethods>`_. At the
C level, they are just one set of methods. At the Python/APSW level,
they are split over the 3 types of object. The leading **x** is
omitted in Python. You can return SQLite error codes (eg
:const:`SQLITE_READONLY`) by raising the appropriate exceptions (eg
:exc:`ReadOnlyError`). :meth:`exceptionfor` is a useful helper
function to do the mapping.
VTModule class
==============
.. class:: VTModule
.. note::
There is no actual *VTModule* class - it is just shown this way
for documentation convenience. Your module instance should implement
all the methods documented here.
A module instance is used to create the virtual tables. Once you have
a module object, you register it with a connection by calling
:meth:`Connection.createmodule`::
# make an instance
mymod=MyModuleClass()
# register the vtable on connection con
con.createmodule("modulename", mymod)
# tell SQLite about the table
con.cursor().execute("create VIRTUAL table tablename USING modulename('arg1', 2)")
The create step is to tell SQLite about the existence of the table.
Any number of tables referring to the same module can be made this
way. Note the (optional) arguments which are passed to the module.
.. method:: VTModule.Connect(connection, modulename, databasename, tablename, *args) -> [ sql string, table object ]
The parameters and return are identical to
:meth:`~VTModule.Create`. This method is called
when there are additional references to the table. :meth:`~VTModule.Create` will be called the first time and
:meth:`~VTModule.Connect` after that.
The advise is to create caches, generated data and other
heavyweight processing on :meth:`~VTModule.Create` calls and then
find and reuse that on the subsequent :meth:`~VTModule.Connect`
calls.
The corresponding call is :meth:`VTTable.Disconnect`. If you have a simple virtual table implemtation, then just
set :meth:`~VTModule.Connect` to be the same as :meth:`~VTModule.Create`::
class MyModule:
def Create(self, connection, modulename, databasename, tablename, *args):
# do lots of hard work
Connect=Create
.. method:: VTModule.Create(connection, modulename, databasename, tablename, *args) -> [ sql string, table object ]
Called when a table is first created on a :class:`connection
<Connection>`.
:param connection: An instance of :class:`Connection`
:param modulename: The string name under which the module was :meth:`registered <Connection.createmodule>`
:param databasename: The name of the database. This will be ``main`` for directly opened files and the name specified in
`ATTACH <http://www.sqlite.org/lang_attach.html>`_ statements.
:param tablename: Name of the table the user wants to create.
:param args: Any arguments that were specified in the `create virtual table <http://www.sqlite.org/lang_createvtab.html>`_ statement.
:returns: A list of two items. The first is a SQL `create table <http://www.sqlite.org/lang_createtable.html>`_ statement. The
columns are parsed so that SQLite knows what columns and declared types exist for the table. The second item
is an object that implements the :class:`table <VTTable>` methods.
The corresponding call is :meth:`VTTable.Destroy`.
VTTable class
=============
.. class:: VTTable
.. note::
There is no actual *VTTable* class - it is just shown this way for
documentation convenience. Your table instance should implement
the methods documented here.
The :class:`VTTable` object contains knowledge of the indices, makes
cursors and can perform transactions.
.. _vtablestructure:
A virtual table is structured as a series of rows, each of which has
the same columns. The value in a column must be one of the `5
supported types <http://www.sqlite.org/datatype3.html>`_, but the
type can be different between rows for the same column. The virtual
table routines identify the columns by number, starting at zero.
Each row has a **unique** 64 bit integer `rowid
<http://www.sqlite.org/autoinc.html>`_ with the :class:`Cursor
<VTCursor>` routines operating on this number, as well as some of
the :class:`Table <VTTable>` routines such as :meth:`UpdateChangeRow
<VTTable.UpdateChangeRow>`.
.. method:: VTTable.Begin()
This function is used as part of transactions. At the time of
writing it is undocumented in `SQLite
<http://www.sqlite.org/cvstrac/wiki?p=VirtualTableMethods>`_. You
do not have to provide the method.
.. method:: VTTable.BestIndex(constraints, orderbys)
This is a complex method and even has its `own page
<http://www.sqlite.org/cvstrac/wiki?p=VirtualTableBestIndexMethod>`_
in the SQLite documentation. To get going initially, just return
:const:`None` and you will be fine. Implementing this method reduces
the number of rows scanned in your table to satisfy queries, but
only if you have an index or index like mechanism available.
.. note::
The implementation of this method differs slightly from the
`SQLite documentation
<http://www.sqlite.org/cvstrac/wiki?p=VirtualTableBestIndexMethod>`_
for the C API. You are not passed "unusable" constraints. The
argv/constraintarg positions are not off by one. In the C api, you
have to return position 1 to get something passed to
:meth:`VTCursor.Filter` in position 0. With the APSW
implementation, you return position 0 to get Filter arg 0,
position 1 to get Filter arg 1 etc.
The purpose of this method is to ask if you have the ability to
determine if a row meets certain constraints that doesn't involve
visiting every row. An example constraint is ``price > 74.99``. In a
traditional SQL database, queries with constraints can be speeded up
`with indices <http://www.sqlite.org/lang_createindex.html>`_. If
you return None, then SQLite will visit every row in your table and
evaluate the constraint itself. Your index choice returned from
BestIndex will also be passed to the :meth:`~VTCursor.Filter` method on your cursor
object. Note that SQLite may call this method multiple times trying
to find the most efficient way of answering a complex query.
**constraints**
You will be passed the contraints as a sequence of tuples containing two
items. The first item is the column number and the second item is
the operation.
Example query: ``select * from foo where price > 74.99 and
quantity<=10 and customer='Acme Widgets'``
If customer is column 0, price column 2 and quantity column 5
then the constraints will be::
(2, apsw.SQLITE_INDEX_CONSTRAINT_GT),
(5, apsw.SQLITE_INDEX_CONSTRAINT_LE),
(0, apsw.SQLITE_INDEX_CONSTRAINT_EQ)
Note that you do not get the value of the constraint (ie "Acme
Widgets", 74.99 and 10 in this example).
If you do have any suitable indices then you return a sequence the
same length as constraints with the members mapping to the
constraints in order. Each can be one of None, an integer or a tuple
of an integer and a boolean. Conceptually SQLite is giving you a
list of constraints and you are returning a list of the same length
describing how you could satisfy each one.
Each list item returned corresponding to a constraint is one of:
None
This means you have no index for that constraint. SQLite
will have to iterate over every row for it.
integer
This is the argument number for the constraintargs being passed
into the :meth:`~VTCursor.Filter` function of your
:class:`cursor <VTCursor>` (the values "Acme Widgets", 74.99
and 10 in the example).
(integer, boolean)
By default SQLite will check what you return. For example if
you said that you had an index on price, SQLite will still
check that each row you returned is greater than 74.99. If you
set the boolean to False then SQLite won't do that double
checking.
Example query: ``select * from foo where price > 74.99 and
quantity<=10 and customer=='Acme Widgets'``. customer is column 0,
price column 2 and quantity column 5. You can index on customer
equality and price.
+----------------------------------------+--------------------------------+
| Constraints (in) | Constraints used (out) |
+========================================+================================+
| :: | :: |
| | |
| (2, apsw.SQLITE_INDEX_CONSTRAINT_GT), | 1, |
| (5, apsw.SQLITE_INDEX_CONSTRAINT_LE), | None, |
| (0, apsw.SQLITE_INDEX_CONSTRAINT_EQ) | 0 |
| | |
+----------------------------------------+--------------------------------+
When your :class:`~VTCursor.Filter` method in the cursor is called,
constraintarg[0] will be "Acme Widgets" (customer constraint value)
and constraintarg[1] will be 74.99 (price constraint value). You can
also return an index number (integer) and index string to use
(SQLite attaches no significance to these values - they are passed
as is to your :meth:`VTCursor.Filter` method as a way for the
BestIndex method to let the :meth:`~VTCursor.Filter` method know
which of your indices or similar mechanism to use.
**orderbys**
The second argument to BestIndex is a sequence of orderbys because
the query requested the results in a certain order. If your data is
already in that order then SQLite can give the results back as
is. If not, then SQLite will have to sort the results first.
Example query: ``select * from foo order by price desc, quantity asc``
Price is column 2, quantity column 5 so orderbys will be::
(2, True), # True means descending, False is ascending
(5, False)
**Return**
You should return up to 5 items. Items not present in the return have a default value.
0: constraints used (default None)
This must either be None or a sequence the same length as
constraints passed in. Each item should be as specified above
saying if that constraint is used, and if so which constraintarg
to make the value be in your :meth:`VTCursor.Filter` function.
1: index number (default zero)
This value is passed as is to :meth:`VTCursor.Filter`
2: index string (default None)
This value is passed as is to :meth:`VTCursor.Filter`
3: orderby consumed (default False)
Return True if your output will be in exactly the same order as the orderbys passed in
4: estimated cost (default a huge number)
Approximately how many disk operations are needed to provide the
results. SQLite uses the cost to optimise queries. For example if
the query includes *A or B* and A has 2,000 operations and B has 100
then it is best to evaluate B before A.
**A complete example**
Query is ``select * from foo where price>74.99 and quantity<=10 and
customer=="Acme Widgets" order by price desc, quantity asc``.
Customer is column 0, price column 2 and quantity column 5. You can
index on customer equality and price.
::
BestIndex(constraints, orderbys)
constraints= ( (2, apsw.SQLITE_INDEX_CONSTRAINT_GT),
(5, apsw.SQLITE_INDEX_CONSTRAINT_LE),
(0, apsw.SQLITE_INDEX_CONSTRAINT_EQ) )
orderbys= ( (2, True), (5, False) )
# You return
( (1, None, 0), # constraints used
27, # index number
"idx_pr_cust", # index name
False, # results are not in orderbys order
1000 # about 1000 disk operations to access index
)
# Your Cursor.Filter method will be called with:
27, # index number you returned
"idx_pr_cust", # index name you returned
"Acme Widgets", # constraintarg[0] - customer
74.99 # constraintarg[1] - price
.. method:: VTTable.Commit()
This function is used as part of transactions. At the time of
writing it is undocumented in `SQLite
<http://www.sqlite.org/cvstrac/wiki?p=VirtualTableMethods>`_. You
do not have to provide the method.
.. method:: VTTable.Destroy()
The opposite of :meth:`VTModule.Create`. This method is called when
the table is no longer used. Note that you must always release
resources even if you intend to return an error, as it will not be
called again on error. SQLite may also :cvstrac:`leak memory
<2099>` if you return an error.
.. method:: VTTable.Disconnect()
The opposite of :meth:`VTModule.Connect`. This method is called when
a reference to a virtual table is no longer used, but :meth:`VTTable.Destroy` will
be called when the table is no longer used.
.. method:: VTTable.FindFunction(name, nargs)
Called to find if the virtual table has its own implementation of a
particular scalar function. You should return the function if you
have it, else return None. You do not have to provide this method.
This method is called while SQLite is `preparing
<http://www.sqlite.org/c3ref/prepare.html>`_ a query. If a query is
in the :ref:`statement cache <statementcache>` then *FindFunction*
won't be called again. If you want to return different
implementations for the same function over time then you will need
to disable the :ref:`statement cache <statementcache>`.
:param name: The function name
:param nargs: How many arguments the function takes
.. seealso::
* :meth:`Connection.overloadfunction`
.. method:: VTTable.Open()
Returns a :class:`cursor <VTCursor>` object.
.. method:: VTTable.Rename(newname)
Notification that the table will be given a new name. If you return
without raising an exception, then SQLite renames the table (you
don't have to do anything). If you raise an exception then the
renaming is prevented. You do not have to provide this method.
.. method:: VTTable.Rollback()
This function is used as part of transactions. At the time of
writing it is undocumented in `SQLite
<http://www.sqlite.org/cvstrac/wiki?p=VirtualTableMethods>`_. You
do not have to provide the method.
.. method:: VTTable.Sync()
This function is used as part of transactions. At the time of
writing it is undocumented in `SQLite
<http://www.sqlite.org/cvstrac/wiki?p=VirtualTableMethods>`_. You
do not have to provide the method.
.. method:: VTTable.UpdateChangeRow(row, newrowid, fields)
Change an existing row. You may also need to change the rowid - for example if the query was
``UPDATE table SET rowid=rowid+100 WHERE ...``
:param row: The existing 64 bit integer rowid
:param newrowid: If not the same as *row* then also change the rowid to this.
:param fields: A tuple of values the same length and order as columns in your table
.. method:: VTTable.UpdateDeleteRow(rowid)
Delete the row with the specified *rowid*.
:param rowid: 64 bit integer
.. method:: VTTable.UpdateInsertRow(rowid, fields) -> newrowid
Insert a row with the specified *rowid*.
:param rowid: :const:`None` if you should choose the rowid yourself, else a 64 bit integer
:param fields: A tuple of values the same length and order as columns in your table
:returns: If *rowid* was :const:`None` then return the id you assigned
to the row. If *rowid* was not :const:`None` then the return value
is ignored.
VTCursor class
==============
.. class:: VTCursor
.. note::
There is no actual *VTCursor* class - it is just shown this
way for documentation convenience. Your cursor instance should
implement all the methods documented here.
The :class:`VTCursor` object is used for iterating over a table.
There may be many cursors simultaneously so each one needs to keep
track of where it is.
.. seealso::
:ref:`Virtual table structure <vtablestructure>`
.. method:: VTCursor.Close()
This is the destructor for the cursor. Note that you must
cleanup. The method will not be called again if you raise an
exception.
.. method:: VTCursor.Column(number)
Requests the value of the specified column *number* of the current
row. If *number* is -1 then return the rowid.
:returns: Must be one one of the :ref:`5
supported types <types>`
.. method:: VTCursor.Eof() -> bool
Called to ask if we are at the end of the table. It is called after each call to Filter and Next.
:returns: False if the cursor is at a valid row of data, else True
.. note::
This method can only return True or False to SQLite. If you have
an exception in the method or provide a non-boolean return then
True (no more data) will be returned to SQLite.
.. method:: VTCursor.Filter(indexnum, indexname, constraintargs)
This method is always called first to initialize an iteration to the
first row of the table. The arguments come from the
:meth:`~VTTable.BestIndex` method in the :class:`table <VTTable>`
object with constraintargs being a tuple of the constraints you
requested. If you always return None in BestIndex then indexnum will
be zero, indexstring will be None and constraintargs will be empty).
.. method:: VTCursor.Next()
Move the cursor to the next row. Do not have an exception if there
is no next row. Instead return False when :meth:`~VTCursor.Eof` is
subsequently called.
If you said you had indices in your :meth:`VTTable.BestIndex`
return, and they were selected for use as provided in the parameters
to :meth:`~VTCursor.Filter` then you should move to the next
appropriate indexed and constrained row.
.. method:: VTCursor.Rowid() -> 64 bit integer
Return the current rowid.
Troubleshooting virtual tables
==============================
Virtual Tables are a relatively recent addition to SQLite and haven't
been widely used yet. They do work well if all your routines work
perfectly.
A big help is using the local variables recipe as described in
:ref:`augmented stack traces <augmentedstacktraces>` which will give
you more details in errors, and shows an example with the complex
:meth:`~VTTable.BestIndex` function.
You may also find errors compounding. For
example if you have an error in the Filter method of a cursor, SQLite
then closes the cursor. If you also return an error in the Close
method then the first error may mask the second or vice versa.
.. note::
SQLite may ignore responses from your methods if they don't make
sense. For example in BestIndex, if you set multiple arguments to
have the same constraintargs position then your Filter won't
receive any constraintargs at all.
|