/usr/share/doc/python-apsw/html/_sources/vfs.txt is in python-apsw-doc 3.7.6.3-r1-1build1.
This file is owned by root:root, with mode 0o644.
The actual contents of the file can be viewed below.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433 434 435 436 437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486 487 488 489 490 491 492 493 494 495 496 497 498 | .. Automatically generated by code2rst.py
code2rst.py src/vfs.c doc/vfs.rst
Edit src/vfs.c not this file!
.. currentmodule:: apsw
.. _vfs:
Virtual File System (VFS)
*************************
SQLite 3.6 has new `VFS functionality
<http://sqlite.org/c3ref/vfs.html>`_ which defines the interface
between the SQLite core and the underlying operating system. The
majority of the functionality deals with files. APSW exposes this
functionality letting you provide your own routines. You can also
*inherit* from an existing vfs making it easy to augment or override
specific routines. For example you could obfuscate your database by
XORing the data implemented by augmenting the read and write
methods. The method names are exactly the same as SQLite uses making
it easier to read the SQLite documentation, trouble tickets, web
searches or mailing lists. The SQLite convention results in names like
xAccess, xCurrentTime and xWrite.
You specify which VFS to use as a parameter to the :class:`Connection`
constructor.
.. code-block:: python
db=apsw.Connection("file", vfs="myvfs")
The easiest way to get started is to make a :class:`VFS` derived class
that inherits from the default vfs. Then override methods you want to
change behaviour of. If you want to just change how file operations
are done then you have to override :meth:`VFS.xOpen` to return a file
instance that has your overridden :class:`VFSFile` methods. The
:ref:`example <example-vfs>` demonstrates obfuscating the database
file contents.
.. note::
All strings supplied and returned to :class:`VFS`/:class:`VFSFile`
routines are treated as Unicode.
Exceptions and errors
=====================
To return an error from any routine you should raise an exception. The
exception will be translated into the appropriate SQLite error code
for SQLite. To return a specific SQLite error code use
:meth:`exceptionfor`. If the exception does not map to any specific
error code then :const:`SQLITE_ERROR` which corresponds to
:exc:`SQLError` is returned to SQLite.
The SQLite code that deals with VFS errors behaves in varying
ways. Some routines have no way to return an error (eg `xDlOpen
<http://www.sqlite.org/c3ref/vfs.html>`_ just returns zero/NULL on
being unable to load a library, `xSleep
<http://www.sqlite.org/c3ref/vfs.html>`_ has no error return
parameter), others have any error values ignored (eg
:cvstrac:`xCurrentTime <3394>`), others are unified (eg almost any
error in xWrite will be returned to the user as disk full
error). Sometimes errors are ignored as they are harmless such as when
a journal can't be deleted after a commit (the journal is marked as
obsolete before being deleted). Simple operations such as opening a
database can result in many different VFS function calls such as hot
journals being detected, locking, and read/writes for
playback/rollback.
To avoid confusion with exceptions being raised in the VFS and
exceptions from normal code to open Connections or execute SQL
queries, VFS exceptions are not raised in the normal way. (If they
were, only one could be raised and it would obscure whatever
exceptions the :class:`Connection` open or SQL query execute wanted to
raise.) Instead the :meth:`VFS.excepthook` or
:meth:`VFSFile.excepthook` method is called with a tuple of exception
type, exception value and exception traceback. The default
implementation of ``excepthook`` calls ``sys.excepthook()`` which
under Python 2 shows the stack trace and under Python 3 merely prints
the exception value. (If ``sys.excepthook`` fails then
``PyErr_Display()`` is called.)
In normal VFS usage there will be no exceptions raised, or specific
expected ones which APSW clears after noting them and returning the
appropriate value back to SQLite. The exception hooking behaviour
helps you find issues in your code or unexpected behaviour of the
external environment. Remember that :ref:`augmented stack traces
<augmentedstacktraces>` are available which significantly increase
detail about the exceptions.
As an example, lets say you have a divide by zero error in your xWrite
routine. The table below shows what happens with time going down and
across.
+----------------------------------------------+--------------------------------+---------------------------------------------+
| Python Query Code | SQLite and APSW C code | Python VFS code |
+==============================================+================================+=============================================+
| ``cursor.execute("update table set foo=3")`` | | |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| | SQLite starts executing query | |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| | | Your VFS routines are called |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| | | Your xWrite divides by zero |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| | | :meth:`VFSFile.excepthook` is called with |
| | | ZeroDivision exception |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| | :const:`SQLITE_ERROR` (closest | |
| | matching SQLite error code) is | |
| | returned to SQLite by APSW | |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| | SQLite error handling and | More VFS routines are called. Any |
| | recovery operates which calls | exceptions in these routines will result in |
| | more VFS routines. | :meth:`VFSFile.excepthook` being called with|
| | | them. |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| | SQLite returns | |
| | :const:`SQLITE_FULL` to APSW | |
+----------------------------------------------+--------------------------------+---------------------------------------------+
| APSW returns :class:`apsw.FullError` | | |
+----------------------------------------------+--------------------------------+---------------------------------------------+
VFS class
=========
.. index:: sqlite3_vfs_register, sqlite3_vfs_find
.. class:: VFS(name[, base=None, makedefault=False, maxpathname=1024])
Provides operating system access. You can get an overview in the
`SQLite documentation <http://sqlite.org/c3ref/vfs.html>`_. To
create a VFS your Python class must inherit from :class:`VFS`.
:param name: The name to register this vfs under. If the name
already exists then this vfs will replace the prior one of the
same name. Use :meth:`apsw.vfsnames` to get a list of
registered vfs names.
:param base: If you would like to inherit behaviour from an
already registered vfs then give their name. To inherit from the
default vfs, use a zero length string ``""`` as the name.
:param makedefault: If true then this vfs will be registered as
the default, and will be used by any opens that don't specify a
vfs.
:param maxpathname: The maximum length of database name in bytes
when represented in UTF-8. If a pathname is passed in longer than
this value then SQLite will not :cvstrac:`be able to open it <3373>`.
:raises ValueError: If *base* is not :const:`None` and the named vfs is not
currently registered.
Calls:
* :sqliteapi:`sqlite3_vfs_register <vfs_find>`
* :sqliteapi:`sqlite3_vfs_find <vfs_find>`
.. method:: VFS.excepthook(etype, evalue, etraceback)
Called when there has been an exception in a :class:`VFS` routine.
The default implementation calls ``sys.excepthook`` and if that
fails then ``PyErr_Display``. The three arguments correspond to
what ``sys.exc_info()`` would return.
:param etype: The exception type
:param evalue: The exception value
:param etraceback: The exception traceback. Note this
includes all frames all the way up to the thread being started.
.. index:: sqlite3_vfs_unregister
.. method:: VFS.unregister()
Unregisters the VFS making it unavailable to future database
opens. You do not need to call this as the VFS is automatically
unregistered by when the VFS has no more references or open
datatabases using it. It is however useful to call if you have made
your VFS be the default and wish to immediately make it be
unavailable. It is safe to call this routine multiple times.
Calls: :sqliteapi:`sqlite3_vfs_unregister <vfs_find>`
.. method:: VFS.xAccess(pathname, flags) -> bool
SQLite wants to check access permissions. Return True or False
accordingly.
:param pathname: File or directory to check
:param flags: One of the `access flags <http://sqlite.org/c3ref/c_access_exists.html>`_
.. method:: VFS.xCurrentTime() -> float
Return the `Julian Day Number
<http://en.wikipedia.org/wiki/Julian_day>`_ as a floating point
number where the integer portion is the day and the fractional part
is the time. Do not adjust for timezone (ie use `UTC
<http://en.wikipedia.org/wiki/Universal_Time>`_). Although SQLite
allows for an error return, that is :cvstrac:`ignored <3394>`.
.. method:: VFS.xDelete(filename, syncdir)
Delete the named file.
:param filename: File to delete
:param syncdir: If True then the directory should be synced
ensuring that the file deletion has been recorded on the disk
platters. ie if there was an immediate power failure after this
call returns, on a reboot the file would still be deleted.
.. method:: VFS.xDlClose(handle)
Close and unload the library corresponding to the handle you
returned from :meth:`~VFS.xDlOpen`. You can use ctypes to do
this::
def xDlClose(handle):
# Note leading underscore in _ctypes
_ctypes.dlclose(handle) # Linux/Mac/Unix
_ctypes.FreeLibrary(handle) # Windows
.. method:: VFS.xDlError() -> string
Return an error string describing the last error of
:meth:`~VFS.xDlOpen` or :meth:`~VFS.xDlSym` (ie they returned
zero/NULL). If you do not supply this routine then SQLite provides
a generic message. To implement this method, catch exceptions in
:meth:`~VFS.xDlOpen` or :meth:`~VFS.xDlSym`, turn them into
strings, save them, and return them in this routine. Note that the
message may be truncated to 255 characters - see
:cvstrac:`3305`. If you have an error in this routine or return
None then SQLite's generic message will be used.
.. method:: VFS.xDlOpen(filename) -> number
Load the shared library. You should return a number which will be
treated as a void pointer at the C level. On error you should
return 0 (NULL). The number is passed as is to
:meth:`~VFS.xDlSym`/:meth:`~VFS.xDlClose` so it can represent
anything that is convenient for you (eg an index into an
array). You can use ctypes to load a library::
def xDlOpen(name):
return ctypes.cdll.LoadLibrary(name)._handle
.. method:: VFS.xDlSym(handle, symbol) -> address
Returns the address of the named symbol which will be called by
SQLite. On error you should return 0 (NULL). You can use ctypes::
def xDlSym(ptr, name):
return _ctypes.dlsym (ptr, name) # Linux/Unix/Mac etc (note leading underscore)
return ctypes.win32.kernel32.GetProcAddress (ptr, name) # Windows
:param handle: The value returned from an earlier :meth:`~VFS.xDlOpen` call
:param symbol: A string
:rtype: An int/long with the symbol address
.. method:: VFS.xFullPathname(name) -> string
Return the absolute pathname for name. You can use ``os.path.abspath`` to do this.
.. method:: VFS.xGetLastError() -> string
This method is to return text describing the last error that
happened in this thread. If not implemented SQLite's more generic
message is used. However the method is :cvstrac:`never called
<3337>` by SQLite.
.. method:: VFS.xGetSystemCall(name) -> int
Returns a pointer for the current method implementing the named
system call. Return None if the call does not exist.
.. method:: VFS.xNextSystemCall(name) -> String or None
This method is repeatedly called to iterate over all of the system
calls in the vfs. When called with None you should return the
name of the first system call. In subsequent calls return the
name after the one passed in. If name is the last system call
then return None.
.. note::
Because of internal SQLite implementation semantics memory will
be leaked on each call to this function. Consequently you
should build up the list of call names once rather than
repeatedly doing it.
.. method:: VFS.xOpen(name, flags) -> VFSFile or similar object
This method should return a new file object based on name. You
can return a :class:`VFSFile` from a completely different VFS.
:param name: File to open. Note that *name* may be :const:`None` in which
case you should open a temporary file with a name of your
choosing.
:param flags: A list of two integers ``[inputflags,
outputflags]``. Each integer is one or more of the `open flags
<http://www.sqlite.org/c3ref/c_open_create.html>`_ binary orred
together. The ``inputflags`` tells you what SQLite wants. For
example :const:`SQLITE_OPEN_DELETEONCLOSE` means the file should
be automatically deleted when closed. The ``outputflags``
describes how you actually did open the file. For example if you
opened it read only then :const:`SQLITE_OPEN_READONLY` should be
set.
.. method:: VFS.xRandomness(numbytes) -> bytes
This method is called once when SQLite needs to seed the random
number generator. It is called on the default VFS only. It is not
called again, even across :meth:`apsw.shutdown` calls. You can
return less than the number of bytes requested including None. If
you return more then the surplus is ignored.
:rtype: (Python 2) string, buffer (Python 3) bytes, buffer
.. method:: VFS.xSetSystemCall(name, pointer) -> bool
Change a system call used by the VFS. This is useful for testing
and some other scenarios such as sandboxing.
:param name: The string name of the system call
:param pointer: A pointer provided as an int/long. There is no
reference counting or other memory tracking of the pointer. If
you provide one you need to ensure it is around for the lifetime
of this and any other related VFS.
Raise an exception to return an error. If the system call does
not exist then raise :exc:`NotFoundError`.
:returns: True if the system call was set. False if the system
call is not known.
.. method:: VFS.xSleep(microseconds) -> integer
Pause exection of the thread for at least the specified number of
microseconds (millionths of a second). This routine is typically called from the busy handler.
:returns: How many microseconds you actually requested the
operating system to sleep for. For example if your operating
system sleep call only takes seconds then you would have to have
rounded the microseconds number up to the nearest second and
should return that rounded up value.
VFSFile class
=============
.. class:: VFSFile(vfs, name, flags)
Wraps access to a file. You only need to derive from this class
if you want the file object returned from :meth:`VFS.xOpen` to
inherit from an existing VFS implementation.
.. note::
All file sizes and offsets are 64 bit quantities even on 32 bit
operating systems.
:param vfs: The vfs you want to inherit behaviour from. You can
use an empty string ``""`` to inherit from the default vfs.
:param name: The name of the file being opened.
:param flags: A two list ``[inflags, outflags]`` as detailed in :meth:`VFS.xOpen`.
:raises ValueError: If the named VFS is not registered.
.. note::
If the VFS that you inherit from supports :ref:`write ahead
logging <wal>` then your :class:`VFSFile` will also support the
xShm methods necessary to implement wal.
.. seealso::
:meth:`VFS.xOpen`
.. method:: VFSFile.excepthook(etype, evalue, etraceback)
Called when there has been an exception in a :class:`VFSFile`
routine. The default implementation calls ``sys.excepthook`` and
if that fails then ``PyErr_Display``. The three arguments
correspond to what ``sys.exc_info()`` would return.
:param etype: The exception type
:param evalue: The exception value
:param etraceback: The exception traceback. Note this
includes all frames all the way up to the thread being started.
.. method:: VFSFile.xCheckReservedLock()
Returns True if any database connection (in this or another process)
has a lock other than `SQLITE_LOCK_NONE or SQLITE_LOCK_SHARED
<http://sqlite.org/c3ref/c_lock_exclusive.html>`_.
.. method:: VFSFile.xClose()
Close the database. Note that even if you return an error you should
still close the file. It is safe to call this method mutliple
times.
.. method:: VFSFile.xDeviceCharacteristics() -> int
Return `I/O capabilities
<http://sqlite.org/c3ref/c_iocap_atomic.html>`_ (bitwise or of
appropriate values). If you do not implement the function or have an
error then 0 (the SQLite default) is returned.
.. method:: VFSFile.xFileControl(op, ptr) -> bool
Receives `file control
<http://sqlite.org/c3ref/file_control.html>`_ request typically
issued by :meth:`Connection.filecontrol`. See
:meth:`Connection.filecontrol` for an example of how to pass a
Python object to this routine.
:param op: A numeric code. Codes below 100 are reserved for SQLite
internal use.
:param ptr: An integer corresponding to a pointer at the C level.
:returns: A boolean indicating if the op was understood
As of SQLite 3.6.10, this method is called by SQLite if you have
inherited from an underlying VFSFile. Consequently ensure you pass
any unrecognised codes through to your super class. For example::
def xFileControl(self, op, ptr):
if op==1027:
process_quick(ptr)
elif op==1028:
obj=ctypes.py_object.from_address(ptr).value
else:
# this ensures superclass implementation is called
return super(MyFile, self).xFileControl(op, ptr)
# we understood the op
return True
.. method:: VFSFile.xFileSize() -> int
Return the size of the file in bytes. Remember that file sizes are
64 bit quantities even on 32 bit operating systems.
.. method:: VFSFile.xLock(level)
Increase the lock to the level specified which is one of the
`SQLITE_LOCK <http://sqlite.org/c3ref/c_lock_exclusive.html>`_
family of constants. If you can't increase the lock level because
someone else has locked it, then raise :exc:`BusyError`.
.. method:: VFSFile.xRead(amount, offset) -> bytes
Read the specified *amount* of data starting at *offset*. You
should make every effort to read all the data requested, or return
an error. If you have the file open for non-blocking I/O or if
signals happen then it is possible for the underlying operating
system to do a partial read. You will need to request the
remaining data. Except for empty files SQLite considers short
reads to be a fatal error.
:param amount: Number of bytes to read
:param offset: Where to start reading. This number may be 64 bit once the database is larger than 2GB.
:rtype: (Python 2) string, buffer. (Python 3) bytes, buffer
.. method:: VFSFile.xSectorSize() -> int
Return the native underlying sector size. SQLite uses the value
returned in determining the default database page size. If you do
not implement the function or have an error then 512 (the SQLite
default) is returned.
.. method:: VFSFile.xSync(flags)
Ensure data is on the disk platters (ie could survive a power
failure immediately after the call returns) with the `sync flags
<http://sqlite.org/c3ref/c_sync_dataonly.html>`_ detailing what
needs to be synced. You can sync more than what is requested.
.. method:: VFSFile.xTruncate(newsize)
Set the file length to *newsize* (which may be more or less than the
current length).
.. method:: VFSFile.xUnlock(level)
Decrease the lock to the level specified which is one of the
`SQLITE_LOCK <http://sqlite.org/c3ref/c_lock_exclusive.html>`_
family of constants.
.. method:: VFSFile.xWrite(data, offset)
Write the *data* starting at absolute *offset*. You must write all the data
requested, or return an error. If you have the file open for
non-blocking I/O or if signals happen then it is possible for the
underlying operating system to do a partial write. You will need to
write the remaining data.
:param offset: Where to start writing. This number may be 64 bit once the database is larger than 2GB.
:param data: (Python 2) string, (Python 3) bytes
|