python-zodb 1:3.9.7-2 / usr / share / pyshared / ZEO / tests / invalidation-age.txt

Invalidation age
================

When a ZEO client with a non-empty cache connects to the server, it
needs to verify whether the data in its cache is current.  It does
this in one of 2 ways:

quick verification
   It gets a list of invalidations from the server since the last
   transaction the client has seen and applies those to it's disk and
   in-memory caches.  This is only possible if there haven't been too
   many transactions since the client was last connected.

full verification
   If quick verification isn't possible, the client iterates through
   it's disk cache asking the server to verify whether each current
   entry is valid.

Unfortunately, for large caches, full verification is soooooo not
quick that it is impractical.  Quick verificatioin is highly
desireable.

To support quick verification, the server keeps a list of recent
invalidations. The size of this list is controlled by the
invalidation_queue_size parameter.  If there is a lot of database
activity, the size might need to be quite large to support having
clients be disconnected for more than a few minutes.  A very large
invalidation queue size can use a lot of memory.

To suppliment the invalidation queue, you can also specify an
invalidation_age parameter.  When a client connects and presents the
last transaction id it has seen, we first check to see if the
invalidation queue has that transaction id. It it does, then we send
all transactions since that id.  Otherwise, we check to see if the
difference between storage's last transaction id and the given id is
less than or equal to the invalidation age.  If it is, then we iterate
over the storage, starting with the given id, to get the invalidations
since the given id.

NOTE: This assumes that iterating from a point near the "end" of a
database is inexpensive. Don't use this option for a storage for which
that is not the case.

Here's an example.  We set up a server, using an
invalidation-queue-size of 5:

    >>> addr, admin = start_server(zeo_conf=dict(invalidation_queue_size=5),
    ...                            keep=True)

Now, we'll open a client with a persistent cache, set up some data,
and  then close client:

    >>> import ZEO, transaction
    >>> db = ZEO.DB(addr, client='test')
    >>> conn = db.open()
    >>> for i in range(9):
    ...     conn.root()[i] = conn.root().__class__()
    ...     conn.root()[i].x = 0
    >>> transaction.commit()
    >>> db.close()

We'll open another client, and commit some transactions:

    >>> db = ZEO.DB(addr)
    >>> conn = db.open()
    >>> import transaction
    >>> for i in range(2):
    ...     conn.root()[i].x = 1
    ...     transaction.commit()
    >>> db.close()

If we reopen the first client, we'll do quick verification.  We'll
turn on logging so we can see this:

    >>> import logging, sys
    >>> old_logging_level = logging.getLogger().getEffectiveLevel()
    >>> logging.getLogger().setLevel(logging.INFO)
    >>> handler = logging.StreamHandler(sys.stdout)
    >>> logging.getLogger().addHandler(handler)

    >>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS
    ('localhost', ...
    ('localhost', ...) Recovering 2 invalidations

    >>> logging.getLogger().removeHandler(handler)

    >>> [v.x for v in db.open().root().values()]
    [1, 1, 0, 0, 0, 0, 0, 0, 0]

Now, if we disconnect and commit more than 5 transactions, we'll see
that verification is necessary:

    >>> db.close()
    >>> db = ZEO.DB(addr)
    >>> conn = db.open()
    >>> import transaction
    >>> for i in range(9):
    ...     conn.root()[i].x = 2
    ...     transaction.commit()
    >>> db.close()

    >>> logging.getLogger().addHandler(handler)
    >>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS
    ('localhost', ...
    ('localhost', ...) Verifying cache
    ('localhost', ...) endVerify finishing
    ('localhost', ...) endVerify finished

    >>> logging.getLogger().removeHandler(handler)

    >>> [v.x for v in db.open().root().values()]
    [2, 2, 2, 2, 2, 2, 2, 2, 2]

    >>> db.close()

But if we restart the server with invalidation-age set, we can
do quick verification:

    >>> stop_server(admin)
    >>> addr, admin = start_server(zeo_conf=dict(invalidation_queue_size=5,
    ...                                          invalidation_age=100))
    >>> db = ZEO.DB(addr)
    >>> conn = db.open()
    >>> import transaction
    >>> for i in range(9):
    ...     conn.root()[i].x = 3
    ...     transaction.commit()
    >>> db.close()


    >>> logging.getLogger().addHandler(handler)
    >>> db = ZEO.DB(addr, client='test') # doctest: +ELLIPSIS
    ('localhost', ...
    ('localhost', ...) Recovering 9 invalidations

    >>> logging.getLogger().removeHandler(handler)

    >>> [v.x for v in db.open().root().values()]
    [3, 3, 3, 3, 3, 3, 3, 3, 3]

    >>> db.close()

    >>> logging.getLogger().setLevel(old_logging_level)