~bzr-pqm/bzr/bzr.dev

# Copyright (C) 2006 Canonical Ltd
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

"""On-disk mutex protecting a resource

bzr on-disk objects are locked by the existence of a directory with a
particular name within the control directory.  We use this rather than OS
internal locks (such as flock etc) because they can be seen across all
transports, including http.

Objects can be read if there is only physical read access; therefore 
readers can never be required to create a lock, though they will
check whether a writer is using the lock.  Writers can't detect
whether anyone else is reading from the resource as they write.
This works because of ordering constraints that make sure readers
see a consistent view of existing data.

Waiting for a lock must be done by polling; this can be aborted after
a timeout.

Locks must always be explicitly released, typically from a try/finally
block -- they are not released from a finalizer or when Python
exits.

Locks may fail to be released if the process is abruptly terminated
(machine stop, SIGKILL) or if a remote transport becomes permanently
disconnected.  There is therefore a method to break an existing lock.
This should rarely be used, and generally only with user approval.
Locks contain some information on when the lock was taken and by who
which may guide in deciding whether it can safely be broken.  (This is
similar to the messages displayed by emacs and vim.) Note that if the
lock holder is still alive they will get no notification that the lock
has been broken and will continue their work -- so it is important to be
sure they are actually dead.

A lock is represented on disk by a directory of a particular name,
containing an information file.  Taking a lock is done by renaming a
temporary directory into place.  We use temporary directories because
for all known transports and filesystems we believe that exactly one
attempt to claim the lock will succeed and the others will fail.  (Files
won't do because some filesystems or transports only have
rename-and-overwrite, making it hard to tell who won.)

The desired characteristics are:

* Locks are not reentrant.  (That is, a client that tries to take a 
  lock it already holds may deadlock or fail.)
* Stale locks can be guessed at by a heuristic
* Lost locks can be broken by any client
* Failed lock operations leave little or no mess
* Deadlocks are avoided by having a timeout always in use, clients
  desiring indefinite waits can retry or set a silly big timeout.

Storage formats use the locks, and also need to consider concurrency
issues underneath the lock.  A format may choose not to use a lock
at all for some operations.

LockDirs always operate over a Transport.  The transport may be readonly, in
which case the lock can be queried but not acquired.

Locks are identified by a path name, relative to a base transport.

Calling code will typically want to make sure there is exactly one LockDir
object per actual lock on disk.  This module does nothing to prevent aliasing
and deadlocks will likely occur if the locks are aliased.

In the future we may add a "freshen" method which can be called
by a lock holder to check that their lock has not been broken, and to 
update the timestamp within it.

Example usage:

>>> from bzrlib.transport.memory import MemoryTransport
>>> # typically will be obtained from a BzrDir, Branch, etc
>>> t = MemoryTransport()
>>> l = LockDir(t, 'sample-lock')
>>> l.create()
>>> l.wait_lock()
>>> # do something here
>>> l.unlock()

"""

import os
import time
from cStringIO import StringIO

from bzrlib import (
    errors,
    )
import bzrlib.config
from bzrlib.errors import (
        DirectoryNotEmpty,
        FileExists,
        LockBreakMismatch,
        LockBroken,
        LockContention,
        LockNotHeld,
        NoSuchFile,
        PathError,
        ResourceBusy,
        UnlockableTransport,
        )
from bzrlib.trace import mutter, note
from bzrlib.transport import Transport
from bzrlib.osutils import rand_chars, format_delta
from bzrlib.rio import read_stanza, Stanza
import bzrlib.ui


# XXX: At the moment there is no consideration of thread safety on LockDir
# objects.  This should perhaps be updated - e.g. if two threads try to take a
# lock at the same time they should *both* get it.  But then that's unlikely
# to be a good idea.

# TODO: Perhaps store some kind of note like the bzr command line in the lock
# info?

# TODO: Some kind of callback run while polling a lock to show progress
# indicators.

# TODO: Make sure to pass the right file and directory mode bits to all
# files/dirs created.


_DEFAULT_TIMEOUT_SECONDS = 300
_DEFAULT_POLL_SECONDS = 1.0


class LockDir(object):
    """Write-lock guarding access to data."""

    __INFO_NAME = '/info'

    def __init__(self, transport, path, file_modebits=0644, dir_modebits=0755):
        """Create a new LockDir object.

        The LockDir is initially unlocked - this just creates the object.

        :param transport: Transport which will contain the lock

        :param path: Path to the lock within the base directory of the 
            transport.
        """
        assert isinstance(transport, Transport), \
            ("not a transport: %r" % transport)
        self.transport = transport
        self.path = path
        self._lock_held = False
        self._locked_via_token = False
        self._fake_read_lock = False
        self._held_dir = path + '/held'
        self._held_info_path = self._held_dir + self.__INFO_NAME
        self._file_modebits = file_modebits
        self._dir_modebits = dir_modebits

        self._report_function = note

    def __repr__(self):
        return '%s(%s%s)' % (self.__class__.__name__,
                             self.transport.base,
                             self.path)

    is_held = property(lambda self: self._lock_held)

    def create(self, mode=None):
        """Create the on-disk lock.

        This is typically only called when the object/directory containing the 
        directory is first created.  The lock is not held when it's created.
        """
        if self.transport.is_readonly():
            raise UnlockableTransport(self.transport)
        self.transport.mkdir(self.path, mode=mode)

    def attempt_lock(self):
        """Take the lock; fail if it's already held.
        
        If you wish to block until the lock can be obtained, call wait_lock()
        instead.
        """
        if self._fake_read_lock:
            raise LockContention(self)
        if self.transport.is_readonly():
            raise UnlockableTransport(self.transport)
        try:
            tmpname = '%s/pending.%s.tmp' % (self.path, rand_chars(20))
            try:
                self.transport.mkdir(tmpname)
            except NoSuchFile:
                # This may raise a FileExists exception
                # which is okay, it will be caught later and determined
                # to be a LockContention.
                self.create(mode=self._dir_modebits)
                
                # After creating the lock directory, try again
                self.transport.mkdir(tmpname)

            self.nonce = rand_chars(20)
            info_bytes = self._prepare_info()
            # We use put_file_non_atomic because we just created a new unique
            # directory so we don't have to worry about files existing there.
            # We'll rename the whole directory into place to get atomic
            # properties
            self.transport.put_bytes_non_atomic(tmpname + self.__INFO_NAME,
                                                info_bytes)

            self.transport.rename(tmpname, self._held_dir)
            self._lock_held = True
            self.confirm()
        except errors.PermissionDenied:
            raise
        except (PathError, DirectoryNotEmpty, FileExists, ResourceBusy), e:
            mutter("contention on %r: %s", self, e)
            raise LockContention(self)

    def unlock(self):
        """Release a held lock
        """
        if self._fake_read_lock:
            self._fake_read_lock = False
            return
        if not self._lock_held:
            raise LockNotHeld(self)
        if self._locked_via_token:
            self._locked_via_token = False
            self._lock_held = False
        else:
            # rename before deleting, because we can't atomically remove the
            # whole tree
            tmpname = '%s/releasing.%s.tmp' % (self.path, rand_chars(20))
            # gotta own it to unlock
            self.confirm()
            self.transport.rename(self._held_dir, tmpname)
            self._lock_held = False
            self.transport.delete(tmpname + self.__INFO_NAME)
            self.transport.rmdir(tmpname)

    def break_lock(self):
        """Break a lock not held by this instance of LockDir.

        This is a UI centric function: it uses the bzrlib.ui.ui_factory to
        prompt for input if a lock is detected and there is any doubt about
        it possibly being still active.
        """
        self._check_not_locked()
        holder_info = self.peek()
        if holder_info is not None:
            lock_info = '\n'.join(self._format_lock_info(holder_info))
            if bzrlib.ui.ui_factory.get_boolean("Break %s" % lock_info):
                self.force_break(holder_info)
        
    def force_break(self, dead_holder_info):
        """Release a lock held by another process.

        WARNING: This should only be used when the other process is dead; if
        it still thinks it has the lock there will be two concurrent writers.
        In general the user's approval should be sought for lock breaks.

        dead_holder_info must be the result of a previous LockDir.peek() call;
        this is used to check that it's still held by the same process that
        the user decided was dead.  If this is not the current holder,
        LockBreakMismatch is raised.

        After the lock is broken it will not be held by any process.
        It is possible that another process may sneak in and take the 
        lock before the breaking process acquires it.
        """
        if not isinstance(dead_holder_info, dict):
            raise ValueError("dead_holder_info: %r" % dead_holder_info)
        self._check_not_locked()
        current_info = self.peek()
        if current_info is None:
            # must have been recently released
            return
        if current_info != dead_holder_info:
            raise LockBreakMismatch(self, current_info, dead_holder_info)
        tmpname = '%s/broken.%s.tmp' % (self.path, rand_chars(20))
        self.transport.rename(self._held_dir, tmpname)
        # check that we actually broke the right lock, not someone else;
        # there's a small race window between checking it and doing the 
        # rename.
        broken_info_path = tmpname + self.__INFO_NAME
        broken_info = self._read_info_file(broken_info_path)
        if broken_info != dead_holder_info:
            raise LockBreakMismatch(self, broken_info, dead_holder_info)
        self.transport.delete(broken_info_path)
        self.transport.rmdir(tmpname)

    def _check_not_locked(self):
        """If the lock is held by this instance, raise an error."""
        if self._lock_held:
            raise AssertionError("can't break own lock: %r" % self)

    def confirm(self):
        """Make sure that the lock is still held by this locker.

        This should only fail if the lock was broken by user intervention,
        or if the lock has been affected by a bug.

        If the lock is not thought to be held, raises LockNotHeld.  If
        the lock is thought to be held but has been broken, raises 
        LockBroken.
        """
        if not self._lock_held:
            raise LockNotHeld(self)
        info = self.peek()
        if info is None:
            # no lock there anymore!
            raise LockBroken(self)
        if info.get('nonce') != self.nonce:
            # there is a lock, but not ours
            raise LockBroken(self)
        
    def _read_info_file(self, path):
        """Read one given info file.

        peek() reads the info file of the lock holder, if any.
        """
        return self._parse_info(self.transport.get(path))

    def peek(self):
        """Check if the lock is held by anyone.
        
        If it is held, this returns the lock info structure as a rio Stanza,
        which contains some information about the current lock holder.
        Otherwise returns None.
        """
        try:
            info = self._read_info_file(self._held_info_path)
            assert isinstance(info, dict), \
                    "bad parse result %r" % info
            return info
        except NoSuchFile, e:
            return None

    def _prepare_info(self):
        """Write information about a pending lock to a temporary file.
        """
        import socket
        # XXX: is creating this here inefficient?
        config = bzrlib.config.GlobalConfig()
        try:
            user = config.user_email()
        except errors.NoEmailInUsername:
            user = config.username()
        s = Stanza(hostname=socket.gethostname(),
                   pid=str(os.getpid()),
                   start_time=str(int(time.time())),
                   nonce=self.nonce,
                   user=user,
                   )
        return s.to_string()

    def _parse_info(self, info_file):
        return read_stanza(info_file.readlines()).as_dict()

    def wait_lock(self, timeout=None, poll=None):
        """Wait a certain period for a lock.

        If the lock can be acquired within the bounded time, it
        is taken and this returns.  Otherwise, LockContention
        is raised.  Either way, this function should return within
        approximately `timeout` seconds.  (It may be a bit more if
        a transport operation takes a long time to complete.)
        """
        if timeout is None:
            timeout = _DEFAULT_TIMEOUT_SECONDS
        if poll is None:
            poll = _DEFAULT_POLL_SECONDS

        # XXX: the transport interface doesn't let us guard 
        # against operations there taking a long time.
        deadline = time.time() + timeout
        deadline_str = None
        last_info = None
        while True:
            try:
                self.attempt_lock()
                return
            except LockContention:
                pass
            new_info = self.peek()
            mutter('last_info: %s, new info: %s', last_info, new_info)
            if new_info is not None and new_info != last_info:
                if last_info is None:
                    start = 'Unable to obtain'
                else:
                    start = 'Lock owner changed for'
                last_info = new_info
                formatted_info = self._format_lock_info(new_info)
                if deadline_str is None:
                    deadline_str = time.strftime('%H:%M:%S',
                                                 time.localtime(deadline))
                self._report_function('%s %s\n'
                                      '%s\n' # held by
                                      '%s\n' # locked ... ago
                                      'Will continue to try until %s\n',
                                      start,
                                      formatted_info[0],
                                      formatted_info[1],
                                      formatted_info[2],
                                      deadline_str)

            if time.time() + poll < deadline:
                time.sleep(poll)
            else:
                raise LockContention(self)
    
    def leave_in_place(self):
        self._locked_via_token = True

    def dont_leave_in_place(self):
        self._locked_via_token = False

    def lock_write(self, token=None):
        """Wait for and acquire the lock.
        
        :param token: if this is already locked, then lock_write will fail
            unless the token matches the existing lock.
        :returns: a token if this instance supports tokens, otherwise None.
        :raises TokenLockingNotSupported: when a token is given but this
            instance doesn't support using token locks.
        :raises MismatchedToken: if the specified token doesn't match the token
            of the existing lock.

        A token should be passed in if you know that you have locked the object
        some other way, and need to synchronise this object's state with that
        fact.
         
        XXX: docstring duplicated from LockableFiles.lock_write.
        """
        if token is not None:
            self.validate_token(token)
            self.nonce = token
            self._lock_held = True
            self._locked_via_token = True
            return token
        else:
            self.wait_lock()
            return self.peek().get('nonce')

    def lock_read(self):
        """Compatibility-mode shared lock.

        LockDir doesn't support shared read-only locks, so this 
        just pretends that the lock is taken but really does nothing.
        """
        # At the moment Branches are commonly locked for read, but 
        # we can't rely on that remotely.  Once this is cleaned up,
        # reenable this warning to prevent it coming back in 
        # -- mbp 20060303
        ## warn("LockDir.lock_read falls back to write lock")
        if self._lock_held or self._fake_read_lock:
            raise LockContention(self)
        self._fake_read_lock = True

    def wait(self, timeout=20, poll=0.5):
        """Wait a certain period for a lock to be released."""
        # XXX: the transport interface doesn't let us guard 
        # against operations there taking a long time.
        deadline = time.time() + timeout
        while True:
            if self.peek():
                return
            if time.time() + poll < deadline:
                time.sleep(poll)
            else:
                raise LockContention(self)

    def _format_lock_info(self, info):
        """Turn the contents of peek() into something for the user"""
        lock_url = self.transport.abspath(self.path)
        delta = time.time() - int(info['start_time'])
        return [
            'lock %s' % (lock_url,),
            'held by %(user)s on host %(hostname)s [process #%(pid)s]' % info,
            'locked %s' % (format_delta(delta),),
            ]

    def validate_token(self, token):
        if token is not None:
            info = self.peek()
            if info is None:
                # Lock isn't held
                lock_token = None
            else:
                lock_token = info.get('nonce')
            if token != lock_token:
                raise errors.TokenMismatch(token, lock_token)