~abentley/bzrtools/bzrtools.dev

Hello Aaron. Here's yet another bug report we received at Debian's BTS. :)

Thanks.

----- Forwarded message from Loïc Minier <lool@dooz.org> -----

From: Loïc Minier <lool@dooz.org>

To: Debian Bug Tracking System <submit@bugs.debian.org>

Date: Mon, 30 Jul 2007 12:27:11 +0200

Subject: Bug#435240: multi-pull is slow because it doesn't reuse connections

Package: bzrtools

Version: 0.18.0-1

Severity: wishlist

Tags: patch

        Hi,

 multi-pull is relatively slow because it opens a connection for each

 branch (especially ssh connections are slow).  I'm attaching a patch to

 reuse the connection for all branches with the same URL base.  I hope

 it's not so ugly that you poke yourself an eye out.

 The old behavior isn't available anymore with the patch, but perhaps

 this is a problem for some transports?  I have no idea.

   Bye,

-- System Information:

Debian Release: lenny/sid

  APT prefers unstable

  APT policy: (500, 'unstable'), (1, 'experimental')

Architecture: i386 (i686)

Kernel: Linux 2.6.22-1-686 (SMP w/2 CPU cores)

Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8)

Shell: /bin/sh linked to /bin/dash

Versions of packages bzrtools depends on:

ii  bzr                           0.18-1     Bazaar, the next-generation distri

ii  patch                         2.5.9-4    Apply a diff file to an original

ii  python                        2.4.4-6    An interactive high-level object-o

ii  python-central                0.5.14     register and build utility for Pyt

Versions of packages bzrtools recommends:

ii  graphviz                      2.12-3     rich set of graph drawing tools

ii  rsync                         2.6.9-3    fast remote file copy program (lik

-- no debconf information

-- 

Loïc Minier

--- bzrtools-0.18.0/debian/changelog

+++ bzrtools-0.18.0/debian/changelog

@@ -1,3 +1,11 @@

+bzrtools (0.18.0-1.1) UNRELEASED; urgency=low

+

+  * Non-maintainer upload.

+  * Group branches by URL base and reuse one transport per URL base to avoid

+    reopening a connection for each pull in multi-pull.

+

+ -- Loic Minier <lool@dooz.org>  Mon, 30 Jul 2007 12:06:43 +0200

+

 bzrtools (0.18.0-1) unstable; urgency=low

   [ Arnaud Fontaine ]

--- bzrtools-0.18.0.orig/__init__.py

+++ bzrtools-0.18.0/__init__.py

@@ -503,27 +503,72 @@

         if not t.listable():

             print "Can't list this type of location."

             return 3

-        for branch, wt in iter_branch_tree(t):

-            if wt is None:

-                pullable = branch

-            else:

-                pullable = wt

-            parent = branch.get_parent()

-            if parent is None:

-                continue

-            if wt is not None:

-                base = wt.basedir

-            else:

-                base = branch.base

-            if base.startswith(t.base):

-                relpath = base[len(t.base):].rstrip('/')

-            else:

-                relpath = base

-            print "Pulling %s from %s" % (relpath, parent)

-            try:

-                pullable.pull(Branch.open(parent))

-            except Exception, e:

-                print e

+        print "Grouping branches by URL"

+        by_urlbase = pullable_infos_by_urlbase(t)

+        for urlbase in by_urlbase:

+            print "Processing branches for %s/" % urlbase

+            urlbase_transport = get_transport(urlbase)

+            for pi in by_urlbase[urlbase]:

+                pullable = pi.get_pullable()

+                relpath = get_relpath(t.base, pi.get_base())

+                parent = pi.get_parent()

+                from bzrtools import bzrdir_from_transport

+                pull_transport = urlbase_transport.clone(get_relpath(urlbase, parent))

+                bzrdir = bzrdir_from_transport(pull_transport)

+                pull_branch = bzrdir.open_branch()

+                print "Pulling %s from %s" % (relpath, parent)

+                try:

+                    pullable.pull(pull_branch)

+                except Exception, e:

+                    print e

+

+

+def get_relpath(base, path):

+    if path.startswith(base):

+        return path[len(base):].rstrip('/')

+    else:

+        return path

+

+

+class PullableInfo:

+    def __init__(self, branch, wt):

+        self.branch = branch

+        self.wt = wt

+

+    def get_pullable(self):

+        if self.wt is None:

+            return self.branch

+        return self.wt

+

+    def get_parent(self):

+        return self.branch.get_parent()

+

+    def get_base(self):

+        if self.wt is not None:

+            return self.wt.basedir

+        return self.branch.base

+

+    def get_urlbase(self):

+        import re

+        # always matches at least the empty string

+        urlbase_pattern = re.compile("^(([^:]*://)?([^/]*))")

+        return urlbase_pattern.match(self.get_parent()).groups()[0]

+

+

+def pullable_infos_by_urlbase(t):

+    pullables_by_urlbase = {}

+    from bzrtools import iter_branch_tree

+    for branch, wt in iter_branch_tree(t):

+        parent = branch.get_parent()

+        if parent is None:

+            continue

+        pullable_info = PullableInfo(branch, wt)

+        urlbase = pullable_info.get_urlbase()

+        try:

+            pullables_by_urlbase[urlbase] += (pullable_info, )

+        except KeyError:

+            pullables_by_urlbase[urlbase] = (pullable_info, )

+    return pullables_by_urlbase

 class cmd_branch_mark(bzrlib.commands.Command):