~bzr-pqm/bzr/bzr.dev

974.1.27 by aaron.bentley at utoronto
Initial greedy fetch work
1
# Copyright (C) 2005 by Canonical Ltd
2
3
# This program is free software; you can redistribute it and/or modify
4
# it under the terms of the GNU General Public License as published by
5
# the Free Software Foundation; either version 2 of the License, or
6
# (at your option) any later version.
7
8
# This program is distributed in the hope that it will be useful,
9
# but WITHOUT ANY WARRANTY; without even the implied warranty of
10
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11
# GNU General Public License for more details.
12
13
# You should have received a copy of the GNU General Public License
14
# along with this program; if not, write to the Free Software
15
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
1218 by Martin Pool
- fix up import
16
1231 by Martin Pool
- more progress on fetch on top of weaves
17
18
"""Copying of history from one branch to another.
19
20
The basic plan is that every branch knows the history of everything
21
that has merged into it.  As the first step of a merge, pull, or
22
branch operation we copy history from the source into the destination
23
branch.
24
25
The copying is done in a slightly complicated order.  We don't want to
26
add a revision to the store until everything it refers to is also
27
stored, so that if a revision is present we can totally recreate it.
28
However, we can't know what files are included in a revision until we
29
read its inventory.  Therefore, we first pull the XML and hold it in
30
memory until we've updated all of the files referenced.
31
"""
32
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
33
import bzrlib
34
import bzrlib.errors as errors
35
from bzrlib.errors import (InstallFailed, NoSuchRevision, WeaveError,
36
                           MissingText)
37
from bzrlib.trace import mutter
38
from bzrlib.progress import ProgressBar
39
from bzrlib.revision import NULL_REVISION
40
from bzrlib.symbol_versioning import *
41
42
1231 by Martin Pool
- more progress on fetch on top of weaves
43
# TODO: Avoid repeatedly opening weaves so many times.
974.1.27 by aaron.bentley at utoronto
Initial greedy fetch work
44
1238 by Martin Pool
- remove a lot of dead code from fetch
45
# XXX: This doesn't handle ghost (not present in branch) revisions at
1240 by Martin Pool
- clean up fetch code and add progress bar
46
# all yet.  I'm not sure they really should be supported.
47
1262 by Martin Pool
- fetch should also copy ancestry records
48
# NOTE: This doesn't copy revisions which may be present but not
49
# merged into the last revision.  I'm not sure we want to do that.
1238 by Martin Pool
- remove a lot of dead code from fetch
50
51
# - get a list of revisions that need to be pulled in
52
# - for each one, pull in that revision file
53
#   and get the inventory, and store the inventory with right
54
#   parents.
55
# - and get the ancestry, and store that with right parents too
56
# - and keep a note of all file ids and version seen
57
# - then go through all files; for each one get the weave,
58
#   and add in all file versions
59
60
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
61
@deprecated_function(zero_eight)
1260 by Martin Pool
- some updates for fetch/update function
62
def greedy_fetch(to_branch, from_branch, revision=None, pb=None):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
63
    """Legacy API, please see branch.fetch(from_branch, last_revision, pb)."""
1219 by Martin Pool
- BROKEN: start refactoring fetch code to work well with weaves
64
    f = Fetcher(to_branch, from_branch, revision, pb)
65
    return f.count_copied, f.failed_revisions
66
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
67
fetch = greedy_fetch
68
1219 by Martin Pool
- BROKEN: start refactoring fetch code to work well with weaves
69
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
70
class RepoFetcher(object):
71
    """Pull revisions and texts from one repository to another.
72
73
    last_revision
74
        if set, try to limit to the data this revision references.
75
76
    after running:
1260 by Martin Pool
- some updates for fetch/update function
77
    count_copied -- number of revisions copied
1534.1.33 by Robert Collins
Move copy_content_into into InterRepository and InterWeaveRepo, and disable the default codepath test as we have optimised paths for all current combinations.
78
79
    This should not be used directory, its essential a object to encapsulate
80
    the logic in InterRepository.fetch().
1260 by Martin Pool
- some updates for fetch/update function
81
    """
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
82
    def __init__(self, to_repository, from_repository, last_revision=None, pb=None):
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
83
        # result variables.
84
        self.failed_revisions = []
85
        self.count_copied = 0
1556.1.4 by Robert Collins
Add a new format for what will become knit, and the surrounding logic to upgrade repositories within metadirs, and tests for the same.
86
        if to_repository.control_files._transport.base == from_repository.control_files._transport.base:
1534.6.13 by Robert Collins
Allow push/pull and branch between branches in the same shared repository.
87
            # check that last_revision is in 'from' and then return a no-operation.
88
            if last_revision not in (None, NULL_REVISION):
89
                from_repository.get_revision(last_revision)
90
            return
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
91
        self.to_repository = to_repository
92
        self.from_repository = from_repository
93
        # must not mutate self._last_revision as its potentially a shared instance
1185.65.27 by Robert Collins
Tweak storage towards mergability.
94
        self._last_revision = last_revision
95
        if pb is None:
96
            self.pb = bzrlib.ui.ui_factory.progress_bar()
97
        else:
98
            self.pb = pb
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
99
        self.from_repository.lock_read()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
100
        try:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
101
            self.to_repository.lock_write()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
102
            try:
103
                self.__fetch()
104
            finally:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
105
                self.to_repository.unlock()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
106
        finally:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
107
            self.from_repository.unlock()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
108
109
    def __fetch(self):
110
        """Primary worker function.
111
112
        This initialises all the needed variables, and then fetches the 
113
        requested revisions, finally clearing the progress bar.
114
        """
1185.67.2 by Aaron Bentley
Renamed Branch.storage to Branch.repository
115
        self.to_weaves = self.to_repository.weave_store
116
        self.to_control = self.to_repository.control_weaves
117
        self.from_weaves = self.from_repository.weave_store
118
        self.from_control = self.from_repository.control_weaves
1240 by Martin Pool
- clean up fetch code and add progress bar
119
        self.count_total = 0
1185.33.55 by Martin Pool
[patch] weave fetch optimizations (Goffredo Baroncelli)
120
        self.file_ids_names = {}
1392 by Robert Collins
reinstate testfetch test case
121
        try:
1185.65.30 by Robert Collins
Merge integration.
122
            revs = self._revids_to_fetch()
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
123
            # nothing to do
124
            if revs: 
1534.1.11 by Robert Collins
Format tweaks for latest merges.
125
                self._fetch_weave_texts(revs)
126
                self._fetch_inventory_weave(revs)
127
                self._fetch_revision_texts(revs)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
128
                self.count_copied += len(revs)
1185.12.15 by Aaron Bentley
Cleared progress bar properly after fetch
129
        finally:
130
            self.pb.clear()
1231 by Martin Pool
- more progress on fetch on top of weaves
131
1185.65.30 by Robert Collins
Merge integration.
132
    def _revids_to_fetch(self):
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
133
        self.pb.update('get destination history')
1185.65.27 by Robert Collins
Tweak storage towards mergability.
134
        mutter('fetch up to rev {%s}', self._last_revision)
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
135
        if self._last_revision is NULL_REVISION:
136
            # explicit limit of no revisions needed
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
137
            return None
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
138
        if (self._last_revision != None and
1185.65.27 by Robert Collins
Tweak storage towards mergability.
139
            self.to_repository.has_revision(self._last_revision)):
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
140
            return None
141
            
1417.1.13 by Robert Collins
do not download remote ancestry.weave if the target revision we are stopping at is in our local store
142
        try:
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
143
            return self.to_repository.missing_revision_ids(self.from_repository,
144
                                                           self._last_revision)
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
145
        except errors.NoSuchRevision:
1185.65.27 by Robert Collins
Tweak storage towards mergability.
146
            raise InstallFailed([self._last_revision])
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
147
1534.1.11 by Robert Collins
Format tweaks for latest merges.
148
    def _fetch_revision_texts(self, revs):
1185.65.30 by Robert Collins
Merge integration.
149
        self.to_repository.revision_store.copy_multi(
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
150
            self.from_repository.revision_store,
151
            revs,
152
            pb=self.pb)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
153
1534.1.11 by Robert Collins
Format tweaks for latest merges.
154
    def _fetch_weave_texts(self, revs):
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
155
        file_ids = self.from_repository.fileid_involved_by_set(revs)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
156
        count = 0
157
        num_file_ids = len(file_ids)
158
        for file_id in file_ids:
1185.77.1 by John Arbash Meinel
Simplifying what bzr displays when merging
159
            self.pb.update("merge weaves", count, num_file_ids)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
160
            count +=1
161
            to_weave = self.to_weaves.get_weave_or_empty(file_id,
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
162
                self.to_repository.get_transaction())
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
163
            from_weave = self.from_weaves.get_weave(file_id,
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
164
                self.from_repository.get_transaction())
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
165
166
            if to_weave.numversions() > 0:
167
                # destination has contents, must merge
168
                try:
169
                    to_weave.join(from_weave)
170
                except errors.WeaveParentMismatch:
171
                    to_weave.reweave(from_weave)
172
            else:
173
                # destination is empty, just replace it
1534.1.11 by Robert Collins
Format tweaks for latest merges.
174
                to_weave = from_weave.copy()
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
175
176
            self.to_weaves.put_weave(file_id, to_weave,
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
177
                self.to_repository.get_transaction())
1534.1.11 by Robert Collins
Format tweaks for latest merges.
178
        self.pb.clear()
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
179
1534.1.11 by Robert Collins
Format tweaks for latest merges.
180
    def _fetch_inventory_weave(self, revs):
1551.3.11 by Aaron Bentley
Merge from Robert
181
        self.pb.update("inventory fetch", 0, 2)
1185.65.30 by Robert Collins
Merge integration.
182
        from_weave = self.from_repository.get_inventory_weave()
183
        to_weave = self.to_repository.get_inventory_weave()
1551.3.11 by Aaron Bentley
Merge from Robert
184
        self.pb.update("inventory fetch", 1, 2)
185
        to_weave = self.to_control.get_weave('inventory',
1534.4.43 by Robert Collins
Merge from integration
186
                self.to_repository.get_transaction())
1551.3.11 by Aaron Bentley
Merge from Robert
187
        self.pb.update("inventory fetch", 2, 2)
1185.64.4 by Goffredo Baroncelli
Apply some style changes on the basis of Robert suggest
188
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
189
        if to_weave.numversions() > 0:
190
            # destination has contents, must merge
191
            try:
1551.3.11 by Aaron Bentley
Merge from Robert
192
                to_weave.join(from_weave, pb=self.pb, msg='merge inventory')
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
193
            except errors.WeaveParentMismatch:
1551.3.11 by Aaron Bentley
Merge from Robert
194
                to_weave.reweave(from_weave, pb=self.pb, msg='reweave inventory')
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
195
        else:
196
            # destination is empty, just replace it
1534.1.11 by Robert Collins
Format tweaks for latest merges.
197
            to_weave = from_weave.copy()
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
198
199
        self.to_control.put_weave('inventory', to_weave,
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
200
            self.to_repository.get_transaction())
1185.64.4 by Goffredo Baroncelli
Apply some style changes on the basis of Robert suggest
201
1534.1.11 by Robert Collins
Format tweaks for latest merges.
202
        self.pb.clear()
1240 by Martin Pool
- clean up fetch code and add progress bar
203
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
204
205
class Fetcher(object):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
206
    """Backwards compatability glue for branch.fetch()."""
207
208
    @deprecated_method(zero_eight)
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
209
    def __init__(self, to_branch, from_branch, last_revision=None, pb=None):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
210
        """Please see branch.fetch()."""
211
        to_branch.fetch(from_branch, last_revision, pb)