~bzr-pqm/bzr/bzr.dev

1615.1.2 by Martin Pool
[merge] robert's knit-performance work
1
# Copyright (C) 2005, 2006 by Canonical Ltd
974.1.27 by aaron.bentley at utoronto
Initial greedy fetch work
2
3
# This program is free software; you can redistribute it and/or modify
4
# it under the terms of the GNU General Public License as published by
5
# the Free Software Foundation; either version 2 of the License, or
6
# (at your option) any later version.
7
8
# This program is distributed in the hope that it will be useful,
9
# but WITHOUT ANY WARRANTY; without even the implied warranty of
10
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11
# GNU General Public License for more details.
12
13
# You should have received a copy of the GNU General Public License
14
# along with this program; if not, write to the Free Software
15
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
1218 by Martin Pool
- fix up import
16
1231 by Martin Pool
- more progress on fetch on top of weaves
17
18
"""Copying of history from one branch to another.
19
20
The basic plan is that every branch knows the history of everything
21
that has merged into it.  As the first step of a merge, pull, or
22
branch operation we copy history from the source into the destination
23
branch.
24
25
The copying is done in a slightly complicated order.  We don't want to
26
add a revision to the store until everything it refers to is also
27
stored, so that if a revision is present we can totally recreate it.
28
However, we can't know what files are included in a revision until we
1563.2.34 by Robert Collins
Remove the commit and rollback transaction methods as misleading, and implement a WriteTransaction
29
read its inventory.  So we query the inventory store of the source for
30
the ids we need, and then pull those ids and finally actually join
31
the inventories.
1231 by Martin Pool
- more progress on fetch on top of weaves
32
"""
33
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
34
import bzrlib
35
import bzrlib.errors as errors
1563.2.10 by Robert Collins
Change weave store to be a versioned store, using WeaveFiles which maintain integrity without needing explicit 'put' operations.
36
from bzrlib.errors import (InstallFailed, NoSuchRevision,
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
37
                           MissingText)
38
from bzrlib.trace import mutter
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
39
from bzrlib.progress import ProgressBar, ProgressPhase
1570.1.14 by Robert Collins
Enforce repository consistency during 'fetch' operations.
40
from bzrlib.reconcile import RepoReconciler
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
41
from bzrlib.revision import NULL_REVISION
42
from bzrlib.symbol_versioning import *
43
44
1231 by Martin Pool
- more progress on fetch on top of weaves
45
# TODO: Avoid repeatedly opening weaves so many times.
974.1.27 by aaron.bentley at utoronto
Initial greedy fetch work
46
1238 by Martin Pool
- remove a lot of dead code from fetch
47
# XXX: This doesn't handle ghost (not present in branch) revisions at
1240 by Martin Pool
- clean up fetch code and add progress bar
48
# all yet.  I'm not sure they really should be supported.
49
1262 by Martin Pool
- fetch should also copy ancestry records
50
# NOTE: This doesn't copy revisions which may be present but not
51
# merged into the last revision.  I'm not sure we want to do that.
1238 by Martin Pool
- remove a lot of dead code from fetch
52
53
# - get a list of revisions that need to be pulled in
54
# - for each one, pull in that revision file
55
#   and get the inventory, and store the inventory with right
56
#   parents.
57
# - and get the ancestry, and store that with right parents too
58
# - and keep a note of all file ids and version seen
59
# - then go through all files; for each one get the weave,
60
#   and add in all file versions
61
62
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
63
@deprecated_function(zero_eight)
1260 by Martin Pool
- some updates for fetch/update function
64
def greedy_fetch(to_branch, from_branch, revision=None, pb=None):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
65
    """Legacy API, please see branch.fetch(from_branch, last_revision, pb)."""
1219 by Martin Pool
- BROKEN: start refactoring fetch code to work well with weaves
66
    f = Fetcher(to_branch, from_branch, revision, pb)
67
    return f.count_copied, f.failed_revisions
68
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
69
fetch = greedy_fetch
70
1219 by Martin Pool
- BROKEN: start refactoring fetch code to work well with weaves
71
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
72
class RepoFetcher(object):
73
    """Pull revisions and texts from one repository to another.
74
75
    last_revision
76
        if set, try to limit to the data this revision references.
77
78
    after running:
1260 by Martin Pool
- some updates for fetch/update function
79
    count_copied -- number of revisions copied
1534.1.33 by Robert Collins
Move copy_content_into into InterRepository and InterWeaveRepo, and disable the default codepath test as we have optimised paths for all current combinations.
80
81
    This should not be used directory, its essential a object to encapsulate
82
    the logic in InterRepository.fetch().
1260 by Martin Pool
- some updates for fetch/update function
83
    """
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
84
    def __init__(self, to_repository, from_repository, last_revision=None, pb=None):
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
85
        # result variables.
86
        self.failed_revisions = []
87
        self.count_copied = 0
1556.1.4 by Robert Collins
Add a new format for what will become knit, and the surrounding logic to upgrade repositories within metadirs, and tests for the same.
88
        if to_repository.control_files._transport.base == from_repository.control_files._transport.base:
1534.6.13 by Robert Collins
Allow push/pull and branch between branches in the same shared repository.
89
            # check that last_revision is in 'from' and then return a no-operation.
90
            if last_revision not in (None, NULL_REVISION):
91
                from_repository.get_revision(last_revision)
92
            return
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
93
        self.to_repository = to_repository
94
        self.from_repository = from_repository
95
        # must not mutate self._last_revision as its potentially a shared instance
1185.65.27 by Robert Collins
Tweak storage towards mergability.
96
        self._last_revision = last_revision
97
        if pb is None:
1594.1.3 by Robert Collins
Fixup pb usage to use nested_progress_bar.
98
            self.pb = bzrlib.ui.ui_factory.nested_progress_bar()
99
            self.nested_pb = self.pb
1185.65.27 by Robert Collins
Tweak storage towards mergability.
100
        else:
101
            self.pb = pb
1594.1.3 by Robert Collins
Fixup pb usage to use nested_progress_bar.
102
            self.nested_pb = None
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
103
        self.from_repository.lock_read()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
104
        try:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
105
            self.to_repository.lock_write()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
106
            try:
107
                self.__fetch()
108
            finally:
1594.1.3 by Robert Collins
Fixup pb usage to use nested_progress_bar.
109
                if self.nested_pb is not None:
110
                    self.nested_pb.finished()
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
111
                self.to_repository.unlock()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
112
        finally:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
113
            self.from_repository.unlock()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
114
115
    def __fetch(self):
116
        """Primary worker function.
117
118
        This initialises all the needed variables, and then fetches the 
119
        requested revisions, finally clearing the progress bar.
120
        """
1185.67.2 by Aaron Bentley
Renamed Branch.storage to Branch.repository
121
        self.to_weaves = self.to_repository.weave_store
122
        self.to_control = self.to_repository.control_weaves
123
        self.from_weaves = self.from_repository.weave_store
124
        self.from_control = self.from_repository.control_weaves
1240 by Martin Pool
- clean up fetch code and add progress bar
125
        self.count_total = 0
1185.33.55 by Martin Pool
[patch] weave fetch optimizations (Goffredo Baroncelli)
126
        self.file_ids_names = {}
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
127
        pp = ProgressPhase('fetch phase', 4, self.pb)
1392 by Robert Collins
reinstate testfetch test case
128
        try:
1185.65.30 by Robert Collins
Merge integration.
129
            revs = self._revids_to_fetch()
1596.2.22 by Robert Collins
Fetch changes to use new pb.
130
            # something to do ?
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
131
            if revs:
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
132
                pp.next_phase()
1534.1.11 by Robert Collins
Format tweaks for latest merges.
133
                self._fetch_weave_texts(revs)
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
134
                pp.next_phase()
1534.1.11 by Robert Collins
Format tweaks for latest merges.
135
                self._fetch_inventory_weave(revs)
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
136
                pp.next_phase()
1534.1.11 by Robert Collins
Format tweaks for latest merges.
137
                self._fetch_revision_texts(revs)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
138
                self.count_copied += len(revs)
1185.12.15 by Aaron Bentley
Cleared progress bar properly after fetch
139
        finally:
140
            self.pb.clear()
1231 by Martin Pool
- more progress on fetch on top of weaves
141
1185.65.30 by Robert Collins
Merge integration.
142
    def _revids_to_fetch(self):
1185.65.27 by Robert Collins
Tweak storage towards mergability.
143
        mutter('fetch up to rev {%s}', self._last_revision)
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
144
        if self._last_revision is NULL_REVISION:
145
            # explicit limit of no revisions needed
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
146
            return None
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
147
        if (self._last_revision != None and
1185.65.27 by Robert Collins
Tweak storage towards mergability.
148
            self.to_repository.has_revision(self._last_revision)):
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
149
            return None
150
            
1417.1.13 by Robert Collins
do not download remote ancestry.weave if the target revision we are stopping at is in our local store
151
        try:
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
152
            return self.to_repository.missing_revision_ids(self.from_repository,
153
                                                           self._last_revision)
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
154
        except errors.NoSuchRevision:
1185.65.27 by Robert Collins
Tweak storage towards mergability.
155
            raise InstallFailed([self._last_revision])
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
156
1534.1.11 by Robert Collins
Format tweaks for latest merges.
157
    def _fetch_weave_texts(self, revs):
1596.2.22 by Robert Collins
Fetch changes to use new pb.
158
        texts_pb = bzrlib.ui.ui_factory.nested_progress_bar()
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
159
        try:
1694.2.6 by Martin Pool
[merge] bzr.dev
160
            file_ids = self.from_repository.fileids_altered_by_revision_ids(revs)
1596.2.22 by Robert Collins
Fetch changes to use new pb.
161
            count = 0
162
            num_file_ids = len(file_ids)
1694.2.6 by Martin Pool
[merge] bzr.dev
163
            for file_id, required_versions in file_ids.items():
1596.2.22 by Robert Collins
Fetch changes to use new pb.
164
                texts_pb.update("fetch texts", count, num_file_ids)
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
165
                count +=1
1666.1.8 by Robert Collins
Remove unneeded optimisation in fetch_texts - the interversioned file optimisers now do this.
166
                to_weave = self.to_weaves.get_weave_or_empty(file_id,
167
                    self.to_repository.get_transaction())
168
                from_weave = self.from_weaves.get_weave(file_id,
169
                    self.from_repository.get_transaction())
170
                # we fetch all the texts, because texts do
171
                # not reference anything, and its cheap enough
1753.1.2 by Robert Collins
Shrink memory usage in branch.
172
                to_weave.join(from_weave, version_ids=required_versions) 
1759.2.2 by Jelmer Vernooij
Revert some of my spelling fixes and fix some typos after review by Aaron.
173
                # we don't need *all* of this data anymore, but we dont know
1753.1.2 by Robert Collins
Shrink memory usage in branch.
174
                # what we do. This cache clearing will result in a new read 
175
                # of the knit data when we do the checkout, but probably we
176
                # want to emit the needed data on the fly rather than at the
177
                # end anyhow.
178
                # the from weave should know not to cache data being joined,
179
                # but its ok to ask it to clear.
180
                from_weave.clear_cache()
181
                to_weave.clear_cache()
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
182
        finally:
1596.2.22 by Robert Collins
Fetch changes to use new pb.
183
            texts_pb.finished()
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
184
1534.1.11 by Robert Collins
Format tweaks for latest merges.
185
    def _fetch_inventory_weave(self, revs):
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
186
        pb = bzrlib.ui.ui_factory.nested_progress_bar()
187
        try:
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
188
            pb.update("fetch inventory", 0, 2)
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
189
            to_weave = self.to_control.get_weave('inventory',
190
                    self.to_repository.get_transaction())
1596.2.22 by Robert Collins
Fetch changes to use new pb.
191
    
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
192
            child_pb = bzrlib.ui.ui_factory.nested_progress_bar()
193
            try:
1759.2.2 by Jelmer Vernooij
Revert some of my spelling fixes and fix some typos after review by Aaron.
194
                # just merge, this is optimisable and its means we don't
1551.2.37 by Aaron Bentley
Merge from bzr.ab.integration
195
                # copy unreferenced data such as not-needed inventories.
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
196
                pb.update("fetch inventory", 1, 3)
1551.2.37 by Aaron Bentley
Merge from bzr.ab.integration
197
                from_weave = self.from_repository.get_inventory_weave()
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
198
                pb.update("fetch inventory", 2, 3)
1551.2.37 by Aaron Bentley
Merge from bzr.ab.integration
199
                # we fetch only the referenced inventories because we do not
200
                # know for unselected inventories whether all their required
201
                # texts are present in the other repository - it could be
202
                # corrupt.
203
                to_weave.join(from_weave, pb=child_pb, msg='merge inventory',
204
                              version_ids=revs)
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
205
            finally:
206
                child_pb.finished()
207
        finally:
208
            pb.finished()
1240 by Martin Pool
- clean up fetch code and add progress bar
209
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
210
1563.2.31 by Robert Collins
Convert Knit repositories to use knits.
211
class GenericRepoFetcher(RepoFetcher):
212
    """This is a generic repo to repo fetcher.
213
214
    This makes minimal assumptions about repo layout and contents.
215
    It triggers a reconciliation after fetching to ensure integrity.
216
    """
217
218
    def _fetch_revision_texts(self, revs):
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
219
        """Fetch revision object texts"""
1596.2.22 by Robert Collins
Fetch changes to use new pb.
220
        rev_pb = bzrlib.ui.ui_factory.nested_progress_bar()
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
221
        try:
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
222
            to_txn = self.to_transaction = self.to_repository.get_transaction()
1596.2.22 by Robert Collins
Fetch changes to use new pb.
223
            count = 0
224
            total = len(revs)
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
225
            to_store = self.to_repository._revision_store
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
226
            for rev in revs:
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
227
                pb = bzrlib.ui.ui_factory.nested_progress_bar()
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
228
                try:
1615.1.2 by Martin Pool
[merge] robert's knit-performance work
229
                    pb.update('copying revisions', count, total)
230
                    try:
231
                        sig_text = self.from_repository.get_signature_text(rev)
232
                        to_store.add_revision_signature_text(rev, sig_text, to_txn)
233
                    except errors.NoSuchRevision:
234
                        # not signed.
235
                        pass
236
                    to_store.add_revision(self.from_repository.get_revision(rev),
237
                                          to_txn)
238
                    count += 1
239
                finally:
240
                    pb.finished()
1596.2.22 by Robert Collins
Fetch changes to use new pb.
241
            # fixup inventory if needed: 
242
            # this is expensive because we have no inverse index to current ghosts.
243
            # but on local disk its a few seconds and sftp push is already insane.
244
            # so we just-do-it.
245
            # FIXME: repository should inform if this is needed.
246
            self.to_repository.reconcile()
1551.2.36 by Aaron Bentley
Make pull update the progress bar more nicely
247
        finally:
1596.2.22 by Robert Collins
Fetch changes to use new pb.
248
            rev_pb.finished()
1563.2.31 by Robert Collins
Convert Knit repositories to use knits.
249
    
250
251
class KnitRepoFetcher(RepoFetcher):
252
    """This is a knit format repository specific fetcher.
253
254
    This differs from the GenericRepoFetcher by not doing a 
255
    reconciliation after copying, and using knit joining to
256
    copy revision texts.
257
    """
258
259
    def _fetch_revision_texts(self, revs):
260
        # may need to be a InterRevisionStore call here.
261
        from_transaction = self.from_repository.get_transaction()
262
        to_transaction = self.to_repository.get_transaction()
1563.2.35 by Robert Collins
cleanup deprecation warnings and finish conversion so the inventory is knit based too.
263
        to_sf = self.to_repository._revision_store.get_signature_file(
1563.2.31 by Robert Collins
Convert Knit repositories to use knits.
264
            to_transaction)
1563.2.35 by Robert Collins
cleanup deprecation warnings and finish conversion so the inventory is knit based too.
265
        from_sf = self.from_repository._revision_store.get_signature_file(
1563.2.31 by Robert Collins
Convert Knit repositories to use knits.
266
            from_transaction)
1596.2.22 by Robert Collins
Fetch changes to use new pb.
267
        to_sf.join(from_sf, version_ids=revs, ignore_missing=True)
1563.2.31 by Robert Collins
Convert Knit repositories to use knits.
268
        to_rf = self.to_repository._revision_store.get_revision_file(
269
            to_transaction)
270
        from_rf = self.from_repository._revision_store.get_revision_file(
271
            from_transaction)
1596.2.22 by Robert Collins
Fetch changes to use new pb.
272
        to_rf.join(from_rf, version_ids=revs)
1563.2.31 by Robert Collins
Convert Knit repositories to use knits.
273
274
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
275
class Fetcher(object):
1759.2.1 by Jelmer Vernooij
Fix some types (found using aspell).
276
    """Backwards compatibility glue for branch.fetch()."""
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
277
278
    @deprecated_method(zero_eight)
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
279
    def __init__(self, to_branch, from_branch, last_revision=None, pb=None):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
280
        """Please see branch.fetch()."""
281
        to_branch.fetch(from_branch, last_revision, pb)