~bzr-pqm/bzr/bzr.dev

974.1.27 by aaron.bentley at utoronto
Initial greedy fetch work
1
# Copyright (C) 2005 by Canonical Ltd
2
3
# This program is free software; you can redistribute it and/or modify
4
# it under the terms of the GNU General Public License as published by
5
# the Free Software Foundation; either version 2 of the License, or
6
# (at your option) any later version.
7
8
# This program is distributed in the hope that it will be useful,
9
# but WITHOUT ANY WARRANTY; without even the implied warranty of
10
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11
# GNU General Public License for more details.
12
13
# You should have received a copy of the GNU General Public License
14
# along with this program; if not, write to the Free Software
15
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
1218 by Martin Pool
- fix up import
16
1231 by Martin Pool
- more progress on fetch on top of weaves
17
18
"""Copying of history from one branch to another.
19
20
The basic plan is that every branch knows the history of everything
21
that has merged into it.  As the first step of a merge, pull, or
22
branch operation we copy history from the source into the destination
23
branch.
24
25
The copying is done in a slightly complicated order.  We don't want to
26
add a revision to the store until everything it refers to is also
27
stored, so that if a revision is present we can totally recreate it.
28
However, we can't know what files are included in a revision until we
29
read its inventory.  Therefore, we first pull the XML and hold it in
30
memory until we've updated all of the files referenced.
31
"""
32
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
33
import bzrlib
34
import bzrlib.errors as errors
1563.2.10 by Robert Collins
Change weave store to be a versioned store, using WeaveFiles which maintain integrity without needing explicit 'put' operations.
35
from bzrlib.errors import (InstallFailed, NoSuchRevision,
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
36
                           MissingText)
37
from bzrlib.trace import mutter
38
from bzrlib.progress import ProgressBar
39
from bzrlib.revision import NULL_REVISION
40
from bzrlib.symbol_versioning import *
41
42
1231 by Martin Pool
- more progress on fetch on top of weaves
43
# TODO: Avoid repeatedly opening weaves so many times.
974.1.27 by aaron.bentley at utoronto
Initial greedy fetch work
44
1238 by Martin Pool
- remove a lot of dead code from fetch
45
# XXX: This doesn't handle ghost (not present in branch) revisions at
1240 by Martin Pool
- clean up fetch code and add progress bar
46
# all yet.  I'm not sure they really should be supported.
47
1262 by Martin Pool
- fetch should also copy ancestry records
48
# NOTE: This doesn't copy revisions which may be present but not
49
# merged into the last revision.  I'm not sure we want to do that.
1238 by Martin Pool
- remove a lot of dead code from fetch
50
51
# - get a list of revisions that need to be pulled in
52
# - for each one, pull in that revision file
53
#   and get the inventory, and store the inventory with right
54
#   parents.
55
# - and get the ancestry, and store that with right parents too
56
# - and keep a note of all file ids and version seen
57
# - then go through all files; for each one get the weave,
58
#   and add in all file versions
59
60
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
61
@deprecated_function(zero_eight)
1260 by Martin Pool
- some updates for fetch/update function
62
def greedy_fetch(to_branch, from_branch, revision=None, pb=None):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
63
    """Legacy API, please see branch.fetch(from_branch, last_revision, pb)."""
1219 by Martin Pool
- BROKEN: start refactoring fetch code to work well with weaves
64
    f = Fetcher(to_branch, from_branch, revision, pb)
65
    return f.count_copied, f.failed_revisions
66
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
67
fetch = greedy_fetch
68
1219 by Martin Pool
- BROKEN: start refactoring fetch code to work well with weaves
69
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
70
class RepoFetcher(object):
71
    """Pull revisions and texts from one repository to another.
72
73
    last_revision
74
        if set, try to limit to the data this revision references.
75
76
    after running:
1260 by Martin Pool
- some updates for fetch/update function
77
    count_copied -- number of revisions copied
1534.1.33 by Robert Collins
Move copy_content_into into InterRepository and InterWeaveRepo, and disable the default codepath test as we have optimised paths for all current combinations.
78
79
    This should not be used directory, its essential a object to encapsulate
80
    the logic in InterRepository.fetch().
1260 by Martin Pool
- some updates for fetch/update function
81
    """
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
82
    def __init__(self, to_repository, from_repository, last_revision=None, pb=None):
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
83
        # result variables.
84
        self.failed_revisions = []
85
        self.count_copied = 0
1556.1.4 by Robert Collins
Add a new format for what will become knit, and the surrounding logic to upgrade repositories within metadirs, and tests for the same.
86
        if to_repository.control_files._transport.base == from_repository.control_files._transport.base:
1534.6.13 by Robert Collins
Allow push/pull and branch between branches in the same shared repository.
87
            # check that last_revision is in 'from' and then return a no-operation.
88
            if last_revision not in (None, NULL_REVISION):
89
                from_repository.get_revision(last_revision)
90
            return
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
91
        self.to_repository = to_repository
92
        self.from_repository = from_repository
93
        # must not mutate self._last_revision as its potentially a shared instance
1185.65.27 by Robert Collins
Tweak storage towards mergability.
94
        self._last_revision = last_revision
95
        if pb is None:
96
            self.pb = bzrlib.ui.ui_factory.progress_bar()
97
        else:
98
            self.pb = pb
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
99
        self.from_repository.lock_read()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
100
        try:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
101
            self.to_repository.lock_write()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
102
            try:
103
                self.__fetch()
104
            finally:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
105
                self.to_repository.unlock()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
106
        finally:
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
107
            self.from_repository.unlock()
1185.65.27 by Robert Collins
Tweak storage towards mergability.
108
109
    def __fetch(self):
110
        """Primary worker function.
111
112
        This initialises all the needed variables, and then fetches the 
113
        requested revisions, finally clearing the progress bar.
114
        """
1185.67.2 by Aaron Bentley
Renamed Branch.storage to Branch.repository
115
        self.to_weaves = self.to_repository.weave_store
116
        self.to_control = self.to_repository.control_weaves
117
        self.from_weaves = self.from_repository.weave_store
118
        self.from_control = self.from_repository.control_weaves
1240 by Martin Pool
- clean up fetch code and add progress bar
119
        self.count_total = 0
1185.33.55 by Martin Pool
[patch] weave fetch optimizations (Goffredo Baroncelli)
120
        self.file_ids_names = {}
1392 by Robert Collins
reinstate testfetch test case
121
        try:
1185.65.30 by Robert Collins
Merge integration.
122
            revs = self._revids_to_fetch()
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
123
            # nothing to do
124
            if revs: 
1534.1.11 by Robert Collins
Format tweaks for latest merges.
125
                self._fetch_weave_texts(revs)
126
                self._fetch_inventory_weave(revs)
127
                self._fetch_revision_texts(revs)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
128
                self.count_copied += len(revs)
1185.12.15 by Aaron Bentley
Cleared progress bar properly after fetch
129
        finally:
130
            self.pb.clear()
1231 by Martin Pool
- more progress on fetch on top of weaves
131
1185.65.30 by Robert Collins
Merge integration.
132
    def _revids_to_fetch(self):
1534.1.31 by Robert Collins
Deprecated fetch.fetch and fetch.greedy_fetch for branch.fetch, and move the Repository.fetch internals to InterRepo and InterWeaveRepo.
133
        self.pb.update('get destination history')
1185.65.27 by Robert Collins
Tweak storage towards mergability.
134
        mutter('fetch up to rev {%s}', self._last_revision)
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
135
        if self._last_revision is NULL_REVISION:
136
            # explicit limit of no revisions needed
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
137
            return None
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
138
        if (self._last_revision != None and
1185.65.27 by Robert Collins
Tweak storage towards mergability.
139
            self.to_repository.has_revision(self._last_revision)):
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
140
            return None
141
            
1417.1.13 by Robert Collins
do not download remote ancestry.weave if the target revision we are stopping at is in our local store
142
        try:
1534.4.50 by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running.
143
            return self.to_repository.missing_revision_ids(self.from_repository,
144
                                                           self._last_revision)
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
145
        except errors.NoSuchRevision:
1185.65.27 by Robert Collins
Tweak storage towards mergability.
146
            raise InstallFailed([self._last_revision])
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
147
1534.1.11 by Robert Collins
Format tweaks for latest merges.
148
    def _fetch_revision_texts(self, revs):
1185.65.30 by Robert Collins
Merge integration.
149
        self.to_repository.revision_store.copy_multi(
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
150
            self.from_repository.revision_store,
151
            revs,
152
            pb=self.pb)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
153
1534.1.11 by Robert Collins
Format tweaks for latest merges.
154
    def _fetch_weave_texts(self, revs):
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
155
        file_ids = self.from_repository.fileid_involved_by_set(revs)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
156
        count = 0
157
        num_file_ids = len(file_ids)
158
        for file_id in file_ids:
1185.77.1 by John Arbash Meinel
Simplifying what bzr displays when merging
159
            self.pb.update("merge weaves", count, num_file_ids)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
160
            count +=1
161
            to_weave = self.to_weaves.get_weave_or_empty(file_id,
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
162
                self.to_repository.get_transaction())
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
163
164
            if to_weave.numversions() > 0:
165
                # destination has contents, must merge
1563.2.10 by Robert Collins
Change weave store to be a versioned store, using WeaveFiles which maintain integrity without needing explicit 'put' operations.
166
                from_weave = self.from_weaves.get_weave(file_id,
167
                    self.from_repository.get_transaction())
1563.2.11 by Robert Collins
Consolidate reweave and join as we have no separate usage, make reweave tests apply to all versionedfile implementations and deprecate the old reweave apis.
168
                to_weave.join(from_weave)
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
169
            else:
170
                # destination is empty, just replace it
1563.2.10 by Robert Collins
Change weave store to be a versioned store, using WeaveFiles which maintain integrity without needing explicit 'put' operations.
171
                self.to_weaves.copy_multi(self.from_weaves, [file_id], self.pb)
1534.1.11 by Robert Collins
Format tweaks for latest merges.
172
        self.pb.clear()
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
173
1534.1.11 by Robert Collins
Format tweaks for latest merges.
174
    def _fetch_inventory_weave(self, revs):
1551.3.11 by Aaron Bentley
Merge from Robert
175
        self.pb.update("inventory fetch", 0, 2)
176
        to_weave = self.to_control.get_weave('inventory',
1534.4.43 by Robert Collins
Merge from integration
177
                self.to_repository.get_transaction())
1185.64.4 by Goffredo Baroncelli
Apply some style changes on the basis of Robert suggest
178
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
179
        if to_weave.numversions() > 0:
180
            # destination has contents, must merge
1563.2.10 by Robert Collins
Change weave store to be a versioned store, using WeaveFiles which maintain integrity without needing explicit 'put' operations.
181
            self.pb.update("inventory fetch", 1, 2)
182
            from_weave = self.from_repository.get_inventory_weave()
183
            self.pb.update("inventory fetch", 2, 2)
1563.2.11 by Robert Collins
Consolidate reweave and join as we have no separate usage, make reweave tests apply to all versionedfile implementations and deprecate the old reweave apis.
184
            to_weave.join(from_weave, pb=self.pb, msg='merge inventory')
1185.64.3 by Goffredo Baroncelli
This patch changes the fetch code. Before, the original code expanded every inventory and
185
        else:
186
            # destination is empty, just replace it
1563.2.10 by Robert Collins
Change weave store to be a versioned store, using WeaveFiles which maintain integrity without needing explicit 'put' operations.
187
            self.to_control.copy_multi(self.from_control,
188
                                       ['inventory'],
189
                                       self.pb,
190
                                       self.from_repository.get_transaction())
1185.64.4 by Goffredo Baroncelli
Apply some style changes on the basis of Robert suggest
191
1534.1.11 by Robert Collins
Format tweaks for latest merges.
192
        self.pb.clear()
1240 by Martin Pool
- clean up fetch code and add progress bar
193
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
194
195
class Fetcher(object):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
196
    """Backwards compatability glue for branch.fetch()."""
197
198
    @deprecated_method(zero_eight)
1534.4.41 by Robert Collins
Branch now uses BzrDir reasonably sanely.
199
    def __init__(self, to_branch, from_branch, last_revision=None, pb=None):
1534.1.32 by Robert Collins
Deprecate direct use of fetch.Fetcher.
200
        """Please see branch.fetch()."""
201
        to_branch.fetch(from_branch, last_revision, pb)