4763.2.4
by John Arbash Meinel
merge bzr.2.1 in preparation for NEWS entry. |
1 |
# Copyright (C) 2006-2010 Canonical Ltd
|
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
2 |
#
|
3 |
# This program is free software; you can redistribute it and/or modify
|
|
4 |
# it under the terms of the GNU General Public License as published by
|
|
5 |
# the Free Software Foundation; either version 2 of the License, or
|
|
6 |
# (at your option) any later version.
|
|
7 |
#
|
|
8 |
# This program is distributed in the hope that it will be useful,
|
|
9 |
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
10 |
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
11 |
# GNU General Public License for more details.
|
|
12 |
#
|
|
13 |
# You should have received a copy of the GNU General Public License
|
|
14 |
# along with this program; if not, write to the Free Software
|
|
4183.7.1
by Sabin Iacob
update FSF mailing address |
15 |
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
16 |
|
6379.6.7
by Jelmer Vernooij
Move importing from future until after doc string, otherwise the doc string will disappear. |
17 |
"""Server-side repository related request implementations."""
|
18 |
||
6379.6.3
by Jelmer Vernooij
Use absolute_import. |
19 |
from __future__ import absolute_import |
20 |
||
3211.5.2
by Robert Collins
Change RemoteRepository.get_parent_map to use bz2 not gzip for compression. |
21 |
import bz2 |
2571.2.2
by Ian Clatworthy
use basename as poolie recommended |
22 |
import os |
4032.3.7
by Robert Collins
Move write locking and write group responsibilities into the Sink objects themselves, allowing complete avoidance of unnecessary calls when the sink is a RemoteSink. |
23 |
import Queue |
2018.18.2
by Martin Pool
smart method Repository.tarball actually returns the tarball |
24 |
import sys |
25 |
import tempfile |
|
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
26 |
import threading |
6280.9.4
by Jelmer Vernooij
use zlib instead. |
27 |
import zlib |
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
28 |
|
3638.3.2
by Vincent Ladeuil
Fix all calls to tempfile.mkdtemp to osutils.mkdtemp. |
29 |
from bzrlib import ( |
2694.5.4
by Jelmer Vernooij
Move bzrlib.util.bencode to bzrlib._bencode_py. |
30 |
bencode, |
3638.3.2
by Vincent Ladeuil
Fix all calls to tempfile.mkdtemp to osutils.mkdtemp. |
31 |
errors, |
6118.2.9
by John Arbash Meinel
Add some tests for how the estimator works. |
32 |
estimate_compressed_size, |
6282.6.6
by Jelmer Vernooij
Implement server side. |
33 |
inventory as _mod_inventory, |
34 |
inventory_delta, |
|
3638.3.2
by Vincent Ladeuil
Fix all calls to tempfile.mkdtemp to osutils.mkdtemp. |
35 |
osutils, |
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
36 |
pack, |
6118.2.1
by John Arbash Meinel
Refactor the Remote get_parent_map loop, and change how we compute how much prefetch to do. |
37 |
trace, |
4634.124.5
by Martin Pool
Warn about inventory-delta streams when encoding for the network |
38 |
ui, |
6341.1.4
by Jelmer Vernooij
Move more functionality to vf_search. |
39 |
vf_search, |
3638.3.2
by Vincent Ladeuil
Fix all calls to tempfile.mkdtemp to osutils.mkdtemp. |
40 |
)
|
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
41 |
from bzrlib.bzrdir import BzrDir |
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
42 |
from bzrlib.smart.request import ( |
43 |
FailedSmartServerResponse, |
|
44 |
SmartServerRequest, |
|
45 |
SuccessfulSmartServerResponse, |
|
46 |
)
|
|
4419.2.5
by Andrew Bennetts
Add Repository.get_rev_id_for_revno, and use it both as the _ensure_real fallback and as the server-side implementation. |
47 |
from bzrlib.repository import _strip_NULL_ghosts, network_format_registry |
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
48 |
from bzrlib import revision as _mod_revision |
4476.3.15
by Andrew Bennetts
Partially working fallback for pre-1.17 servers. |
49 |
from bzrlib.versionedfile import ( |
6282.6.31
by Jelmer Vernooij
Use record streams in get_inventories call. |
50 |
ChunkedContentFactory, |
4476.3.15
by Andrew Bennetts
Partially working fallback for pre-1.17 servers. |
51 |
NetworkRecordStream, |
52 |
record_to_fulltext_bytes, |
|
53 |
)
|
|
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
54 |
|
55 |
||
2018.5.56
by Robert Collins
Factor out code we expect to be common in SmartServerRequestHasRevision to SmartServerRepositoryRequest (Robert Collins, Vincent Ladeuil). |
56 |
class SmartServerRepositoryRequest(SmartServerRequest): |
57 |
"""Common base class for Repository requests."""
|
|
58 |
||
59 |
def do(self, path, *args): |
|
60 |
"""Execute a repository request.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
61 |
|
2692.1.10
by Andrew Bennetts
More docstring polish |
62 |
All Repository requests take a path to the repository as their first
|
63 |
argument. The repository must be at the exact path given by the
|
|
64 |
client - no searching is done.
|
|
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
65 |
|
2018.5.56
by Robert Collins
Factor out code we expect to be common in SmartServerRequestHasRevision to SmartServerRepositoryRequest (Robert Collins, Vincent Ladeuil). |
66 |
The actual logic is delegated to self.do_repository_request.
|
67 |
||
2692.1.10
by Andrew Bennetts
More docstring polish |
68 |
:param client_path: The path for the repository as received from the
|
69 |
client.
|
|
70 |
:return: A SmartServerResponse from self.do_repository_request().
|
|
2018.5.56
by Robert Collins
Factor out code we expect to be common in SmartServerRequestHasRevision to SmartServerRepositoryRequest (Robert Collins, Vincent Ladeuil). |
71 |
"""
|
2692.1.1
by Andrew Bennetts
Add translate_client_path method to SmartServerRequest. |
72 |
transport = self.transport_from_client_path(path) |
2018.5.56
by Robert Collins
Factor out code we expect to be common in SmartServerRequestHasRevision to SmartServerRepositoryRequest (Robert Collins, Vincent Ladeuil). |
73 |
bzrdir = BzrDir.open_from_transport(transport) |
3184.1.10
by Robert Collins
Change the smart server verb for Repository.stream_revisions_chunked to use SearchResults as the request mechanism for downloads. |
74 |
# Save the repository for use with do_body.
|
75 |
self._repository = bzrdir.open_repository() |
|
76 |
return self.do_repository_request(self._repository, *args) |
|
77 |
||
78 |
def do_repository_request(self, repository, *args): |
|
79 |
"""Override to provide an implementation for a verb."""
|
|
80 |
# No-op for verbs that take bodies (None as a result indicates a body
|
|
81 |
# is expected)
|
|
82 |
return None |
|
2018.5.56
by Robert Collins
Factor out code we expect to be common in SmartServerRequestHasRevision to SmartServerRepositoryRequest (Robert Collins, Vincent Ladeuil). |
83 |
|
4332.2.1
by Robert Collins
Fix bug 360791 by not raising an error when a smart server is asked for more content than it has locally; the client is assumed to be monitoring what it gets. |
84 |
def recreate_search(self, repository, search_bytes, discard_excess=False): |
85 |
"""Recreate a search from its serialised form.
|
|
86 |
||
87 |
:param discard_excess: If True, and the search refers to data we don't
|
|
88 |
have, just silently accept that fact - the verb calling
|
|
89 |
recreate_search trusts that clients will look for missing things
|
|
90 |
they expected and get it from elsewhere.
|
|
91 |
"""
|
|
5539.2.14
by Andrew Bennetts
Don't add a new verb; instead just teach the client to fallback if it gets a BadSearch error. |
92 |
if search_bytes == 'everything': |
6341.1.4
by Jelmer Vernooij
Move more functionality to vf_search. |
93 |
return vf_search.EverythingResult(repository), None |
4070.9.5
by Andrew Bennetts
Better wire protocol: don't shoehorn MiniSearchResult serialisation into previous serialisation format. |
94 |
lines = search_bytes.split('\n') |
95 |
if lines[0] == 'ancestry-of': |
|
4070.9.14
by Andrew Bennetts
Tweaks requested by Robert's review. |
96 |
heads = lines[1:] |
6341.1.4
by Jelmer Vernooij
Move more functionality to vf_search. |
97 |
search_result = vf_search.PendingAncestryResult(heads, repository) |
4070.9.5
by Andrew Bennetts
Better wire protocol: don't shoehorn MiniSearchResult serialisation into previous serialisation format. |
98 |
return search_result, None |
99 |
elif lines[0] == 'search': |
|
4332.2.1
by Robert Collins
Fix bug 360791 by not raising an error when a smart server is asked for more content than it has locally; the client is assumed to be monitoring what it gets. |
100 |
return self.recreate_search_from_recipe(repository, lines[1:], |
101 |
discard_excess=discard_excess) |
|
4070.9.5
by Andrew Bennetts
Better wire protocol: don't shoehorn MiniSearchResult serialisation into previous serialisation format. |
102 |
else: |
103 |
return (None, FailedSmartServerResponse(('BadSearch',))) |
|
104 |
||
4332.2.1
by Robert Collins
Fix bug 360791 by not raising an error when a smart server is asked for more content than it has locally; the client is assumed to be monitoring what it gets. |
105 |
def recreate_search_from_recipe(self, repository, lines, |
106 |
discard_excess=False): |
|
107 |
"""Recreate a specific revision search (vs a from-tip search).
|
|
108 |
||
109 |
:param discard_excess: If True, and the search refers to data we don't
|
|
110 |
have, just silently accept that fact - the verb calling
|
|
111 |
recreate_search trusts that clients will look for missing things
|
|
112 |
they expected and get it from elsewhere.
|
|
113 |
"""
|
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
114 |
start_keys = set(lines[0].split(' ')) |
115 |
exclude_keys = set(lines[1].split(' ')) |
|
116 |
revision_count = int(lines[2]) |
|
117 |
repository.lock_read() |
|
118 |
try: |
|
119 |
search = repository.get_graph()._make_breadth_first_searcher( |
|
120 |
start_keys) |
|
121 |
while True: |
|
122 |
try: |
|
123 |
next_revs = search.next() |
|
124 |
except StopIteration: |
|
125 |
break
|
|
126 |
search.stop_searching_any(exclude_keys.intersection(next_revs)) |
|
6341.1.5
by Jelmer Vernooij
Fix get_state(). |
127 |
(started_keys, excludes, included_keys) = search.get_state() |
128 |
if (not discard_excess and len(included_keys) != revision_count): |
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
129 |
# we got back a different amount of data than expected, this
|
130 |
# gets reported as NoSuchRevision, because less revisions
|
|
131 |
# indicates missing revisions, and more should never happen as
|
|
132 |
# the excludes list considers ghosts and ensures that ghost
|
|
133 |
# filling races are not a problem.
|
|
134 |
return (None, FailedSmartServerResponse(('NoSuchRevision',))) |
|
6341.1.5
by Jelmer Vernooij
Fix get_state(). |
135 |
search_result = vf_search.SearchResult(started_keys, excludes, |
136 |
len(included_keys), included_keys) |
|
4070.9.2
by Andrew Bennetts
Rough prototype of allowing a SearchResult to be passed to fetch, and using that to improve network conversations. |
137 |
return (search_result, None) |
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
138 |
finally: |
139 |
repository.unlock() |
|
140 |
||
2018.5.56
by Robert Collins
Factor out code we expect to be common in SmartServerRequestHasRevision to SmartServerRepositoryRequest (Robert Collins, Vincent Ladeuil). |
141 |
|
3287.6.1
by Robert Collins
* ``VersionedFile.get_graph`` is deprecated, with no replacement method. |
142 |
class SmartServerRepositoryReadLocked(SmartServerRepositoryRequest): |
143 |
"""Calls self.do_readlocked_repository_request."""
|
|
144 |
||
145 |
def do_repository_request(self, repository, *args): |
|
146 |
"""Read lock a repository for do_readlocked_repository_request."""
|
|
147 |
repository.lock_read() |
|
148 |
try: |
|
149 |
return self.do_readlocked_repository_request(repository, *args) |
|
150 |
finally: |
|
151 |
repository.unlock() |
|
152 |
||
6280.4.2
by Jelmer Vernooij
Provide server side of Repository.break_lock HPSS call. |
153 |
|
154 |
class SmartServerRepositoryBreakLock(SmartServerRepositoryRequest): |
|
155 |
"""Break a repository lock."""
|
|
156 |
||
157 |
def do_repository_request(self, repository): |
|
158 |
repository.break_lock() |
|
159 |
return SuccessfulSmartServerResponse(('ok', )) |
|
160 |
||
161 |
||
6118.2.4
by John Arbash Meinel
debugging. Use lsprof to determine where we are spending our time. |
162 |
_lsprof_count = 0 |
3287.6.1
by Robert Collins
* ``VersionedFile.get_graph`` is deprecated, with no replacement method. |
163 |
|
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
164 |
class SmartServerRepositoryGetParentMap(SmartServerRepositoryRequest): |
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
165 |
"""Bzr 1.2+ - get parent data for revisions during a graph search."""
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
166 |
|
4035.2.1
by Andrew Bennetts
Fix unnecessary get_parent_map calls after insert_stream during push. |
167 |
no_extra_results = False |
168 |
||
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
169 |
def do_repository_request(self, repository, *revision_ids): |
170 |
"""Get parent details for some revisions.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
171 |
|
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
172 |
All the parents for revision_ids are returned. Additionally up to 64KB
|
173 |
of additional parent data found by performing a breadth first search
|
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
174 |
from revision_ids is returned. The verb takes a body containing the
|
175 |
current search state, see do_body for details.
|
|
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
176 |
|
4190.1.5
by Robert Collins
Review tweaks. |
177 |
If 'include-missing:' is in revision_ids, ghosts encountered in the
|
4190.1.3
by Robert Collins
Allow optional inclusion of ghost data in server get_parent_map calls. |
178 |
graph traversal for getting parent data are included in the result with
|
179 |
a prefix of 'missing:'.
|
|
180 |
||
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
181 |
:param repository: The repository to query in.
|
3172.5.8
by Robert Collins
Review feedback. |
182 |
:param revision_ids: The utf8 encoded revision_id to answer for.
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
183 |
"""
|
184 |
self._revision_ids = revision_ids |
|
185 |
return None # Signal that we want a body. |
|
186 |
||
187 |
def do_body(self, body_bytes): |
|
188 |
"""Process the current search state and perform the parent lookup.
|
|
189 |
||
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
190 |
:return: A smart server response where the body contains an utf8
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
191 |
encoded flattened list of the parents of the revisions (the same
|
3211.5.3
by Robert Collins
Adjust size of batch and change gzip comments to bzip2. |
192 |
format as Repository.get_revision_graph) which has been bz2
|
193 |
compressed.
|
|
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
194 |
"""
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
195 |
repository = self._repository |
196 |
repository.lock_read() |
|
197 |
try: |
|
198 |
return self._do_repository_request(body_bytes) |
|
199 |
finally: |
|
200 |
repository.unlock() |
|
201 |
||
6118.2.1
by John Arbash Meinel
Refactor the Remote get_parent_map loop, and change how we compute how much prefetch to do. |
202 |
def _expand_requested_revs(self, repo_graph, revision_ids, client_seen_revs, |
203 |
include_missing, max_size=65536): |
|
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
204 |
result = {} |
205 |
queried_revs = set() |
|
6118.2.9
by John Arbash Meinel
Add some tests for how the estimator works. |
206 |
estimator = estimate_compressed_size.ZLibEstimator(max_size) |
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
207 |
next_revs = revision_ids |
208 |
first_loop_done = False |
|
209 |
while next_revs: |
|
210 |
queried_revs.update(next_revs) |
|
211 |
parent_map = repo_graph.get_parent_map(next_revs) |
|
4190.1.3
by Robert Collins
Allow optional inclusion of ghost data in server get_parent_map calls. |
212 |
current_revs = next_revs |
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
213 |
next_revs = set() |
4190.1.3
by Robert Collins
Allow optional inclusion of ghost data in server get_parent_map calls. |
214 |
for revision_id in current_revs: |
215 |
missing_rev = False |
|
216 |
parents = parent_map.get(revision_id) |
|
217 |
if parents is not None: |
|
218 |
# adjust for the wire
|
|
219 |
if parents == (_mod_revision.NULL_REVISION,): |
|
220 |
parents = () |
|
221 |
# prepare the next query
|
|
222 |
next_revs.update(parents) |
|
223 |
encoded_id = revision_id |
|
224 |
else: |
|
225 |
missing_rev = True |
|
226 |
encoded_id = "missing:" + revision_id |
|
227 |
parents = [] |
|
228 |
if (revision_id not in client_seen_revs and |
|
229 |
(not missing_rev or include_missing)): |
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
230 |
# Client does not have this revision, give it to it.
|
231 |
# add parents to the result
|
|
4190.1.3
by Robert Collins
Allow optional inclusion of ghost data in server get_parent_map calls. |
232 |
result[encoded_id] = parents |
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
233 |
# Approximate the serialized cost of this revision_id.
|
6118.2.1
by John Arbash Meinel
Refactor the Remote get_parent_map loop, and change how we compute how much prefetch to do. |
234 |
line = '%s %s\n' % (encoded_id, ' '.join(parents)) |
6118.2.3
by John Arbash Meinel
An 'entropy' computation. |
235 |
estimator.add_content(line) |
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
236 |
# get all the directly asked for parents, and then flesh out to
|
237 |
# 64K (compressed) or so. We do one level of depth at a time to
|
|
3211.5.3
by Robert Collins
Adjust size of batch and change gzip comments to bzip2. |
238 |
# stay in sync with the client. The 250000 magic number is
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
239 |
# estimated compression ratio taken from bzr.dev itself.
|
6118.2.3
by John Arbash Meinel
An 'entropy' computation. |
240 |
if self.no_extra_results or (first_loop_done and estimator.full()): |
6118.2.6
by John Arbash Meinel
Updates to ZLibEstimator. |
241 |
trace.mutter('size: %d, z_size: %d' |
6118.2.3
by John Arbash Meinel
An 'entropy' computation. |
242 |
% (estimator._uncompressed_size_added, |
6118.2.6
by John Arbash Meinel
Updates to ZLibEstimator. |
243 |
estimator._compressed_size_added)) |
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
244 |
next_revs = set() |
245 |
break
|
|
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
246 |
# don't query things we've already queried
|
5609.56.1
by John Arbash Meinel
Repository.get_parent_map was doing an inefficient set op. |
247 |
next_revs = next_revs.difference(queried_revs) |
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
248 |
first_loop_done = True |
6118.2.1
by John Arbash Meinel
Refactor the Remote get_parent_map loop, and change how we compute how much prefetch to do. |
249 |
return result |
250 |
||
251 |
def _do_repository_request(self, body_bytes): |
|
252 |
repository = self._repository |
|
253 |
revision_ids = set(self._revision_ids) |
|
254 |
include_missing = 'include-missing:' in revision_ids |
|
255 |
if include_missing: |
|
256 |
revision_ids.remove('include-missing:') |
|
257 |
body_lines = body_bytes.split('\n') |
|
258 |
search_result, error = self.recreate_search_from_recipe( |
|
259 |
repository, body_lines) |
|
260 |
if error is not None: |
|
261 |
return error |
|
262 |
# TODO might be nice to start up the search again; but thats not
|
|
263 |
# written or tested yet.
|
|
264 |
client_seen_revs = set(search_result.get_keys()) |
|
265 |
# Always include the requested ids.
|
|
266 |
client_seen_revs.difference_update(revision_ids) |
|
267 |
||
268 |
repo_graph = repository.get_graph() |
|
269 |
result = self._expand_requested_revs(repo_graph, revision_ids, |
|
270 |
client_seen_revs, include_missing) |
|
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
271 |
|
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
272 |
# sorting trivially puts lexographically similar revision ids together.
|
273 |
# Compression FTW.
|
|
6118.2.1
by John Arbash Meinel
Refactor the Remote get_parent_map loop, and change how we compute how much prefetch to do. |
274 |
lines = [] |
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
275 |
for revision, parents in sorted(result.items()): |
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
276 |
lines.append(' '.join((revision, ) + tuple(parents))) |
277 |
||
3211.5.1
by Robert Collins
Change the smart server get_parents method to take a graph search to exclude already recieved parents from. This prevents history shortcuts causing huge numbers of duplicates. |
278 |
return SuccessfulSmartServerResponse( |
3211.5.2
by Robert Collins
Change RemoteRepository.get_parent_map to use bz2 not gzip for compression. |
279 |
('ok', ), bz2.compress('\n'.join(lines))) |
3172.5.6
by Robert Collins
Create new smart server verb Repository.get_parent_map. |
280 |
|
281 |
||
3287.6.1
by Robert Collins
* ``VersionedFile.get_graph`` is deprecated, with no replacement method. |
282 |
class SmartServerRepositoryGetRevisionGraph(SmartServerRepositoryReadLocked): |
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
283 |
|
3287.6.1
by Robert Collins
* ``VersionedFile.get_graph`` is deprecated, with no replacement method. |
284 |
def do_readlocked_repository_request(self, repository, revision_id): |
2018.5.67
by Wouter van Heyst
Implement RemoteRepository.get_revision_graph (Wouter van Heyst, Robert Collins) |
285 |
"""Return the result of repository.get_revision_graph(revision_id).
|
3287.6.1
by Robert Collins
* ``VersionedFile.get_graph`` is deprecated, with no replacement method. |
286 |
|
287 |
Deprecated as of bzr 1.4, but supported for older clients.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
288 |
|
2018.5.67
by Wouter van Heyst
Implement RemoteRepository.get_revision_graph (Wouter van Heyst, Robert Collins) |
289 |
:param repository: The repository to query in.
|
290 |
:param revision_id: The utf8 encoded revision_id to get a graph from.
|
|
291 |
:return: A smart server response where the body contains an utf8
|
|
292 |
encoded flattened list of the revision graph.
|
|
293 |
"""
|
|
2018.5.83
by Andrew Bennetts
Fix some test failures caused by the switch from unicode to UTF-8-encoded strs for revision IDs. |
294 |
if not revision_id: |
295 |
revision_id = None |
|
2018.5.67
by Wouter van Heyst
Implement RemoteRepository.get_revision_graph (Wouter van Heyst, Robert Collins) |
296 |
|
297 |
lines = [] |
|
3287.6.1
by Robert Collins
* ``VersionedFile.get_graph`` is deprecated, with no replacement method. |
298 |
graph = repository.get_graph() |
299 |
if revision_id: |
|
300 |
search_ids = [revision_id] |
|
301 |
else: |
|
302 |
search_ids = repository.all_revision_ids() |
|
303 |
search = graph._make_breadth_first_searcher(search_ids) |
|
304 |
transitive_ids = set() |
|
305 |
map(transitive_ids.update, list(search)) |
|
306 |
parent_map = graph.get_parent_map(transitive_ids) |
|
3287.6.8
by Robert Collins
Reduce code duplication as per review. |
307 |
revision_graph = _strip_NULL_ghosts(parent_map) |
3287.6.1
by Robert Collins
* ``VersionedFile.get_graph`` is deprecated, with no replacement method. |
308 |
if revision_id and revision_id not in revision_graph: |
2018.14.1
by Andrew Bennetts
Update to current hpss branch? Fix lots of test failures. |
309 |
# Note that we return an empty body, rather than omitting the body.
|
310 |
# This way the client knows that it can always expect to find a body
|
|
311 |
# in the response for this method, even in the error case.
|
|
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
312 |
return FailedSmartServerResponse(('nosuchrevision', revision_id), '') |
2018.5.67
by Wouter van Heyst
Implement RemoteRepository.get_revision_graph (Wouter van Heyst, Robert Collins) |
313 |
|
314 |
for revision, parents in revision_graph.items(): |
|
2592.3.50
by Robert Collins
Merge bzr.dev. |
315 |
lines.append(' '.join((revision, ) + tuple(parents))) |
2018.5.67
by Wouter van Heyst
Implement RemoteRepository.get_revision_graph (Wouter van Heyst, Robert Collins) |
316 |
|
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
317 |
return SuccessfulSmartServerResponse(('ok', ), '\n'.join(lines)) |
2018.5.67
by Wouter van Heyst
Implement RemoteRepository.get_revision_graph (Wouter van Heyst, Robert Collins) |
318 |
|
319 |
||
4419.2.4
by Andrew Bennetts
Add Repository.get_rev_id_for_revno RPC, removes VFS calls from 'pull -r 123' case. |
320 |
class SmartServerRepositoryGetRevIdForRevno(SmartServerRepositoryReadLocked): |
321 |
||
322 |
def do_readlocked_repository_request(self, repository, revno, |
|
323 |
known_pair): |
|
324 |
"""Find the revid for a given revno, given a known revno/revid pair.
|
|
325 |
|
|
4419.2.16
by Andrew Bennetts
New in 1.17, not 1.16. |
326 |
New in 1.17.
|
4419.2.4
by Andrew Bennetts
Add Repository.get_rev_id_for_revno RPC, removes VFS calls from 'pull -r 123' case. |
327 |
"""
|
4419.2.6
by Andrew Bennetts
Add tests for server-side logic, and fix the bugs exposed by those tests. |
328 |
try: |
329 |
found_flag, result = repository.get_rev_id_for_revno(revno, known_pair) |
|
330 |
except errors.RevisionNotPresent, err: |
|
331 |
if err.revision_id != known_pair[1]: |
|
332 |
raise AssertionError( |
|
333 |
'get_rev_id_for_revno raised RevisionNotPresent for '
|
|
334 |
'non-initial revision: ' + err.revision_id) |
|
335 |
return FailedSmartServerResponse( |
|
336 |
('nosuchrevision', err.revision_id)) |
|
4419.2.5
by Andrew Bennetts
Add Repository.get_rev_id_for_revno, and use it both as the _ensure_real fallback and as the server-side implementation. |
337 |
if found_flag: |
338 |
return SuccessfulSmartServerResponse(('ok', result)) |
|
339 |
else: |
|
340 |
earliest_revno, earliest_revid = result |
|
4419.2.4
by Andrew Bennetts
Add Repository.get_rev_id_for_revno RPC, removes VFS calls from 'pull -r 123' case. |
341 |
return SuccessfulSmartServerResponse( |
4419.2.5
by Andrew Bennetts
Add Repository.get_rev_id_for_revno, and use it both as the _ensure_real fallback and as the server-side implementation. |
342 |
('history-incomplete', earliest_revno, earliest_revid)) |
4419.2.4
by Andrew Bennetts
Add Repository.get_rev_id_for_revno RPC, removes VFS calls from 'pull -r 123' case. |
343 |
|
344 |
||
6280.5.2
by Jelmer Vernooij
New HPSS call VersionedFileRepository.get_serializer_format. |
345 |
class SmartServerRepositoryGetSerializerFormat(SmartServerRepositoryRequest): |
346 |
||
347 |
def do_repository_request(self, repository): |
|
348 |
"""Return the serializer format for this repository.
|
|
349 |
||
350 |
New in 2.5.0.
|
|
351 |
||
352 |
:param repository: The repository to query
|
|
353 |
:return: A smart server response ('ok', FORMAT)
|
|
354 |
"""
|
|
355 |
serializer = repository.get_serializer_format() |
|
356 |
return SuccessfulSmartServerResponse(('ok', serializer)) |
|
357 |
||
358 |
||
2018.5.56
by Robert Collins
Factor out code we expect to be common in SmartServerRequestHasRevision to SmartServerRepositoryRequest (Robert Collins, Vincent Ladeuil). |
359 |
class SmartServerRequestHasRevision(SmartServerRepositoryRequest): |
360 |
||
361 |
def do_repository_request(self, repository, revision_id): |
|
362 |
"""Return ok if a specific revision is in the repository at path.
|
|
363 |
||
364 |
:param repository: The repository to query in.
|
|
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
365 |
:param revision_id: The utf8 encoded revision_id to lookup.
|
6265.2.1
by Jelmer Vernooij
fix docstring |
366 |
:return: A smart server response of ('yes', ) if the revision is
|
367 |
present. ('no', ) if it is missing.
|
|
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
368 |
"""
|
2018.5.83
by Andrew Bennetts
Fix some test failures caused by the switch from unicode to UTF-8-encoded strs for revision IDs. |
369 |
if repository.has_revision(revision_id): |
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
370 |
return SuccessfulSmartServerResponse(('yes', )) |
2018.5.40
by Robert Collins
Implement a remote Repository.has_revision method. |
371 |
else: |
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
372 |
return SuccessfulSmartServerResponse(('no', )) |
2018.5.57
by Robert Collins
Implement RemoteRepository.is_shared (Robert Collins, Vincent Ladeuil). |
373 |
|
374 |
||
6265.1.1
by Jelmer Vernooij
Add new HPSS call ``Repository.has_signature_for_revision_id``. |
375 |
class SmartServerRequestHasSignatureForRevisionId( |
376 |
SmartServerRepositoryRequest): |
|
377 |
||
378 |
def do_repository_request(self, repository, revision_id): |
|
379 |
"""Return ok if a signature is present for a revision.
|
|
380 |
||
6265.1.2
by Jelmer Vernooij
Document when Repository.has_signature_for_revision_id was introduced. |
381 |
Introduced in bzr 2.5.0.
|
382 |
||
6265.1.1
by Jelmer Vernooij
Add new HPSS call ``Repository.has_signature_for_revision_id``. |
383 |
:param repository: The repository to query in.
|
384 |
:param revision_id: The utf8 encoded revision_id to lookup.
|
|
6265.2.1
by Jelmer Vernooij
fix docstring |
385 |
:return: A smart server response of ('yes', ) if a
|
386 |
signature for the revision is present,
|
|
387 |
('no', ) if it is missing.
|
|
6265.1.1
by Jelmer Vernooij
Add new HPSS call ``Repository.has_signature_for_revision_id``. |
388 |
"""
|
389 |
try: |
|
390 |
if repository.has_signature_for_revision_id(revision_id): |
|
391 |
return SuccessfulSmartServerResponse(('yes', )) |
|
392 |
else: |
|
393 |
return SuccessfulSmartServerResponse(('no', )) |
|
394 |
except errors.NoSuchRevision: |
|
395 |
return FailedSmartServerResponse( |
|
6265.1.5
by Jelmer Vernooij
Fix capitalization - NoSuchRevision is for branches. |
396 |
('nosuchrevision', revision_id)) |
6265.1.1
by Jelmer Vernooij
Add new HPSS call ``Repository.has_signature_for_revision_id``. |
397 |
|
398 |
||
2018.10.2
by v.ladeuil+lp at free
gather_stats server side and request registration |
399 |
class SmartServerRepositoryGatherStats(SmartServerRepositoryRequest): |
400 |
||
401 |
def do_repository_request(self, repository, revid, committers): |
|
402 |
"""Return the result of repository.gather_stats().
|
|
403 |
||
404 |
:param repository: The repository to query in.
|
|
405 |
:param revid: utf8 encoded rev id or an empty string to indicate None
|
|
406 |
:param committers: 'yes' or 'no'.
|
|
407 |
||
408 |
:return: A SmartServerResponse ('ok',), a encoded body looking like
|
|
409 |
committers: 1
|
|
410 |
firstrev: 1234.230 0
|
|
411 |
latestrev: 345.700 3600
|
|
412 |
revisions: 2
|
|
413 |
||
414 |
But containing only fields returned by the gather_stats() call
|
|
415 |
"""
|
|
416 |
if revid == '': |
|
417 |
decoded_revision_id = None |
|
418 |
else: |
|
2018.5.83
by Andrew Bennetts
Fix some test failures caused by the switch from unicode to UTF-8-encoded strs for revision IDs. |
419 |
decoded_revision_id = revid |
2018.10.2
by v.ladeuil+lp at free
gather_stats server side and request registration |
420 |
if committers == 'yes': |
421 |
decoded_committers = True |
|
422 |
else: |
|
423 |
decoded_committers = None |
|
6291.1.1
by Jelmer Vernooij
Cope with missing revision ids being specified to |
424 |
try: |
425 |
stats = repository.gather_stats(decoded_revision_id, |
|
426 |
decoded_committers) |
|
427 |
except errors.NoSuchRevision: |
|
428 |
return FailedSmartServerResponse(('nosuchrevision', revid)) |
|
2018.10.2
by v.ladeuil+lp at free
gather_stats server side and request registration |
429 |
|
430 |
body = '' |
|
431 |
if stats.has_key('committers'): |
|
432 |
body += 'committers: %d\n' % stats['committers'] |
|
433 |
if stats.has_key('firstrev'): |
|
434 |
body += 'firstrev: %.3f %d\n' % stats['firstrev'] |
|
435 |
if stats.has_key('latestrev'): |
|
436 |
body += 'latestrev: %.3f %d\n' % stats['latestrev'] |
|
437 |
if stats.has_key('revisions'): |
|
438 |
body += 'revisions: %d\n' % stats['revisions'] |
|
439 |
if stats.has_key('size'): |
|
440 |
body += 'size: %d\n' % stats['size'] |
|
441 |
||
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
442 |
return SuccessfulSmartServerResponse(('ok', ), body) |
2018.10.2
by v.ladeuil+lp at free
gather_stats server side and request registration |
443 |
|
444 |
||
6263.3.2
by Jelmer Vernooij
Add new HPSS call 'Repository.get_revision_signature_text'. |
445 |
class SmartServerRepositoryGetRevisionSignatureText( |
446 |
SmartServerRepositoryRequest): |
|
6263.3.3
by Jelmer Vernooij
Add test for presence. |
447 |
"""Return the signature text of a revision.
|
448 |
||
449 |
New in 2.5.
|
|
450 |
"""
|
|
6263.3.2
by Jelmer Vernooij
Add new HPSS call 'Repository.get_revision_signature_text'. |
451 |
|
452 |
def do_repository_request(self, repository, revision_id): |
|
453 |
"""Return the result of repository.get_signature_text().
|
|
454 |
||
455 |
:param repository: The repository to query in.
|
|
456 |
:return: A smart server response of with the signature text as
|
|
457 |
body.
|
|
458 |
"""
|
|
6263.3.4
by Jelmer Vernooij
Fix NoSuchRevision error. |
459 |
try: |
460 |
text = repository.get_signature_text(revision_id) |
|
461 |
except errors.NoSuchRevision, err: |
|
462 |
return FailedSmartServerResponse( |
|
463 |
('nosuchrevision', err.revision)) |
|
6263.3.2
by Jelmer Vernooij
Add new HPSS call 'Repository.get_revision_signature_text'. |
464 |
return SuccessfulSmartServerResponse(('ok', ), text) |
465 |
||
466 |
||
2018.5.57
by Robert Collins
Implement RemoteRepository.is_shared (Robert Collins, Vincent Ladeuil). |
467 |
class SmartServerRepositoryIsShared(SmartServerRepositoryRequest): |
468 |
||
469 |
def do_repository_request(self, repository): |
|
470 |
"""Return the result of repository.is_shared().
|
|
471 |
||
472 |
:param repository: The repository to query in.
|
|
473 |
:return: A smart server response of ('yes', ) if the repository is
|
|
474 |
shared, and ('no', ) if it is not.
|
|
475 |
"""
|
|
476 |
if repository.is_shared(): |
|
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
477 |
return SuccessfulSmartServerResponse(('yes', )) |
2018.5.57
by Robert Collins
Implement RemoteRepository.is_shared (Robert Collins, Vincent Ladeuil). |
478 |
else: |
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
479 |
return SuccessfulSmartServerResponse(('no', )) |
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
480 |
|
481 |
||
6263.2.1
by Jelmer Vernooij
Add hpss call ``Repository.make_working_trees`` |
482 |
class SmartServerRepositoryMakeWorkingTrees(SmartServerRepositoryRequest): |
483 |
||
484 |
def do_repository_request(self, repository): |
|
485 |
"""Return the result of repository.make_working_trees().
|
|
486 |
||
6263.2.2
by Jelmer Vernooij
Document when Repository.make_working_trees was introduced. |
487 |
Introduced in bzr 2.5.0.
|
488 |
||
6263.2.1
by Jelmer Vernooij
Add hpss call ``Repository.make_working_trees`` |
489 |
:param repository: The repository to query in.
|
490 |
:return: A smart server response of ('yes', ) if the repository uses
|
|
491 |
working trees, and ('no', ) if it is not.
|
|
492 |
"""
|
|
493 |
if repository.make_working_trees(): |
|
494 |
return SuccessfulSmartServerResponse(('yes', )) |
|
495 |
else: |
|
496 |
return SuccessfulSmartServerResponse(('no', )) |
|
497 |
||
498 |
||
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
499 |
class SmartServerRepositoryLockWrite(SmartServerRepositoryRequest): |
500 |
||
2018.5.79
by Andrew Bennetts
Implement RemoteBranch.lock_write/unlock as smart operations. |
501 |
def do_repository_request(self, repository, token=''): |
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
502 |
# XXX: this probably should not have a token.
|
503 |
if token == '': |
|
504 |
token = None |
|
505 |
try: |
|
5200.3.3
by Robert Collins
Lock methods on ``Tree``, ``Branch`` and ``Repository`` are now |
506 |
token = repository.lock_write(token=token).repository_token |
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
507 |
except errors.LockContention, e: |
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
508 |
return FailedSmartServerResponse(('LockContention',)) |
2018.5.95
by Andrew Bennetts
Add a Transport.is_readonly remote call, let {Branch,Repository}.lock_write remote call return UnlockableTransport, and miscellaneous test fixes. |
509 |
except errors.UnlockableTransport: |
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
510 |
return FailedSmartServerResponse(('UnlockableTransport',)) |
2872.5.3
by Martin Pool
Pass back LockFailed from smart server lock methods |
511 |
except errors.LockFailed, e: |
512 |
return FailedSmartServerResponse(('LockFailed', |
|
513 |
str(e.lock), str(e.why))) |
|
3015.2.7
by Robert Collins
In the RemoteServer repository methods handle repositories that cannot be remotely locked like pack repositories, and add a read lock in SmartServerRepositoryStreamKnitDataForRevisions. |
514 |
if token is not None: |
515 |
repository.leave_lock_in_place() |
|
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
516 |
repository.unlock() |
517 |
if token is None: |
|
518 |
token = '' |
|
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
519 |
return SuccessfulSmartServerResponse(('ok', token)) |
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
520 |
|
521 |
||
4060.1.5
by Robert Collins
Verb change name requested by Andrew. |
522 |
class SmartServerRepositoryGetStream(SmartServerRepositoryRequest): |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
523 |
|
524 |
def do_repository_request(self, repository, to_network_name): |
|
525 |
"""Get a stream for inserting into a to_format repository.
|
|
526 |
||
5539.2.14
by Andrew Bennetts
Don't add a new verb; instead just teach the client to fallback if it gets a BadSearch error. |
527 |
The request body is 'search_bytes', a description of the revisions
|
528 |
being requested.
|
|
529 |
||
530 |
In 2.3 this verb added support for search_bytes == 'everything'. Older
|
|
531 |
implementations will respond with a BadSearch error, and clients should
|
|
532 |
catch this and fallback appropriately.
|
|
533 |
||
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
534 |
:param repository: The repository to stream from.
|
535 |
:param to_network_name: The network name of the format of the target
|
|
536 |
repository.
|
|
537 |
"""
|
|
538 |
self._to_format = network_format_registry.get(to_network_name) |
|
4476.3.29
by Andrew Bennetts
Add Repository.get_stream_1.18 verb. |
539 |
if self._should_fake_unknown(): |
540 |
return FailedSmartServerResponse( |
|
541 |
('UnknownMethod', 'Repository.get_stream')) |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
542 |
return None # Signal that we want a body. |
543 |
||
4476.3.29
by Andrew Bennetts
Add Repository.get_stream_1.18 verb. |
544 |
def _should_fake_unknown(self): |
4476.3.80
by Andrew Bennetts
Comment/docstring tweaks prompted by review. |
545 |
"""Return True if we should return UnknownMethod to the client.
|
546 |
|
|
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
547 |
This is a workaround for bugs in pre-1.19 clients that claim to
|
548 |
support receiving streams of CHK repositories. The pre-1.19 client
|
|
4476.3.80
by Andrew Bennetts
Comment/docstring tweaks prompted by review. |
549 |
expects inventory records to be serialized in the format defined by
|
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
550 |
to_network_name, but in pre-1.19 (at least) that format definition
|
4476.3.80
by Andrew Bennetts
Comment/docstring tweaks prompted by review. |
551 |
tries to use the xml5 serializer, which does not correctly handle
|
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
552 |
rich-roots. After 1.19 the client can also accept inventory-deltas
|
4476.3.80
by Andrew Bennetts
Comment/docstring tweaks prompted by review. |
553 |
(which avoids this issue), and those clients will use the
|
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
554 |
Repository.get_stream_1.19 verb instead of this one.
|
4476.3.80
by Andrew Bennetts
Comment/docstring tweaks prompted by review. |
555 |
So: if this repository is CHK, and the to_format doesn't match,
|
556 |
we should just fake an UnknownSmartMethod error so that the client
|
|
557 |
will fallback to VFS, rather than sending it a stream we know it
|
|
558 |
cannot handle.
|
|
559 |
"""
|
|
4476.3.29
by Andrew Bennetts
Add Repository.get_stream_1.18 verb. |
560 |
from_format = self._repository._format |
561 |
to_format = self._to_format |
|
562 |
if not from_format.supports_chks: |
|
563 |
# Source not CHK: that's ok
|
|
564 |
return False |
|
565 |
if (to_format.supports_chks and |
|
566 |
from_format.repository_class is to_format.repository_class and |
|
567 |
from_format._serializer == to_format._serializer): |
|
568 |
# Source is CHK, but target matches: that's ok
|
|
569 |
# (e.g. 2a->2a, or CHK2->2a)
|
|
570 |
return False |
|
571 |
# Source is CHK, and target is not CHK or incompatible CHK. We can't
|
|
572 |
# generate a compatible stream.
|
|
573 |
return True |
|
574 |
||
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
575 |
def do_body(self, body_bytes): |
576 |
repository = self._repository |
|
577 |
repository.lock_read() |
|
578 |
try: |
|
4332.2.1
by Robert Collins
Fix bug 360791 by not raising an error when a smart server is asked for more content than it has locally; the client is assumed to be monitoring what it gets. |
579 |
search_result, error = self.recreate_search(repository, body_bytes, |
580 |
discard_excess=True) |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
581 |
if error is not None: |
582 |
repository.unlock() |
|
583 |
return error |
|
584 |
source = repository._get_source(self._to_format) |
|
4070.9.2
by Andrew Bennetts
Rough prototype of allowing a SearchResult to be passed to fetch, and using that to improve network conversations. |
585 |
stream = source.get_stream(search_result) |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
586 |
except Exception: |
587 |
exc_info = sys.exc_info() |
|
588 |
try: |
|
589 |
# On non-error, unlocking is done by the body stream handler.
|
|
590 |
repository.unlock() |
|
591 |
finally: |
|
592 |
raise exc_info[0], exc_info[1], exc_info[2] |
|
593 |
return SuccessfulSmartServerResponse(('ok',), |
|
594 |
body_stream=self.body_stream(stream, repository)) |
|
595 |
||
596 |
def body_stream(self, stream, repository): |
|
597 |
byte_stream = _stream_to_byte_stream(stream, repository._format) |
|
598 |
try: |
|
599 |
for bytes in byte_stream: |
|
600 |
yield bytes |
|
601 |
except errors.RevisionNotPresent, e: |
|
602 |
# This shouldn't be able to happen, but as we don't buffer
|
|
603 |
# everything it can in theory happen.
|
|
604 |
repository.unlock() |
|
605 |
yield FailedSmartServerResponse(('NoSuchRevision', e.revision_id)) |
|
606 |
else: |
|
607 |
repository.unlock() |
|
608 |
||
609 |
||
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
610 |
class SmartServerRepositoryGetStream_1_19(SmartServerRepositoryGetStream): |
5539.2.14
by Andrew Bennetts
Don't add a new verb; instead just teach the client to fallback if it gets a BadSearch error. |
611 |
"""The same as Repository.get_stream, but will return stream CHK formats to
|
612 |
clients.
|
|
613 |
||
614 |
See SmartServerRepositoryGetStream._should_fake_unknown.
|
|
615 |
|
|
616 |
New in 1.19.
|
|
617 |
"""
|
|
4476.3.29
by Andrew Bennetts
Add Repository.get_stream_1.18 verb. |
618 |
|
619 |
def _should_fake_unknown(self): |
|
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
620 |
"""Returns False; we don't need to workaround bugs in 1.19+ clients."""
|
4476.3.29
by Andrew Bennetts
Add Repository.get_stream_1.18 verb. |
621 |
return False |
622 |
||
623 |
||
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
624 |
def _stream_to_byte_stream(stream, src_format): |
625 |
"""Convert a record stream to a self delimited byte stream."""
|
|
626 |
pack_writer = pack.ContainerSerialiser() |
|
627 |
yield pack_writer.begin() |
|
628 |
yield pack_writer.bytes_record(src_format.network_name(), '') |
|
629 |
for substream_type, substream in stream: |
|
630 |
for record in substream: |
|
631 |
if record.storage_kind in ('chunked', 'fulltext'): |
|
632 |
serialised = record_to_fulltext_bytes(record) |
|
4392.2.2
by John Arbash Meinel
Add tests that ensure we can fetch branches with ghosts in their ancestry. |
633 |
elif record.storage_kind == 'absent': |
634 |
raise ValueError("Absent factory for %s" % (record.key,)) |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
635 |
else: |
636 |
serialised = record.get_bytes_as(record.storage_kind) |
|
637 |
if serialised: |
|
638 |
# Some streams embed the whole stream into the wire
|
|
639 |
# representation of the first record, which means that
|
|
640 |
# later records have no wire representation: we skip them.
|
|
641 |
yield pack_writer.bytes_record(serialised, [(substream_type,)]) |
|
642 |
yield pack_writer.end() |
|
643 |
||
644 |
||
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
645 |
class _ByteStreamDecoder(object): |
646 |
"""Helper for _byte_stream_to_stream.
|
|
647 |
||
4634.19.2
by Robert Collins
Review feedback. |
648 |
The expected usage of this class is via the function _byte_stream_to_stream
|
649 |
which creates a _ByteStreamDecoder, pops off the stream format and then
|
|
650 |
yields the output of record_stream(), the main entry point to
|
|
651 |
_ByteStreamDecoder.
|
|
652 |
||
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
653 |
Broadly this class has to unwrap two layers of iterators:
|
654 |
(type, substream)
|
|
655 |
(substream details)
|
|
656 |
||
657 |
This is complicated by wishing to return type, iterator_for_type, but
|
|
658 |
getting the data for iterator_for_type when we find out type: we can't
|
|
659 |
simply pass a generator down to the NetworkRecordStream parser, instead
|
|
660 |
we have a little local state to seed each NetworkRecordStream instance,
|
|
661 |
and gather the type that we'll be yielding.
|
|
662 |
||
663 |
:ivar byte_stream: The byte stream being decoded.
|
|
664 |
:ivar stream_decoder: A pack parser used to decode the bytestream
|
|
665 |
:ivar current_type: The current type, used to join adjacent records of the
|
|
666 |
same type into a single stream.
|
|
667 |
:ivar first_bytes: The first bytes to give the next NetworkRecordStream.
|
|
668 |
"""
|
|
669 |
||
5195.3.23
by Parth Malwankar
moved progress bar logic to SourceStream. |
670 |
def __init__(self, byte_stream, record_counter): |
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
671 |
"""Create a _ByteStreamDecoder."""
|
672 |
self.stream_decoder = pack.ContainerPushParser() |
|
673 |
self.current_type = None |
|
674 |
self.first_bytes = None |
|
675 |
self.byte_stream = byte_stream |
|
5195.3.27
by Parth Malwankar
code cleanup and comments. |
676 |
self._record_counter = record_counter |
5195.3.23
by Parth Malwankar
moved progress bar logic to SourceStream. |
677 |
self.key_count = 0 |
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
678 |
|
679 |
def iter_stream_decoder(self): |
|
680 |
"""Iterate the contents of the pack from stream_decoder."""
|
|
681 |
# dequeue pending items
|
|
682 |
for record in self.stream_decoder.read_pending_records(): |
|
683 |
yield record |
|
684 |
# Pull bytes of the wire, decode them to records, yield those records.
|
|
685 |
for bytes in self.byte_stream: |
|
686 |
self.stream_decoder.accept_bytes(bytes) |
|
687 |
for record in self.stream_decoder.read_pending_records(): |
|
688 |
yield record |
|
689 |
||
690 |
def iter_substream_bytes(self): |
|
691 |
if self.first_bytes is not None: |
|
692 |
yield self.first_bytes |
|
693 |
# If we run out of pack records, single the outer layer to stop.
|
|
694 |
self.first_bytes = None |
|
695 |
for record in self.iter_pack_records: |
|
696 |
record_names, record_bytes = record |
|
697 |
record_name, = record_names |
|
698 |
substream_type = record_name[0] |
|
699 |
if substream_type != self.current_type: |
|
700 |
# end of a substream, seed the next substream.
|
|
701 |
self.current_type = substream_type |
|
702 |
self.first_bytes = record_bytes |
|
703 |
return
|
|
704 |
yield record_bytes |
|
705 |
||
706 |
def record_stream(self): |
|
707 |
"""Yield substream_type, substream from the byte stream."""
|
|
5195.3.27
by Parth Malwankar
code cleanup and comments. |
708 |
def wrap_and_count(pb, rc, substream): |
709 |
"""Yield records from stream while showing progress."""
|
|
710 |
counter = 0 |
|
711 |
if rc: |
|
712 |
if self.current_type != 'revisions' and self.key_count != 0: |
|
713 |
# As we know the number of revisions now (in self.key_count)
|
|
714 |
# we can setup and use record_counter (rc).
|
|
715 |
if not rc.is_initialized(): |
|
716 |
rc.setup(self.key_count, self.key_count) |
|
717 |
for record in substream.read(): |
|
718 |
if rc: |
|
719 |
if rc.is_initialized() and counter == rc.STEP: |
|
720 |
rc.increment(counter) |
|
721 |
pb.update('Estimate', rc.current, rc.max) |
|
722 |
counter = 0 |
|
723 |
if self.current_type == 'revisions': |
|
724 |
# Total records is proportional to number of revs
|
|
725 |
# to fetch. With remote, we used self.key_count to
|
|
726 |
# track the number of revs. Once we have the revs
|
|
727 |
# counts in self.key_count, the progress bar changes
|
|
728 |
# from 'Estimating..' to 'Estimate' above.
|
|
729 |
self.key_count += 1 |
|
730 |
if counter == rc.STEP: |
|
731 |
pb.update('Estimating..', self.key_count) |
|
732 |
counter = 0 |
|
733 |
counter += 1 |
|
734 |
yield record |
|
735 |
||
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
736 |
self.seed_state() |
5195.3.23
by Parth Malwankar
moved progress bar logic to SourceStream. |
737 |
pb = ui.ui_factory.nested_progress_bar() |
5195.3.27
by Parth Malwankar
code cleanup and comments. |
738 |
rc = self._record_counter |
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
739 |
# Make and consume sub generators, one per substream type:
|
740 |
while self.first_bytes is not None: |
|
741 |
substream = NetworkRecordStream(self.iter_substream_bytes()) |
|
742 |
# after substream is fully consumed, self.current_type is set to
|
|
743 |
# the next type, and self.first_bytes is set to the matching bytes.
|
|
5195.3.27
by Parth Malwankar
code cleanup and comments. |
744 |
yield self.current_type, wrap_and_count(pb, rc, substream) |
745 |
if rc: |
|
746 |
pb.update('Done', rc.max, rc.max) |
|
5195.3.23
by Parth Malwankar
moved progress bar logic to SourceStream. |
747 |
pb.finished() |
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
748 |
|
749 |
def seed_state(self): |
|
750 |
"""Prepare the _ByteStreamDecoder to decode from the pack stream."""
|
|
751 |
# Set a single generator we can use to get data from the pack stream.
|
|
752 |
self.iter_pack_records = self.iter_stream_decoder() |
|
753 |
# Seed the very first subiterator with content; after this each one
|
|
754 |
# seeds the next.
|
|
755 |
list(self.iter_substream_bytes()) |
|
756 |
||
757 |
||
5195.3.23
by Parth Malwankar
moved progress bar logic to SourceStream. |
758 |
def _byte_stream_to_stream(byte_stream, record_counter=None): |
4060.1.5
by Robert Collins
Verb change name requested by Andrew. |
759 |
"""Convert a byte stream into a format and a stream.
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
760 |
|
761 |
:param byte_stream: A bytes iterator, as output by _stream_to_byte_stream.
|
|
762 |
:return: (RepositoryFormat, stream_generator)
|
|
763 |
"""
|
|
5195.3.23
by Parth Malwankar
moved progress bar logic to SourceStream. |
764 |
decoder = _ByteStreamDecoder(byte_stream, record_counter) |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
765 |
for bytes in byte_stream: |
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
766 |
decoder.stream_decoder.accept_bytes(bytes) |
767 |
for record in decoder.stream_decoder.read_pending_records(max=1): |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
768 |
record_names, src_format_name = record |
769 |
src_format = network_format_registry.get(src_format_name) |
|
4634.19.1
by Robert Collins
Combine adjacent substreams of the same type in bzrlib.smart.repository._byte_stream_to_stream. |
770 |
return src_format, decoder.record_stream() |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
771 |
|
772 |
||
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
773 |
class SmartServerRepositoryUnlock(SmartServerRepositoryRequest): |
774 |
||
775 |
def do_repository_request(self, repository, token): |
|
776 |
try: |
|
777 |
repository.lock_write(token=token) |
|
778 |
except errors.TokenMismatch, e: |
|
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
779 |
return FailedSmartServerResponse(('TokenMismatch',)) |
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
780 |
repository.dont_leave_lock_in_place() |
781 |
repository.unlock() |
|
2432.4.5
by Robert Collins
Make using SuccessfulSmartServerResponse and FailedSmartServerResponse mandatory rather than optional in smart server logic. |
782 |
return SuccessfulSmartServerResponse(('ok',)) |
2018.5.78
by Andrew Bennetts
Implement RemoteRepository.lock_write/unlock to expect and send tokens over the |
783 |
|
2018.18.1
by Martin Pool
Add stub Repository.tarball smart method |
784 |
|
6280.6.2
by Jelmer Vernooij
Add HPSS calls Repository.get_physical_lock_status and Branch.get_physical_lock_status. |
785 |
class SmartServerRepositoryGetPhysicalLockStatus(SmartServerRepositoryRequest): |
786 |
"""Get the physical lock status for a repository.
|
|
787 |
||
788 |
New in 2.5.
|
|
789 |
"""
|
|
790 |
||
791 |
def do_repository_request(self, repository): |
|
792 |
if repository.get_physical_lock_status(): |
|
793 |
return SuccessfulSmartServerResponse(('yes', )) |
|
794 |
else: |
|
795 |
return SuccessfulSmartServerResponse(('no', )) |
|
796 |
||
797 |
||
4017.3.4
by Robert Collins
Create a verb for Repository.set_make_working_trees. |
798 |
class SmartServerRepositorySetMakeWorkingTrees(SmartServerRepositoryRequest): |
799 |
||
800 |
def do_repository_request(self, repository, str_bool_new_value): |
|
801 |
if str_bool_new_value == 'True': |
|
802 |
new_value = True |
|
803 |
else: |
|
804 |
new_value = False |
|
805 |
repository.set_make_working_trees(new_value) |
|
806 |
return SuccessfulSmartServerResponse(('ok',)) |
|
807 |
||
808 |
||
2018.18.1
by Martin Pool
Add stub Repository.tarball smart method |
809 |
class SmartServerRepositoryTarball(SmartServerRepositoryRequest): |
2018.18.11
by Martin Pool
merge hpss changes |
810 |
"""Get the raw repository files as a tarball.
|
811 |
||
812 |
The returned tarball contains a .bzr control directory which in turn
|
|
813 |
contains a repository.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
814 |
|
815 |
This takes one parameter, compression, which currently must be
|
|
2018.18.2
by Martin Pool
smart method Repository.tarball actually returns the tarball |
816 |
"", "gz", or "bz2".
|
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
817 |
|
818 |
This is used to implement the Repository.copy_content_into operation.
|
|
2018.18.1
by Martin Pool
Add stub Repository.tarball smart method |
819 |
"""
|
820 |
||
2018.18.2
by Martin Pool
smart method Repository.tarball actually returns the tarball |
821 |
def do_repository_request(self, repository, compression): |
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
822 |
tmp_dirname, tmp_repo = self._copy_to_tempdir(repository) |
2018.18.5
by Martin Pool
Repository.tarball locks repository while running for consistency |
823 |
try: |
2018.18.10
by Martin Pool
copy_content_into from Remote repositories by using temporary directories on both ends. |
824 |
controldir_name = tmp_dirname + '/.bzr' |
825 |
return self._tarfile_response(controldir_name, compression) |
|
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
826 |
finally: |
827 |
osutils.rmtree(tmp_dirname) |
|
828 |
||
829 |
def _copy_to_tempdir(self, from_repo): |
|
3638.3.2
by Vincent Ladeuil
Fix all calls to tempfile.mkdtemp to osutils.mkdtemp. |
830 |
tmp_dirname = osutils.mkdtemp(prefix='tmpbzrclone') |
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
831 |
tmp_bzrdir = from_repo.bzrdir._format.initialize(tmp_dirname) |
832 |
tmp_repo = from_repo._format.initialize(tmp_bzrdir) |
|
833 |
from_repo.copy_content_into(tmp_repo) |
|
834 |
return tmp_dirname, tmp_repo |
|
835 |
||
2018.18.10
by Martin Pool
copy_content_into from Remote repositories by using temporary directories on both ends. |
836 |
def _tarfile_response(self, tmp_dirname, compression): |
2018.18.2
by Martin Pool
smart method Repository.tarball actually returns the tarball |
837 |
temp = tempfile.NamedTemporaryFile() |
838 |
try: |
|
2557.1.1
by Martin Pool
[BUG 119330] Fix tempfile permissions error in smart server tar bundling (under windows) (Martin_) |
839 |
self._tarball_of_dir(tmp_dirname, compression, temp.file) |
2018.18.2
by Martin Pool
smart method Repository.tarball actually returns the tarball |
840 |
# all finished; write the tempfile out to the network
|
841 |
temp.seek(0) |
|
2420.2.2
by Andrew Bennetts
Merge tarball branch that's already with PQM, resolving conflicts. |
842 |
return SuccessfulSmartServerResponse(('ok',), temp.read()) |
3638.3.2
by Vincent Ladeuil
Fix all calls to tempfile.mkdtemp to osutils.mkdtemp. |
843 |
# FIXME: Don't read the whole thing into memory here; rather stream
|
844 |
# it out from the file onto the network. mbp 20070411
|
|
2018.18.2
by Martin Pool
smart method Repository.tarball actually returns the tarball |
845 |
finally: |
846 |
temp.close() |
|
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
847 |
|
2557.1.1
by Martin Pool
[BUG 119330] Fix tempfile permissions error in smart server tar bundling (under windows) (Martin_) |
848 |
def _tarball_of_dir(self, dirname, compression, ofile): |
5017.2.4
by Martin Pool
Move or remove some unconditionally loaded code |
849 |
import tarfile |
2571.2.2
by Ian Clatworthy
use basename as poolie recommended |
850 |
filename = os.path.basename(ofile.name) |
851 |
tarball = tarfile.open(fileobj=ofile, name=filename, |
|
2571.2.1
by Ian Clatworthy
fix #123485 - selftest broken under Python 2.5.1 because of tafile API change |
852 |
mode='w|' + compression) |
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
853 |
try: |
854 |
# The tarball module only accepts ascii names, and (i guess)
|
|
855 |
# packs them with their 8bit names. We know all the files
|
|
856 |
# within the repository have ASCII names so the should be safe
|
|
857 |
# to pack in.
|
|
2018.18.10
by Martin Pool
copy_content_into from Remote repositories by using temporary directories on both ends. |
858 |
dirname = dirname.encode(sys.getfilesystemencoding()) |
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
859 |
# python's tarball module includes the whole path by default so
|
860 |
# override it
|
|
3376.2.4
by Martin Pool
Remove every assert statement from bzrlib! |
861 |
if not dirname.endswith('.bzr'): |
862 |
raise ValueError(dirname) |
|
2018.18.10
by Martin Pool
copy_content_into from Remote repositories by using temporary directories on both ends. |
863 |
tarball.add(dirname, '.bzr') # recursive by default |
2018.18.9
by Martin Pool
remote Repository.tarball builds a temporary directory and tars that |
864 |
finally: |
865 |
tarball.close() |
|
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
866 |
|
867 |
||
4144.3.1
by Andrew Bennetts
Add Repository.insert_stream_locked server-side implementation, plus tests for server-side _translate_error. |
868 |
class SmartServerRepositoryInsertStreamLocked(SmartServerRepositoryRequest): |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
869 |
"""Insert a record stream from a RemoteSink into a repository.
|
870 |
||
871 |
This gets bytes pushed to it by the network infrastructure and turns that
|
|
872 |
into a bytes iterator using a thread. That is then processed by
|
|
873 |
_byte_stream_to_stream.
|
|
4144.3.1
by Andrew Bennetts
Add Repository.insert_stream_locked server-side implementation, plus tests for server-side _translate_error. |
874 |
|
875 |
New in 1.14.
|
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
876 |
"""
|
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
877 |
|
4144.3.1
by Andrew Bennetts
Add Repository.insert_stream_locked server-side implementation, plus tests for server-side _translate_error. |
878 |
def do_repository_request(self, repository, resume_tokens, lock_token): |
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
879 |
"""StreamSink.insert_stream for a remote repository."""
|
4144.3.1
by Andrew Bennetts
Add Repository.insert_stream_locked server-side implementation, plus tests for server-side _translate_error. |
880 |
repository.lock_write(token=lock_token) |
881 |
self.do_insert_stream_request(repository, resume_tokens) |
|
882 |
||
883 |
def do_insert_stream_request(self, repository, resume_tokens): |
|
4029.2.1
by Robert Collins
Support streaming push to stacked branches. |
884 |
tokens = [token for token in resume_tokens.split(' ') if token] |
4032.3.7
by Robert Collins
Move write locking and write group responsibilities into the Sink objects themselves, allowing complete avoidance of unnecessary calls when the sink is a RemoteSink. |
885 |
self.tokens = tokens |
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
886 |
self.repository = repository |
887 |
self.queue = Queue.Queue() |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
888 |
self.insert_thread = threading.Thread(target=self._inserter_thread) |
889 |
self.insert_thread.start() |
|
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
890 |
|
891 |
def do_chunk(self, body_stream_chunk): |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
892 |
self.queue.put(body_stream_chunk) |
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
893 |
|
894 |
def _inserter_thread(self): |
|
4032.3.7
by Robert Collins
Move write locking and write group responsibilities into the Sink objects themselves, allowing complete avoidance of unnecessary calls when the sink is a RemoteSink. |
895 |
try: |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
896 |
src_format, stream = _byte_stream_to_stream( |
897 |
self.blocking_byte_stream()) |
|
4032.3.7
by Robert Collins
Move write locking and write group responsibilities into the Sink objects themselves, allowing complete avoidance of unnecessary calls when the sink is a RemoteSink. |
898 |
self.insert_result = self.repository._get_sink().insert_stream( |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
899 |
stream, src_format, self.tokens) |
4032.3.7
by Robert Collins
Move write locking and write group responsibilities into the Sink objects themselves, allowing complete avoidance of unnecessary calls when the sink is a RemoteSink. |
900 |
self.insert_ok = True |
901 |
except: |
|
902 |
self.insert_exception = sys.exc_info() |
|
903 |
self.insert_ok = False |
|
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
904 |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
905 |
def blocking_byte_stream(self): |
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
906 |
while True: |
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
907 |
bytes = self.queue.get() |
908 |
if bytes is StopIteration: |
|
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
909 |
return
|
910 |
else: |
|
4060.1.4
by Robert Collins
Streaming fetch from remote servers. |
911 |
yield bytes |
4022.1.6
by Robert Collins
Cherrypick and polish the RemoteSink for streaming push. |
912 |
|
913 |
def do_end(self): |
|
914 |
self.queue.put(StopIteration) |
|
915 |
if self.insert_thread is not None: |
|
916 |
self.insert_thread.join() |
|
4032.3.7
by Robert Collins
Move write locking and write group responsibilities into the Sink objects themselves, allowing complete avoidance of unnecessary calls when the sink is a RemoteSink. |
917 |
if not self.insert_ok: |
918 |
exc_info = self.insert_exception |
|
919 |
raise exc_info[0], exc_info[1], exc_info[2] |
|
920 |
write_group_tokens, missing_keys = self.insert_result |
|
921 |
if write_group_tokens or missing_keys: |
|
922 |
# bzip needed? missing keys should typically be a small set.
|
|
923 |
# Should this be a streaming body response ?
|
|
924 |
missing_keys = sorted(missing_keys) |
|
925 |
bytes = bencode.bencode((write_group_tokens, missing_keys)) |
|
926 |
self.repository.unlock() |
|
927 |
return SuccessfulSmartServerResponse(('missing-basis', bytes)) |
|
4029.2.1
by Robert Collins
Support streaming push to stacked branches. |
928 |
else: |
4032.3.7
by Robert Collins
Move write locking and write group responsibilities into the Sink objects themselves, allowing complete avoidance of unnecessary calls when the sink is a RemoteSink. |
929 |
self.repository.unlock() |
930 |
return SuccessfulSmartServerResponse(('ok', )) |
|
4144.3.1
by Andrew Bennetts
Add Repository.insert_stream_locked server-side implementation, plus tests for server-side _translate_error. |
931 |
|
932 |
||
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
933 |
class SmartServerRepositoryInsertStream_1_19(SmartServerRepositoryInsertStreamLocked): |
4476.3.15
by Andrew Bennetts
Partially working fallback for pre-1.17 servers. |
934 |
"""Insert a record stream from a RemoteSink into a repository.
|
935 |
||
936 |
Same as SmartServerRepositoryInsertStreamLocked, except:
|
|
937 |
- the lock token argument is optional
|
|
938 |
- servers that implement this verb accept 'inventory-delta' records in the
|
|
939 |
stream.
|
|
940 |
||
4476.3.82
by Andrew Bennetts
Mention another bug fix in NEWS, and update verb name, comments, and NEWS additions for landing on 1.19 rather than 1.18. |
941 |
New in 1.19.
|
4476.3.15
by Andrew Bennetts
Partially working fallback for pre-1.17 servers. |
942 |
"""
|
943 |
||
944 |
def do_repository_request(self, repository, resume_tokens, lock_token=None): |
|
945 |
"""StreamSink.insert_stream for a remote repository."""
|
|
946 |
SmartServerRepositoryInsertStreamLocked.do_repository_request( |
|
947 |
self, repository, resume_tokens, lock_token) |
|
948 |
||
949 |
||
4144.3.1
by Andrew Bennetts
Add Repository.insert_stream_locked server-side implementation, plus tests for server-side _translate_error. |
950 |
class SmartServerRepositoryInsertStream(SmartServerRepositoryInsertStreamLocked): |
951 |
"""Insert a record stream from a RemoteSink into an unlocked repository.
|
|
952 |
||
953 |
This is the same as SmartServerRepositoryInsertStreamLocked, except it
|
|
954 |
takes no lock_tokens; i.e. it works with an unlocked (or lock-free, e.g.
|
|
955 |
like pack format) repository.
|
|
956 |
||
957 |
New in 1.13.
|
|
958 |
"""
|
|
959 |
||
960 |
def do_repository_request(self, repository, resume_tokens): |
|
961 |
"""StreamSink.insert_stream for a remote repository."""
|
|
962 |
repository.lock_write() |
|
963 |
self.do_insert_stream_request(repository, resume_tokens) |
|
964 |
||
965 |
||
6268.1.2
by Jelmer Vernooij
Initial work on Repository.add_signature_text. |
966 |
class SmartServerRepositoryAddSignatureText(SmartServerRepositoryRequest): |
967 |
"""Add a revision signature text.
|
|
968 |
||
969 |
New in 2.5.
|
|
970 |
"""
|
|
971 |
||
6280.10.32
by Jelmer Vernooij
Merge bzr.dev. |
972 |
def do_repository_request(self, repository, lock_token, revision_id, |
973 |
*write_group_tokens): |
|
6268.1.7
by Jelmer Vernooij
Docstrings. |
974 |
"""Add a revision signature text.
|
975 |
||
976 |
:param repository: Repository to operate on
|
|
977 |
:param lock_token: Lock token
|
|
6280.10.32
by Jelmer Vernooij
Merge bzr.dev. |
978 |
:param revision_id: Revision for which to add signature
|
6268.1.7
by Jelmer Vernooij
Docstrings. |
979 |
:param write_group_tokens: Write group tokens
|
980 |
"""
|
|
6268.1.6
by Jelmer Vernooij
Fix add_signature_text. |
981 |
self._lock_token = lock_token |
6280.10.32
by Jelmer Vernooij
Merge bzr.dev. |
982 |
self._revision_id = revision_id |
6268.1.6
by Jelmer Vernooij
Fix add_signature_text. |
983 |
self._write_group_tokens = write_group_tokens |
6268.1.2
by Jelmer Vernooij
Initial work on Repository.add_signature_text. |
984 |
return None |
985 |
||
986 |
def do_body(self, body_bytes): |
|
6268.1.7
by Jelmer Vernooij
Docstrings. |
987 |
"""Add a signature text.
|
988 |
||
989 |
:param body_bytes: GPG signature text
|
|
990 |
:return: SuccessfulSmartServerResponse with arguments 'ok' and
|
|
991 |
the list of new write group tokens.
|
|
992 |
"""
|
|
6268.1.6
by Jelmer Vernooij
Fix add_signature_text. |
993 |
self._repository.lock_write(token=self._lock_token) |
994 |
try: |
|
995 |
self._repository.resume_write_group(self._write_group_tokens) |
|
996 |
try: |
|
6280.10.35
by Jelmer Vernooij
Fix formatting. |
997 |
self._repository.add_signature_text(self._revision_id, |
998 |
body_bytes) |
|
6268.1.6
by Jelmer Vernooij
Fix add_signature_text. |
999 |
finally: |
1000 |
new_write_group_tokens = self._repository.suspend_write_group() |
|
1001 |
finally: |
|
1002 |
self._repository.unlock() |
|
6280.10.35
by Jelmer Vernooij
Fix formatting. |
1003 |
return SuccessfulSmartServerResponse( |
1004 |
('ok', ) + tuple(new_write_group_tokens)) |
|
6268.1.4
by Jelmer Vernooij
Merge write group improvements. |
1005 |
|
1006 |
||
6280.7.2
by Jelmer Vernooij
Add HPSS calls ``Repository.start_write_group``, ``Repository.abort_write_group`` and ``Repository.commit_write_group``. |
1007 |
class SmartServerRepositoryStartWriteGroup(SmartServerRepositoryRequest): |
1008 |
"""Start a write group.
|
|
1009 |
||
1010 |
New in 2.5.
|
|
1011 |
"""
|
|
1012 |
||
1013 |
def do_repository_request(self, repository, lock_token): |
|
1014 |
"""Start a write group."""
|
|
1015 |
repository.lock_write(token=lock_token) |
|
1016 |
try: |
|
1017 |
repository.start_write_group() |
|
6280.7.9
by Jelmer Vernooij
test repositories with unsuspendable write groups. |
1018 |
try: |
1019 |
tokens = repository.suspend_write_group() |
|
1020 |
except errors.UnsuspendableWriteGroup: |
|
1021 |
return FailedSmartServerResponse(('UnsuspendableWriteGroup',)) |
|
6280.7.2
by Jelmer Vernooij
Add HPSS calls ``Repository.start_write_group``, ``Repository.abort_write_group`` and ``Repository.commit_write_group``. |
1022 |
finally: |
1023 |
repository.unlock() |
|
6280.7.6
by Jelmer Vernooij
Fix remaining tests. |
1024 |
return SuccessfulSmartServerResponse(('ok', tokens)) |
6280.7.2
by Jelmer Vernooij
Add HPSS calls ``Repository.start_write_group``, ``Repository.abort_write_group`` and ``Repository.commit_write_group``. |
1025 |
|
1026 |
||
1027 |
class SmartServerRepositoryCommitWriteGroup(SmartServerRepositoryRequest): |
|
1028 |
"""Commit a write group.
|
|
1029 |
||
1030 |
New in 2.5.
|
|
1031 |
"""
|
|
1032 |
||
1033 |
def do_repository_request(self, repository, lock_token, |
|
6280.7.4
by Jelmer Vernooij
pass write group tokens as list/tuple. |
1034 |
write_group_tokens): |
6280.7.2
by Jelmer Vernooij
Add HPSS calls ``Repository.start_write_group``, ``Repository.abort_write_group`` and ``Repository.commit_write_group``. |
1035 |
"""Commit a write group."""
|
1036 |
repository.lock_write(token=lock_token) |
|
1037 |
try: |
|
6280.7.5
by Jelmer Vernooij
Bunch of test fixes. |
1038 |
try: |
1039 |
repository.resume_write_group(write_group_tokens) |
|
1040 |
except errors.UnresumableWriteGroup, e: |
|
1041 |
return FailedSmartServerResponse( |
|
1042 |
('UnresumableWriteGroup', e.write_groups, e.reason)) |
|
1043 |
try: |
|
1044 |
repository.commit_write_group() |
|
1045 |
except: |
|
1046 |
write_group_tokens = repository.suspend_write_group() |
|
1047 |
# FIXME JRV 2011-11-19: What if the write_group_tokens
|
|
1048 |
# have changed?
|
|
1049 |
raise
|
|
6280.7.2
by Jelmer Vernooij
Add HPSS calls ``Repository.start_write_group``, ``Repository.abort_write_group`` and ``Repository.commit_write_group``. |
1050 |
finally: |
1051 |
repository.unlock() |
|
1052 |
return SuccessfulSmartServerResponse(('ok', )) |
|
1053 |
||
1054 |
||
1055 |
class SmartServerRepositoryAbortWriteGroup(SmartServerRepositoryRequest): |
|
1056 |
"""Abort a write group.
|
|
1057 |
||
1058 |
New in 2.5.
|
|
1059 |
"""
|
|
1060 |
||
6280.7.5
by Jelmer Vernooij
Bunch of test fixes. |
1061 |
def do_repository_request(self, repository, lock_token, write_group_tokens): |
6280.7.2
by Jelmer Vernooij
Add HPSS calls ``Repository.start_write_group``, ``Repository.abort_write_group`` and ``Repository.commit_write_group``. |
1062 |
"""Abort a write group."""
|
1063 |
repository.lock_write(token=lock_token) |
|
1064 |
try: |
|
6280.7.5
by Jelmer Vernooij
Bunch of test fixes. |
1065 |
try: |
1066 |
repository.resume_write_group(write_group_tokens) |
|
1067 |
except errors.UnresumableWriteGroup, e: |
|
1068 |
return FailedSmartServerResponse( |
|
1069 |
('UnresumableWriteGroup', e.write_groups, e.reason)) |
|
1070 |
repository.abort_write_group() |
|
6280.7.2
by Jelmer Vernooij
Add HPSS calls ``Repository.start_write_group``, ``Repository.abort_write_group`` and ``Repository.commit_write_group``. |
1071 |
finally: |
1072 |
repository.unlock() |
|
1073 |
return SuccessfulSmartServerResponse(('ok', )) |
|
6280.7.6
by Jelmer Vernooij
Fix remaining tests. |
1074 |
|
1075 |
||
1076 |
class SmartServerRepositoryCheckWriteGroup(SmartServerRepositoryRequest): |
|
1077 |
"""Check that a write group is still valid.
|
|
1078 |
||
1079 |
New in 2.5.
|
|
1080 |
"""
|
|
1081 |
||
1082 |
def do_repository_request(self, repository, lock_token, write_group_tokens): |
|
1083 |
"""Abort a write group."""
|
|
1084 |
repository.lock_write(token=lock_token) |
|
1085 |
try: |
|
1086 |
try: |
|
1087 |
repository.resume_write_group(write_group_tokens) |
|
1088 |
except errors.UnresumableWriteGroup, e: |
|
1089 |
return FailedSmartServerResponse( |
|
1090 |
('UnresumableWriteGroup', e.write_groups, e.reason)) |
|
1091 |
else: |
|
1092 |
repository.suspend_write_group() |
|
1093 |
finally: |
|
1094 |
repository.unlock() |
|
1095 |
return SuccessfulSmartServerResponse(('ok', )) |
|
6280.7.13
by Jelmer Vernooij
Merge bzr.dev. |
1096 |
|
6268.1.13
by Jelmer Vernooij
Merge bzr.dev. |
1097 |
|
6280.3.2
by Jelmer Vernooij
Add smart side of RemoteRepository.all_revision_ids(). |
1098 |
class SmartServerRepositoryAllRevisionIds(SmartServerRepositoryRequest): |
1099 |
"""Retrieve all of the revision ids in a repository.
|
|
1100 |
||
1101 |
New in 2.5.
|
|
1102 |
"""
|
|
1103 |
||
1104 |
def do_repository_request(self, repository): |
|
1105 |
revids = repository.all_revision_ids() |
|
1106 |
return SuccessfulSmartServerResponse(("ok", ), "\n".join(revids)) |
|
6280.10.28
by Jelmer Vernooij
merge bzr.dev. |
1107 |
|
1108 |
||
6300.1.2
by Jelmer Vernooij
Add remote side of Repository.reconcile. |
1109 |
class SmartServerRepositoryReconcile(SmartServerRepositoryRequest): |
1110 |
"""Reconcile a repository.
|
|
1111 |
||
1112 |
New in 2.5.
|
|
1113 |
"""
|
|
1114 |
||
6300.1.7
by Jelmer Vernooij
Fix test. |
1115 |
def do_repository_request(self, repository, lock_token): |
1116 |
try: |
|
1117 |
repository.lock_write(token=lock_token) |
|
1118 |
except errors.TokenLockingNotSupported, e: |
|
1119 |
return FailedSmartServerResponse( |
|
1120 |
('TokenLockingNotSupported', )) |
|
1121 |
try: |
|
1122 |
reconciler = repository.reconcile() |
|
1123 |
finally: |
|
1124 |
repository.unlock() |
|
6300.1.4
by Jelmer Vernooij
Add reconcile results. |
1125 |
body = [ |
1126 |
"garbage_inventories: %d\n" % reconciler.garbage_inventories, |
|
1127 |
"inconsistent_parents: %d\n" % reconciler.inconsistent_parents, |
|
1128 |
]
|
|
1129 |
return SuccessfulSmartServerResponse(('ok', ), "".join(body)) |
|
6300.1.5
by Jelmer Vernooij
merge bzr.dev. |
1130 |
|
1131 |
||
6305.2.2
by Jelmer Vernooij
Add smart side of pack. |
1132 |
class SmartServerRepositoryPack(SmartServerRepositoryRequest): |
1133 |
"""Pack a repository.
|
|
1134 |
||
1135 |
New in 2.5.
|
|
1136 |
"""
|
|
1137 |
||
6305.2.4
by Jelmer Vernooij
Fix tests. |
1138 |
def do_repository_request(self, repository, lock_token, clean_obsolete_packs): |
6305.2.3
by Jelmer Vernooij
Store hint in body. |
1139 |
self._repository = repository |
6305.2.4
by Jelmer Vernooij
Fix tests. |
1140 |
self._lock_token = lock_token |
1141 |
if clean_obsolete_packs == 'True': |
|
1142 |
self._clean_obsolete_packs = True |
|
1143 |
else: |
|
1144 |
self._clean_obsolete_packs = False |
|
6305.2.3
by Jelmer Vernooij
Store hint in body. |
1145 |
return None |
1146 |
||
1147 |
def do_body(self, body_bytes): |
|
1148 |
if body_bytes == "": |
|
1149 |
hint = None |
|
1150 |
else: |
|
1151 |
hint = body_bytes.splitlines() |
|
6305.2.4
by Jelmer Vernooij
Fix tests. |
1152 |
self._repository.lock_write(token=self._lock_token) |
1153 |
try: |
|
1154 |
self._repository.pack(hint, self._clean_obsolete_packs) |
|
1155 |
finally: |
|
1156 |
self._repository.unlock() |
|
6305.2.2
by Jelmer Vernooij
Add smart side of pack. |
1157 |
return SuccessfulSmartServerResponse(("ok", ), ) |
6280.10.33
by Jelmer Vernooij
Merge bzr.dev. |
1158 |
|
1159 |
||
6280.10.20
by Jelmer Vernooij
Convert smart to zlib. |
1160 |
class SmartServerRepositoryIterFilesBytes(SmartServerRepositoryRequest): |
6280.10.6
by Jelmer Vernooij
Convert to iter_files_bytes_bz2. |
1161 |
"""Iterate over the contents of files.
|
1162 |
||
1163 |
The client sends a list of desired files to stream, one
|
|
1164 |
per line, and as tuples of file id and revision, separated by
|
|
1165 |
\0.
|
|
6280.10.2
by Jelmer Vernooij
Add Repository.iter_file_bytes. |
1166 |
|
6280.10.15
by Jelmer Vernooij
Document protocol. |
1167 |
The server replies with a stream. Each entry is preceded by a header,
|
1168 |
which can either be:
|
|
1169 |
||
1170 |
* "ok\x00IDX\n" where IDX is the index of the entry in the desired files
|
|
1171 |
list sent by the client. This header is followed by the contents of
|
|
1172 |
the file, bzip2-compressed.
|
|
1173 |
* "absent\x00FILEID\x00REVISION\x00IDX" to indicate a text is missing.
|
|
1174 |
The client can then raise an appropriate RevisionNotPresent error
|
|
1175 |
or check its fallback repositories.
|
|
1176 |
||
6280.10.2
by Jelmer Vernooij
Add Repository.iter_file_bytes. |
1177 |
New in 2.5.
|
1178 |
"""
|
|
1179 |
||
6280.10.6
by Jelmer Vernooij
Convert to iter_files_bytes_bz2. |
1180 |
def body_stream(self, repository, desired_files): |
1181 |
self._repository.lock_read() |
|
1182 |
try: |
|
6280.10.11
by Jelmer Vernooij
mark absent entries. |
1183 |
text_keys = {} |
1184 |
for i, key in enumerate(desired_files): |
|
1185 |
text_keys[key] = i |
|
6280.10.12
by Jelmer Vernooij
Handle stacking. |
1186 |
for record in repository.texts.get_record_stream(text_keys, |
1187 |
'unordered', True): |
|
6280.10.22
by Jelmer Vernooij
Simplify code a bit. |
1188 |
identifier = text_keys[record.key] |
6280.10.11
by Jelmer Vernooij
mark absent entries. |
1189 |
if record.storage_kind == 'absent': |
6280.10.12
by Jelmer Vernooij
Handle stacking. |
1190 |
yield "absent\0%s\0%s\0%d\n" % (record.key[0], |
6280.10.22
by Jelmer Vernooij
Simplify code a bit. |
1191 |
record.key[1], identifier) |
6280.10.11
by Jelmer Vernooij
mark absent entries. |
1192 |
# FIXME: Way to abort early?
|
1193 |
continue
|
|
6280.10.22
by Jelmer Vernooij
Simplify code a bit. |
1194 |
yield "ok\0%d\n" % identifier |
6280.10.20
by Jelmer Vernooij
Convert smart to zlib. |
1195 |
compressor = zlib.compressobj() |
6280.10.11
by Jelmer Vernooij
mark absent entries. |
1196 |
for bytes in record.get_bytes_as('chunked'): |
6280.10.20
by Jelmer Vernooij
Convert smart to zlib. |
1197 |
data = compressor.compress(bytes) |
1198 |
if data: |
|
1199 |
yield data |
|
6280.10.8
by Jelmer Vernooij
Fix iterator handling. |
1200 |
data = compressor.flush() |
1201 |
if data: |
|
1202 |
yield data |
|
6280.10.6
by Jelmer Vernooij
Convert to iter_files_bytes_bz2. |
1203 |
finally: |
1204 |
self._repository.unlock() |
|
1205 |
||
1206 |
def do_body(self, body_bytes): |
|
1207 |
desired_files = [ |
|
6280.10.11
by Jelmer Vernooij
mark absent entries. |
1208 |
tuple(l.split("\0")) for l in body_bytes.splitlines()] |
6280.10.2
by Jelmer Vernooij
Add Repository.iter_file_bytes. |
1209 |
return SuccessfulSmartServerResponse(('ok', ), |
6280.10.6
by Jelmer Vernooij
Convert to iter_files_bytes_bz2. |
1210 |
body_stream=self.body_stream(self._repository, desired_files)) |
1211 |
||
1212 |
def do_repository_request(self, repository): |
|
1213 |
# Signal that we want a body
|
|
1214 |
return None |
|
6280.10.38
by Jelmer Vernooij
Merge bzr.dev. |
1215 |
|
1216 |
||
6280.9.2
by Jelmer Vernooij
Add smart side. |
1217 |
class SmartServerRepositoryIterRevisions(SmartServerRepositoryRequest): |
1218 |
"""Stream a list of revisions.
|
|
1219 |
||
1220 |
The client sends a list of newline-separated revision ids in the
|
|
1221 |
body of the request and the server replies with the serializer format,
|
|
1222 |
and a stream of bzip2-compressed revision texts (using the specified
|
|
1223 |
serializer format).
|
|
1224 |
||
1225 |
Any revisions the server does not have are omitted from the stream.
|
|
1226 |
||
1227 |
New in 2.5.
|
|
1228 |
"""
|
|
1229 |
||
1230 |
def do_repository_request(self, repository): |
|
1231 |
self._repository = repository |
|
1232 |
# Signal there is a body
|
|
1233 |
return None |
|
1234 |
||
1235 |
def do_body(self, body_bytes): |
|
1236 |
revision_ids = body_bytes.split("\n") |
|
1237 |
return SuccessfulSmartServerResponse( |
|
1238 |
('ok', self._repository.get_serializer_format()), |
|
1239 |
body_stream=self.body_stream(self._repository, revision_ids)) |
|
1240 |
||
1241 |
def body_stream(self, repository, revision_ids): |
|
1242 |
self._repository.lock_read() |
|
1243 |
try: |
|
1244 |
for record in repository.revisions.get_record_stream( |
|
1245 |
[(revid,) for revid in revision_ids], 'unordered', True): |
|
1246 |
if record.storage_kind == 'absent': |
|
1247 |
continue
|
|
6280.9.6
by Jelmer Vernooij
simplify server side implementation - just zlib in one go. |
1248 |
yield zlib.compress(record.get_bytes_as('fulltext')) |
6280.9.2
by Jelmer Vernooij
Add smart side. |
1249 |
finally: |
1250 |
self._repository.unlock() |
|
6282.6.6
by Jelmer Vernooij
Implement server side. |
1251 |
|
1252 |
||
6282.6.28
by Jelmer Vernooij
Rename VersionedFileRepository.iter_inventories to VersionedFileRepository.get_inventories. |
1253 |
class SmartServerRepositoryGetInventories(SmartServerRepositoryRequest): |
1254 |
"""Get the inventory deltas for a set of revision ids.
|
|
6282.6.6
by Jelmer Vernooij
Implement server side. |
1255 |
|
6282.6.24
by Jelmer Vernooij
Update docstring. |
1256 |
This accepts a list of revision ids, and then sends a chain
|
1257 |
of deltas for the inventories of those revisions. The first
|
|
1258 |
revision will be empty.
|
|
6282.6.6
by Jelmer Vernooij
Implement server side. |
1259 |
|
1260 |
The server writes back zlibbed serialized inventory deltas,
|
|
6282.6.24
by Jelmer Vernooij
Update docstring. |
1261 |
in the ordering specified. The base for each delta is the
|
1262 |
inventory generated by the previous delta.
|
|
6282.6.6
by Jelmer Vernooij
Implement server side. |
1263 |
|
1264 |
New in 2.5.
|
|
1265 |
"""
|
|
1266 |
||
6282.6.31
by Jelmer Vernooij
Use record streams in get_inventories call. |
1267 |
def _inventory_delta_stream(self, repository, ordering, revids): |
1268 |
prev_inv = _mod_inventory.Inventory(root_id=None, |
|
1269 |
revision_id=_mod_revision.NULL_REVISION) |
|
6282.6.6
by Jelmer Vernooij
Implement server side. |
1270 |
serializer = inventory_delta.InventoryDeltaSerializer( |
1271 |
repository.supports_rich_root(), |
|
6282.6.7
by Jelmer Vernooij
Add basic server side test. |
1272 |
repository._format.supports_tree_reference) |
6282.6.31
by Jelmer Vernooij
Use record streams in get_inventories call. |
1273 |
repository.lock_read() |
6282.6.6
by Jelmer Vernooij
Implement server side. |
1274 |
try: |
6282.6.9
by Jelmer Vernooij
More tests. |
1275 |
for inv, revid in repository._iter_inventories(revids, ordering): |
6282.6.10
by Jelmer Vernooij
Fix smart tests. |
1276 |
if inv is None: |
1277 |
continue
|
|
6282.6.9
by Jelmer Vernooij
More tests. |
1278 |
inv_delta = inv._make_delta(prev_inv) |
1279 |
lines = serializer.delta_to_lines( |
|
1280 |
prev_inv.revision_id, inv.revision_id, inv_delta) |
|
6282.6.36
by Jelmer Vernooij
Fix smart server tests. |
1281 |
yield ChunkedContentFactory(inv.revision_id, None, None, lines) |
6282.6.6
by Jelmer Vernooij
Implement server side. |
1282 |
prev_inv = inv |
1283 |
finally: |
|
6282.6.31
by Jelmer Vernooij
Use record streams in get_inventories call. |
1284 |
repository.unlock() |
1285 |
||
1286 |
def body_stream(self, repository, ordering, revids): |
|
1287 |
substream = self._inventory_delta_stream(repository, |
|
1288 |
ordering, revids) |
|
6282.6.38
by Jelmer Vernooij
Fix typo in stream name. |
1289 |
return _stream_to_byte_stream([('inventory-deltas', substream)], |
6282.6.31
by Jelmer Vernooij
Use record streams in get_inventories call. |
1290 |
repository._format) |
6282.6.6
by Jelmer Vernooij
Implement server side. |
1291 |
|
1292 |
def do_body(self, body_bytes): |
|
1293 |
return SuccessfulSmartServerResponse(('ok', ), |
|
6282.6.9
by Jelmer Vernooij
More tests. |
1294 |
body_stream=self.body_stream(self._repository, self._ordering, |
1295 |
body_bytes.splitlines())) |
|
6282.6.6
by Jelmer Vernooij
Implement server side. |
1296 |
|
1297 |
def do_repository_request(self, repository, ordering): |
|
1298 |
if ordering == 'unordered': |
|
1299 |
# inventory deltas for a topologically sorted stream
|
|
1300 |
# are likely to be smaller
|
|
1301 |
ordering = 'topological' |
|
1302 |
self._ordering = ordering |
|
1303 |
# Signal that we want a body
|
|
1304 |
return None |