4763.2.4
by John Arbash Meinel
merge bzr.2.1 in preparation for NEWS entry. |
1 |
# Copyright (C) 2005-2010 Canonical Ltd
|
1540.3.18
by Martin Pool
Style review fixes (thanks robertc) |
2 |
#
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
3 |
# This program is free software; you can redistribute it and/or modify
|
4 |
# it under the terms of the GNU General Public License as published by
|
|
5 |
# the Free Software Foundation; either version 2 of the License, or
|
|
6 |
# (at your option) any later version.
|
|
1540.3.18
by Martin Pool
Style review fixes (thanks robertc) |
7 |
#
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
8 |
# This program is distributed in the hope that it will be useful,
|
9 |
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
10 |
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
11 |
# GNU General Public License for more details.
|
|
1540.3.18
by Martin Pool
Style review fixes (thanks robertc) |
12 |
#
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
13 |
# You should have received a copy of the GNU General Public License
|
14 |
# along with this program; if not, write to the Free Software
|
|
4183.7.1
by Sabin Iacob
update FSF mailing address |
15 |
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
1540.3.3
by Martin Pool
Review updates of pycurl transport |
16 |
|
17 |
"""Base implementation of Transport over http.
|
|
18 |
||
19 |
There are separate implementation modules for each http client implementation.
|
|
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
20 |
"""
|
21 |
||
1711.4.14
by John Arbash Meinel
Custom HttpRequestHandler which treats all paths as utf8 encoded |
22 |
from cStringIO import StringIO |
1540.3.23
by Martin Pool
Allow urls like http+pycurl://host/ to use a particular impl |
23 |
import re |
1540.3.3
by Martin Pool
Review updates of pycurl transport |
24 |
import urlparse |
25 |
import urllib |
|
2172.3.2
by v.ladeuil+lp at free
Fix the missing import and typos in comments. |
26 |
import sys |
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
27 |
import weakref |
1786.1.6
by John Arbash Meinel
Missed a couple of imports |
28 |
|
2485.8.24
by Vincent Ladeuil
Finish http refactoring. Test suite passing. |
29 |
from bzrlib import ( |
3675.1.1
by Martin Pool
Merge and update log+ transport decorator |
30 |
debug, |
2485.8.24
by Vincent Ladeuil
Finish http refactoring. Test suite passing. |
31 |
errors, |
5609.9.1
by Martin
Blindly change all users of get_transport to address the function via the transport module |
32 |
transport, |
2485.8.24
by Vincent Ladeuil
Finish http refactoring. Test suite passing. |
33 |
ui, |
34 |
urlutils, |
|
35 |
)
|
|
2400.1.3
by Andrew Bennetts
Split smart transport code into several separate modules. |
36 |
from bzrlib.smart import medium |
1185.11.1
by John Arbash Meinel
(broken) Transport work is merged in. Tests do not pass yet. |
37 |
from bzrlib.trace import mutter |
2018.2.2
by Andrew Bennetts
Implement HTTP smart server. |
38 |
from bzrlib.transport import ( |
2485.8.16
by Vincent Ladeuil
Create a new, empty, ConnectedTransport class. |
39 |
ConnectedTransport, |
2018.2.2
by Andrew Bennetts
Implement HTTP smart server. |
40 |
)
|
1540.3.6
by Martin Pool
[merge] update from bzr.dev |
41 |
|
2004.1.9
by vila
Takes jam's remarks into account when possible, add TODOs for the rest. |
42 |
# TODO: This is not used anymore by HttpTransport_urllib
|
43 |
# (extracting the auth info and prompting the user for a password
|
|
44 |
# have been split), only the tests still use it. It should be
|
|
45 |
# deleted and the tests rewritten ASAP to stay in sync.
|
|
1185.40.20
by Robey Pointer
allow user:pass@ info in http urls to be used for auth; this should be easily expandable later to use auth config files |
46 |
def extract_auth(url, password_manager): |
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
47 |
"""Extract auth parameters from am HTTP/HTTPS url and add them to the given
|
1185.40.20
by Robey Pointer
allow user:pass@ info in http urls to be used for auth; this should be easily expandable later to use auth config files |
48 |
password manager. Return the url, minus those auth parameters (which
|
49 |
confuse urllib2).
|
|
50 |
"""
|
|
3376.2.4
by Martin Pool
Remove every assert statement from bzrlib! |
51 |
if not re.match(r'^(https?)(\+\w+)?://', url): |
52 |
raise ValueError( |
|
53 |
'invalid absolute url %r' % (url,)) |
|
1540.2.1
by Röbey Pointer
change http url parsing to use urlparse, and use the ui_factory to ask for a password if necessary |
54 |
scheme, netloc, path, query, fragment = urlparse.urlsplit(url) |
2004.3.1
by vila
Test ConnectionError exceptions. |
55 |
|
1540.2.1
by Röbey Pointer
change http url parsing to use urlparse, and use the ui_factory to ask for a password if necessary |
56 |
if '@' in netloc: |
57 |
auth, netloc = netloc.split('@', 1) |
|
1185.40.20
by Robey Pointer
allow user:pass@ info in http urls to be used for auth; this should be easily expandable later to use auth config files |
58 |
if ':' in auth: |
59 |
username, password = auth.split(':', 1) |
|
60 |
else: |
|
61 |
username, password = auth, None |
|
1540.2.1
by Röbey Pointer
change http url parsing to use urlparse, and use the ui_factory to ask for a password if necessary |
62 |
if ':' in netloc: |
63 |
host = netloc.split(':', 1)[0] |
|
64 |
else: |
|
65 |
host = netloc |
|
66 |
username = urllib.unquote(username) |
|
1185.40.20
by Robey Pointer
allow user:pass@ info in http urls to be used for auth; this should be easily expandable later to use auth config files |
67 |
if password is not None: |
68 |
password = urllib.unquote(password) |
|
1540.2.1
by Röbey Pointer
change http url parsing to use urlparse, and use the ui_factory to ask for a password if necessary |
69 |
else: |
2094.3.6
by John Arbash Meinel
[merge] bzr.dev 2158 |
70 |
password = ui.ui_factory.get_password( |
5923.1.2
by Vincent Ladeuil
Fix some more prompts to be unicode. |
71 |
prompt=u'HTTP %(user)s@%(host)s password', |
2004.2.1
by John Arbash Meinel
Cleanup of urllib functions |
72 |
user=username, host=host) |
1540.2.1
by Röbey Pointer
change http url parsing to use urlparse, and use the ui_factory to ask for a password if necessary |
73 |
password_manager.add_password(None, host, username, password) |
74 |
url = urlparse.urlunsplit((scheme, netloc, path, query, fragment)) |
|
1185.40.20
by Robey Pointer
allow user:pass@ info in http urls to be used for auth; this should be easily expandable later to use auth config files |
75 |
return url |
1553.1.5
by James Henstridge
Make HTTP transport has() method do HEAD requests, and update test to |
76 |
|
1185.50.83
by John Arbash Meinel
[merge] James Henstridge: Set Agent string in http headers, add tests for it. |
77 |
|
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
78 |
class HttpTransportBase(ConnectedTransport): |
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
79 |
"""Base class for http implementations.
|
80 |
||
1540.3.23
by Martin Pool
Allow urls like http+pycurl://host/ to use a particular impl |
81 |
Does URL parsing, etc, but not any network IO.
|
82 |
||
83 |
The protocol can be given as e.g. http+urllib://host/ to use a particular
|
|
84 |
implementation.
|
|
85 |
"""
|
|
86 |
||
2485.8.24
by Vincent Ladeuil
Finish http refactoring. Test suite passing. |
87 |
# _unqualified_scheme: "http" or "https"
|
88 |
# _scheme: may have "+pycurl", etc
|
|
1540.3.24
by Martin Pool
Add new protocol 'http+pycurl' that always uses PyCurl. |
89 |
|
3878.4.2
by Vincent Ladeuil
Fix bug #265070 by providing a finer sieve for accepted redirections. |
90 |
def __init__(self, base, _impl_name, _from_transport=None): |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
91 |
"""Set the base path where files will be stored."""
|
1540.3.23
by Martin Pool
Allow urls like http+pycurl://host/ to use a particular impl |
92 |
proto_match = re.match(r'^(https?)(\+\w+)?://', base) |
93 |
if not proto_match: |
|
94 |
raise AssertionError("not a http url: %r" % base) |
|
2485.8.24
by Vincent Ladeuil
Finish http refactoring. Test suite passing. |
95 |
self._unqualified_scheme = proto_match.group(1) |
3878.4.2
by Vincent Ladeuil
Fix bug #265070 by providing a finer sieve for accepted redirections. |
96 |
self._impl_name = _impl_name |
2485.8.59
by Vincent Ladeuil
Update from review comments. |
97 |
super(HttpTransportBase, self).__init__(base, |
98 |
_from_transport=_from_transport) |
|
3734.3.2
by Vincent Ladeuil
Fix another SmartHTTPMedium refactoring bit. |
99 |
self._medium = None |
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
100 |
# range hint is handled dynamically throughout the life
|
2363.4.9
by Vincent Ladeuil
Catch first succesful authentification to avoid further 401 |
101 |
# of the transport object. We start by trying multi-range
|
102 |
# requests and if the server returns bogus results, we
|
|
103 |
# retry with single range requests and, finally, we
|
|
104 |
# forget about range if the server really can't
|
|
105 |
# understand. Once acquired, this piece of info is
|
|
106 |
# propagated to clones.
|
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
107 |
if _from_transport is not None: |
108 |
self._range_hint = _from_transport._range_hint |
|
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
109 |
else: |
110 |
self._range_hint = 'multi' |
|
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
111 |
|
112 |
def has(self, relpath): |
|
1540.3.15
by Martin Pool
[merge] large merge to sync with bzr.dev |
113 |
raise NotImplementedError("has() is abstract on %r" % self) |
114 |
||
2164.2.15
by Vincent Ladeuil
Http redirections are not followed by default. Do not use hints |
115 |
def get(self, relpath): |
1594.2.5
by Robert Collins
Readv patch from Johan Rydberg giving knits partial download support. |
116 |
"""Get the file at the given relative path.
|
117 |
||
118 |
:param relpath: The relative path to the file
|
|
119 |
"""
|
|
3945.1.5
by Vincent Ladeuil
Start implementing http activity reporting at socket level. |
120 |
code, response_file = self._get(relpath, None) |
3059.2.2
by Vincent Ladeuil
Read http responses on demand without buffering the whole body |
121 |
# FIXME: some callers want an iterable... One step forward, three steps
|
3059.2.6
by Vincent Ladeuil
Light modifications after a failed attempt at making RangeFile iterable. |
122 |
# backwards :-/ And not only an iterable, but an iterable that can be
|
123 |
# seeked backwards, so we will never be able to do that. One such
|
|
124 |
# known client is bzrlib.bundle.serializer.v4.get_bundle_reader. At the
|
|
125 |
# time of this writing it's even the only known client -- vila20071203
|
|
3945.1.5
by Vincent Ladeuil
Start implementing http activity reporting at socket level. |
126 |
return StringIO(response_file.read()) |
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
127 |
|
2164.2.15
by Vincent Ladeuil
Http redirections are not followed by default. Do not use hints |
128 |
def _get(self, relpath, ranges, tail_amount=0): |
1540.3.27
by Martin Pool
Integrate http range support for pycurl |
129 |
"""Get a file, or part of a file.
|
130 |
||
131 |
:param relpath: Path relative to transport base URL
|
|
2164.2.1
by v.ladeuil+lp at free
First rough http branch redirection implementation. |
132 |
:param ranges: None to get the whole file;
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
133 |
or a list of _CoalescedOffset to fetch parts of a file.
|
2164.2.26
by Vincent Ladeuil
Delete obsolete note in doc string. |
134 |
:param tail_amount: The amount to get from the end of the file.
|
1540.3.27
by Martin Pool
Integrate http range support for pycurl |
135 |
|
136 |
:returns: (http_code, result_file)
|
|
137 |
"""
|
|
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
138 |
raise NotImplementedError(self._get) |
1594.2.5
by Robert Collins
Readv patch from Johan Rydberg giving knits partial download support. |
139 |
|
3133.1.2
by Vincent Ladeuil
Fix #177643 by making pycurl handle url-embedded credentials again. |
140 |
def _remote_path(self, relpath): |
141 |
"""See ConnectedTransport._remote_path.
|
|
142 |
||
143 |
user and passwords are not embedded in the path provided to the server.
|
|
144 |
"""
|
|
5268.7.19
by Jelmer Vernooij
Use urlutils.URL in bzrlib.transport.http. |
145 |
url = self._parsed_url.clone(relpath) |
146 |
url.user = url.quoted_user = None |
|
147 |
url.password = url.quoted_password = None |
|
148 |
url.scheme = self._unqualified_scheme |
|
149 |
return str(url) |
|
3133.1.2
by Vincent Ladeuil
Fix #177643 by making pycurl handle url-embedded credentials again. |
150 |
|
151 |
def _create_auth(self): |
|
4795.4.4
by Vincent Ladeuil
Protect more access to 'user' and 'password' auth attributes. |
152 |
"""Returns a dict containing the credentials provided at build time."""
|
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
153 |
auth = dict(host=self._parsed_url.host, port=self._parsed_url.port, |
154 |
user=self._parsed_url.user, password=self._parsed_url.password, |
|
3133.1.2
by Vincent Ladeuil
Fix #177643 by making pycurl handle url-embedded credentials again. |
155 |
protocol=self._unqualified_scheme, |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
156 |
path=self._parsed_url.path) |
3133.1.2
by Vincent Ladeuil
Fix #177643 by making pycurl handle url-embedded credentials again. |
157 |
return auth |
158 |
||
2018.2.3
by Andrew Bennetts
Starting factoring out the smart server client "medium" from the protocol. |
159 |
def get_smart_medium(self): |
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
160 |
"""See Transport.get_smart_medium."""
|
161 |
if self._medium is None: |
|
162 |
# Since medium holds some state (smart server probing at least), we
|
|
163 |
# need to keep it around. Note that this is needed because medium
|
|
164 |
# has the same 'base' attribute as the transport so it can't be
|
|
165 |
# shared between transports having different bases.
|
|
166 |
self._medium = SmartClientHTTPMedium(self) |
|
167 |
return self._medium |
|
2018.2.3
by Andrew Bennetts
Starting factoring out the smart server client "medium" from the protocol. |
168 |
|
2520.2.2
by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request |
169 |
def _degrade_range_hint(self, relpath, ranges, exc_info): |
2000.3.9
by v.ladeuil+lp at free
The tests that would have help avoid bug #73948 and all that mess :) |
170 |
if self._range_hint == 'multi': |
171 |
self._range_hint = 'single' |
|
2520.2.2
by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request |
172 |
mutter('Retry "%s" with single range request' % relpath) |
2000.3.9
by v.ladeuil+lp at free
The tests that would have help avoid bug #73948 and all that mess :) |
173 |
elif self._range_hint == 'single': |
174 |
self._range_hint = None |
|
2520.2.2
by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request |
175 |
mutter('Retry "%s" without ranges' % relpath) |
2000.3.9
by v.ladeuil+lp at free
The tests that would have help avoid bug #73948 and all that mess :) |
176 |
else: |
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
177 |
# We tried all the tricks, but nothing worked. We re-raise the
|
178 |
# original exception; the 'mutter' calls above will indicate that
|
|
179 |
# further tries were unsuccessful
|
|
2172.3.1
by v.ladeuil+lp at free
Merge a recent bzr.dev (2172) and takes John's remarks into account. |
180 |
raise exc_info[0], exc_info[1], exc_info[2] |
2000.3.9
by v.ladeuil+lp at free
The tests that would have help avoid bug #73948 and all that mess :) |
181 |
|
2520.2.2
by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request |
182 |
# _coalesce_offsets is a helper for readv, it try to combine ranges without
|
183 |
# degrading readv performances. _bytes_to_read_before_seek is the value
|
|
184 |
# used for the limit parameter and has been tuned for other transports. For
|
|
185 |
# HTTP, the name is inappropriate but the parameter is still useful and
|
|
186 |
# helps reduce the number of chunks in the response. The overhead for a
|
|
187 |
# chunk (headers, length, footer around the data itself is variable but
|
|
188 |
# around 50 bytes. We use 128 to reduce the range specifiers that appear in
|
|
189 |
# the header, some servers (notably Apache) enforce a maximum length for a
|
|
190 |
# header and issue a '400: Bad request' error when too much ranges are
|
|
191 |
# specified.
|
|
192 |
_bytes_to_read_before_seek = 128 |
|
193 |
# No limit on the offset number that get combined into one, we are trying
|
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
194 |
# to avoid downloading the whole file.
|
3024.2.1
by Vincent Ladeuil
Fix 165061 by using the correct _max_readv_combine attribute. |
195 |
_max_readv_combine = 0 |
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
196 |
# By default Apache has a limit of ~400 ranges before replying with a 400
|
197 |
# Bad Request. So we go underneath that amount to be safe.
|
|
198 |
_max_get_ranges = 200 |
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
199 |
# We impose no limit on the range size. But see _pycurl.py for a different
|
200 |
# use.
|
|
201 |
_get_max_size = 0 |
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
202 |
|
2745.5.1
by Robert Collins
* New parameter on ``bzrlib.transport.Transport.readv`` |
203 |
def _readv(self, relpath, offsets): |
1594.2.5
by Robert Collins
Readv patch from Johan Rydberg giving knits partial download support. |
204 |
"""Get parts of the file at the given relative path.
|
205 |
||
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
206 |
:param offsets: A list of (offset, size) tuples.
|
1540.3.27
by Martin Pool
Integrate http range support for pycurl |
207 |
:param return: A list or generator of (offset, data) tuples
|
1594.2.5
by Robert Collins
Readv patch from Johan Rydberg giving knits partial download support. |
208 |
"""
|
3059.2.18
by Vincent Ladeuil
Take spiv review comments into account. |
209 |
# offsets may be a generator, we will iterate it several times, so
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
210 |
# build a list
|
211 |
offsets = list(offsets) |
|
212 |
||
213 |
try_again = True |
|
3146.3.2
by Vincent Ladeuil
Fix #179368 by keeping the current range hint on ShortReadvErrors. |
214 |
retried_offset = None |
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
215 |
while try_again: |
216 |
try_again = False |
|
217 |
||
218 |
# Coalesce the offsets to minimize the GET requests issued
|
|
219 |
sorted_offsets = sorted(offsets) |
|
220 |
coalesced = self._coalesce_offsets( |
|
221 |
sorted_offsets, limit=self._max_readv_combine, |
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
222 |
fudge_factor=self._bytes_to_read_before_seek, |
223 |
max_size=self._get_max_size) |
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
224 |
|
225 |
# Turn it into a list, we will iterate it several times
|
|
226 |
coalesced = list(coalesced) |
|
3675.1.1
by Martin Pool
Merge and update log+ transport decorator |
227 |
if 'http' in debug.debug_flags: |
228 |
mutter('http readv of %s offsets => %s collapsed %s', |
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
229 |
relpath, len(offsets), len(coalesced)) |
230 |
||
231 |
# Cache the data read, but only until it's been used
|
|
232 |
data_map = {} |
|
233 |
# We will iterate on the data received from the GET requests and
|
|
3059.2.18
by Vincent Ladeuil
Take spiv review comments into account. |
234 |
# serve the corresponding offsets respecting the initial order. We
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
235 |
# need an offset iterator for that.
|
236 |
iter_offsets = iter(offsets) |
|
237 |
cur_offset_and_size = iter_offsets.next() |
|
238 |
||
239 |
try: |
|
3059.2.10
by Vincent Ladeuil
Jam's review feedback. |
240 |
for cur_coal, rfile in self._coalesce_readv(relpath, coalesced): |
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
241 |
# Split the received chunk
|
242 |
for offset, size in cur_coal.ranges: |
|
3052.3.2
by Vincent Ladeuil
Add tests and fix trivial bugs and other typos. |
243 |
start = cur_coal.start + offset |
3059.2.10
by Vincent Ladeuil
Jam's review feedback. |
244 |
rfile.seek(start, 0) |
245 |
data = rfile.read(size) |
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
246 |
data_len = len(data) |
247 |
if data_len != size: |
|
248 |
raise errors.ShortReadvError(relpath, start, size, |
|
249 |
actual=data_len) |
|
3059.2.5
by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix). |
250 |
if (start, size) == cur_offset_and_size: |
251 |
# The offset requested are sorted as the coalesced
|
|
3059.2.11
by Vincent Ladeuil
Fix typos mentioned by spiv. |
252 |
# ones, no need to cache. Win !
|
3059.2.5
by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix). |
253 |
yield cur_offset_and_size[0], data |
254 |
cur_offset_and_size = iter_offsets.next() |
|
255 |
else: |
|
256 |
# Different sorting. We need to cache.
|
|
257 |
data_map[(start, size)] = data |
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
258 |
|
259 |
# Yield everything we can
|
|
260 |
while cur_offset_and_size in data_map: |
|
261 |
# Clean the cached data since we use it
|
|
262 |
# XXX: will break if offsets contains duplicates --
|
|
263 |
# vila20071129
|
|
264 |
this_data = data_map.pop(cur_offset_and_size) |
|
265 |
yield cur_offset_and_size[0], this_data |
|
266 |
cur_offset_and_size = iter_offsets.next() |
|
267 |
||
3059.2.2
by Vincent Ladeuil
Read http responses on demand without buffering the whole body |
268 |
except (errors.ShortReadvError, errors.InvalidRange, |
5609.52.1
by Martin Pool
Cope with buggy squids interrupting the response before a mime multipart boundary |
269 |
errors.InvalidHttpRange, errors.HttpBoundaryMissing), e: |
3146.3.2
by Vincent Ladeuil
Fix #179368 by keeping the current range hint on ShortReadvErrors. |
270 |
mutter('Exception %r: %s during http._readv',e, e) |
271 |
if (not isinstance(e, errors.ShortReadvError) |
|
272 |
or retried_offset == cur_offset_and_size): |
|
273 |
# We don't degrade the range hint for ShortReadvError since
|
|
274 |
# they do not indicate a problem with the server ability to
|
|
275 |
# handle ranges. Except when we fail to get back a required
|
|
276 |
# offset twice in a row. In that case, falling back to
|
|
277 |
# single range or whole file should help or end up in a
|
|
278 |
# fatal exception.
|
|
279 |
self._degrade_range_hint(relpath, coalesced, sys.exc_info()) |
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
280 |
# Some offsets may have been already processed, so we retry
|
281 |
# only the unsuccessful ones.
|
|
3052.3.2
by Vincent Ladeuil
Add tests and fix trivial bugs and other typos. |
282 |
offsets = [cur_offset_and_size] + [o for o in iter_offsets] |
3146.3.2
by Vincent Ladeuil
Fix #179368 by keeping the current range hint on ShortReadvErrors. |
283 |
retried_offset = cur_offset_and_size |
3052.3.2
by Vincent Ladeuil
Add tests and fix trivial bugs and other typos. |
284 |
try_again = True |
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
285 |
|
286 |
def _coalesce_readv(self, relpath, coalesced): |
|
287 |
"""Issue several GET requests to satisfy the coalesced offsets"""
|
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
288 |
|
289 |
def get_and_yield(relpath, coalesced): |
|
290 |
if coalesced: |
|
291 |
# Note that the _get below may raise
|
|
3059.2.18
by Vincent Ladeuil
Take spiv review comments into account. |
292 |
# errors.InvalidHttpRange. It's the caller's responsibility to
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
293 |
# decide how to retry since it may provide different coalesced
|
294 |
# offsets.
|
|
295 |
code, rfile = self._get(relpath, coalesced) |
|
296 |
for coal in coalesced: |
|
297 |
yield coal, rfile |
|
298 |
||
299 |
if self._range_hint is None: |
|
300 |
# Download whole file
|
|
301 |
for c, rfile in get_and_yield(relpath, coalesced): |
|
302 |
yield c, rfile |
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
303 |
else: |
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
304 |
total = len(coalesced) |
305 |
if self._range_hint == 'multi': |
|
306 |
max_ranges = self._max_get_ranges |
|
3059.2.18
by Vincent Ladeuil
Take spiv review comments into account. |
307 |
elif self._range_hint == 'single': |
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
308 |
max_ranges = total |
3059.2.18
by Vincent Ladeuil
Take spiv review comments into account. |
309 |
else: |
310 |
raise AssertionError("Unknown _range_hint %r" |
|
311 |
% (self._range_hint,)) |
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
312 |
# TODO: Some web servers may ignore the range requests and return
|
313 |
# the whole file, we may want to detect that and avoid further
|
|
314 |
# requests.
|
|
315 |
# Hint: test_readv_multiple_get_requests will fail once we do that
|
|
316 |
cumul = 0 |
|
317 |
ranges = [] |
|
318 |
for coal in coalesced: |
|
319 |
if ((self._get_max_size > 0 |
|
320 |
and cumul + coal.length > self._get_max_size) |
|
321 |
or len(ranges) >= max_ranges): |
|
322 |
# Get that much and yield
|
|
323 |
for c, rfile in get_and_yield(relpath, ranges): |
|
324 |
yield c, rfile |
|
325 |
# Restart with the current offset
|
|
326 |
ranges = [coal] |
|
327 |
cumul = coal.length |
|
328 |
else: |
|
329 |
ranges.append(coal) |
|
330 |
cumul += coal.length |
|
331 |
# Get the rest and yield
|
|
332 |
for c, rfile in get_and_yield(relpath, ranges): |
|
333 |
yield c, rfile |
|
1786.1.5
by John Arbash Meinel
Move the common Multipart stuff into plain http, and wrap pycurl response so that it matches the urllib response object. |
334 |
|
2671.3.1
by Robert Collins
* New method ``bzrlib.transport.Transport.get_recommended_page_size``. |
335 |
def recommended_page_size(self): |
336 |
"""See Transport.recommended_page_size().
|
|
337 |
||
338 |
For HTTP we suggest a large page size to reduce the overhead
|
|
339 |
introduced by latency.
|
|
340 |
"""
|
|
341 |
return 64 * 1024 |
|
342 |
||
2018.2.10
by Andrew Bennetts
Tidy up TODOs, further testing and fixes for SmartServerRequestProtocolOne, and remove a read_bytes(1) call. |
343 |
def _post(self, body_bytes): |
344 |
"""POST body_bytes to .bzr/smart on this transport.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
345 |
|
2018.2.10
by Andrew Bennetts
Tidy up TODOs, further testing and fixes for SmartServerRequestProtocolOne, and remove a read_bytes(1) call. |
346 |
:returns: (response code, response body file-like object).
|
347 |
"""
|
|
348 |
# TODO: Requiring all the body_bytes to be available at the beginning of
|
|
349 |
# the POST may require large client buffers. It would be nice to have
|
|
350 |
# an interface that allows streaming via POST when possible (and
|
|
351 |
# degrades to a local buffer when not).
|
|
352 |
raise NotImplementedError(self._post) |
|
353 |
||
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
354 |
def put_file(self, relpath, f, mode=None): |
355 |
"""Copy the file-like object into the location.
|
|
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
356 |
|
357 |
:param relpath: Location to put the contents, relative to base.
|
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
358 |
:param f: File-like object.
|
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
359 |
"""
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
360 |
raise errors.TransportNotPossible('http PUT not supported') |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
361 |
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
362 |
def mkdir(self, relpath, mode=None): |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
363 |
"""Create a directory at the given path."""
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
364 |
raise errors.TransportNotPossible('http does not support mkdir()') |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
365 |
|
1534.4.15
by Robert Collins
Remove shutil dependency in upgrade - create a delete_tree method for transports. |
366 |
def rmdir(self, relpath): |
367 |
"""See Transport.rmdir."""
|
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
368 |
raise errors.TransportNotPossible('http does not support rmdir()') |
1534.4.15
by Robert Collins
Remove shutil dependency in upgrade - create a delete_tree method for transports. |
369 |
|
1955.3.15
by John Arbash Meinel
Deprecate 'Transport.append' in favor of Transport.append_file or Transport.append_bytes |
370 |
def append_file(self, relpath, f, mode=None): |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
371 |
"""Append the text in the file-like object into the final
|
372 |
location.
|
|
373 |
"""
|
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
374 |
raise errors.TransportNotPossible('http does not support append()') |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
375 |
|
376 |
def copy(self, rel_from, rel_to): |
|
377 |
"""Copy the item at rel_from to the location at rel_to"""
|
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
378 |
raise errors.TransportNotPossible('http does not support copy()') |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
379 |
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
380 |
def copy_to(self, relpaths, other, mode=None, pb=None): |
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
381 |
"""Copy a set of entries from self into another Transport.
|
382 |
||
383 |
:param relpaths: A list/generator of entries to be copied.
|
|
907.1.50
by John Arbash Meinel
Removed encode/decode from Transport.put/get, added more exceptions that can be thrown. |
384 |
|
385 |
TODO: if other is LocalTransport, is it possible to
|
|
386 |
do better than put(get())?
|
|
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
387 |
"""
|
907.1.29
by John Arbash Meinel
Fixing small bug in HttpTransport.copy_to |
388 |
# At this point HttpTransport might be able to check and see if
|
389 |
# the remote location is the same, and rather than download, and
|
|
390 |
# then upload, it could just issue a remote copy_this command.
|
|
1540.3.6
by Martin Pool
[merge] update from bzr.dev |
391 |
if isinstance(other, HttpTransportBase): |
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
392 |
raise errors.TransportNotPossible( |
393 |
'http cannot be the target of copy_to()') |
|
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
394 |
else: |
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
395 |
return super(HttpTransportBase, self).\ |
396 |
copy_to(relpaths, other, mode=mode, pb=pb) |
|
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
397 |
|
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
398 |
def move(self, rel_from, rel_to): |
399 |
"""Move the item at rel_from to the location at rel_to"""
|
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
400 |
raise errors.TransportNotPossible('http does not support move()') |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
401 |
|
402 |
def delete(self, relpath): |
|
403 |
"""Delete the item at relpath"""
|
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
404 |
raise errors.TransportNotPossible('http does not support delete()') |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
405 |
|
2634.1.1
by Robert Collins
(robertc) Reinstate the accidentally backed out external_url patch. |
406 |
def external_url(self): |
407 |
"""See bzrlib.transport.Transport.external_url."""
|
|
3878.4.6
by Vincent Ladeuil
Fix bug #270863 by preserving 'bzr+http[s]' decorator. |
408 |
# HTTP URL's are externally usable as long as they don't mention their
|
409 |
# implementation qualifier
|
|
5268.7.18
by Jelmer Vernooij
Use urlutils.URL in bzrlib.transport.http. |
410 |
url = self._parsed_url.clone() |
411 |
url.scheme = self._unqualified_scheme |
|
412 |
return str(url) |
|
2634.1.1
by Robert Collins
(robertc) Reinstate the accidentally backed out external_url patch. |
413 |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
414 |
def is_readonly(self): |
415 |
"""See Transport.is_readonly."""
|
|
416 |
return True |
|
417 |
||
1400.1.1
by Robert Collins
implement a basic test for the ui branch command from http servers |
418 |
def listable(self): |
419 |
"""See Transport.listable."""
|
|
420 |
return False |
|
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
421 |
|
422 |
def stat(self, relpath): |
|
423 |
"""Return the stat information for a file.
|
|
424 |
"""
|
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
425 |
raise errors.TransportNotPossible('http does not support stat()') |
907.1.21
by John Arbash Meinel
Adding http transport as a valid transport protocol. |
426 |
|
907.1.24
by John Arbash Meinel
Remote functionality work. |
427 |
def lock_read(self, relpath): |
428 |
"""Lock the given file for shared (read) access.
|
|
429 |
:return: A lock object, which should be passed to Transport.unlock()
|
|
430 |
"""
|
|
431 |
# The old RemoteBranch ignore lock for reading, so we will
|
|
432 |
# continue that tradition and return a bogus lock object.
|
|
433 |
class BogusLock(object): |
|
434 |
def __init__(self, path): |
|
435 |
self.path = path |
|
436 |
def unlock(self): |
|
437 |
pass
|
|
438 |
return BogusLock(relpath) |
|
439 |
||
440 |
def lock_write(self, relpath): |
|
441 |
"""Lock the given file for exclusive (write) access.
|
|
442 |
WARNING: many transports do not support this, so trying avoid using it
|
|
443 |
||
444 |
:return: A lock object, which should be passed to Transport.unlock()
|
|
445 |
"""
|
|
2004.1.25
by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :) |
446 |
raise errors.TransportNotPossible('http does not support lock_write()') |
1530.1.1
by Robert Collins
Minimal infrastructure to test TransportTestProviderAdapter. |
447 |
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
448 |
def _attempted_range_header(self, offsets, tail_amount): |
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
449 |
"""Prepare a HTTP Range header at a level the server should accept.
|
450 |
||
451 |
:return: the range header representing offsets/tail_amount or None if
|
|
452 |
no header can be built.
|
|
453 |
"""
|
|
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
454 |
|
455 |
if self._range_hint == 'multi': |
|
3024.2.3
by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug. |
456 |
# Generate the header describing all offsets
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
457 |
return self._range_header(offsets, tail_amount) |
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
458 |
elif self._range_hint == 'single': |
459 |
# Combine all the requested ranges into a single
|
|
460 |
# encompassing one
|
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
461 |
if len(offsets) > 0: |
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
462 |
if tail_amount not in (0, None): |
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
463 |
# Nothing we can do here to combine ranges with tail_amount
|
464 |
# in a single range, just returns None. The whole file
|
|
465 |
# should be downloaded.
|
|
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
466 |
return None |
467 |
else: |
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
468 |
start = offsets[0].start |
469 |
last = offsets[-1] |
|
470 |
end = last.start + last.length - 1 |
|
471 |
whole = self._coalesce_offsets([(start, end - start + 1)], |
|
472 |
limit=0, fudge_factor=0) |
|
473 |
return self._range_header(list(whole), 0) |
|
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
474 |
else: |
475 |
# Only tail_amount, requested, leave range_header
|
|
476 |
# do its work
|
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
477 |
return self._range_header(offsets, tail_amount) |
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
478 |
else: |
479 |
return None |
|
480 |
||
1786.1.27
by John Arbash Meinel
Fix up the http transports so that tests pass with the new configuration. |
481 |
@staticmethod
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
482 |
def _range_header(ranges, tail_amount): |
1750.1.2
by Michael Ellerman
Add support for HTTP multipart ranges and hook it into http+urllib. |
483 |
"""Turn a list of bytes ranges into a HTTP Range header value.
|
484 |
||
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
485 |
:param ranges: A list of _CoalescedOffset
|
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
486 |
:param tail_amount: The amount to get from the end of the file.
|
1750.1.2
by Michael Ellerman
Add support for HTTP multipart ranges and hook it into http+urllib. |
487 |
|
488 |
:return: HTTP range header string.
|
|
2004.1.30
by v.ladeuil+lp at free
Fix #62276 and #62029 by providing a more robust http range handling. |
489 |
|
490 |
At least a non-empty ranges *or* a tail_amount must be
|
|
491 |
provided.
|
|
1750.1.2
by Michael Ellerman
Add support for HTTP multipart ranges and hook it into http+urllib. |
492 |
"""
|
493 |
strings = [] |
|
2520.2.1
by Vincent Ladeuil
First step to fix #115209 use _coalesce_offsets like other transports. |
494 |
for offset in ranges: |
495 |
strings.append('%d-%d' % (offset.start, |
|
496 |
offset.start + offset.length - 1)) |
|
1750.1.2
by Michael Ellerman
Add support for HTTP multipart ranges and hook it into http+urllib. |
497 |
|
1786.1.8
by John Arbash Meinel
[merge] Johan Rydberg test updates |
498 |
if tail_amount: |
499 |
strings.append('-%d' % tail_amount) |
|
500 |
||
1786.1.36
by John Arbash Meinel
pycurl expects us to just set the range of bytes, not including bytes= |
501 |
return ','.join(strings) |
1750.1.2
by Michael Ellerman
Add support for HTTP multipart ranges and hook it into http+urllib. |
502 |
|
3878.4.5
by Vincent Ladeuil
Don't use the exception as a parameter for _redirected_to. |
503 |
def _redirected_to(self, source, target): |
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
504 |
"""Returns a transport suitable to re-issue a redirected request.
|
505 |
||
3878.4.5
by Vincent Ladeuil
Don't use the exception as a parameter for _redirected_to. |
506 |
:param source: The source url as returned by the server.
|
507 |
:param target: The target url as returned by the server.
|
|
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
508 |
|
509 |
The redirection can be handled only if the relpath involved is not
|
|
510 |
renamed by the redirection.
|
|
511 |
||
512 |
:returns: A transport or None.
|
|
513 |
"""
|
|
6145.1.2
by Jelmer Vernooij
Some refactoring. |
514 |
parsed_source = self._split_url(source) |
6145.1.3
by Jelmer Vernooij
Fix redirecting to other transports. |
515 |
parsed_target = self._split_url(target) |
6145.1.2
by Jelmer Vernooij
Some refactoring. |
516 |
pl = len(self._parsed_url.path) |
6145.1.4
by Jelmer Vernooij
Some more comments. |
517 |
# determine the excess tail - the relative path that was in
|
518 |
# the original request but not part of this transports' URL.
|
|
6145.1.3
by Jelmer Vernooij
Fix redirecting to other transports. |
519 |
excess_tail = parsed_source.path[pl:].strip("/") |
520 |
if not target.endswith(excess_tail): |
|
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
521 |
# The final part of the url has been renamed, we can't handle the
|
522 |
# redirection.
|
|
523 |
return None |
|
6145.1.3
by Jelmer Vernooij
Fix redirecting to other transports. |
524 |
|
525 |
target_path = parsed_target.path |
|
526 |
if excess_tail: |
|
527 |
# Drop the tail that was in the redirect but not part of
|
|
528 |
# the path of this transport.
|
|
529 |
target_path = target_path[:-len(excess_tail)] |
|
530 |
||
6145.1.2
by Jelmer Vernooij
Some refactoring. |
531 |
if parsed_target.scheme in ('http', 'https'): |
3878.4.7
by Vincent Ladeuil
Fixed as per Robert's review. |
532 |
# Same protocol family (i.e. http[s]), we will preserve the same
|
533 |
# http client implementation when a redirection occurs from one to
|
|
534 |
# the other (otherwise users may be surprised that bzr switches
|
|
535 |
# from one implementation to the other, and devs may suffer
|
|
536 |
# debugging it).
|
|
6145.1.2
by Jelmer Vernooij
Some refactoring. |
537 |
if (parsed_target.scheme == self._unqualified_scheme |
538 |
and parsed_target.host == self._parsed_url.host |
|
539 |
and parsed_target.port == self._parsed_url.port |
|
540 |
and (parsed_target.user is None or |
|
541 |
parsed_target.user == self._parsed_url.user)): |
|
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
542 |
# If a user is specified, it should match, we don't care about
|
543 |
# passwords, wrong passwords will be rejected anyway.
|
|
6145.1.3
by Jelmer Vernooij
Fix redirecting to other transports. |
544 |
return self.clone(target_path) |
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
545 |
else: |
3878.4.2
by Vincent Ladeuil
Fix bug #265070 by providing a finer sieve for accepted redirections. |
546 |
# Rebuild the url preserving the scheme qualification and the
|
547 |
# credentials (if they don't apply, the redirected to server
|
|
548 |
# will tell us, but if they do apply, we avoid prompting the
|
|
549 |
# user)
|
|
6145.1.2
by Jelmer Vernooij
Some refactoring. |
550 |
redir_scheme = parsed_target.scheme + '+' + self._impl_name |
3878.4.2
by Vincent Ladeuil
Fix bug #265070 by providing a finer sieve for accepted redirections. |
551 |
new_url = self._unsplit_url(redir_scheme, |
6145.1.3
by Jelmer Vernooij
Fix redirecting to other transports. |
552 |
self._parsed_url.user, |
553 |
self._parsed_url.password, |
|
554 |
parsed_target.host, parsed_target.port, |
|
555 |
target_path) |
|
556 |
return transport.get_transport_from_url(new_url) |
|
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
557 |
else: |
558 |
# Redirected to a different protocol
|
|
6145.1.3
by Jelmer Vernooij
Fix redirecting to other transports. |
559 |
new_url = self._unsplit_url(parsed_target.scheme, |
560 |
parsed_target.user, |
|
561 |
parsed_target.password, |
|
562 |
parsed_target.host, parsed_target.port, |
|
563 |
target_path) |
|
564 |
return transport.get_transport_from_url(new_url) |
|
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
565 |
|
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
566 |
|
567 |
# TODO: May be better located in smart/medium.py with the other
|
|
568 |
# SmartMedium classes
|
|
569 |
class SmartClientHTTPMedium(medium.SmartClientMedium): |
|
570 |
||
571 |
def __init__(self, http_transport): |
|
572 |
super(SmartClientHTTPMedium, self).__init__(http_transport.base) |
|
573 |
# We don't want to create a circular reference between the http
|
|
574 |
# transport and its associated medium. Since the transport will live
|
|
575 |
# longer than the medium, the medium keep only a weak reference to its
|
|
576 |
# transport.
|
|
577 |
self._http_transport_ref = weakref.ref(http_transport) |
|
578 |
||
579 |
def get_request(self): |
|
580 |
return SmartClientHTTPMediumRequest(self) |
|
2018.2.8
by Andrew Bennetts
Make HttpTransportBase.get_smart_client return self again. |
581 |
|
3245.4.47
by Andrew Bennetts
Don't automatically send 'hello' requests from RemoteBzrDirFormat.probe_transport unless we have to (i.e. the transport is HTTP). |
582 |
def should_probe(self): |
583 |
return True |
|
584 |
||
3431.3.11
by Andrew Bennetts
Push remote_path_from_transport logic into SmartClientMedium, removing special-casing of bzr+http from _SmartClient. |
585 |
def remote_path_from_transport(self, transport): |
586 |
# Strip the optional 'bzr+' prefix from transport so it will have the
|
|
587 |
# same scheme as self.
|
|
588 |
transport_base = transport.base |
|
589 |
if transport_base.startswith('bzr+'): |
|
590 |
transport_base = transport_base[4:] |
|
591 |
rel_url = urlutils.relative_url(self.base, transport_base) |
|
592 |
return urllib.unquote(rel_url) |
|
593 |
||
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
594 |
def send_http_smart_request(self, bytes): |
595 |
try: |
|
596 |
# Get back the http_transport hold by the weak reference
|
|
597 |
t = self._http_transport_ref() |
|
598 |
code, body_filelike = t._post(bytes) |
|
599 |
if code != 200: |
|
6123.2.1
by Jelmer Vernooij
Remove unused imports, fix import of error. |
600 |
raise errors.InvalidHttpResponse( |
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
601 |
t._remote_path('.bzr/smart'), |
602 |
'Expected 200 response code, got %r' % (code,)) |
|
4628.1.2
by Vincent Ladeuil
More complete fix. |
603 |
except (errors.InvalidHttpResponse, errors.ConnectionReset), e: |
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
604 |
raise errors.SmartProtocolError(str(e)) |
605 |
return body_filelike |
|
606 |
||
3958.1.1
by Andrew Bennetts
Report traffic on smart media as transport activity. |
607 |
def _report_activity(self, bytes, direction): |
608 |
"""See SmartMedium._report_activity.
|
|
609 |
||
610 |
Does nothing; the underlying plain HTTP transport will report the
|
|
611 |
activity that this medium would report.
|
|
612 |
"""
|
|
613 |
pass
|
|
614 |
||
5247.2.12
by Vincent Ladeuil
Ensure that all transports close their underlying connection. |
615 |
def disconnect(self): |
616 |
"""See SmartClientMedium.disconnect()."""
|
|
617 |
t = self._http_transport_ref() |
|
618 |
t.disconnect() |
|
619 |
||
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
620 |
|
621 |
# TODO: May be better located in smart/medium.py with the other
|
|
622 |
# SmartMediumRequest classes
|
|
2018.5.2
by Andrew Bennetts
Start splitting bzrlib/transport/smart.py into a package. |
623 |
class SmartClientHTTPMediumRequest(medium.SmartClientMediumRequest): |
2018.2.8
by Andrew Bennetts
Make HttpTransportBase.get_smart_client return self again. |
624 |
"""A SmartClientMediumRequest that works with an HTTP medium."""
|
625 |
||
2018.5.2
by Andrew Bennetts
Start splitting bzrlib/transport/smart.py into a package. |
626 |
def __init__(self, client_medium): |
627 |
medium.SmartClientMediumRequest.__init__(self, client_medium) |
|
2018.2.8
by Andrew Bennetts
Make HttpTransportBase.get_smart_client return self again. |
628 |
self._buffer = '' |
629 |
||
630 |
def _accept_bytes(self, bytes): |
|
631 |
self._buffer += bytes |
|
632 |
||
633 |
def _finished_writing(self): |
|
634 |
data = self._medium.send_http_smart_request(self._buffer) |
|
635 |
self._response_body = data |
|
636 |
||
637 |
def _read_bytes(self, count): |
|
3565.1.2
by Andrew Bennetts
Delete some more code, fix some bugs, add more comments. |
638 |
"""See SmartClientMediumRequest._read_bytes."""
|
2018.2.8
by Andrew Bennetts
Make HttpTransportBase.get_smart_client return self again. |
639 |
return self._response_body.read(count) |
2004.1.28
by v.ladeuil+lp at free
Merge bzr.dev. Including http modifications by "smart" related code |
640 |
|
3606.4.1
by Andrew Bennetts
Fix NotImplementedError when probing for smart protocol via HTTP. |
641 |
def _read_line(self): |
642 |
line, excess = medium._get_line(self._response_body.read) |
|
643 |
if excess != '': |
|
644 |
raise AssertionError( |
|
645 |
'_get_line returned excess bytes, but this mediumrequest '
|
|
646 |
'cannot handle excess. (%r)' % (excess,)) |
|
647 |
return line |
|
648 |
||
2018.2.8
by Andrew Bennetts
Make HttpTransportBase.get_smart_client return self again. |
649 |
def _finished_reading(self): |
650 |
"""See SmartClientMediumRequest._finished_reading."""
|
|
651 |
pass
|
|
4912.2.1
by Martin Pool
Add unhtml_roughly |
652 |
|
653 |
||
4912.2.4
by Martin Pool
Add test for unhtml_roughly, and truncate at 1000 bytes |
654 |
def unhtml_roughly(maybe_html, length_limit=1000): |
4912.2.1
by Martin Pool
Add unhtml_roughly |
655 |
"""Very approximate html->text translation, for presenting error bodies.
|
656 |
||
4912.2.4
by Martin Pool
Add test for unhtml_roughly, and truncate at 1000 bytes |
657 |
:param length_limit: Truncate the result to this many characters.
|
658 |
||
4912.2.1
by Martin Pool
Add unhtml_roughly |
659 |
>>> unhtml_roughly("<b>bad</b> things happened\\n")
|
660 |
' bad things happened '
|
|
661 |
"""
|
|
4912.2.4
by Martin Pool
Add test for unhtml_roughly, and truncate at 1000 bytes |
662 |
return re.subn(r"(<[^>]*>|\n| )", " ", maybe_html)[0][:length_limit] |