5752.3.8
by John Arbash Meinel
Merge bzr.dev 5764 to resolve release-notes (aka NEWS) conflicts |
1 |
# Copyright (C) 2005-2011 Canonical Ltd
|
1887.1.1
by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines, |
2 |
#
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
3 |
# This program is free software; you can redistribute it and/or modify
|
4 |
# it under the terms of the GNU General Public License as published by
|
|
5 |
# the Free Software Foundation; either version 2 of the License, or
|
|
6 |
# (at your option) any later version.
|
|
1887.1.1
by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines, |
7 |
#
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
8 |
# This program is distributed in the hope that it will be useful,
|
9 |
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
10 |
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
11 |
# GNU General Public License for more details.
|
|
1887.1.1
by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines, |
12 |
#
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
13 |
# You should have received a copy of the GNU General Public License
|
14 |
# along with this program; if not, write to the Free Software
|
|
4183.7.1
by Sabin Iacob
update FSF mailing address |
15 |
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
16 |
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
17 |
"""Transport is an abstraction layer to handle file access.
|
18 |
||
19 |
The abstraction is to allow access from the local filesystem, as well
|
|
20 |
as remote (such as http or sftp).
|
|
1540.3.12
by Martin Pool
Multiple transports can be registered for any protocol, and they are |
21 |
|
22 |
Transports are constructed from a string, being a URL or (as a degenerate
|
|
23 |
case) a local filesystem path. This is typically the top directory of
|
|
24 |
a bzrdir, repository, or similar object we are interested in working with.
|
|
25 |
The Transport returned has methods to read, write and manipulate files within
|
|
26 |
it.
|
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
27 |
"""
|
28 |
||
6379.6.1
by Jelmer Vernooij
Import absolute_import in a few places. |
29 |
from __future__ import absolute_import |
30 |
||
1996.3.6
by John Arbash Meinel
Find a few places that weren't importing their dependencies. |
31 |
from cStringIO import StringIO |
32 |
import sys |
|
33 |
||
34 |
from bzrlib.lazy_import import lazy_import |
|
35 |
lazy_import(globals(), """ |
|
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
36 |
import errno
|
1773.4.1
by Martin Pool
Add pyflakes makefile target; fix many warnings |
37 |
from stat import S_ISDIR
|
1636.1.1
by Robert Collins
Fix calling relpath() and abspath() on transports at their root. |
38 |
import urlparse
|
1530.1.1
by Robert Collins
Minimal infrastructure to test TransportTestProviderAdapter. |
39 |
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
40 |
from bzrlib import (
|
41 |
errors,
|
|
42 |
osutils,
|
|
43 |
symbol_versioning,
|
|
3882.7.5
by Martin Pool
Further mockup of transport-based activity indicator. |
44 |
ui,
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
45 |
urlutils,
|
46 |
)
|
|
1996.3.6
by John Arbash Meinel
Find a few places that weren't importing their dependencies. |
47 |
""") |
48 |
||
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
49 |
from bzrlib.symbol_versioning import ( |
5753.2.2
by Jelmer Vernooij
Remove some unnecessary imports, clean up lazy imports. |
50 |
DEPRECATED_PARAMETER, |
51 |
)
|
|
5436.3.3
by Martin
Minor changes from review and revert fiddling with mutter |
52 |
from bzrlib.trace import ( |
53 |
mutter, |
|
54 |
)
|
|
5436.3.1
by Martin
Create new post_connect hook for transports |
55 |
from bzrlib import ( |
56 |
hooks, |
|
57 |
registry, |
|
2164.2.21
by Vincent Ladeuil
Take bundles into account. |
58 |
)
|
2018.5.104
by Andrew Bennetts
Completely rework chrooted transports. |
59 |
|
60 |
||
2671.3.2
by Robert Collins
Start open_file_stream logic. |
61 |
# a dictionary of open file streams. Keys are absolute paths, values are
|
62 |
# transport defined.
|
|
63 |
_file_streams = {} |
|
64 |
||
65 |
||
1530.1.11
by Robert Collins
Push the transport permutations list into each transport module allowing for automatic testing of new modules that are registered as transports. |
66 |
def _get_protocol_handlers(): |
1540.3.12
by Martin Pool
Multiple transports can be registered for any protocol, and they are |
67 |
"""Return a dictionary of {urlprefix: [factory]}"""
|
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
68 |
return transport_list_registry |
1530.1.11
by Robert Collins
Push the transport permutations list into each transport module allowing for automatic testing of new modules that are registered as transports. |
69 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
70 |
|
1530.1.11
by Robert Collins
Push the transport permutations list into each transport module allowing for automatic testing of new modules that are registered as transports. |
71 |
def _set_protocol_handlers(new_handlers): |
72 |
"""Replace the current protocol handlers dictionary.
|
|
73 |
||
74 |
WARNING this will remove all build in protocols. Use with care.
|
|
75 |
"""
|
|
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
76 |
global transport_list_registry |
77 |
transport_list_registry = new_handlers |
|
1530.1.11
by Robert Collins
Push the transport permutations list into each transport module allowing for automatic testing of new modules that are registered as transports. |
78 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
79 |
|
1540.3.12
by Martin Pool
Multiple transports can be registered for any protocol, and they are |
80 |
def _clear_protocol_handlers(): |
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
81 |
global transport_list_registry |
82 |
transport_list_registry = TransportListRegistry() |
|
1540.3.12
by Martin Pool
Multiple transports can be registered for any protocol, and they are |
83 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
84 |
|
1530.1.11
by Robert Collins
Push the transport permutations list into each transport module allowing for automatic testing of new modules that are registered as transports. |
85 |
def _get_transport_modules(): |
86 |
"""Return a list of the modules providing transports."""
|
|
87 |
modules = set() |
|
3882.1.1
by Martin Pool
Don't call iteritems on transport_list_registry, because it may change during iteration |
88 |
for prefix, factory_list in transport_list_registry.items(): |
1540.3.12
by Martin Pool
Multiple transports can be registered for any protocol, and they are |
89 |
for factory in factory_list: |
5676.1.6
by Jelmer Vernooij
Add _ObjGetter.get_module. |
90 |
modules.add(factory.get_module()) |
4634.44.1
by Andrew Bennetts
First draft of a generic path-filtering transport decorator. |
91 |
# Add chroot and pathfilter directly, because there is no handler
|
92 |
# registered for it.
|
|
2018.5.104
by Andrew Bennetts
Completely rework chrooted transports. |
93 |
modules.add('bzrlib.transport.chroot') |
4634.44.1
by Andrew Bennetts
First draft of a generic path-filtering transport decorator. |
94 |
modules.add('bzrlib.transport.pathfilter') |
1530.1.19
by Robert Collins
Make transport test adapter tests reliable. |
95 |
result = list(modules) |
96 |
result.sort() |
|
97 |
return result |
|
1530.1.11
by Robert Collins
Push the transport permutations list into each transport module allowing for automatic testing of new modules that are registered as transports. |
98 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
99 |
|
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
100 |
class TransportListRegistry(registry.Registry): |
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
101 |
"""A registry which simplifies tracking available Transports.
|
102 |
||
3764.1.1
by Vincent Ladeuil
Fix typo |
103 |
A registration of a new protocol requires two steps:
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
104 |
1) register the prefix with the function register_transport( )
|
105 |
2) register the protocol provider with the function
|
|
106 |
register_transport_provider( ) ( and the "lazy" variant )
|
|
107 |
||
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
108 |
This is needed because:
|
4764.1.1
by Vincent Ladeuil
Fix typos. |
109 |
a) a single provider can support multiple protocols ( like the ftp
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
110 |
provider which supports both the ftp:// and the aftp:// protocols )
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
111 |
b) a single protocol can have multiple providers ( like the http://
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
112 |
protocol which is supported by both the urllib and pycurl provider )
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
113 |
"""
|
114 |
||
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
115 |
def register_transport_provider(self, key, obj): |
116 |
self.get(key).insert(0, registry._ObjectGetter(obj)) |
|
117 |
||
118 |
def register_lazy_transport_provider(self, key, module_name, member_name): |
|
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
119 |
self.get(key).insert(0, |
2241.2.1
by ghigo
Add the TransportRegistry class |
120 |
registry._LazyObjectGetter(module_name, member_name)) |
121 |
||
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
122 |
def register_transport(self, key, help=None): |
123 |
self.register(key, [], help) |
|
2241.2.1
by ghigo
Add the TransportRegistry class |
124 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
125 |
|
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
126 |
transport_list_registry = TransportListRegistry() |
2241.2.1
by ghigo
Add the TransportRegistry class |
127 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
128 |
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
129 |
def register_transport_proto(prefix, help=None, info=None, |
2919.2.1
by John Arbash Meinel
Register netloc protocols as soon as the protocol is registered. |
130 |
register_netloc=False): |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
131 |
transport_list_registry.register_transport(prefix, help) |
2919.2.1
by John Arbash Meinel
Register netloc protocols as soon as the protocol is registered. |
132 |
if register_netloc: |
3376.2.4
by Martin Pool
Remove every assert statement from bzrlib! |
133 |
if not prefix.endswith('://'): |
134 |
raise ValueError(prefix) |
|
2919.2.1
by John Arbash Meinel
Register netloc protocols as soon as the protocol is registered. |
135 |
register_urlparse_netloc_protocol(prefix[:-3]) |
2241.2.1
by ghigo
Add the TransportRegistry class |
136 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
137 |
|
2241.2.1
by ghigo
Add the TransportRegistry class |
138 |
def register_lazy_transport(prefix, module, classname): |
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
139 |
if not prefix in transport_list_registry: |
2241.2.2
by ghigo
Create the TransportList class |
140 |
register_transport_proto(prefix) |
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
141 |
transport_list_registry.register_lazy_transport_provider(prefix, module, classname) |
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
142 |
|
143 |
||
2241.2.1
by ghigo
Add the TransportRegistry class |
144 |
def register_transport(prefix, klass, override=DEPRECATED_PARAMETER): |
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
145 |
if not prefix in transport_list_registry: |
2241.2.2
by ghigo
Create the TransportList class |
146 |
register_transport_proto(prefix) |
2241.2.3
by ghigo
removed the the registration of the 'None' transport, instead we use the set_default_transport function |
147 |
transport_list_registry.register_transport_provider(prefix, klass) |
2241.2.1
by ghigo
Add the TransportRegistry class |
148 |
|
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
149 |
|
1636.1.2
by Robert Collins
More review fixen to the relpath at '/' fixes. |
150 |
def register_urlparse_netloc_protocol(protocol): |
1636.1.1
by Robert Collins
Fix calling relpath() and abspath() on transports at their root. |
151 |
"""Ensure that protocol is setup to be used with urlparse netloc parsing."""
|
152 |
if protocol not in urlparse.uses_netloc: |
|
153 |
urlparse.uses_netloc.append(protocol) |
|
154 |
||
2241.3.4
by ghigo
Updates on the basis of John Arbash Meinel comments |
155 |
|
2485.8.61
by Vincent Ladeuil
From review comments, use a private scheme for testing. |
156 |
def _unregister_urlparse_netloc_protocol(protocol): |
157 |
"""Remove protocol from urlparse netloc parsing.
|
|
158 |
||
159 |
Except for tests, you should never use that function. Using it with 'http',
|
|
160 |
for example, will break all http transports.
|
|
161 |
"""
|
|
162 |
if protocol in urlparse.uses_netloc: |
|
163 |
urlparse.uses_netloc.remove(protocol) |
|
164 |
||
165 |
||
2241.3.5
by ghigo
update to the latest bzr.dev |
166 |
def unregister_transport(scheme, factory): |
167 |
"""Unregister a transport."""
|
|
168 |
l = transport_list_registry.get(scheme) |
|
169 |
for i in l: |
|
170 |
o = i.get_obj( ) |
|
171 |
if o == factory: |
|
172 |
transport_list_registry.get(scheme).remove(i) |
|
173 |
break
|
|
174 |
if len(l) == 0: |
|
175 |
transport_list_registry.remove(scheme) |
|
176 |
||
177 |
||
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
178 |
class _CoalescedOffset(object): |
179 |
"""A data container for keeping track of coalesced offsets."""
|
|
180 |
||
181 |
__slots__ = ['start', 'length', 'ranges'] |
|
182 |
||
183 |
def __init__(self, start, length, ranges): |
|
184 |
self.start = start |
|
185 |
self.length = length |
|
186 |
self.ranges = ranges |
|
187 |
||
188 |
def __cmp__(self, other): |
|
189 |
return cmp((self.start, self.length, self.ranges), |
|
190 |
(other.start, other.length, other.ranges)) |
|
191 |
||
3059.2.2
by Vincent Ladeuil
Read http responses on demand without buffering the whole body |
192 |
def __repr__(self): |
193 |
return '%s(%r, %r, %r)' % (self.__class__.__name__, |
|
194 |
self.start, self.length, self.ranges) |
|
195 |
||
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
196 |
|
2052.6.1
by Robert Collins
``Transport.get`` has had its interface made more clear for ease of use. |
197 |
class LateReadError(object): |
198 |
"""A helper for transports which pretends to be a readable file.
|
|
199 |
||
200 |
When read() is called, errors.ReadError is raised.
|
|
201 |
"""
|
|
202 |
||
203 |
def __init__(self, path): |
|
204 |
self._path = path |
|
205 |
||
206 |
def close(self): |
|
207 |
"""a no-op - do nothing."""
|
|
208 |
||
2052.6.2
by Robert Collins
Merge bzr.dev. |
209 |
def _fail(self): |
2052.6.1
by Robert Collins
``Transport.get`` has had its interface made more clear for ease of use. |
210 |
"""Raise ReadError."""
|
211 |
raise errors.ReadError(self._path) |
|
212 |
||
2052.6.2
by Robert Collins
Merge bzr.dev. |
213 |
def __iter__(self): |
214 |
self._fail() |
|
215 |
||
216 |
def read(self, count=-1): |
|
217 |
self._fail() |
|
218 |
||
219 |
def readlines(self): |
|
220 |
self._fail() |
|
221 |
||
2052.6.1
by Robert Collins
``Transport.get`` has had its interface made more clear for ease of use. |
222 |
|
2671.3.6
by Robert Collins
Review feedback. |
223 |
class FileStream(object): |
224 |
"""Base class for FileStreams."""
|
|
225 |
||
226 |
def __init__(self, transport, relpath): |
|
227 |
"""Create a FileStream for relpath on transport."""
|
|
228 |
self.transport = transport |
|
229 |
self.relpath = relpath |
|
230 |
||
231 |
def _close(self): |
|
232 |
"""A hook point for subclasses that need to take action on close."""
|
|
233 |
||
6006.4.5
by Martin Pool
Flush pack, index, and dirstate files to disk on closing |
234 |
def close(self, want_fdatasync=False): |
235 |
if want_fdatasync: |
|
236 |
try: |
|
237 |
self.fdatasync() |
|
238 |
except errors.TransportNotPossible: |
|
239 |
pass
|
|
2671.3.6
by Robert Collins
Review feedback. |
240 |
self._close() |
241 |
del _file_streams[self.transport.abspath(self.relpath)] |
|
242 |
||
6006.4.3
by Martin Pool
Add FileStream.fdatasync |
243 |
def fdatasync(self): |
244 |
"""Force data out to physical disk if possible.
|
|
245 |
||
246 |
:raises TransportNotPossible: If this transport has no way to
|
|
247 |
flush to disk.
|
|
248 |
"""
|
|
6006.4.5
by Martin Pool
Flush pack, index, and dirstate files to disk on closing |
249 |
raise errors.TransportNotPossible( |
6006.4.3
by Martin Pool
Add FileStream.fdatasync |
250 |
"%s cannot fdatasync" % (self.transport,)) |
251 |
||
2671.3.6
by Robert Collins
Review feedback. |
252 |
|
253 |
class FileFileStream(FileStream): |
|
2671.3.9
by Robert Collins
Review feedback and fix VFat emulated transports to not claim to have unix permissions. |
254 |
"""A file stream object returned by open_write_stream.
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
255 |
|
2671.3.6
by Robert Collins
Review feedback. |
256 |
This version uses a file like object to perform writes.
|
257 |
"""
|
|
258 |
||
259 |
def __init__(self, transport, relpath, file_handle): |
|
260 |
FileStream.__init__(self, transport, relpath) |
|
261 |
self.file_handle = file_handle |
|
262 |
||
263 |
def _close(self): |
|
264 |
self.file_handle.close() |
|
265 |
||
6006.4.3
by Martin Pool
Add FileStream.fdatasync |
266 |
def fdatasync(self): |
267 |
"""Force data out to physical disk if possible."""
|
|
268 |
self.file_handle.flush() |
|
6006.4.5
by Martin Pool
Flush pack, index, and dirstate files to disk on closing |
269 |
try: |
270 |
fileno = self.file_handle.fileno() |
|
271 |
except AttributeError: |
|
272 |
raise errors.TransportNotPossible() |
|
6006.4.8
by Martin Pool
Use fsync when fdatasync is not available |
273 |
osutils.fdatasync(fileno) |
6006.4.3
by Martin Pool
Add FileStream.fdatasync |
274 |
|
2671.3.6
by Robert Collins
Review feedback. |
275 |
def write(self, bytes): |
3635.1.2
by Robert Collins
Add osutils.pump_string_file helper function. |
276 |
osutils.pump_string_file(bytes, self.file_handle) |
2671.3.6
by Robert Collins
Review feedback. |
277 |
|
278 |
||
279 |
class AppendBasedFileStream(FileStream): |
|
2671.3.9
by Robert Collins
Review feedback and fix VFat emulated transports to not claim to have unix permissions. |
280 |
"""A file stream object returned by open_write_stream.
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
281 |
|
2671.3.6
by Robert Collins
Review feedback. |
282 |
This version uses append on a transport to perform writes.
|
283 |
"""
|
|
284 |
||
285 |
def write(self, bytes): |
|
286 |
self.transport.append_bytes(self.relpath, bytes) |
|
287 |
||
288 |
||
5436.3.1
by Martin
Create new post_connect hook for transports |
289 |
class TransportHooks(hooks.Hooks): |
290 |
"""Mapping of hook names to registered callbacks for transport hooks"""
|
|
291 |
def __init__(self): |
|
292 |
super(TransportHooks, self).__init__() |
|
5436.3.7
by Jelmer Vernooij
Merge bzr.dev, update to use new hooks. |
293 |
self.add_hook("post_connect", |
5436.3.1
by Martin
Create new post_connect hook for transports |
294 |
"Called after a new connection is established or a reconnect "
|
5436.3.10
by Martin Packman
Accept and document passing the medium rather than transport for smart connections |
295 |
"occurs. The sole argument passed is either the connected "
|
296 |
"transport or smart medium instance.", (2, 5)) |
|
5436.3.1
by Martin
Create new post_connect hook for transports |
297 |
|
298 |
||
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
299 |
class Transport(object): |
300 |
"""This class encapsulates methods for retrieving or putting a file
|
|
301 |
from/to a storage location.
|
|
302 |
||
303 |
Most functions have a _multi variant, which allows you to queue up
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
304 |
multiple requests. They generally have a dumb base implementation
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
305 |
which just iterates over the arguments, but smart Transport
|
306 |
implementations can do pipelining.
|
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
307 |
In general implementations should support having a generator or a list
|
308 |
as an argument (ie always iterate, never index)
|
|
1910.7.17
by Andrew Bennetts
Various cosmetic changes. |
309 |
|
310 |
:ivar base: Base URL for the transport; should always end in a slash.
|
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
311 |
"""
|
312 |
||
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
313 |
# implementations can override this if it is more efficient
|
314 |
# for them to combine larger read chunks together
|
|
1864.5.4
by John Arbash Meinel
play around with tuning the partial reads. |
315 |
_max_readv_combine = 50 |
1864.5.3
by John Arbash Meinel
Allow collapsing ranges even if they are just 'close' |
316 |
# It is better to read this much more data in order, rather
|
317 |
# than doing another seek. Even for the local filesystem,
|
|
318 |
# there is a benefit in just reading.
|
|
319 |
# TODO: jam 20060714 Do some real benchmarking to figure out
|
|
320 |
# where the biggest benefit between combining reads and
|
|
1864.5.8
by John Arbash Meinel
Cleanup and NEWS |
321 |
# and seeking is. Consider a runtime auto-tune.
|
1864.5.4
by John Arbash Meinel
play around with tuning the partial reads. |
322 |
_bytes_to_read_before_seek = 0 |
5436.3.1
by Martin
Create new post_connect hook for transports |
323 |
|
324 |
hooks = TransportHooks() |
|
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
325 |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
326 |
def __init__(self, base): |
3734.2.3
by Vincent Ladeuil
Don't use multiple inheritance for http smart medium since we |
327 |
super(Transport, self).__init__() |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
328 |
self.base = base |
6240.2.3
by Jelmer Vernooij
Actually modify transport rather than just URLs. |
329 |
(self._raw_base, self._segment_parameters) = ( |
330 |
urlutils.split_segment_parameters(base)) |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
331 |
|
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
332 |
def _translate_error(self, e, path, raise_generic=True): |
333 |
"""Translate an IOError or OSError into an appropriate bzr error.
|
|
334 |
||
335 |
This handles things like ENOENT, ENOTDIR, EEXIST, and EACCESS
|
|
336 |
"""
|
|
1963.2.6
by Robey Pointer
pychecker is on crack; go back to using 'is None'. |
337 |
if getattr(e, 'errno', None) is not None: |
6015.3.7
by Martin Pool
log a message when we get a EINVAL on a transport operation |
338 |
if e.errno in (errno.ENOENT, errno.ENOTDIR): |
339 |
raise errors.NoSuchFile(path, extra=e) |
|
340 |
elif e.errno == errno.EINVAL: |
|
6015.26.1
by IWATA Hidetaka
Avoid UnicodeError when reporting EINVAL error |
341 |
mutter("EINVAL returned on path %s: %r" % (path, e)) |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
342 |
raise errors.NoSuchFile(path, extra=e) |
1185.31.58
by John Arbash Meinel
Updating for new transport tests so that they pass on win32 |
343 |
# I would rather use errno.EFOO, but there doesn't seem to be
|
344 |
# any matching for 267
|
|
345 |
# This is the error when doing a listdir on a file:
|
|
346 |
# WindowsError: [Errno 267] The directory name is invalid
|
|
347 |
if sys.platform == 'win32' and e.errno in (errno.ESRCH, 267): |
|
348 |
raise errors.NoSuchFile(path, extra=e) |
|
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
349 |
if e.errno == errno.EEXIST: |
350 |
raise errors.FileExists(path, extra=e) |
|
351 |
if e.errno == errno.EACCES: |
|
352 |
raise errors.PermissionDenied(path, extra=e) |
|
1553.5.10
by Martin Pool
New DirectoryNotEmpty exception, and raise this from local and memory |
353 |
if e.errno == errno.ENOTEMPTY: |
354 |
raise errors.DirectoryNotEmpty(path, extra=e) |
|
1558.10.1
by Aaron Bentley
Handle lockdirs over NFS properly |
355 |
if e.errno == errno.EBUSY: |
356 |
raise errors.ResourceBusy(path, extra=e) |
|
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
357 |
if raise_generic: |
358 |
raise errors.TransportError(orig_error=e) |
|
359 |
||
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
360 |
def clone(self, offset=None): |
361 |
"""Return a new Transport object, cloned from the current location,
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
362 |
using a subdirectory or parent directory. This allows connections
|
1185.11.6
by John Arbash Meinel
Made HttpTransport handle a request for a parent directory differently. |
363 |
to be pooled, rather than a new one needed for each subdir.
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
364 |
"""
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
365 |
raise NotImplementedError(self.clone) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
366 |
|
4294.2.1
by Robert Collins
Move directory checking for bzr push options into Branch.create_clone_on_transport. |
367 |
def create_prefix(self): |
4294.2.10
by Robert Collins
Review feedback. |
368 |
"""Create all the directories leading down to self.base."""
|
4294.2.1
by Robert Collins
Move directory checking for bzr push options into Branch.create_clone_on_transport. |
369 |
cur_transport = self |
370 |
needed = [cur_transport] |
|
371 |
# Recurse upwards until we can create a directory successfully
|
|
372 |
while True: |
|
373 |
new_transport = cur_transport.clone('..') |
|
374 |
if new_transport.base == cur_transport.base: |
|
375 |
raise errors.BzrCommandError( |
|
376 |
"Failed to create path prefix for %s." |
|
377 |
% cur_transport.base) |
|
378 |
try: |
|
379 |
new_transport.mkdir('.') |
|
380 |
except errors.NoSuchFile: |
|
381 |
needed.append(new_transport) |
|
382 |
cur_transport = new_transport |
|
383 |
except errors.FileExists: |
|
384 |
break
|
|
385 |
else: |
|
386 |
break
|
|
387 |
# Now we only need to create child directories
|
|
388 |
while needed: |
|
389 |
cur_transport = needed.pop() |
|
390 |
cur_transport.ensure_base() |
|
391 |
||
2475.3.2
by John Arbash Meinel
Add Transport.ensure_base() |
392 |
def ensure_base(self): |
393 |
"""Ensure that the directory this transport references exists.
|
|
394 |
||
395 |
This will create a directory if it doesn't exist.
|
|
396 |
:return: True if the directory was created, False otherwise.
|
|
397 |
"""
|
|
398 |
# The default implementation just uses "Easier to ask for forgiveness
|
|
399 |
# than permission". We attempt to create the directory, and just
|
|
2805.3.1
by Martin Pool
Transport.ensure_base should ignore PermissionDenied when trying to create the directory (Axel Kollmorgen) |
400 |
# suppress FileExists and PermissionDenied (for Windows) exceptions.
|
2475.3.2
by John Arbash Meinel
Add Transport.ensure_base() |
401 |
try: |
402 |
self.mkdir('.') |
|
2805.3.1
by Martin Pool
Transport.ensure_base should ignore PermissionDenied when trying to create the directory (Axel Kollmorgen) |
403 |
except (errors.FileExists, errors.PermissionDenied): |
2475.3.2
by John Arbash Meinel
Add Transport.ensure_base() |
404 |
return False |
405 |
else: |
|
406 |
return True |
|
407 |
||
2634.1.1
by Robert Collins
(robertc) Reinstate the accidentally backed out external_url patch. |
408 |
def external_url(self): |
409 |
"""Return a URL for self that can be given to an external process.
|
|
410 |
||
411 |
There is no guarantee that the URL can be accessed from a different
|
|
412 |
machine - e.g. file:/// urls are only usable on the local machine,
|
|
413 |
sftp:/// urls when the server is only bound to localhost are only
|
|
414 |
usable from localhost etc.
|
|
415 |
||
416 |
NOTE: This method may remove security wrappers (e.g. on chroot
|
|
417 |
transports) and thus should *only* be used when the result will not
|
|
418 |
be used to obtain a new transport within bzrlib. Ideally chroot
|
|
419 |
transports would know enough to cause the external url to be the exact
|
|
420 |
one used that caused the chrooting in the first place, but that is not
|
|
421 |
currently the case.
|
|
422 |
||
423 |
:return: A URL that can be given to another process.
|
|
424 |
:raises InProcessTransport: If the transport is one that cannot be
|
|
425 |
accessed out of the current process (e.g. a MemoryTransport)
|
|
426 |
then InProcessTransport is raised.
|
|
427 |
"""
|
|
428 |
raise NotImplementedError(self.external_url) |
|
429 |
||
5268.7.5
by Jelmer Vernooij
Use accessor for segment parameters. |
430 |
def get_segment_parameters(self): |
431 |
"""Return the segment parameters for the top segment of the URL.
|
|
432 |
"""
|
|
433 |
return self._segment_parameters |
|
434 |
||
6240.2.3
by Jelmer Vernooij
Actually modify transport rather than just URLs. |
435 |
def set_segment_parameter(self, name, value): |
436 |
"""Set a segment parameter.
|
|
437 |
||
6240.2.11
by Jelmer Vernooij
Clarify that segment parameter names/values should be urlencoded. |
438 |
:param name: Segment parameter name (urlencoded string)
|
439 |
:param value: Segment parameter value (urlencoded string)
|
|
6240.2.3
by Jelmer Vernooij
Actually modify transport rather than just URLs. |
440 |
"""
|
6240.2.4
by Jelmer Vernooij
Allow removing parameters. |
441 |
if value is None: |
6240.2.5
by Jelmer Vernooij
Add Transport.set_segment_parameter. |
442 |
try: |
443 |
del self._segment_parameters[name] |
|
444 |
except KeyError: |
|
445 |
pass
|
|
6240.2.4
by Jelmer Vernooij
Allow removing parameters. |
446 |
else: |
447 |
self._segment_parameters[name] = value |
|
6240.2.3
by Jelmer Vernooij
Actually modify transport rather than just URLs. |
448 |
self.base = urlutils.join_segment_parameters( |
449 |
self._raw_base, self._segment_parameters) |
|
450 |
||
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
451 |
def _pump(self, from_file, to_file): |
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
452 |
"""Most children will need to copy from one file-like
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
453 |
object or string to another one.
|
454 |
This just gives them something easy to call.
|
|
455 |
"""
|
|
2745.5.2
by Robert Collins
* ``bzrlib.transport.Transport.put_file`` now returns the number of bytes |
456 |
return osutils.pumpfile(from_file, to_file) |
1948.3.8
by Vincent LADEUIL
_pump accepts strings finally :) |
457 |
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
458 |
def _get_total(self, multi): |
459 |
"""Try to figure out how many entries are in multi,
|
|
460 |
but if not possible, return None.
|
|
461 |
"""
|
|
462 |
try: |
|
463 |
return len(multi) |
|
464 |
except TypeError: # We can't tell how many, because relpaths is a generator |
|
465 |
return None |
|
466 |
||
3882.7.1
by Martin Pool
Add stub _report_activity method as a transport callback |
467 |
def _report_activity(self, bytes, direction): |
468 |
"""Notify that this transport has activity.
|
|
469 |
||
470 |
Implementations should call this from all methods that actually do IO.
|
|
471 |
Be careful that it's not called twice, if one method is implemented on
|
|
472 |
top of another.
|
|
473 |
||
474 |
:param bytes: Number of bytes read or written.
|
|
475 |
:param direction: 'read' or 'write' or None.
|
|
476 |
"""
|
|
3882.7.5
by Martin Pool
Further mockup of transport-based activity indicator. |
477 |
ui.ui_factory.report_transport_activity(self, bytes, direction) |
3882.7.1
by Martin Pool
Add stub _report_activity method as a transport callback |
478 |
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
479 |
def _update_pb(self, pb, msg, count, total): |
480 |
"""Update the progress bar based on the current count
|
|
481 |
and total available, total may be None if it was
|
|
482 |
not possible to determine.
|
|
483 |
"""
|
|
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
484 |
if pb is None: |
485 |
return
|
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
486 |
if total is None: |
487 |
pb.update(msg, count, count+1) |
|
488 |
else: |
|
489 |
pb.update(msg, count, total) |
|
490 |
||
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
491 |
def _iterate_over(self, multi, func, pb, msg, expand=True): |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
492 |
"""Iterate over all entries in multi, passing them to func,
|
493 |
and update the progress bar as you go along.
|
|
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
494 |
|
495 |
:param expand: If True, the entries will be passed to the function
|
|
496 |
by expanding the tuple. If False, it will be passed
|
|
497 |
as a single parameter.
|
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
498 |
"""
|
499 |
total = self._get_total(multi) |
|
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
500 |
result = [] |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
501 |
count = 0 |
502 |
for entry in multi: |
|
503 |
self._update_pb(pb, msg, count, total) |
|
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
504 |
if expand: |
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
505 |
result.append(func(*entry)) |
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
506 |
else: |
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
507 |
result.append(func(entry)) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
508 |
count += 1 |
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
509 |
return tuple(result) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
510 |
|
907.1.34
by John Arbash Meinel
Fixing append(), cleaning up function locations. |
511 |
def abspath(self, relpath): |
512 |
"""Return the full url to the given relative path.
|
|
1910.16.6
by Andrew Bennetts
Update Transport.abspath docstring. |
513 |
|
514 |
:param relpath: a string of a relative path
|
|
1910.16.2
by Andrew Bennetts
Reduce transport code duplication by creating a '_combine_paths' method to Transport. |
515 |
"""
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
516 |
|
1910.16.2
by Andrew Bennetts
Reduce transport code duplication by creating a '_combine_paths' method to Transport. |
517 |
# XXX: Robert Collins 20051016 - is this really needed in the public
|
518 |
# interface ?
|
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
519 |
raise NotImplementedError(self.abspath) |
907.1.34
by John Arbash Meinel
Fixing append(), cleaning up function locations. |
520 |
|
2671.3.1
by Robert Collins
* New method ``bzrlib.transport.Transport.get_recommended_page_size``. |
521 |
def recommended_page_size(self): |
522 |
"""Return the recommended page size for this transport.
|
|
523 |
||
524 |
This is potentially different for every path in a given namespace.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
525 |
For example, local transports might use an operating system call to
|
2671.3.1
by Robert Collins
* New method ``bzrlib.transport.Transport.get_recommended_page_size``. |
526 |
get the block size for a given path, which can vary due to mount
|
527 |
points.
|
|
528 |
||
529 |
:return: The page size in bytes.
|
|
530 |
"""
|
|
531 |
return 4 * 1024 |
|
532 |
||
907.1.34
by John Arbash Meinel
Fixing append(), cleaning up function locations. |
533 |
def relpath(self, abspath): |
534 |
"""Return the local path portion from a given absolute path.
|
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
535 |
|
536 |
This default implementation is not suitable for filesystems with
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
537 |
aliasing, such as that given by symlinks, where a path may not
|
538 |
start with our base, but still be a relpath once aliasing is
|
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
539 |
resolved.
|
907.1.34
by John Arbash Meinel
Fixing append(), cleaning up function locations. |
540 |
"""
|
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
541 |
# TODO: This might want to use bzrlib.osutils.relpath
|
542 |
# but we have to watch out because of the prefix issues
|
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
543 |
if not (abspath == self.base[:-1] or abspath.startswith(self.base)): |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
544 |
raise errors.PathNotChild(abspath, self.base) |
1442.1.44
by Robert Collins
Many transport related tweaks: |
545 |
pl = len(self.base) |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
546 |
return abspath[pl:].strip('/') |
907.1.34
by John Arbash Meinel
Fixing append(), cleaning up function locations. |
547 |
|
1685.1.9
by John Arbash Meinel
Updated LocalTransport so that it's base is now a URL rather than a local path. This helps consistency with all other functions. To do so, I added local_abspath() which returns the local path, and local_path_to/from_url |
548 |
def local_abspath(self, relpath): |
549 |
"""Return the absolute path on the local filesystem.
|
|
550 |
||
551 |
This function will only be defined for Transports which have a
|
|
552 |
physical local filesystem representation.
|
|
5200.2.1
by Robert Collins
Minor local_abspath docstring improvement. |
553 |
|
554 |
:raises errors.NotLocalUrl: When no local path representation is
|
|
555 |
available.
|
|
1685.1.9
by John Arbash Meinel
Updated LocalTransport so that it's base is now a URL rather than a local path. This helps consistency with all other functions. To do so, I added local_abspath() which returns the local path, and local_path_to/from_url |
556 |
"""
|
2018.18.4
by Martin Pool
Change Transport.local_abspath to raise NotLocalUrl, and test. |
557 |
raise errors.NotLocalUrl(self.abspath(relpath)) |
1685.1.9
by John Arbash Meinel
Updated LocalTransport so that it's base is now a URL rather than a local path. This helps consistency with all other functions. To do so, I added local_abspath() which returns the local path, and local_path_to/from_url |
558 |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
559 |
def has(self, relpath): |
1442.1.44
by Robert Collins
Many transport related tweaks: |
560 |
"""Does the file relpath exist?
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
561 |
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
562 |
Note that some transports MAY allow querying on directories, but this
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
563 |
is not part of the protocol. In other words, the results of
|
1786.1.8
by John Arbash Meinel
[merge] Johan Rydberg test updates |
564 |
t.has("a_directory_name") are undefined.
|
1910.7.17
by Andrew Bennetts
Various cosmetic changes. |
565 |
|
566 |
:rtype: bool
|
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
567 |
"""
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
568 |
raise NotImplementedError(self.has) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
569 |
|
907.1.36
by John Arbash Meinel
Moving the multi-get functionality higher up into the Branch class. |
570 |
def has_multi(self, relpaths, pb=None): |
907.1.34
by John Arbash Meinel
Fixing append(), cleaning up function locations. |
571 |
"""Return True/False for each entry in relpaths"""
|
572 |
total = self._get_total(relpaths) |
|
573 |
count = 0 |
|
574 |
for relpath in relpaths: |
|
907.1.36
by John Arbash Meinel
Moving the multi-get functionality higher up into the Branch class. |
575 |
self._update_pb(pb, 'has', count, total) |
907.1.34
by John Arbash Meinel
Fixing append(), cleaning up function locations. |
576 |
yield self.has(relpath) |
577 |
count += 1 |
|
578 |
||
1185.16.155
by John Arbash Meinel
Added a has_any function to the Transport API |
579 |
def has_any(self, relpaths): |
580 |
"""Return True if any of the paths exist."""
|
|
581 |
for relpath in relpaths: |
|
582 |
if self.has(relpath): |
|
583 |
return True |
|
584 |
return False |
|
585 |
||
1442.1.44
by Robert Collins
Many transport related tweaks: |
586 |
def iter_files_recursive(self): |
587 |
"""Iter the relative paths of files in the transports sub-tree.
|
|
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
588 |
|
589 |
*NOTE*: This only lists *files*, not subdirectories!
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
590 |
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
591 |
As with other listing functions, only some transports implement this,.
|
3484.1.1
by Vincent Ladeuil
Trivial drive-by cleanups. |
592 |
you may check via listable() to determine if it will.
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
593 |
"""
|
1530.1.4
by Robert Collins
integrate Memory tests into transport interface tests. |
594 |
raise errors.TransportNotPossible("This transport has not " |
1530.1.21
by Robert Collins
Review feedback fixes. |
595 |
"implemented iter_files_recursive "
|
1530.1.4
by Robert Collins
integrate Memory tests into transport interface tests. |
596 |
"(but must claim to be listable "
|
597 |
"to trigger this error).") |
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
598 |
|
2164.2.15
by Vincent Ladeuil
Http redirections are not followed by default. Do not use hints |
599 |
def get(self, relpath): |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
600 |
"""Get the file at the given relative path.
|
907.1.20
by John Arbash Meinel
Removed Transport.open(), making get + put encode/decode to utf-8 |
601 |
|
2052.6.1
by Robert Collins
``Transport.get`` has had its interface made more clear for ease of use. |
602 |
This may fail in a number of ways:
|
603 |
- HTTP servers may return content for a directory. (unexpected
|
|
604 |
content failure)
|
|
605 |
- FTP servers may indicate NoSuchFile for a directory.
|
|
606 |
- SFTP servers may give a file handle for a directory that will
|
|
607 |
fail on read().
|
|
608 |
||
609 |
For correct use of the interface, be sure to catch errors.PathError
|
|
610 |
when calling it and catch errors.ReadError when reading from the
|
|
611 |
returned object.
|
|
612 |
||
907.1.20
by John Arbash Meinel
Removed Transport.open(), making get + put encode/decode to utf-8 |
613 |
:param relpath: The relative path to the file
|
1910.19.2
by Andrew Bennetts
Add a new method ``Transport.get_smart_client()``. This is provided to allow |
614 |
:rtype: File-like object.
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
615 |
"""
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
616 |
raise NotImplementedError(self.get) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
617 |
|
1955.3.3
by John Arbash Meinel
Implement and test 'get_bytes' |
618 |
def get_bytes(self, relpath): |
619 |
"""Get a raw string of the bytes for a file at the given location.
|
|
620 |
||
621 |
:param relpath: The relative path to the file
|
|
622 |
"""
|
|
3882.7.3
by Martin Pool
transport.get should specifically close the file handle |
623 |
f = self.get(relpath) |
624 |
try: |
|
625 |
return f.read() |
|
626 |
finally: |
|
627 |
f.close() |
|
1910.19.2
by Andrew Bennetts
Add a new method ``Transport.get_smart_client()``. This is provided to allow |
628 |
|
2018.2.3
by Andrew Bennetts
Starting factoring out the smart server client "medium" from the protocol. |
629 |
def get_smart_medium(self): |
630 |
"""Return a smart client medium for this transport if possible.
|
|
631 |
||
632 |
A smart medium doesn't imply the presence of a smart server: it implies
|
|
633 |
that the smart protocol can be tunnelled via this transport.
|
|
634 |
||
635 |
:raises NoSmartMedium: if no smart server medium is available.
|
|
636 |
"""
|
|
637 |
raise errors.NoSmartMedium(self) |
|
638 |
||
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
639 |
def readv(self, relpath, offsets, adjust_for_latency=False, |
640 |
upper_limit=None): |
|
2745.5.1
by Robert Collins
* New parameter on ``bzrlib.transport.Transport.readv`` |
641 |
"""Get parts of the file at the given relative path.
|
642 |
||
643 |
:param relpath: The path to read data from.
|
|
644 |
:param offsets: A list of (offset, size) tuples.
|
|
3024.2.1
by Vincent Ladeuil
Fix 165061 by using the correct _max_readv_combine attribute. |
645 |
:param adjust_for_latency: Adjust the requested offsets to accomodate
|
2745.5.1
by Robert Collins
* New parameter on ``bzrlib.transport.Transport.readv`` |
646 |
transport latency. This may re-order the offsets, expand them to
|
647 |
grab adjacent data when there is likely a high cost to requesting
|
|
648 |
data relative to delivering it.
|
|
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
649 |
:param upper_limit: When adjust_for_latency is True setting upper_limit
|
650 |
allows the caller to tell the transport about the length of the
|
|
651 |
file, so that requests are not issued for ranges beyond the end of
|
|
652 |
the file. This matters because some servers and/or transports error
|
|
653 |
in such a case rather than just satisfying the available ranges.
|
|
654 |
upper_limit should always be provided when adjust_for_latency is
|
|
655 |
True, and should be the size of the file in bytes.
|
|
2745.5.1
by Robert Collins
* New parameter on ``bzrlib.transport.Transport.readv`` |
656 |
:return: A list or generator of (offset, data) tuples
|
657 |
"""
|
|
658 |
if adjust_for_latency: |
|
2890.1.3
by Robert Collins
Review feedback and discussion with Martin - split out the readv offset adjustment into a new helper and document where the design might/should go next. |
659 |
# Design note: We may wish to have different algorithms for the
|
660 |
# expansion of the offsets per-transport. E.g. for local disk to
|
|
3024.2.1
by Vincent Ladeuil
Fix 165061 by using the correct _max_readv_combine attribute. |
661 |
# use page-aligned expansion. If that is the case consider the
|
662 |
# following structure:
|
|
663 |
# - a test that transport.readv uses self._offset_expander or some
|
|
664 |
# similar attribute, to do the expansion
|
|
665 |
# - a test for each transport that it has some known-good offset
|
|
666 |
# expander
|
|
2890.1.3
by Robert Collins
Review feedback and discussion with Martin - split out the readv offset adjustment into a new helper and document where the design might/should go next. |
667 |
# - unit tests for each offset expander
|
668 |
# - a set of tests for the offset expander interface, giving
|
|
669 |
# baseline behaviour (which the current transport
|
|
670 |
# adjust_for_latency tests could be repurposed to).
|
|
671 |
offsets = self._sort_expand_and_combine(offsets, upper_limit) |
|
2745.5.1
by Robert Collins
* New parameter on ``bzrlib.transport.Transport.readv`` |
672 |
return self._readv(relpath, offsets) |
673 |
||
674 |
def _readv(self, relpath, offsets): |
|
1594.2.5
by Robert Collins
Readv patch from Johan Rydberg giving knits partial download support. |
675 |
"""Get parts of the file at the given relative path.
|
676 |
||
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
677 |
:param relpath: The path to read.
|
678 |
:param offsets: A list of (offset, size) tuples.
|
|
1594.2.5
by Robert Collins
Readv patch from Johan Rydberg giving knits partial download support. |
679 |
:return: A list or generator of (offset, data) tuples
|
680 |
"""
|
|
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
681 |
if not offsets: |
1594.2.16
by Robert Collins
Coalesce readv requests on file based transports. |
682 |
return
|
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
683 |
|
1864.5.7
by John Arbash Meinel
remove disable prefetch support |
684 |
fp = self.get(relpath) |
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
685 |
return self._seek_and_read(fp, offsets, relpath) |
1864.5.11
by John Arbash Meinel
Factor out the workhorse of Transport.readv() into a helper function, and re-use that function in sftp but with a non-prefetch file. |
686 |
|
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
687 |
def _seek_and_read(self, fp, offsets, relpath='<unknown>'): |
1864.5.11
by John Arbash Meinel
Factor out the workhorse of Transport.readv() into a helper function, and re-use that function in sftp but with a non-prefetch file. |
688 |
"""An implementation of readv that uses fp.seek and fp.read.
|
689 |
||
690 |
This uses _coalesce_offsets to issue larger reads and fewer seeks.
|
|
691 |
||
5807.5.1
by John Arbash Meinel
Fix bug #767177. Be more agressive with file.close() calls. |
692 |
:param fp: A file-like object that supports seek() and read(size).
|
693 |
Note that implementations are allowed to call .close() on this file
|
|
694 |
handle, so don't trust that you can use it for other work.
|
|
1864.5.11
by John Arbash Meinel
Factor out the workhorse of Transport.readv() into a helper function, and re-use that function in sftp but with a non-prefetch file. |
695 |
:param offsets: A list of offsets to be read from the given file.
|
696 |
:return: yield (pos, data) tuples for each request
|
|
697 |
"""
|
|
1864.5.2
by John Arbash Meinel
always read in sorted order, and return in requested order, but only cache what is currently out of order |
698 |
# We are going to iterate multiple times, we need a list
|
699 |
offsets = list(offsets) |
|
700 |
sorted_offsets = sorted(offsets) |
|
701 |
||
702 |
# turn the list of offsets into a stack
|
|
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
703 |
offset_stack = iter(offsets) |
704 |
cur_offset_and_size = offset_stack.next() |
|
1864.5.2
by John Arbash Meinel
always read in sorted order, and return in requested order, but only cache what is currently out of order |
705 |
coalesced = self._coalesce_offsets(sorted_offsets, |
1864.5.3
by John Arbash Meinel
Allow collapsing ranges even if they are just 'close' |
706 |
limit=self._max_readv_combine, |
707 |
fudge_factor=self._bytes_to_read_before_seek) |
|
1864.5.2
by John Arbash Meinel
always read in sorted order, and return in requested order, but only cache what is currently out of order |
708 |
|
709 |
# Cache the results, but only until they have been fulfilled
|
|
710 |
data_map = {} |
|
5807.5.1
by John Arbash Meinel
Fix bug #767177. Be more agressive with file.close() calls. |
711 |
try: |
712 |
for c_offset in coalesced: |
|
713 |
# TODO: jam 20060724 it might be faster to not issue seek if
|
|
714 |
# we are already at the right location. This should be
|
|
715 |
# benchmarked.
|
|
716 |
fp.seek(c_offset.start) |
|
717 |
data = fp.read(c_offset.length) |
|
718 |
if len(data) < c_offset.length: |
|
719 |
raise errors.ShortReadvError(relpath, c_offset.start, |
|
720 |
c_offset.length, actual=len(data)) |
|
721 |
for suboffset, subsize in c_offset.ranges: |
|
722 |
key = (c_offset.start+suboffset, subsize) |
|
723 |
data_map[key] = data[suboffset:suboffset+subsize] |
|
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
724 |
|
5807.5.1
by John Arbash Meinel
Fix bug #767177. Be more agressive with file.close() calls. |
725 |
# Now that we've read some data, see if we can yield anything back
|
726 |
while cur_offset_and_size in data_map: |
|
727 |
this_data = data_map.pop(cur_offset_and_size) |
|
728 |
this_offset = cur_offset_and_size[0] |
|
729 |
try: |
|
730 |
cur_offset_and_size = offset_stack.next() |
|
731 |
except StopIteration: |
|
5887.1.1
by Martin
Try reintroducing fp.close() earlier in Transport._seek_and_read |
732 |
fp.close() |
5807.5.1
by John Arbash Meinel
Fix bug #767177. Be more agressive with file.close() calls. |
733 |
cur_offset_and_size = None |
734 |
yield this_offset, this_data |
|
5887.1.1
by Martin
Try reintroducing fp.close() earlier in Transport._seek_and_read |
735 |
finally: |
5807.5.1
by John Arbash Meinel
Fix bug #767177. Be more agressive with file.close() calls. |
736 |
fp.close() |
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
737 |
|
2890.1.3
by Robert Collins
Review feedback and discussion with Martin - split out the readv offset adjustment into a new helper and document where the design might/should go next. |
738 |
def _sort_expand_and_combine(self, offsets, upper_limit): |
739 |
"""Helper for readv.
|
|
740 |
||
741 |
:param offsets: A readv vector - (offset, length) tuples.
|
|
742 |
:param upper_limit: The highest byte offset that may be requested.
|
|
743 |
:return: A readv vector that will read all the regions requested by
|
|
744 |
offsets, in start-to-end order, with no duplicated regions,
|
|
745 |
expanded by the transports recommended page size.
|
|
746 |
"""
|
|
747 |
offsets = sorted(offsets) |
|
748 |
# short circuit empty requests
|
|
749 |
if len(offsets) == 0: |
|
750 |
def empty_yielder(): |
|
751 |
# Quick thunk to stop this function becoming a generator
|
|
752 |
# itself, rather we return a generator that has nothing to
|
|
753 |
# yield.
|
|
754 |
if False: |
|
755 |
yield None |
|
756 |
return empty_yielder() |
|
757 |
# expand by page size at either end
|
|
758 |
maximum_expansion = self.recommended_page_size() |
|
759 |
new_offsets = [] |
|
760 |
for offset, length in offsets: |
|
761 |
expansion = maximum_expansion - length |
|
762 |
if expansion < 0: |
|
763 |
# we're asking for more than the minimum read anyway.
|
|
764 |
expansion = 0 |
|
765 |
reduction = expansion / 2 |
|
766 |
new_offset = offset - reduction |
|
767 |
new_length = length + expansion |
|
768 |
if new_offset < 0: |
|
769 |
# don't ask for anything < 0
|
|
770 |
new_offset = 0 |
|
771 |
if (upper_limit is not None and |
|
772 |
new_offset + new_length > upper_limit): |
|
773 |
new_length = upper_limit - new_offset |
|
774 |
new_offsets.append((new_offset, new_length)) |
|
775 |
# combine the expanded offsets
|
|
776 |
offsets = [] |
|
777 |
current_offset, current_length = new_offsets[0] |
|
778 |
current_finish = current_length + current_offset |
|
779 |
for offset, length in new_offsets[1:]: |
|
780 |
finish = offset + length |
|
3805.4.2
by John Arbash Meinel
Revert the changes to _sort_expand_and_combine |
781 |
if offset > current_finish: |
782 |
# there is a gap, output the current accumulator and start
|
|
783 |
# a new one for the region we're examining.
|
|
2890.1.3
by Robert Collins
Review feedback and discussion with Martin - split out the readv offset adjustment into a new helper and document where the design might/should go next. |
784 |
offsets.append((current_offset, current_length)) |
785 |
current_offset = offset |
|
786 |
current_length = length |
|
787 |
current_finish = finish |
|
788 |
continue
|
|
789 |
if finish > current_finish: |
|
790 |
# extend the current accumulator to the end of the region
|
|
791 |
# we're examining.
|
|
792 |
current_finish = finish |
|
793 |
current_length = finish - current_offset |
|
794 |
offsets.append((current_offset, current_length)) |
|
795 |
return offsets |
|
796 |
||
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
797 |
@staticmethod
|
3686.1.6
by John Arbash Meinel
Respond to Martin's review comments. |
798 |
def _coalesce_offsets(offsets, limit=0, fudge_factor=0, max_size=0): |
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
799 |
"""Yield coalesced offsets.
|
800 |
||
801 |
With a long list of neighboring requests, combine them
|
|
802 |
into a single large request, while retaining the original
|
|
803 |
offsets.
|
|
804 |
Turns [(15, 10), (25, 10)] => [(15, 20, [(0, 10), (10, 10)])]
|
|
3686.1.6
by John Arbash Meinel
Respond to Martin's review comments. |
805 |
Note that overlapping requests are not permitted. (So [(15, 10), (20,
|
806 |
10)] will raise a ValueError.) This is because the data we access never
|
|
807 |
overlaps, and it allows callers to trust that we only need any byte of
|
|
808 |
data for 1 request (so nothing needs to be buffered to fulfill a second
|
|
809 |
request.)
|
|
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
810 |
|
811 |
:param offsets: A list of (start, length) pairs
|
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
812 |
:param limit: Only combine a maximum of this many pairs Some transports
|
813 |
penalize multiple reads more than others, and sometimes it is
|
|
814 |
better to return early.
|
|
815 |
0 means no limit
|
|
1864.5.3
by John Arbash Meinel
Allow collapsing ranges even if they are just 'close' |
816 |
:param fudge_factor: All transports have some level of 'it is
|
3120.2.1
by John Arbash Meinel
Change the sftp_readv loop to buffer even less. |
817 |
better to read some more data and throw it away rather
|
1864.5.3
by John Arbash Meinel
Allow collapsing ranges even if they are just 'close' |
818 |
than seek', so collapse if we are 'close enough'
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
819 |
:param max_size: Create coalesced offsets no bigger than this size.
|
820 |
When a single offset is bigger than 'max_size', it will keep
|
|
821 |
its size and be alone in the coalesced offset.
|
|
822 |
0 means no maximum size.
|
|
3120.2.1
by John Arbash Meinel
Change the sftp_readv loop to buffer even less. |
823 |
:return: return a list of _CoalescedOffset objects, which have members
|
824 |
for where to start, how much to read, and how to split those chunks
|
|
825 |
back up
|
|
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
826 |
"""
|
827 |
last_end = None |
|
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
828 |
cur = _CoalescedOffset(None, None, []) |
3120.2.1
by John Arbash Meinel
Change the sftp_readv loop to buffer even less. |
829 |
coalesced_offsets = [] |
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
830 |
|
3876.1.1
by John Arbash Meinel
Fix bug #303538 by capping the max per-hunk read size at 100MB. |
831 |
if max_size <= 0: |
832 |
# 'unlimited', but we actually take this to mean 100MB buffer limit
|
|
833 |
max_size = 100*1024*1024 |
|
834 |
||
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
835 |
for start, size in offsets: |
836 |
end = start + size |
|
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
837 |
if (last_end is not None |
1864.5.3
by John Arbash Meinel
Allow collapsing ranges even if they are just 'close' |
838 |
and start <= last_end + fudge_factor |
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
839 |
and start >= cur.start |
3059.2.17
by Vincent Ladeuil
Limit GET requests by body size instead of number of ranges. |
840 |
and (limit <= 0 or len(cur.ranges) < limit) |
841 |
and (max_size <= 0 or end - cur.start <= max_size)): |
|
3686.1.6
by John Arbash Meinel
Respond to Martin's review comments. |
842 |
if start < last_end: |
3686.1.9
by John Arbash Meinel
Overlapping ranges are not allowed anymore. |
843 |
raise ValueError('Overlapping range not allowed:' |
3686.1.6
by John Arbash Meinel
Respond to Martin's review comments. |
844 |
' last range ended at %s, new one starts at %s' |
845 |
% (last_end, start)) |
|
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
846 |
cur.length = end - cur.start |
847 |
cur.ranges.append((start-cur.start, size)) |
|
1594.2.16
by Robert Collins
Coalesce readv requests on file based transports. |
848 |
else: |
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
849 |
if cur.start is not None: |
3120.2.1
by John Arbash Meinel
Change the sftp_readv loop to buffer even less. |
850 |
coalesced_offsets.append(cur) |
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
851 |
cur = _CoalescedOffset(start, size, [(0, size)]) |
1864.5.1
by John Arbash Meinel
Change the readv combining algorithm for one that is easier to test. |
852 |
last_end = end |
853 |
||
1864.5.9
by John Arbash Meinel
Switch to returning an object to make the api more understandable. |
854 |
if cur.start is not None: |
3120.2.1
by John Arbash Meinel
Change the sftp_readv loop to buffer even less. |
855 |
coalesced_offsets.append(cur) |
856 |
return coalesced_offsets |
|
1594.2.5
by Robert Collins
Readv patch from Johan Rydberg giving knits partial download support. |
857 |
|
907.1.50
by John Arbash Meinel
Removed encode/decode from Transport.put/get, added more exceptions that can be thrown. |
858 |
def get_multi(self, relpaths, pb=None): |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
859 |
"""Get a list of file-like objects, one for each entry in relpaths.
|
860 |
||
861 |
:param relpaths: A list of relative paths.
|
|
4463.1.1
by Martin Pool
Update docstrings for recent progress changes |
862 |
:param pb: An optional ProgressTask for indicating percent done.
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
863 |
:return: A list or generator of file-like objects
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
864 |
"""
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
865 |
# TODO: Consider having this actually buffer the requests,
|
866 |
# in the default mode, it probably won't give worse performance,
|
|
867 |
# and all children wouldn't have to implement buffering
|
|
907.1.16
by John Arbash Meinel
Fixing a few cut&paste typos. |
868 |
total = self._get_total(relpaths) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
869 |
count = 0 |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
870 |
for relpath in relpaths: |
907.1.16
by John Arbash Meinel
Fixing a few cut&paste typos. |
871 |
self._update_pb(pb, 'get', count, total) |
907.1.50
by John Arbash Meinel
Removed encode/decode from Transport.put/get, added more exceptions that can be thrown. |
872 |
yield self.get(relpath) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
873 |
count += 1 |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
874 |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
875 |
def put_bytes(self, relpath, bytes, mode=None): |
876 |
"""Atomically put the supplied bytes into the given location.
|
|
877 |
||
878 |
:param relpath: The location to put the contents, relative to the
|
|
879 |
transport base.
|
|
880 |
:param bytes: A bytestring of data.
|
|
881 |
:param mode: Create the file with the given mode.
|
|
882 |
:return: None
|
|
883 |
"""
|
|
2414.1.3
by Andrew Bennetts
Fix test failures under 'python2.4 -O' ('python2.5 -O' was already passing). |
884 |
if not isinstance(bytes, str): |
885 |
raise AssertionError( |
|
886 |
'bytes must be a plain string, not %s' % type(bytes)) |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
887 |
return self.put_file(relpath, StringIO(bytes), mode=mode) |
888 |
||
889 |
def put_bytes_non_atomic(self, relpath, bytes, mode=None, |
|
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
890 |
create_parent_dir=False, |
891 |
dir_mode=None): |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
892 |
"""Copy the string into the target location.
|
893 |
||
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
894 |
This function is not strictly safe to use. See
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
895 |
Transport.put_bytes_non_atomic for more information.
|
896 |
||
897 |
:param relpath: The remote location to put the contents.
|
|
898 |
:param bytes: A string object containing the raw bytes to write into
|
|
899 |
the target file.
|
|
900 |
:param mode: Possible access permissions for new file.
|
|
901 |
None means do not set remote permissions.
|
|
902 |
:param create_parent_dir: If we cannot create the target file because
|
|
903 |
the parent directory does not exist, go ahead and
|
|
904 |
create it, and then try again.
|
|
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
905 |
:param dir_mode: Possible access permissions for new directories.
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
906 |
"""
|
2414.1.3
by Andrew Bennetts
Fix test failures under 'python2.4 -O' ('python2.5 -O' was already passing). |
907 |
if not isinstance(bytes, str): |
908 |
raise AssertionError( |
|
909 |
'bytes must be a plain string, not %s' % type(bytes)) |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
910 |
self.put_file_non_atomic(relpath, StringIO(bytes), mode=mode, |
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
911 |
create_parent_dir=create_parent_dir, |
912 |
dir_mode=dir_mode) |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
913 |
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
914 |
def put_file(self, relpath, f, mode=None): |
915 |
"""Copy the file-like object into the location.
|
|
916 |
||
917 |
:param relpath: Location to put the contents, relative to base.
|
|
918 |
:param f: File-like object.
|
|
919 |
:param mode: The mode for the newly created file,
|
|
920 |
None means just use the default.
|
|
2745.5.2
by Robert Collins
* ``bzrlib.transport.Transport.put_file`` now returns the number of bytes |
921 |
:return: The length of the file that was written.
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
922 |
"""
|
923 |
# We would like to mark this as NotImplemented, but most likely
|
|
924 |
# transports have defined it in terms of the old api.
|
|
925 |
symbol_versioning.warn('Transport %s should implement put_file,' |
|
926 |
' rather than implementing put() as of'
|
|
927 |
' version 0.11.'
|
|
928 |
% (self.__class__.__name__,), |
|
929 |
DeprecationWarning) |
|
930 |
return self.put(relpath, f, mode=mode) |
|
931 |
#raise NotImplementedError(self.put_file)
|
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
932 |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
933 |
def put_file_non_atomic(self, relpath, f, mode=None, |
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
934 |
create_parent_dir=False, |
935 |
dir_mode=None): |
|
1946.1.1
by John Arbash Meinel
Stub out the test and basic implementation for 'non_atomic_put' |
936 |
"""Copy the file-like object into the target location.
|
937 |
||
938 |
This function is not strictly safe to use. It is only meant to
|
|
939 |
be used when you already know that the target does not exist.
|
|
940 |
It is not safe, because it will open and truncate the remote
|
|
941 |
file. So there may be a time when the file has invalid contents.
|
|
942 |
||
943 |
:param relpath: The remote location to put the contents.
|
|
944 |
:param f: File-like object.
|
|
945 |
:param mode: Possible access permissions for new file.
|
|
946 |
None means do not set remote permissions.
|
|
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
947 |
:param create_parent_dir: If we cannot create the target file because
|
948 |
the parent directory does not exist, go ahead and
|
|
949 |
create it, and then try again.
|
|
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
950 |
:param dir_mode: Possible access permissions for new directories.
|
1946.1.1
by John Arbash Meinel
Stub out the test and basic implementation for 'non_atomic_put' |
951 |
"""
|
952 |
# Default implementation just does an atomic put.
|
|
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
953 |
try: |
1955.3.19
by John Arbash Meinel
rename non_atomic_put => non_atomic_put_file |
954 |
return self.put_file(relpath, f, mode=mode) |
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
955 |
except errors.NoSuchFile: |
956 |
if not create_parent_dir: |
|
957 |
raise
|
|
958 |
parent_dir = osutils.dirname(relpath) |
|
959 |
if parent_dir: |
|
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
960 |
self.mkdir(parent_dir, mode=dir_mode) |
1955.3.19
by John Arbash Meinel
rename non_atomic_put => non_atomic_put_file |
961 |
return self.put_file(relpath, f, mode=mode) |
1946.1.1
by John Arbash Meinel
Stub out the test and basic implementation for 'non_atomic_put' |
962 |
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
963 |
def mkdir(self, relpath, mode=None): |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
964 |
"""Create a directory at the given path."""
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
965 |
raise NotImplementedError(self.mkdir) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
966 |
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
967 |
def mkdir_multi(self, relpaths, mode=None, pb=None): |
907.1.47
by John Arbash Meinel
Created mkdir_multi, modified branch to use _multi forms when initializing a branch, creating a full test routine for transports |
968 |
"""Create a group of directories"""
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
969 |
def mkdir(path): |
970 |
self.mkdir(path, mode=mode) |
|
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
971 |
return len(self._iterate_over(relpaths, mkdir, pb, 'mkdir', expand=False)) |
907.1.47
by John Arbash Meinel
Created mkdir_multi, modified branch to use _multi forms when initializing a branch, creating a full test routine for transports |
972 |
|
2671.3.9
by Robert Collins
Review feedback and fix VFat emulated transports to not claim to have unix permissions. |
973 |
def open_write_stream(self, relpath, mode=None): |
974 |
"""Open a writable file stream at relpath.
|
|
2671.3.2
by Robert Collins
Start open_file_stream logic. |
975 |
|
2671.3.9
by Robert Collins
Review feedback and fix VFat emulated transports to not claim to have unix permissions. |
976 |
A file stream is a file like object with a write() method that accepts
|
977 |
bytes to write.. Buffering may occur internally until the stream is
|
|
978 |
closed with stream.close(). Calls to readv or the get_* methods will
|
|
979 |
be synchronised with any internal buffering that may be present.
|
|
2671.3.2
by Robert Collins
Start open_file_stream logic. |
980 |
|
981 |
:param relpath: The relative path to the file.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
982 |
:param mode: The mode for the newly created file,
|
2671.3.3
by Robert Collins
Add mode parameter to Transport.open_file_stream. |
983 |
None means just use the default
|
2671.3.6
by Robert Collins
Review feedback. |
984 |
:return: A FileStream. FileStream objects have two methods, write() and
|
985 |
close(). There is no guarantee that data is committed to the file
|
|
986 |
if close() has not been called (even if get() is called on the same
|
|
987 |
path).
|
|
2671.3.2
by Robert Collins
Start open_file_stream logic. |
988 |
"""
|
2671.3.9
by Robert Collins
Review feedback and fix VFat emulated transports to not claim to have unix permissions. |
989 |
raise NotImplementedError(self.open_write_stream) |
2671.3.2
by Robert Collins
Start open_file_stream logic. |
990 |
|
1955.3.15
by John Arbash Meinel
Deprecate 'Transport.append' in favor of Transport.append_file or Transport.append_bytes |
991 |
def append_file(self, relpath, f, mode=None): |
1910.7.18
by Andrew Bennetts
Merge from bzr.dev |
992 |
"""Append bytes from a file-like object to a file at relpath.
|
993 |
||
994 |
The file is created if it does not already exist.
|
|
995 |
||
996 |
:param f: a file-like object of the bytes to append.
|
|
997 |
:param mode: Unix mode for newly created files. This is not used for
|
|
998 |
existing files.
|
|
999 |
||
1000 |
:returns: the length of relpath before the content was written to it.
|
|
1955.3.15
by John Arbash Meinel
Deprecate 'Transport.append' in favor of Transport.append_file or Transport.append_bytes |
1001 |
"""
|
1002 |
symbol_versioning.warn('Transport %s should implement append_file,' |
|
1003 |
' rather than implementing append() as of'
|
|
1004 |
' version 0.11.'
|
|
1005 |
% (self.__class__.__name__,), |
|
1006 |
DeprecationWarning) |
|
1007 |
return self.append(relpath, f, mode=mode) |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1008 |
|
1955.3.2
by John Arbash Meinel
Implement and test 'Transport.append_bytes', cleanup the tests of plain append |
1009 |
def append_bytes(self, relpath, bytes, mode=None): |
1910.7.18
by Andrew Bennetts
Merge from bzr.dev |
1010 |
"""Append bytes to a file at relpath.
|
1011 |
||
1012 |
The file is created if it does not already exist.
|
|
1013 |
||
1014 |
:type f: str
|
|
1015 |
:param f: a string of the bytes to append.
|
|
1016 |
:param mode: Unix mode for newly created files. This is not used for
|
|
1017 |
existing files.
|
|
1018 |
||
1019 |
:returns: the length of relpath before the content was written to it.
|
|
1955.3.2
by John Arbash Meinel
Implement and test 'Transport.append_bytes', cleanup the tests of plain append |
1020 |
"""
|
3376.2.4
by Martin Pool
Remove every assert statement from bzrlib! |
1021 |
if not isinstance(bytes, str): |
1022 |
raise TypeError( |
|
1023 |
'bytes must be a plain string, not %s' % type(bytes)) |
|
1955.3.15
by John Arbash Meinel
Deprecate 'Transport.append' in favor of Transport.append_file or Transport.append_bytes |
1024 |
return self.append_file(relpath, StringIO(bytes), mode=mode) |
1955.3.2
by John Arbash Meinel
Implement and test 'Transport.append_bytes', cleanup the tests of plain append |
1025 |
|
1185.11.19
by John Arbash Meinel
Testing put and append, also testing agaist file-like objects as well as strings. |
1026 |
def append_multi(self, files, pb=None): |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1027 |
"""Append the text in each file-like or string object to
|
1028 |
the supplied location.
|
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
1029 |
|
1030 |
:param files: A set of (path, f) entries
|
|
4463.1.1
by Martin Pool
Update docstrings for recent progress changes |
1031 |
:param pb: An optional ProgressTask for indicating percent done.
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1032 |
"""
|
1955.3.15
by John Arbash Meinel
Deprecate 'Transport.append' in favor of Transport.append_file or Transport.append_bytes |
1033 |
return self._iterate_over(files, self.append_file, pb, 'append', expand=True) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1034 |
|
1035 |
def copy(self, rel_from, rel_to): |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1036 |
"""Copy the item at rel_from to the location at rel_to.
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1037 |
|
1038 |
Override this for efficiency if a specific transport can do it
|
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1039 |
faster than this default implementation.
|
1040 |
"""
|
|
1955.3.11
by John Arbash Meinel
Clean up the rest of the api calls to deprecated functions in the test suite, and make sure Transport._pump is only accepting files, not strings |
1041 |
self.put_file(rel_to, self.get(rel_from)) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1042 |
|
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
1043 |
def copy_multi(self, relpaths, pb=None): |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1044 |
"""Copy a bunch of entries.
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1045 |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1046 |
:param relpaths: A list of tuples of the form [(from, to), (from, to),...]
|
1047 |
"""
|
|
1048 |
# This is the non-pipelined implementation, so that
|
|
1049 |
# implementors don't have to implement everything.
|
|
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
1050 |
return self._iterate_over(relpaths, self.copy, pb, 'copy', expand=True) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1051 |
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
1052 |
def copy_to(self, relpaths, other, mode=None, pb=None): |
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
1053 |
"""Copy a set of entries from self into another Transport.
|
1054 |
||
1055 |
:param relpaths: A list/generator of entries to be copied.
|
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
1056 |
:param mode: This is the target mode for the newly created files
|
1185.16.156
by John Arbash Meinel
Adding a note about changing copy_to's interface |
1057 |
TODO: This interface needs to be updated so that the target location
|
1058 |
can be different from the source location.
|
|
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
1059 |
"""
|
1060 |
# The dummy implementation just does a simple get + put
|
|
1061 |
def copy_entry(path): |
|
1955.3.11
by John Arbash Meinel
Clean up the rest of the api calls to deprecated functions in the test suite, and make sure Transport._pump is only accepting files, not strings |
1062 |
other.put_file(path, self.get(path), mode=mode) |
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
1063 |
|
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
1064 |
return len(self._iterate_over(relpaths, copy_entry, pb, 'copy_to', expand=False)) |
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
1065 |
|
1534.4.26
by Robert Collins
Move working tree initialisation out from Branch.initialize, deprecated Branch.initialize to Branch.create. |
1066 |
def copy_tree(self, from_relpath, to_relpath): |
1067 |
"""Copy a subtree from one relpath to another.
|
|
1068 |
||
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1069 |
If a faster implementation is available, specific transports should
|
1534.4.26
by Robert Collins
Move working tree initialisation out from Branch.initialize, deprecated Branch.initialize to Branch.create. |
1070 |
implement it.
|
1071 |
"""
|
|
1072 |
source = self.clone(from_relpath) |
|
1073 |
target = self.clone(to_relpath) |
|
4634.148.5
by Martin Pool
Additional fix for setting permissions across ftp (thanks Parth) |
1074 |
|
1075 |
# create target directory with the same rwx bits as source.
|
|
1076 |
# use mask to ensure that bits other than rwx are ignored.
|
|
4634.148.1
by Martin Pool
Backport fix for permissions of backup.bzr |
1077 |
stat = self.stat(from_relpath) |
4634.148.5
by Martin Pool
Additional fix for setting permissions across ftp (thanks Parth) |
1078 |
target.mkdir('.', stat.st_mode & 0777) |
1551.21.5
by Aaron Bentley
Implement copy_tree on copy_tree_to_transport |
1079 |
source.copy_tree_to_transport(target) |
1080 |
||
1081 |
def copy_tree_to_transport(self, to_transport): |
|
1082 |
"""Copy a subtree from one transport to another.
|
|
1083 |
||
1084 |
self.base is used as the source tree root, and to_transport.base
|
|
1085 |
is used as the target. to_transport.base must exist (and be a
|
|
1086 |
directory).
|
|
1087 |
"""
|
|
1534.4.26
by Robert Collins
Move working tree initialisation out from Branch.initialize, deprecated Branch.initialize to Branch.create. |
1088 |
files = [] |
1089 |
directories = ['.'] |
|
1090 |
while directories: |
|
1091 |
dir = directories.pop() |
|
1092 |
if dir != '.': |
|
1551.21.5
by Aaron Bentley
Implement copy_tree on copy_tree_to_transport |
1093 |
to_transport.mkdir(dir) |
1094 |
for path in self.list_dir(dir): |
|
1534.4.26
by Robert Collins
Move working tree initialisation out from Branch.initialize, deprecated Branch.initialize to Branch.create. |
1095 |
path = dir + '/' + path |
1551.21.5
by Aaron Bentley
Implement copy_tree on copy_tree_to_transport |
1096 |
stat = self.stat(path) |
1534.4.26
by Robert Collins
Move working tree initialisation out from Branch.initialize, deprecated Branch.initialize to Branch.create. |
1097 |
if S_ISDIR(stat.st_mode): |
1098 |
directories.append(path) |
|
1099 |
else: |
|
1100 |
files.append(path) |
|
1551.21.5
by Aaron Bentley
Implement copy_tree on copy_tree_to_transport |
1101 |
self.copy_to(files, to_transport) |
1534.4.26
by Robert Collins
Move working tree initialisation out from Branch.initialize, deprecated Branch.initialize to Branch.create. |
1102 |
|
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
1103 |
def rename(self, rel_from, rel_to): |
1104 |
"""Rename a file or directory.
|
|
1105 |
||
1106 |
This *must* fail if the destination is a nonempty directory - it must
|
|
1107 |
not automatically remove it. It should raise DirectoryNotEmpty, or
|
|
1108 |
some other PathError if the case can't be specifically detected.
|
|
1109 |
||
1110 |
If the destination is an empty directory or a file this function may
|
|
1111 |
either fail or succeed, depending on the underlying transport. It
|
|
1112 |
should not attempt to remove the destination if overwriting is not the
|
|
1113 |
native transport behaviour. If at all possible the transport should
|
|
1114 |
ensure that the rename either completes or not, without leaving the
|
|
1115 |
destination deleted and the new file not moved in place.
|
|
1116 |
||
1117 |
This is intended mainly for use in implementing LockDir.
|
|
1118 |
"""
|
|
1119 |
# transports may need to override this
|
|
1553.5.17
by Martin Pool
Transport.rename should be unimplemented in base class |
1120 |
raise NotImplementedError(self.rename) |
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
1121 |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1122 |
def move(self, rel_from, rel_to): |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1123 |
"""Move the item at rel_from to the location at rel_to.
|
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
1124 |
|
1125 |
The destination is deleted if possible, even if it's a non-empty
|
|
1126 |
directory tree.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1127 |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1128 |
If a transport can directly implement this it is suggested that
|
1129 |
it do so for efficiency.
|
|
1130 |
"""
|
|
1534.4.26
by Robert Collins
Move working tree initialisation out from Branch.initialize, deprecated Branch.initialize to Branch.create. |
1131 |
if S_ISDIR(self.stat(rel_from).st_mode): |
1132 |
self.copy_tree(rel_from, rel_to) |
|
1133 |
self.delete_tree(rel_from) |
|
1134 |
else: |
|
1135 |
self.copy(rel_from, rel_to) |
|
1136 |
self.delete(rel_from) |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1137 |
|
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
1138 |
def move_multi(self, relpaths, pb=None): |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1139 |
"""Move a bunch of entries.
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1140 |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1141 |
:param relpaths: A list of tuples of the form [(from1, to1), (from2, to2),...]
|
1142 |
"""
|
|
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
1143 |
return self._iterate_over(relpaths, self.move, pb, 'move', expand=True) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1144 |
|
1145 |
def move_multi_to(self, relpaths, rel_to): |
|
1146 |
"""Move a bunch of entries to a single location.
|
|
1147 |
This differs from move_multi in that you give a list of from, and
|
|
1148 |
a single destination, rather than multiple destinations.
|
|
1149 |
||
1150 |
:param relpaths: A list of relative paths [from1, from2, from3, ...]
|
|
1151 |
:param rel_to: A directory where each entry should be placed.
|
|
1152 |
"""
|
|
1153 |
# This is not implemented, because you need to do special tricks to
|
|
1154 |
# extract the basename, and add it to rel_to
|
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
1155 |
raise NotImplementedError(self.move_multi_to) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1156 |
|
1157 |
def delete(self, relpath): |
|
1158 |
"""Delete the item at relpath"""
|
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
1159 |
raise NotImplementedError(self.delete) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1160 |
|
907.1.28
by John Arbash Meinel
Added pb to function that were missing, implemented a basic double-dispatch copy_to function. |
1161 |
def delete_multi(self, relpaths, pb=None): |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1162 |
"""Queue up a bunch of deletes to be done.
|
1163 |
"""
|
|
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
1164 |
return self._iterate_over(relpaths, self.delete, pb, 'delete', expand=False) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
1165 |
|
1534.4.15
by Robert Collins
Remove shutil dependency in upgrade - create a delete_tree method for transports. |
1166 |
def delete_tree(self, relpath): |
1167 |
"""Delete an entire tree. This may require a listable transport."""
|
|
1168 |
subtree = self.clone(relpath) |
|
1169 |
files = [] |
|
1170 |
directories = ['.'] |
|
1171 |
pending_rmdirs = [] |
|
1172 |
while directories: |
|
1173 |
dir = directories.pop() |
|
1174 |
if dir != '.': |
|
1175 |
pending_rmdirs.append(dir) |
|
1176 |
for path in subtree.list_dir(dir): |
|
1177 |
path = dir + '/' + path |
|
1178 |
stat = subtree.stat(path) |
|
1179 |
if S_ISDIR(stat.st_mode): |
|
1180 |
directories.append(path) |
|
1181 |
else: |
|
1182 |
files.append(path) |
|
1183 |
subtree.delete_multi(files) |
|
1184 |
pending_rmdirs.reverse() |
|
1185 |
for dir in pending_rmdirs: |
|
1186 |
subtree.rmdir(dir) |
|
1187 |
self.rmdir(relpath) |
|
1188 |
||
1534.5.7
by Robert Collins
Start factoring out the upgrade policy logic. |
1189 |
def __repr__(self): |
1190 |
return "<%s.%s url=%s>" % (self.__module__, self.__class__.__name__, self.base) |
|
1191 |
||
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
1192 |
def stat(self, relpath): |
1193 |
"""Return the stat information for a file.
|
|
1194 |
WARNING: This may not be implementable for all protocols, so use
|
|
1195 |
sparingly.
|
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
1196 |
NOTE: This returns an object with fields such as 'st_size'. It MAY
|
1197 |
or MAY NOT return the literal result of an os.stat() call, so all
|
|
1198 |
access should be via named fields.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1199 |
ALSO NOTE: Stats of directories may not be supported on some
|
1442.1.44
by Robert Collins
Many transport related tweaks: |
1200 |
transports.
|
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
1201 |
"""
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
1202 |
raise NotImplementedError(self.stat) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
1203 |
|
1534.4.15
by Robert Collins
Remove shutil dependency in upgrade - create a delete_tree method for transports. |
1204 |
def rmdir(self, relpath): |
1205 |
"""Remove a directory at the given path."""
|
|
1206 |
raise NotImplementedError |
|
1207 |
||
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
1208 |
def stat_multi(self, relpaths, pb=None): |
1209 |
"""Stat multiple files and return the information.
|
|
1210 |
"""
|
|
1211 |
#TODO: Is it worth making this a generator instead of a
|
|
1212 |
# returning a list?
|
|
1213 |
stats = [] |
|
1214 |
def gather(path): |
|
1215 |
stats.append(self.stat(path)) |
|
1216 |
||
907.1.14
by John Arbash Meinel
Handling some transport functions which take only a single argument. |
1217 |
count = self._iterate_over(relpaths, gather, pb, 'stat', expand=False) |
907.1.2
by John Arbash Meinel
Working on making Branch() do all of it's work over a Transport. |
1218 |
return stats |
1219 |
||
5056.1.4
by Neil Santos
Changed stat() in SFTPTransport and LocalTransport back to calling plain stat(), instead of lstat(). |
1220 |
def readlink(self, relpath): |
1221 |
"""Return a string representing the path to which the symbolic link points."""
|
|
1222 |
raise errors.TransportNotPossible("Dereferencing symlinks is not supported on %s" % self) |
|
1223 |
||
5056.1.9
by Neil Santos
Renamed link() methods to hardlink(), as per mbp's suggestion |
1224 |
def hardlink(self, source, link_name): |
5056.1.4
by Neil Santos
Changed stat() in SFTPTransport and LocalTransport back to calling plain stat(), instead of lstat(). |
1225 |
"""Create a hardlink pointing to source named link_name."""
|
5056.1.1
by Neil Santos
Added default link() and symlink() methods to Transport. |
1226 |
raise errors.TransportNotPossible("Hard links are not supported on %s" % self) |
1227 |
||
1228 |
def symlink(self, source, link_name): |
|
5056.1.4
by Neil Santos
Changed stat() in SFTPTransport and LocalTransport back to calling plain stat(), instead of lstat(). |
1229 |
"""Create a symlink pointing to source named link_name."""
|
5056.1.1
by Neil Santos
Added default link() and symlink() methods to Transport. |
1230 |
raise errors.TransportNotPossible("Symlinks are not supported on %s" % self) |
1231 |
||
1400.1.1
by Robert Collins
implement a basic test for the ui branch command from http servers |
1232 |
def listable(self): |
1233 |
"""Return True if this store supports listing."""
|
|
1540.3.1
by Martin Pool
First-cut implementation of pycurl. Substantially faster than using urllib. |
1234 |
raise NotImplementedError(self.listable) |
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1235 |
|
1236 |
def list_dir(self, relpath): |
|
1237 |
"""Return a list of all files at the given location.
|
|
1238 |
WARNING: many transports do not support this, so trying avoid using
|
|
1239 |
it if at all possible.
|
|
1240 |
"""
|
|
1910.16.1
by Andrew Bennetts
lock_read and lock_write may raise TransportNotPossible. |
1241 |
raise errors.TransportNotPossible("Transport %r has not " |
1530.1.21
by Robert Collins
Review feedback fixes. |
1242 |
"implemented list_dir "
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1243 |
"(but must claim to be listable "
|
1910.16.1
by Andrew Bennetts
lock_read and lock_write may raise TransportNotPossible. |
1244 |
"to trigger this error)."
|
1245 |
% (self)) |
|
907.1.1
by John Arbash Meinel
Reworking the Branch and Store code to support an abstracted filesystem layer. |
1246 |
|
907.1.24
by John Arbash Meinel
Remote functionality work. |
1247 |
def lock_read(self, relpath): |
1248 |
"""Lock the given file for shared (read) access.
|
|
1910.16.1
by Andrew Bennetts
lock_read and lock_write may raise TransportNotPossible. |
1249 |
|
1250 |
WARNING: many transports do not support this, so trying avoid using it.
|
|
1251 |
These methods may be removed in the future.
|
|
1252 |
||
1253 |
Transports may raise TransportNotPossible if OS-level locks cannot be
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1254 |
taken over this transport.
|
907.1.24
by John Arbash Meinel
Remote functionality work. |
1255 |
|
1256 |
:return: A lock object, which should contain an unlock() function.
|
|
1257 |
"""
|
|
1910.16.1
by Andrew Bennetts
lock_read and lock_write may raise TransportNotPossible. |
1258 |
raise errors.TransportNotPossible("transport locks not supported on %s" % self) |
907.1.24
by John Arbash Meinel
Remote functionality work. |
1259 |
|
1260 |
def lock_write(self, relpath): |
|
1261 |
"""Lock the given file for exclusive (write) access.
|
|
1910.16.1
by Andrew Bennetts
lock_read and lock_write may raise TransportNotPossible. |
1262 |
|
1263 |
WARNING: many transports do not support this, so trying avoid using it.
|
|
1264 |
These methods may be removed in the future.
|
|
1265 |
||
1266 |
Transports may raise TransportNotPossible if OS-level locks cannot be
|
|
1267 |
taken over this transport.
|
|
907.1.24
by John Arbash Meinel
Remote functionality work. |
1268 |
|
1269 |
:return: A lock object, which should contain an unlock() function.
|
|
1270 |
"""
|
|
1910.16.1
by Andrew Bennetts
lock_read and lock_write may raise TransportNotPossible. |
1271 |
raise errors.TransportNotPossible("transport locks not supported on %s" % self) |
907.1.24
by John Arbash Meinel
Remote functionality work. |
1272 |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1273 |
def is_readonly(self): |
1274 |
"""Return true if this connection cannot be written to."""
|
|
1275 |
return False |
|
1276 |
||
1608.2.7
by Martin Pool
Rename supports_unix_modebits to _can_roundtrip_unix_modebits for clarity |
1277 |
def _can_roundtrip_unix_modebits(self): |
1608.2.5
by Martin Pool
Add Transport.supports_unix_modebits, so tests can |
1278 |
"""Return true if this transport can store and retrieve unix modebits.
|
1279 |
||
1280 |
(For example, 0700 to make a directory owner-private.)
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1281 |
|
1282 |
Note: most callers will not want to switch on this, but should rather
|
|
1608.2.5
by Martin Pool
Add Transport.supports_unix_modebits, so tests can |
1283 |
just try and set permissions and let them be either stored or not.
|
1284 |
This is intended mainly for the use of the test suite.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1285 |
|
1286 |
Warning: this is not guaranteed to be accurate as sometimes we can't
|
|
1608.2.5
by Martin Pool
Add Transport.supports_unix_modebits, so tests can |
1287 |
be sure: for example with vfat mounted on unix, or a windows sftp
|
1288 |
server."""
|
|
1289 |
# TODO: Perhaps return a e.g. TransportCharacteristics that can answer
|
|
1290 |
# several questions about the transport.
|
|
1291 |
return False |
|
1292 |
||
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1293 |
def _reuse_for(self, other_base): |
2485.8.43
by Vincent Ladeuil
Cleaning. |
1294 |
# This is really needed for ConnectedTransport only, but it's easier to
|
1295 |
# have Transport refuses to be reused than testing that the reuse
|
|
1296 |
# should be asked to ConnectedTransport only.
|
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1297 |
return None |
907.1.24
by John Arbash Meinel
Remote functionality work. |
1298 |
|
5247.2.12
by Vincent Ladeuil
Ensure that all transports close their underlying connection. |
1299 |
def disconnect(self): |
1300 |
# This is really needed for ConnectedTransport only, but it's easier to
|
|
1301 |
# have Transport do nothing than testing that the disconnect should be
|
|
1302 |
# asked to ConnectedTransport only.
|
|
1303 |
pass
|
|
1304 |
||
3878.4.5
by Vincent Ladeuil
Don't use the exception as a parameter for _redirected_to. |
1305 |
def _redirected_to(self, source, target): |
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
1306 |
"""Returns a transport suitable to re-issue a redirected request.
|
1307 |
||
3878.4.5
by Vincent Ladeuil
Don't use the exception as a parameter for _redirected_to. |
1308 |
:param source: The source url as returned by the server.
|
1309 |
:param target: The target url as returned by the server.
|
|
3878.4.1
by Vincent Ladeuil
Fix bug #245964 by preserving decorators during redirections (when |
1310 |
|
1311 |
The redirection can be handled only if the relpath involved is not
|
|
1312 |
renamed by the redirection.
|
|
1313 |
||
1314 |
:returns: A transport or None.
|
|
1315 |
"""
|
|
1316 |
# This returns None by default, meaning the transport can't handle the
|
|
1317 |
# redirection.
|
|
1318 |
return None |
|
1319 |
||
1320 |
||
2485.8.54
by Vincent Ladeuil
Refactor medium uses by making a distinction betweem shared and real medium. |
1321 |
|
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1322 |
class _SharedConnection(object): |
1323 |
"""A connection shared between several transports."""
|
|
1324 |
||
3104.4.1
by Andrew Bennetts
Fix paths sent by bzr+http client to correctly adjust for shared medium. |
1325 |
def __init__(self, connection=None, credentials=None, base=None): |
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1326 |
"""Constructor.
|
1327 |
||
1328 |
:param connection: An opaque object specific to each transport.
|
|
1329 |
||
1330 |
:param credentials: An opaque object containing the credentials used to
|
|
1331 |
create the connection.
|
|
1332 |
"""
|
|
1333 |
self.connection = connection |
|
1334 |
self.credentials = credentials |
|
3104.4.1
by Andrew Bennetts
Fix paths sent by bzr+http client to correctly adjust for shared medium. |
1335 |
self.base = base |
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1336 |
|
1337 |
||
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1338 |
class ConnectedTransport(Transport): |
1339 |
"""A transport connected to a remote server.
|
|
1340 |
||
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1341 |
This class provide the basis to implement transports that need to connect
|
1342 |
to a remote server.
|
|
1343 |
||
1344 |
Host and credentials are available as private attributes, cloning preserves
|
|
1345 |
them and share the underlying, protocol specific, connection.
|
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1346 |
"""
|
1347 |
||
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1348 |
def __init__(self, base, _from_transport=None): |
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1349 |
"""Constructor.
|
1350 |
||
2485.8.62
by Vincent Ladeuil
From review comments, fix typos and deprecate some functions. |
1351 |
The caller should ensure that _from_transport points at the same host
|
1352 |
as the new base.
|
|
1353 |
||
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1354 |
:param base: transport root URL
|
1355 |
||
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1356 |
:param _from_transport: optional transport to build from. The built
|
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1357 |
transport will share the connection with this transport.
|
1358 |
"""
|
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1359 |
if not base.endswith('/'): |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1360 |
base += '/' |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1361 |
self._parsed_url = self._split_url(base) |
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1362 |
if _from_transport is not None: |
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1363 |
# Copy the password as it does not appear in base and will be lost
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1364 |
# otherwise. It can appear in the _split_url above if the user
|
1365 |
# provided it on the command line. Otherwise, daughter classes will
|
|
1366 |
# prompt the user for one when appropriate.
|
|
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1367 |
self._parsed_url.password = _from_transport._parsed_url.password |
6055.2.5
by Jelmer Vernooij
Fix password handling. |
1368 |
self._parsed_url.quoted_password = ( |
1369 |
_from_transport._parsed_url.quoted_password) |
|
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1370 |
|
5268.7.9
by Jelmer Vernooij
Fix tests. |
1371 |
base = str(self._parsed_url) |
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1372 |
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1373 |
super(ConnectedTransport, self).__init__(base) |
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1374 |
if _from_transport is None: |
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1375 |
self._shared_connection = _SharedConnection() |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1376 |
else: |
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1377 |
self._shared_connection = _from_transport._shared_connection |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1378 |
|
6055.2.9
by Jelmer Vernooij
Provide proxy objects. |
1379 |
@property
|
1380 |
def _user(self): |
|
1381 |
return self._parsed_url.user |
|
1382 |
||
1383 |
@property
|
|
1384 |
def _password(self): |
|
1385 |
return self._parsed_url.password |
|
1386 |
||
1387 |
@property
|
|
1388 |
def _host(self): |
|
1389 |
return self._parsed_url.host |
|
1390 |
||
1391 |
@property
|
|
1392 |
def _port(self): |
|
1393 |
return self._parsed_url.port |
|
1394 |
||
1395 |
@property
|
|
1396 |
def _path(self): |
|
1397 |
return self._parsed_url.path |
|
1398 |
||
1399 |
@property
|
|
1400 |
def _scheme(self): |
|
1401 |
return self._parsed_url.scheme |
|
1402 |
||
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1403 |
def clone(self, offset=None): |
1404 |
"""Return a new transport with root at self.base + offset
|
|
1405 |
||
1406 |
We leave the daughter classes take advantage of the hint
|
|
1407 |
that it's a cloning not a raw creation.
|
|
1408 |
"""
|
|
1409 |
if offset is None: |
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1410 |
return self.__class__(self.base, _from_transport=self) |
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1411 |
else: |
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1412 |
return self.__class__(self.abspath(offset), _from_transport=self) |
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1413 |
|
2485.8.47
by Vincent Ladeuil
Remove _initial_split_url hack. |
1414 |
@staticmethod
|
1415 |
def _split_url(url): |
|
6055.2.7
by Jelmer Vernooij
Change parse_url to URL.from_string. |
1416 |
return urlutils.URL.from_string(url) |
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1417 |
|
2485.8.47
by Vincent Ladeuil
Remove _initial_split_url hack. |
1418 |
@staticmethod
|
1419 |
def _unsplit_url(scheme, user, password, host, port, path): |
|
6055.2.4
by Jelmer Vernooij
Fix docstring formatting. |
1420 |
"""Build the full URL for the given already URL encoded path.
|
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1421 |
|
1422 |
user, password, host and path will be quoted if they contain reserved
|
|
1423 |
chars.
|
|
1424 |
||
1425 |
:param scheme: protocol
|
|
1426 |
:param user: login
|
|
1427 |
:param password: associated password
|
|
1428 |
:param host: the server address
|
|
1429 |
:param port: the associated port
|
|
1430 |
:param path: the absolute path on the server
|
|
1431 |
||
1432 |
:return: The corresponding URL.
|
|
1433 |
"""
|
|
6379.4.2
by Jelmer Vernooij
Add urlutils.quote / urlutils.unquote. |
1434 |
netloc = urlutils.quote(host) |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1435 |
if user is not None: |
1436 |
# Note that we don't put the password back even if we
|
|
1437 |
# have one so that it doesn't get accidentally
|
|
1438 |
# exposed.
|
|
6379.4.2
by Jelmer Vernooij
Add urlutils.quote / urlutils.unquote. |
1439 |
netloc = '%s@%s' % (urlutils.quote(user), netloc) |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1440 |
if port is not None: |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1441 |
netloc = '%s:%d' % (netloc, port) |
5268.7.11
by Jelmer Vernooij
revert some unnecessary changes |
1442 |
path = urlutils.escape(path) |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1443 |
return urlparse.urlunparse((scheme, netloc, path, None, None, None)) |
1444 |
||
1445 |
def relpath(self, abspath): |
|
2485.8.29
by Vincent Ladeuil
Cometic changes (and a typo). |
1446 |
"""Return the local path portion from a given absolute path"""
|
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1447 |
parsed_url = self._split_url(abspath) |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1448 |
error = [] |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1449 |
if parsed_url.scheme != self._parsed_url.scheme: |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1450 |
error.append('scheme mismatch') |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1451 |
if parsed_url.user != self._parsed_url.user: |
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1452 |
error.append('user name mismatch') |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1453 |
if parsed_url.host != self._parsed_url.host: |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1454 |
error.append('host mismatch') |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1455 |
if parsed_url.port != self._parsed_url.port: |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1456 |
error.append('port mismatch') |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1457 |
if (not (parsed_url.path == self._parsed_url.path[:-1] or |
1458 |
parsed_url.path.startswith(self._parsed_url.path))): |
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1459 |
error.append('path mismatch') |
1460 |
if error: |
|
1461 |
extra = ', '.join(error) |
|
1462 |
raise errors.PathNotChild(abspath, self.base, extra=extra) |
|
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1463 |
pl = len(self._parsed_url.path) |
1464 |
return parsed_url.path[pl:].strip('/') |
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1465 |
|
1466 |
def abspath(self, relpath): |
|
1467 |
"""Return the full url to the given relative path.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1468 |
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1469 |
:param relpath: the relative path urlencoded
|
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1470 |
|
2485.8.25
by Vincent Ladeuil
Separate abspath from _remote_path, the intents are different. |
1471 |
:returns: the Unicode version of the absolute path for relpath.
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1472 |
"""
|
5268.7.9
by Jelmer Vernooij
Fix tests. |
1473 |
return str(self._parsed_url.clone(relpath)) |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1474 |
|
1475 |
def _remote_path(self, relpath): |
|
1476 |
"""Return the absolute path part of the url to the given relative path.
|
|
2485.8.28
by Vincent Ladeuil
Further simplifications and doc updates. |
1477 |
|
1478 |
This is the path that the remote server expect to receive in the
|
|
1479 |
requests, daughter classes should redefine this method if needed and
|
|
1480 |
use the result to build their requests.
|
|
1481 |
||
1482 |
:param relpath: the path relative to the transport base urlencoded.
|
|
1483 |
||
1484 |
:return: the absolute Unicode path on the server,
|
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1485 |
"""
|
6055.2.18
by Jelmer Vernooij
Use clone. |
1486 |
return self._parsed_url.clone(relpath).path |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1487 |
|
2485.8.54
by Vincent Ladeuil
Refactor medium uses by making a distinction betweem shared and real medium. |
1488 |
def _get_shared_connection(self): |
1489 |
"""Get the object shared amongst cloned transports.
|
|
1490 |
||
1491 |
This should be used only by classes that needs to extend the sharing
|
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1492 |
with objects other than transports.
|
2485.8.54
by Vincent Ladeuil
Refactor medium uses by making a distinction betweem shared and real medium. |
1493 |
|
1494 |
Use _get_connection to get the connection itself.
|
|
1495 |
"""
|
|
1496 |
return self._shared_connection |
|
2485.8.32
by Vincent Ladeuil
Keep credentials used at connection creation for reconnection purposes. |
1497 |
|
1498 |
def _set_connection(self, connection, credentials=None): |
|
2485.8.34
by Vincent Ladeuil
Refactor mutiple connections detection and fix false positives. Only |
1499 |
"""Record a newly created connection with its associated credentials.
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1500 |
|
2485.8.30
by Vincent Ladeuil
Implement reliable connection sharing. |
1501 |
Note: To ensure that connection is still shared after a temporary
|
1502 |
failure and a new one needs to be created, daughter classes should
|
|
2485.8.34
by Vincent Ladeuil
Refactor mutiple connections detection and fix false positives. Only |
1503 |
always call this method to set the connection and do so each time a new
|
1504 |
connection is created.
|
|
2485.8.32
by Vincent Ladeuil
Keep credentials used at connection creation for reconnection purposes. |
1505 |
|
1506 |
:param connection: An opaque object representing the connection used by
|
|
1507 |
the daughter class.
|
|
1508 |
||
1509 |
:param credentials: An opaque object representing the credentials
|
|
1510 |
needed to create the connection.
|
|
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1511 |
"""
|
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1512 |
self._shared_connection.connection = connection |
1513 |
self._shared_connection.credentials = credentials |
|
5436.3.1
by Martin
Create new post_connect hook for transports |
1514 |
for hook in self.hooks["post_connect"]: |
1515 |
hook(self) |
|
2485.8.32
by Vincent Ladeuil
Keep credentials used at connection creation for reconnection purposes. |
1516 |
|
1517 |
def _get_connection(self): |
|
2485.8.30
by Vincent Ladeuil
Implement reliable connection sharing. |
1518 |
"""Returns the transport specific connection object."""
|
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1519 |
return self._shared_connection.connection |
2485.8.30
by Vincent Ladeuil
Implement reliable connection sharing. |
1520 |
|
2485.8.32
by Vincent Ladeuil
Keep credentials used at connection creation for reconnection purposes. |
1521 |
def _get_credentials(self): |
2485.8.34
by Vincent Ladeuil
Refactor mutiple connections detection and fix false positives. Only |
1522 |
"""Returns the credentials used to establish the connection."""
|
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1523 |
return self._shared_connection.credentials |
2485.8.19
by Vincent Ladeuil
Add a new ConnectedTransport class refactored from [s]ftp and http. |
1524 |
|
2485.8.39
by Vincent Ladeuil
Add tests around connection reuse. |
1525 |
def _update_credentials(self, credentials): |
1526 |
"""Update the credentials of the current connection.
|
|
1527 |
||
1528 |
Some protocols can renegociate the credentials within a connection,
|
|
1529 |
this method allows daughter classes to share updated credentials.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1530 |
|
2485.8.39
by Vincent Ladeuil
Add tests around connection reuse. |
1531 |
:param credentials: the updated credentials.
|
1532 |
"""
|
|
1533 |
# We don't want to call _set_connection here as we are only updating
|
|
1534 |
# the credentials not creating a new connection.
|
|
2485.8.55
by Vincent Ladeuil
Cleanup connection accessors. |
1535 |
self._shared_connection.credentials = credentials |
2485.8.39
by Vincent Ladeuil
Add tests around connection reuse. |
1536 |
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1537 |
def _reuse_for(self, other_base): |
1538 |
"""Returns a transport sharing the same connection if possible.
|
|
1539 |
||
1540 |
Note: we share the connection if the expected credentials are the
|
|
1541 |
same: (host, port, user). Some protocols may disagree and redefine the
|
|
1542 |
criteria in daughter classes.
|
|
1543 |
||
1544 |
Note: we don't compare the passwords here because other_base may have
|
|
1545 |
been obtained from an existing transport.base which do not mention the
|
|
1546 |
password.
|
|
1547 |
||
1548 |
:param other_base: the URL we want to share the connection with.
|
|
1549 |
||
1550 |
:return: A new transport or None if the connection cannot be shared.
|
|
1551 |
"""
|
|
2990.2.2
by Vincent Ladeuil
Detect invalid transport reuse attempts by catching invalid URLs. |
1552 |
try: |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1553 |
parsed_url = self._split_url(other_base) |
2990.2.2
by Vincent Ladeuil
Detect invalid transport reuse attempts by catching invalid URLs. |
1554 |
except errors.InvalidURL: |
1555 |
# No hope in trying to reuse an existing transport for an invalid
|
|
1556 |
# URL
|
|
1557 |
return None |
|
1558 |
||
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1559 |
transport = None |
2485.8.39
by Vincent Ladeuil
Add tests around connection reuse. |
1560 |
# Don't compare passwords, they may be absent from other_base or from
|
1561 |
# self and they don't carry more information than user anyway.
|
|
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1562 |
if (parsed_url.scheme == self._parsed_url.scheme |
1563 |
and parsed_url.user == self._parsed_url.user |
|
1564 |
and parsed_url.host == self._parsed_url.host |
|
1565 |
and parsed_url.port == self._parsed_url.port): |
|
1566 |
path = parsed_url.path |
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1567 |
if not path.endswith('/'): |
1568 |
# This normally occurs at __init__ time, but it's easier to do
|
|
2485.8.39
by Vincent Ladeuil
Add tests around connection reuse. |
1569 |
# it now to avoid creating two transports for the same base.
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1570 |
path += '/' |
6055.2.1
by Jelmer Vernooij
Add UnparsedUrl. |
1571 |
if self._parsed_url.path == path: |
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1572 |
# shortcut, it's really the same transport
|
1573 |
return self |
|
1574 |
# We don't call clone here because the intent is different: we
|
|
1575 |
# build a new transport on a different base (which may be totally
|
|
1576 |
# unrelated) but we share the connection.
|
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1577 |
transport = self.__class__(other_base, _from_transport=self) |
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1578 |
return transport |
1579 |
||
5247.2.12
by Vincent Ladeuil
Ensure that all transports close their underlying connection. |
1580 |
def disconnect(self): |
1581 |
"""Disconnect the transport.
|
|
1582 |
||
1583 |
If and when required the transport willl reconnect automatically.
|
|
1584 |
"""
|
|
1585 |
raise NotImplementedError(self.disconnect) |
|
1586 |
||
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1587 |
|
6030.2.1
by Jelmer Vernooij
Create location_to_url. |
1588 |
def location_to_url(location): |
1589 |
"""Determine a fully qualified URL from a location string.
|
|
1590 |
||
6030.2.4
by Jelmer Vernooij
Move local file interpretation to location_to_url. |
1591 |
This will try to interpret location as both a URL and a directory path. It
|
1592 |
will also lookup the location in directories.
|
|
6030.2.7
by Jelmer Vernooij
Review feedback from Riddell: clarify why set_default_transport can be removed and improve the docstring for location_to_url. |
1593 |
|
1594 |
:param location: Unicode or byte string object with a location
|
|
1595 |
:raise InvalidURL: If the location is already a URL, but not valid.
|
|
1596 |
:return: Byte string with resulting URL
|
|
6030.2.1
by Jelmer Vernooij
Create location_to_url. |
1597 |
"""
|
1598 |
if not isinstance(location, basestring): |
|
1599 |
raise AssertionError("location not a byte or unicode string") |
|
1600 |
from bzrlib.directory_service import directories |
|
1601 |
location = directories.dereference(location) |
|
6030.2.3
by Jelmer Vernooij
Move encoding of path to URL. |
1602 |
|
1603 |
# Catch any URLs which are passing Unicode rather than ASCII
|
|
1604 |
try: |
|
1605 |
location = location.encode('ascii') |
|
1606 |
except UnicodeError: |
|
1607 |
if urlutils.is_url(location): |
|
1608 |
raise errors.InvalidURL(path=location, |
|
1609 |
extra='URLs must be properly escaped') |
|
1610 |
location = urlutils.local_path_to_url(location) |
|
1611 |
||
6037.1.1
by Jelmer Vernooij
Allow file: for use with relative local paths. |
1612 |
if location.startswith("file:") and not location.startswith("file://"): |
1613 |
return urlutils.join(urlutils.local_path_to_url("."), location[5:]) |
|
1614 |
||
1615 |
if not urlutils.is_url(location): |
|
1616 |
return urlutils.local_path_to_url(location) |
|
1617 |
||
1618 |
return location |
|
6030.2.1
by Jelmer Vernooij
Create location_to_url. |
1619 |
|
1620 |
||
6039.1.5
by Jelmer Vernooij
Add get_transport_from_url and get_transport_from_path functions. |
1621 |
def get_transport_from_path(path, possible_transports=None): |
1622 |
"""Open a transport for a local path.
|
|
1623 |
||
1624 |
:param path: Local path as byte or unicode string
|
|
1625 |
:return: Transport object for path
|
|
1626 |
"""
|
|
1627 |
return get_transport_from_url(urlutils.local_path_to_url(path), |
|
1628 |
possible_transports) |
|
1629 |
||
1630 |
||
1631 |
def get_transport_from_url(url, possible_transports=None): |
|
1632 |
"""Open a transport to access a URL.
|
|
1633 |
|
|
1634 |
:param base: a URL
|
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1635 |
:param transports: optional reusable transports list. If not None, created
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1636 |
transports will be added to the list.
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1637 |
|
1638 |
:return: A new transport optionally sharing its connection with one of
|
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
1639 |
possible_transports.
|
1185.70.3
by Martin Pool
Various updates to make storage branch mergeable: |
1640 |
"""
|
2476.3.5
by Vincent Ladeuil
Naive implementation of transport reuse by Transport.get_transport(). |
1641 |
transport = None |
1551.18.10
by Aaron Bentley
get_transport appends to possible_transports if it's an empty list |
1642 |
if possible_transports is not None: |
2476.3.8
by Vincent Ladeuil
Mark transports that need to be instrumented or refactored to check |
1643 |
for t in possible_transports: |
6039.1.5
by Jelmer Vernooij
Add get_transport_from_url and get_transport_from_path functions. |
1644 |
t_same_connection = t._reuse_for(url) |
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1645 |
if t_same_connection is not None: |
1646 |
# Add only new transports
|
|
1647 |
if t_same_connection not in possible_transports: |
|
1648 |
possible_transports.append(t_same_connection) |
|
1649 |
return t_same_connection |
|
1650 |
||
6030.2.4
by Jelmer Vernooij
Move local file interpretation to location_to_url. |
1651 |
last_err = None |
3882.1.1
by Martin Pool
Don't call iteritems on transport_list_registry, because it may change during iteration |
1652 |
for proto, factory_list in transport_list_registry.items(): |
6039.1.5
by Jelmer Vernooij
Add get_transport_from_url and get_transport_from_path functions. |
1653 |
if proto is not None and url.startswith(proto): |
1654 |
transport, last_err = _try_transport_factories(url, factory_list) |
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1655 |
if transport: |
1551.18.10
by Aaron Bentley
get_transport appends to possible_transports if it's an empty list |
1656 |
if possible_transports is not None: |
3376.2.4
by Martin Pool
Remove every assert statement from bzrlib! |
1657 |
if transport in possible_transports: |
1658 |
raise AssertionError() |
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1659 |
possible_transports.append(transport) |
1660 |
return transport |
|
6039.1.6
by Jelmer Vernooij
Add more tests. |
1661 |
if not urlutils.is_url(url): |
1662 |
raise errors.InvalidURL(path=url) |
|
6039.1.5
by Jelmer Vernooij
Add get_transport_from_url and get_transport_from_path functions. |
1663 |
raise errors.UnsupportedProtocol(url, last_err) |
1664 |
||
1665 |
||
1666 |
def get_transport(base, possible_transports=None): |
|
1667 |
"""Open a transport to access a URL or directory.
|
|
1668 |
||
1669 |
:param base: either a URL or a directory name.
|
|
1670 |
||
1671 |
:param transports: optional reusable transports list. If not None, created
|
|
1672 |
transports will be added to the list.
|
|
1673 |
||
1674 |
:return: A new transport optionally sharing its connection with one of
|
|
1675 |
possible_transports.
|
|
1676 |
"""
|
|
1677 |
if base is None: |
|
1678 |
base = '.' |
|
1679 |
return get_transport_from_url(location_to_url(base), possible_transports) |
|
2476.3.5
by Vincent Ladeuil
Naive implementation of transport reuse by Transport.get_transport(). |
1680 |
|
6030.2.5
by Jelmer Vernooij
Fix formatting. |
1681 |
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1682 |
def _try_transport_factories(base, factory_list): |
1683 |
last_err = None |
|
1684 |
for factory in factory_list: |
|
1685 |
try: |
|
1686 |
return factory.get_obj()(base), None |
|
1687 |
except errors.DependencyNotPresent, e: |
|
5436.3.3
by Martin
Minor changes from review and revert fiddling with mutter |
1688 |
mutter("failed to instantiate transport %r for %r: %r" % |
1689 |
(factory, base, e)) |
|
2485.8.37
by Vincent Ladeuil
Fix merge multiple connections. Test suite *not* passing (sftp |
1690 |
last_err = e |
1691 |
continue
|
|
1692 |
return None, last_err |
|
1693 |
||
1694 |
||
2164.2.22
by Vincent Ladeuil
Take Aaron's review comments into account. |
1695 |
def do_catching_redirections(action, transport, redirected): |
2164.2.21
by Vincent Ladeuil
Take bundles into account. |
1696 |
"""Execute an action with given transport catching redirections.
|
1697 |
||
1698 |
This is a facility provided for callers needing to follow redirections
|
|
2164.2.25
by Vincent Ladeuil
Fix typos noticed by Aaron. |
1699 |
silently. The silence is relative: it is the caller responsability to
|
2164.2.21
by Vincent Ladeuil
Take bundles into account. |
1700 |
inform the user about each redirection or only inform the user of a user
|
1701 |
via the exception parameter.
|
|
1702 |
||
1703 |
:param action: A callable, what the caller want to do while catching
|
|
1704 |
redirections.
|
|
1705 |
:param transport: The initial transport used.
|
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1706 |
:param redirected: A callable receiving the redirected transport and the
|
2164.2.21
by Vincent Ladeuil
Take bundles into account. |
1707 |
RedirectRequested exception.
|
1708 |
||
1709 |
:return: Whatever 'action' returns
|
|
1710 |
"""
|
|
1711 |
MAX_REDIRECTIONS = 8 |
|
1712 |
||
1713 |
# If a loop occurs, there is little we can do. So we don't try to detect
|
|
1714 |
# them, just getting out if too much redirections occurs. The solution
|
|
1715 |
# is outside: where the loop is defined.
|
|
1716 |
for redirections in range(MAX_REDIRECTIONS): |
|
1717 |
try: |
|
2164.2.22
by Vincent Ladeuil
Take Aaron's review comments into account. |
1718 |
return action(transport) |
2164.2.21
by Vincent Ladeuil
Take bundles into account. |
1719 |
except errors.RedirectRequested, e: |
1720 |
redirection_notice = '%s is%s redirected to %s' % ( |
|
2164.2.22
by Vincent Ladeuil
Take Aaron's review comments into account. |
1721 |
e.source, e.permanently, e.target) |
1722 |
transport = redirected(transport, e, redirection_notice) |
|
2164.2.21
by Vincent Ladeuil
Take bundles into account. |
1723 |
else: |
2164.2.22
by Vincent Ladeuil
Take Aaron's review comments into account. |
1724 |
# Loop exited without resolving redirect ? Either the
|
1725 |
# user has kept a very very very old reference or a loop
|
|
2164.2.25
by Vincent Ladeuil
Fix typos noticed by Aaron. |
1726 |
# occurred in the redirections. Nothing we can cure here:
|
2164.2.22
by Vincent Ladeuil
Take Aaron's review comments into account. |
1727 |
# tell the user. Note that as the user has been informed
|
1728 |
# about each redirection (it is the caller responsibility
|
|
1729 |
# to do that in redirected via the provided
|
|
1730 |
# redirection_notice). The caller may provide more
|
|
2164.2.25
by Vincent Ladeuil
Fix typos noticed by Aaron. |
1731 |
# information if needed (like what file or directory we
|
2164.2.22
by Vincent Ladeuil
Take Aaron's review comments into account. |
1732 |
# were trying to act upon when the redirection loop
|
2164.2.25
by Vincent Ladeuil
Fix typos noticed by Aaron. |
1733 |
# occurred).
|
2164.2.22
by Vincent Ladeuil
Take Aaron's review comments into account. |
1734 |
raise errors.TooManyRedirections |
2164.2.21
by Vincent Ladeuil
Take bundles into account. |
1735 |
|
1736 |
||
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1737 |
class Server(object): |
1530.1.21
by Robert Collins
Review feedback fixes. |
1738 |
"""A Transport Server.
|
3943.8.1
by Marius Kruger
remove all trailing whitespace from bzr source |
1739 |
|
5017.3.1
by Vincent Ladeuil
Create a tests.test_server.TestServer class out of transport.Server (while retaining the later for some special non-tests usages). |
1740 |
The Server interface provides a server for a given transport type.
|
1530.1.21
by Robert Collins
Review feedback fixes. |
1741 |
"""
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1742 |
|
4934.3.3
by Martin Pool
Rename Server.setUp to Server.start_server |
1743 |
def start_server(self): |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1744 |
"""Setup the server to service requests."""
|
1745 |
||
4934.3.1
by Martin Pool
Rename Server.tearDown to .stop_server |
1746 |
def stop_server(self): |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1747 |
"""Remove the server and cleanup any resources it owns."""
|
1748 |
||
1749 |
||
1185.16.81
by mbp at sourcefrog
[merge] robert |
1750 |
# None is the default transport, for things with no url scheme
|
2241.2.9
by ghigo
add the 'file://' prefix to the urlspec topic |
1751 |
register_transport_proto('file://', |
1752 |
help="Access using the standard filesystem (default)") |
|
1185.16.79
by Martin Pool
Load transports when they're first used. |
1753 |
register_lazy_transport('file://', 'bzrlib.transport.local', 'LocalTransport') |
2241.2.4
by ghigo
removed commented line |
1754 |
|
2241.2.5
by ghigo
add the topics transport |
1755 |
register_transport_proto('sftp://', |
2919.2.1
by John Arbash Meinel
Register netloc protocols as soon as the protocol is registered. |
1756 |
help="Access using SFTP (most SSH servers provide SFTP).", |
1757 |
register_netloc=True) |
|
1185.16.79
by Martin Pool
Load transports when they're first used. |
1758 |
register_lazy_transport('sftp://', 'bzrlib.transport.sftp', 'SFTPTransport') |
2164.2.7
by v.ladeuil+lp at free
First implementation of transport hints. |
1759 |
# Decorated http transport
|
2241.2.5
by ghigo
add the topics transport |
1760 |
register_transport_proto('http+urllib://', |
2241.2.11
by ghigo
On the basis of Robert Collins and John Arbash Meinel |
1761 |
# help="Read-only access of branches exported on the web."
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1762 |
register_netloc=True) |
1540.3.23
by Martin Pool
Allow urls like http+pycurl://host/ to use a particular impl |
1763 |
register_lazy_transport('http+urllib://', 'bzrlib.transport.http._urllib', |
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
1764 |
'HttpTransport_urllib') |
2241.2.5
by ghigo
add the topics transport |
1765 |
register_transport_proto('https+urllib://', |
2241.2.11
by ghigo
On the basis of Robert Collins and John Arbash Meinel |
1766 |
# help="Read-only access of branches exported on the web using SSL."
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1767 |
register_netloc=True) |
1540.3.23
by Martin Pool
Allow urls like http+pycurl://host/ to use a particular impl |
1768 |
register_lazy_transport('https+urllib://', 'bzrlib.transport.http._urllib', |
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
1769 |
'HttpTransport_urllib') |
2241.2.5
by ghigo
add the topics transport |
1770 |
register_transport_proto('http+pycurl://', |
2241.2.11
by ghigo
On the basis of Robert Collins and John Arbash Meinel |
1771 |
# help="Read-only access of branches exported on the web."
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1772 |
register_netloc=True) |
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
1773 |
register_lazy_transport('http+pycurl://', 'bzrlib.transport.http._pycurl', |
1774 |
'PyCurlTransport') |
|
2241.2.5
by ghigo
add the topics transport |
1775 |
register_transport_proto('https+pycurl://', |
2241.2.11
by ghigo
On the basis of Robert Collins and John Arbash Meinel |
1776 |
# help="Read-only access of branches exported on the web using SSL."
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1777 |
register_netloc=True) |
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
1778 |
register_lazy_transport('https+pycurl://', 'bzrlib.transport.http._pycurl', |
1779 |
'PyCurlTransport') |
|
2164.2.7
by v.ladeuil+lp at free
First implementation of transport hints. |
1780 |
# Default http transports (last declared wins (if it can be imported))
|
2241.2.5
by ghigo
add the topics transport |
1781 |
register_transport_proto('http://', |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1782 |
help="Read-only access of branches exported on the web.") |
2241.2.5
by ghigo
add the topics transport |
1783 |
register_transport_proto('https://', |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1784 |
help="Read-only access of branches exported on the web using SSL.") |
3882.2.1
by John Arbash Meinel
Use urllib for http connections, pycurl for https connections. |
1785 |
# The default http implementation is urllib, but https is pycurl if available
|
1786 |
register_lazy_transport('http://', 'bzrlib.transport.http._pycurl', |
|
1787 |
'PyCurlTransport') |
|
1540.3.26
by Martin Pool
[merge] bzr.dev; pycurl not updated for readv yet |
1788 |
register_lazy_transport('http://', 'bzrlib.transport.http._urllib', |
1789 |
'HttpTransport_urllib') |
|
1790 |
register_lazy_transport('https://', 'bzrlib.transport.http._urllib', |
|
1791 |
'HttpTransport_urllib') |
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1792 |
register_lazy_transport('https://', 'bzrlib.transport.http._pycurl', |
1793 |
'PyCurlTransport') |
|
2241.2.4
by ghigo
removed commented line |
1794 |
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1795 |
register_transport_proto('ftp://', help="Access using passive FTP.") |
1185.36.4
by Daniel Silverstone
Add FTP transport |
1796 |
register_lazy_transport('ftp://', 'bzrlib.transport.ftp', 'FtpTransport') |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1797 |
register_transport_proto('aftp://', help="Access using active FTP.") |
1185.36.4
by Daniel Silverstone
Add FTP transport |
1798 |
register_lazy_transport('aftp://', 'bzrlib.transport.ftp', 'FtpTransport') |
5270.1.3
by Jelmer Vernooij
Remove unused imports, lazily load kerberos python module. |
1799 |
register_transport_proto('gio+', help="Access using any GIO supported protocols.") |
1800 |
register_lazy_transport('gio+', 'bzrlib.transport.gio_transport', 'GioTransport') |
|
1801 |
||
1802 |
||
1803 |
# Default to trying GSSAPI authentication (if the kerberos module is
|
|
1804 |
# available)
|
|
5270.1.1
by Jelmer Vernooij
Always register GSSAPI proto's, even if we don't have an implementation that uses them. |
1805 |
register_transport_proto('ftp+gssapi://', register_netloc=True) |
1806 |
register_transport_proto('aftp+gssapi://', register_netloc=True) |
|
1807 |
register_transport_proto('ftp+nogssapi://', register_netloc=True) |
|
1808 |
register_transport_proto('aftp+nogssapi://', register_netloc=True) |
|
5270.1.3
by Jelmer Vernooij
Remove unused imports, lazily load kerberos python module. |
1809 |
register_lazy_transport('ftp+gssapi://', 'bzrlib.transport.ftp._gssapi', |
1810 |
'GSSAPIFtpTransport') |
|
1811 |
register_lazy_transport('aftp+gssapi://', 'bzrlib.transport.ftp._gssapi', |
|
1812 |
'GSSAPIFtpTransport') |
|
1813 |
register_lazy_transport('ftp://', 'bzrlib.transport.ftp._gssapi', |
|
1814 |
'GSSAPIFtpTransport') |
|
1815 |
register_lazy_transport('aftp://', 'bzrlib.transport.ftp._gssapi', |
|
1816 |
'GSSAPIFtpTransport') |
|
1817 |
register_lazy_transport('ftp+nogssapi://', 'bzrlib.transport.ftp', |
|
1818 |
'FtpTransport') |
|
1819 |
register_lazy_transport('aftp+nogssapi://', 'bzrlib.transport.ftp', |
|
1820 |
'FtpTransport') |
|
3331.2.4
by Jelmer Vernooij
Move GSSAPI support to a separate file. |
1821 |
|
2241.2.1
by ghigo
Add the TransportRegistry class |
1822 |
register_transport_proto('memory://') |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1823 |
register_lazy_transport('memory://', 'bzrlib.transport.memory', |
1824 |
'MemoryTransport') |
|
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
1825 |
|
2241.2.5
by ghigo
add the topics transport |
1826 |
register_transport_proto('readonly+', |
2241.2.11
by ghigo
On the basis of Robert Collins and John Arbash Meinel |
1827 |
# help="This modifier converts any transport to be readonly."
|
1828 |
)
|
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1829 |
register_lazy_transport('readonly+', 'bzrlib.transport.readonly', |
1830 |
'ReadonlyTransportDecorator') |
|
2555.3.8
by Martin Pool
Add new BrokenRenameTransportDecorator |
1831 |
|
2617.4.5
by Robert Collins
Reinstate the fakenfs+ decorator registration. |
1832 |
register_transport_proto('fakenfs+') |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1833 |
register_lazy_transport('fakenfs+', 'bzrlib.transport.fakenfs', |
1834 |
'FakeNFSTransportDecorator') |
|
2617.4.5
by Robert Collins
Reinstate the fakenfs+ decorator registration. |
1835 |
|
3675.1.1
by Martin Pool
Merge and update log+ transport decorator |
1836 |
register_transport_proto('log+') |
1837 |
register_lazy_transport('log+', 'bzrlib.transport.log', 'TransportLogDecorator') |
|
1838 |
||
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
1839 |
register_transport_proto('trace+') |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1840 |
register_lazy_transport('trace+', 'bzrlib.transport.trace', |
1841 |
'TransportTraceDecorator') |
|
2745.5.3
by Robert Collins
* Move transport logging into a new transport class |
1842 |
|
2617.4.1
by Robert Collins
Add a new transport decorator unlistable+ for testing. |
1843 |
register_transport_proto('unlistable+') |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1844 |
register_lazy_transport('unlistable+', 'bzrlib.transport.unlistable', |
1845 |
'UnlistableTransportDecorator') |
|
2555.3.8
by Martin Pool
Add new BrokenRenameTransportDecorator |
1846 |
|
1847 |
register_transport_proto('brokenrename+') |
|
1848 |
register_lazy_transport('brokenrename+', 'bzrlib.transport.brokenrename', |
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1849 |
'BrokenRenameTransportDecorator') |
2555.3.8
by Martin Pool
Add new BrokenRenameTransportDecorator |
1850 |
|
2241.2.1
by ghigo
Add the TransportRegistry class |
1851 |
register_transport_proto('vfat+') |
1910.19.1
by Andrew Bennetts
Support bzr:// urls to work with the new RPC-based transport which will be used |
1852 |
register_lazy_transport('vfat+', |
1608.2.4
by Martin Pool
[broken] Add FakeFVATTransport |
1853 |
'bzrlib.transport.fakevfat', |
1854 |
'FakeVFATTransportDecorator') |
|
2919.2.1
by John Arbash Meinel
Register netloc protocols as soon as the protocol is registered. |
1855 |
|
3313.3.4
by Andrew Bennetts
Add a 'nosmart+' transport decorator. |
1856 |
register_transport_proto('nosmart+') |
1857 |
register_lazy_transport('nosmart+', 'bzrlib.transport.nosmart', |
|
1858 |
'NoSmartTransportDecorator') |
|
1859 |
||
2241.2.5
by ghigo
add the topics transport |
1860 |
register_transport_proto('bzr://', |
2811.1.1
by Andrew Bennetts
Cherrypick fix proposed for 0.90. |
1861 |
help="Fast access using the Bazaar smart server.", |
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1862 |
register_netloc=True) |
2241.2.5
by ghigo
add the topics transport |
1863 |
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1864 |
register_lazy_transport('bzr://', 'bzrlib.transport.remote', |
2413.2.1
by Andrew Bennetts
Rename Smart.*Transport classes to RemoteTransport, RemoteTCPTransport, etc. |
1865 |
'RemoteTCPTransport') |
3453.5.1
by Andrew Bennetts
Add {bzrdir,repository,branch}_implementations tests for Remote objects using protocol v2 and pre-1.6 RPCs. |
1866 |
register_transport_proto('bzr-v2://', register_netloc=True) |
1867 |
||
1868 |
register_lazy_transport('bzr-v2://', 'bzrlib.transport.remote', |
|
1869 |
'RemoteTCPTransportV2Only') |
|
2241.2.5
by ghigo
add the topics transport |
1870 |
register_transport_proto('bzr+http://', |
2241.2.11
by ghigo
On the basis of Robert Collins and John Arbash Meinel |
1871 |
# help="Fast access using the Bazaar smart server over HTTP."
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1872 |
register_netloc=True) |
1873 |
register_lazy_transport('bzr+http://', 'bzrlib.transport.remote', |
|
2413.2.1
by Andrew Bennetts
Rename Smart.*Transport classes to RemoteTransport, RemoteTCPTransport, etc. |
1874 |
'RemoteHTTPTransport') |
2651.1.1
by John Ferlito
bzr+https support |
1875 |
register_transport_proto('bzr+https://', |
1876 |
# help="Fast access using the Bazaar smart server over HTTPS."
|
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1877 |
register_netloc=True) |
2651.1.1
by John Ferlito
bzr+https support |
1878 |
register_lazy_transport('bzr+https://', |
1879 |
'bzrlib.transport.remote', |
|
1880 |
'RemoteHTTPTransport') |
|
2241.2.5
by ghigo
add the topics transport |
1881 |
register_transport_proto('bzr+ssh://', |
2919.2.1
by John Arbash Meinel
Register netloc protocols as soon as the protocol is registered. |
1882 |
help="Fast access using the Bazaar smart server over SSH.", |
1883 |
register_netloc=True) |
|
3004.2.1
by Vincent Ladeuil
Fix 150860 by leaving port as user specified it. |
1884 |
register_lazy_transport('bzr+ssh://', 'bzrlib.transport.remote', |
2413.2.1
by Andrew Bennetts
Rename Smart.*Transport classes to RemoteTransport, RemoteTCPTransport, etc. |
1885 |
'RemoteSSHTransport') |
4011.4.1
by Jelmer Vernooij
Point out bzr+ssh:// to the user when they use ssh://. |
1886 |
|
4011.4.2
by Jelmer Vernooij
Register ssh: rather than ssh://, avoid referring to smart server, which users may not be familiar with. |
1887 |
register_transport_proto('ssh:') |
4032.1.2
by John Arbash Meinel
Track down a few more files that have trailing whitespace. |
1888 |
register_lazy_transport('ssh:', 'bzrlib.transport.remote', |
4011.4.1
by Jelmer Vernooij
Point out bzr+ssh:// to the user when they use ssh://. |
1889 |
'HintingSSHTransport') |
4370.4.6
by Jelmer Vernooij
Move server protocol registry to bzrlib.transport. |
1890 |
|
1891 |
||
1892 |
transport_server_registry = registry.Registry() |
|
5056.1.4
by Neil Santos
Changed stat() in SFTPTransport and LocalTransport back to calling plain stat(), instead of lstat(). |
1893 |
transport_server_registry.register_lazy('bzr', 'bzrlib.smart.server', |
6259.2.11
by Martin Packman
Revert addition of redundant full stops after brackets |
1894 |
'serve_bzr', help="The Bazaar smart server protocol over TCP. (default port: 4155)") |
4370.4.6
by Jelmer Vernooij
Move server protocol registry to bzrlib.transport. |
1895 |
transport_server_registry.default_key = 'bzr' |