1662.1.12
by Martin Pool
Translate unknown sftp errors to PathError, no NoSuchFile |
1 |
# Copyright (C) 2005 Robey Pointer <robey@lag.net>
|
2221.5.1
by Dmitry Vasiliev
Added support for Putty's SSH implementation |
2 |
# Copyright (C) 2005, 2006, 2007 Canonical Ltd
|
1887.1.1
by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines, |
3 |
#
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
4 |
# This program is free software; you can redistribute it and/or modify
|
5 |
# it under the terms of the GNU General Public License as published by
|
|
6 |
# the Free Software Foundation; either version 2 of the License, or
|
|
7 |
# (at your option) any later version.
|
|
1887.1.1
by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines, |
8 |
#
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
9 |
# This program is distributed in the hope that it will be useful,
|
10 |
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
11 |
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
12 |
# GNU General Public License for more details.
|
|
1887.1.1
by Adeodato Simó
Do not separate paragraphs in the copyright statement with blank lines, |
13 |
#
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
14 |
# You should have received a copy of the GNU General Public License
|
15 |
# along with this program; if not, write to the Free Software
|
|
16 |
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
|
17 |
||
18 |
"""Implementation of Transport over SFTP, using paramiko."""
|
|
19 |
||
1910.7.17
by Andrew Bennetts
Various cosmetic changes. |
20 |
# TODO: Remove the transport-based lock_read and lock_write methods. They'll
|
21 |
# then raise TransportNotPossible, which will break remote access to any
|
|
22 |
# formats which rely on OS-level locks. That should be fine as those formats
|
|
23 |
# are pretty old, but these combinations may have to be removed from the test
|
|
1910.7.19
by Andrew Bennetts
Expand comment about lock_read/lock_write. |
24 |
# suite. Those formats all date back to 0.7; so we should be able to remove
|
25 |
# these methods when we officially drop support for those formats.
|
|
1910.7.17
by Andrew Bennetts
Various cosmetic changes. |
26 |
|
1489
by Robert Collins
Make the paramiko tests pass. Nice huh. |
27 |
import errno |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
28 |
import os |
1540.2.2
by Röbey Pointer
allow forcing the use of paramiko via environ var; use prefetch on paramiko >= 1.5.2 |
29 |
import random |
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
30 |
import select |
1806.1.1
by Robey Pointer
absorb a broken-pipe exception from paramiko when running the unit tests. it's okay for the client to vanish abruptly and paramiko really ought to mask the exception itself. |
31 |
import socket |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
32 |
import stat |
33 |
import sys |
|
1540.2.2
by Röbey Pointer
allow forcing the use of paramiko via environ var; use prefetch on paramiko >= 1.5.2 |
34 |
import time |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
35 |
import urllib |
1185.48.1
by James Henstridge
Use /usr/bin/ssh if we can. |
36 |
import urlparse |
2711.1.1
by Martin Pool
Suppress warning about Paramiko use of struct.pack |
37 |
import warnings |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
38 |
|
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
39 |
from bzrlib import ( |
40 |
errors, |
|
41 |
urlutils, |
|
42 |
)
|
|
1996.2.1
by Andrew Bennetts
Fix NameError reported by Alexander Belchenko. |
43 |
from bzrlib.errors import (FileExists, |
1951.1.6
by Andrew Bennetts
Tidy up some unused imports. |
44 |
NoSuchFile, PathNotChild, |
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
45 |
TransportError, |
1996.2.1
by Andrew Bennetts
Fix NameError reported by Alexander Belchenko. |
46 |
LockError, |
1662.1.12
by Martin Pool
Translate unknown sftp errors to PathError, no NoSuchFile |
47 |
PathError, |
48 |
ParamikoNotPresent, |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
49 |
)
|
1711.5.1
by John Arbash Meinel
Get most SFTP tests to pass. StubSFTPServer now talks the same path protocol that SFTPTransport talks. on win32 |
50 |
from bzrlib.osutils import pathjoin, fancy_rename, getcwd |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
51 |
from bzrlib.symbol_versioning import ( |
52 |
deprecated_function, |
|
2687.2.1
by Martin Pool
Rename upcoming release from 0.19 to 0.90 |
53 |
zero_ninety, |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
54 |
)
|
1951.1.6
by Andrew Bennetts
Tidy up some unused imports. |
55 |
from bzrlib.trace import mutter, warning |
1636.1.2
by Robert Collins
More review fixen to the relpath at '/' fixes. |
56 |
from bzrlib.transport import ( |
2671.3.6
by Robert Collins
Review feedback. |
57 |
FileFileStream, |
2671.3.2
by Robert Collins
Start open_file_stream logic. |
58 |
_file_streams, |
2018.5.114
by Robert Collins
Commit current test pass improvements. |
59 |
local, |
1636.1.2
by Robert Collins
More review fixen to the relpath at '/' fixes. |
60 |
Server, |
1951.1.4
by Andrew Bennetts
Start moving SSH connection code into bzrlib/transport/ssh.py |
61 |
ssh, |
2485.8.16
by Vincent Ladeuil
Create a new, empty, ConnectedTransport class. |
62 |
ConnectedTransport, |
1636.1.2
by Robert Collins
More review fixen to the relpath at '/' fixes. |
63 |
)
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
64 |
|
2711.1.1
by Martin Pool
Suppress warning about Paramiko use of struct.pack |
65 |
# Disable one particular warning that comes from paramiko in Python2.5; if
|
66 |
# this is emitted at the wrong time it tends to cause spurious test failures
|
|
67 |
# or at least noise in the test case::
|
|
68 |
#
|
|
69 |
# [1770/7639 in 86s, 1 known failures, 50 skipped, 2 missing features]
|
|
70 |
# test_permissions.TestSftpPermissions.test_new_files
|
|
71 |
# /var/lib/python-support/python2.5/paramiko/message.py:226: DeprecationWarning: integer argument expected, got float
|
|
72 |
# self.packet.write(struct.pack('>I', n))
|
|
73 |
warnings.filterwarnings('ignore', |
|
74 |
'integer argument expected, got float', |
|
75 |
category=DeprecationWarning, |
|
76 |
module='paramiko.message') |
|
77 |
||
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
78 |
try: |
79 |
import paramiko |
|
1185.62.24
by John Arbash Meinel
Changing the exception that sftp.py throws when it can't find paramiko, so that the test suite can handle it. |
80 |
except ImportError, e: |
81 |
raise ParamikoNotPresent(e) |
|
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
82 |
else: |
83 |
from paramiko.sftp import (SFTP_FLAG_WRITE, SFTP_FLAG_CREATE, |
|
84 |
SFTP_FLAG_EXCL, SFTP_FLAG_TRUNC, |
|
85 |
CMD_HANDLE, CMD_OPEN) |
|
86 |
from paramiko.sftp_attr import SFTPAttributes |
|
87 |
from paramiko.sftp_file import SFTPFile |
|
1636.1.1
by Robert Collins
Fix calling relpath() and abspath() on transports at their root. |
88 |
|
1883.1.3
by Stefan (metze) Metzmacher
- fix typo |
89 |
|
1711.6.2
by John Arbash Meinel
Clean up paramiko agent support. |
90 |
_paramiko_version = getattr(paramiko, '__version_info__', (0, 0, 0)) |
91 |
# don't use prefetch unless paramiko version >= 1.5.5 (there were bugs earlier)
|
|
92 |
_default_do_prefetch = (_paramiko_version >= (1, 5, 5)) |
|
1540.2.3
by Robey Pointer
move the paramiko version check up to the top and use getattr instead of hasattr. use BZR_SSH instead of BZR_USE_PARAMIKO, to allow overriding the ssh vendor check in a general way. |
93 |
|
1185.48.1
by James Henstridge
Use /usr/bin/ssh if we can. |
94 |
|
2687.2.1
by Martin Pool
Rename upcoming release from 0.19 to 0.90 |
95 |
@deprecated_function(zero_ninety) |
1607.1.13
by Robert Collins
Add a clear_connection_cache to the SFTP transport and use it in fixing the bound branch test speed performance problem which was cause by the server thread timing out - it was blocked on a read and closing the client side causes the server to unblock and exit. |
96 |
def clear_connection_cache(): |
97 |
"""Remove all hosts from the SFTP connection cache.
|
|
98 |
||
99 |
Primarily useful for test cases wanting to force garbage collection.
|
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
100 |
We don't have a global connection cache anymore.
|
1607.1.13
by Robert Collins
Add a clear_connection_cache to the SFTP transport and use it in fixing the bound branch test speed performance problem which was cause by the server thread timing out - it was blocked on a read and closing the client side causes the server to unblock and exit. |
101 |
"""
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
102 |
|
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
103 |
class SFTPLock(object): |
1910.7.17
by Andrew Bennetts
Various cosmetic changes. |
104 |
"""This fakes a lock in a remote location.
|
105 |
|
|
106 |
A present lock is indicated just by the existence of a file. This
|
|
107 |
doesn't work well on all transports and they are only used in
|
|
108 |
deprecated storage formats.
|
|
109 |
"""
|
|
110 |
||
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
111 |
__slots__ = ['path', 'lock_path', 'lock_file', 'transport'] |
1910.7.17
by Andrew Bennetts
Various cosmetic changes. |
112 |
|
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
113 |
def __init__(self, path, transport): |
114 |
assert isinstance(transport, SFTPTransport) |
|
115 |
||
116 |
self.lock_file = None |
|
117 |
self.path = path |
|
118 |
self.lock_path = path + '.write-lock' |
|
119 |
self.transport = transport |
|
120 |
try: |
|
1530.1.7
by Robert Collins
merge integration. |
121 |
# RBC 20060103 FIXME should we be using private methods here ?
|
122 |
abspath = transport._remote_path(self.lock_path) |
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
123 |
self.lock_file = transport._sftp_open_exclusive(abspath) |
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
124 |
except FileExists: |
125 |
raise LockError('File %r already locked' % (self.path,)) |
|
126 |
||
127 |
def __del__(self): |
|
128 |
"""Should this warn, or actually try to cleanup?"""
|
|
129 |
if self.lock_file: |
|
1547.1.3
by Robey Pointer
spell warning correctly :) and move recv_ready close to recv for readability |
130 |
warning("SFTPLock %r not explicitly unlocked" % (self.path,)) |
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
131 |
self.unlock() |
132 |
||
133 |
def unlock(self): |
|
134 |
if not self.lock_file: |
|
135 |
return
|
|
136 |
self.lock_file.close() |
|
137 |
self.lock_file = None |
|
138 |
try: |
|
139 |
self.transport.delete(self.lock_path) |
|
140 |
except (NoSuchFile,): |
|
141 |
# What specific errors should we catch here?
|
|
142 |
pass
|
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
143 |
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
144 |
|
2485.8.27
by Vincent Ladeuil
Hearing jam saying "vila, you're trying too hard", I simplified again. |
145 |
class SFTPTransport(ConnectedTransport): |
1752.2.40
by Andrew Bennetts
Merge from bzr.dev. |
146 |
"""Transport implementation for SFTP access."""
|
1864.5.2
by John Arbash Meinel
always read in sorted order, and return in requested order, but only cache what is currently out of order |
147 |
|
1540.2.3
by Robey Pointer
move the paramiko version check up to the top and use getattr instead of hasattr. use BZR_SSH instead of BZR_USE_PARAMIKO, to allow overriding the ssh vendor check in a general way. |
148 |
_do_prefetch = _default_do_prefetch |
1864.5.6
by John Arbash Meinel
Setting defaults based on benchmarking data. |
149 |
# TODO: jam 20060717 Conceivably these could be configurable, either
|
150 |
# by auto-tuning at run-time, or by a configuration (per host??)
|
|
151 |
# but the performance curve is pretty flat, so just going with
|
|
152 |
# reasonable defaults.
|
|
1864.5.18
by John Arbash Meinel
Remove benchmarking stipple. |
153 |
_max_readv_combine = 200 |
1864.5.6
by John Arbash Meinel
Setting defaults based on benchmarking data. |
154 |
# Having to round trip to the server means waiting for a response,
|
155 |
# so it is better to download extra bytes.
|
|
156 |
# 8KiB had good performance for both local and remote network operations
|
|
1864.5.18
by John Arbash Meinel
Remove benchmarking stipple. |
157 |
_bytes_to_read_before_seek = 8192 |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
158 |
|
1864.5.20
by John Arbash Meinel
Some small cleanups. |
159 |
# The sftp spec says that implementations SHOULD allow reads
|
160 |
# to be at least 32K. paramiko.readv() does an async request
|
|
161 |
# for the chunks. So we need to keep it within a single request
|
|
162 |
# size for paramiko <= 1.6.1. paramiko 1.6.2 will probably chop
|
|
163 |
# up the request itself, rather than us having to worry about it
|
|
164 |
_max_request_size = 32768 |
|
1864.5.17
by John Arbash Meinel
A little cleanup, and restrict to 32K to conform better to sftp spec |
165 |
|
2485.8.59
by Vincent Ladeuil
Update from review comments. |
166 |
def __init__(self, base, _from_transport=None): |
2485.8.23
by Vincent Ladeuil
Assert the accepted schemes for sftp and ftp. |
167 |
assert base.startswith('sftp://') |
2485.8.59
by Vincent Ladeuil
Update from review comments. |
168 |
super(SFTPTransport, self).__init__(base, |
169 |
_from_transport=_from_transport) |
|
2485.8.27
by Vincent Ladeuil
Hearing jam saying "vila, you're trying too hard", I simplified again. |
170 |
|
171 |
def _remote_path(self, relpath): |
|
172 |
"""Return the path to be passed along the sftp protocol for relpath.
|
|
173 |
|
|
174 |
:param relpath: is a urlencoded string.
|
|
175 |
"""
|
|
176 |
relative = urlutils.unescape(relpath).encode('utf-8') |
|
177 |
remote_path = self._combine_paths(self._path, relative) |
|
178 |
# the initial slash should be removed from the path, and treated as a
|
|
179 |
# homedir relative path (the path begins with a double slash if it is
|
|
180 |
# absolute). see draft-ietf-secsh-scp-sftp-ssh-uri-03.txt
|
|
181 |
# RBC 20060118 we are not using this as its too user hostile. instead
|
|
182 |
# we are following lftp and using /~/foo to mean '~/foo'
|
|
183 |
# vila--20070602 and leave absolute paths begin with a single slash.
|
|
184 |
if remote_path.startswith('/~/'): |
|
185 |
remote_path = remote_path[3:] |
|
186 |
elif remote_path == '/~': |
|
187 |
remote_path = '' |
|
188 |
return remote_path |
|
189 |
||
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
190 |
def _create_connection(self, credentials=None): |
191 |
"""Create a new connection with the provided credentials.
|
|
192 |
||
193 |
:param credentials: The credentials needed to establish the connection.
|
|
194 |
||
195 |
:return: The created connection and its associated credentials.
|
|
196 |
||
197 |
The credentials are only the password as it may have been entered
|
|
198 |
interactively by the user and may be different from the one provided
|
|
199 |
in base url at transport creation time.
|
|
200 |
"""
|
|
201 |
if credentials is None: |
|
202 |
password = self._password |
|
203 |
else: |
|
204 |
password = credentials |
|
205 |
||
206 |
vendor = ssh._get_ssh_vendor() |
|
207 |
connection = vendor.connect_sftp(self._user, password, |
|
208 |
self._host, self._port) |
|
209 |
return connection, password |
|
210 |
||
211 |
def _get_sftp(self): |
|
212 |
"""Ensures that a connection is established"""
|
|
213 |
connection = self._get_connection() |
|
214 |
if connection is None: |
|
215 |
# First connection ever
|
|
216 |
connection, credentials = self._create_connection() |
|
217 |
self._set_connection(connection, credentials) |
|
218 |
return connection |
|
219 |
||
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
220 |
def has(self, relpath): |
221 |
"""
|
|
222 |
Does the target location exist?
|
|
223 |
"""
|
|
224 |
try: |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
225 |
self._get_sftp().stat(self._remote_path(relpath)) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
226 |
return True |
227 |
except IOError: |
|
228 |
return False |
|
229 |
||
2164.2.15
by Vincent Ladeuil
Http redirections are not followed by default. Do not use hints |
230 |
def get(self, relpath): |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
231 |
"""
|
232 |
Get the file at the given relative path.
|
|
233 |
||
234 |
:param relpath: The relative path to the file
|
|
235 |
"""
|
|
236 |
try: |
|
1524.1.1
by Robert Collins
Test sftp with relative, absolute-in-homedir and absolute-not-in-homedir |
237 |
path = self._remote_path(relpath) |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
238 |
f = self._get_sftp().file(path, mode='rb') |
1540.2.3
by Robey Pointer
move the paramiko version check up to the top and use getattr instead of hasattr. use BZR_SSH instead of BZR_USE_PARAMIKO, to allow overriding the ssh vendor check in a general way. |
239 |
if self._do_prefetch and (getattr(f, 'prefetch', None) is not None): |
1185.40.1
by Robey Pointer
prefetch files under paramiko 1.5.1 for improved speed |
240 |
f.prefetch() |
241 |
return f |
|
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
242 |
except (IOError, paramiko.SSHException), e: |
2052.6.1
by Robert Collins
``Transport.get`` has had its interface made more clear for ease of use. |
243 |
self._translate_io_exception(e, path, ': error retrieving', |
2052.6.4
by Robert Collins
Tidier SFTP code (feedback from Aaron). |
244 |
failure_exc=errors.ReadError) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
245 |
|
2745.5.1
by Robert Collins
* New parameter on ``bzrlib.transport.Transport.readv`` |
246 |
def _readv(self, relpath, offsets): |
1864.5.11
by John Arbash Meinel
Factor out the workhorse of Transport.readv() into a helper function, and re-use that function in sftp but with a non-prefetch file. |
247 |
"""See Transport.readv()"""
|
248 |
# We overload the default readv() because we want to use a file
|
|
249 |
# that does not have prefetch enabled.
|
|
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
250 |
# Also, if we have a new paramiko, it implements an async readv()
|
1864.5.11
by John Arbash Meinel
Factor out the workhorse of Transport.readv() into a helper function, and re-use that function in sftp but with a non-prefetch file. |
251 |
if not offsets: |
252 |
return
|
|
253 |
||
254 |
try: |
|
255 |
path = self._remote_path(relpath) |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
256 |
fp = self._get_sftp().file(path, mode='rb') |
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
257 |
readv = getattr(fp, 'readv', None) |
1864.5.18
by John Arbash Meinel
Remove benchmarking stipple. |
258 |
if readv: |
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
259 |
return self._sftp_readv(fp, offsets, relpath) |
1864.5.16
by John Arbash Meinel
instrument to enable/disable readv support for testing |
260 |
mutter('seek and read %s offsets', len(offsets)) |
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
261 |
return self._seek_and_read(fp, offsets, relpath) |
1864.5.11
by John Arbash Meinel
Factor out the workhorse of Transport.readv() into a helper function, and re-use that function in sftp but with a non-prefetch file. |
262 |
except (IOError, paramiko.SSHException), e: |
263 |
self._translate_io_exception(e, path, ': error retrieving') |
|
264 |
||
2671.3.1
by Robert Collins
* New method ``bzrlib.transport.Transport.get_recommended_page_size``. |
265 |
def recommended_page_size(self): |
266 |
"""See Transport.recommended_page_size().
|
|
267 |
||
268 |
For SFTP we suggest a large page size to reduce the overhead
|
|
269 |
introduced by latency.
|
|
270 |
"""
|
|
271 |
return 64 * 1024 |
|
272 |
||
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
273 |
def _sftp_readv(self, fp, offsets, relpath='<unknown>'): |
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
274 |
"""Use the readv() member of fp to do async readv.
|
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
275 |
|
276 |
And then read them using paramiko.readv(). paramiko.readv()
|
|
277 |
does not support ranges > 64K, so it caps the request size, and
|
|
278 |
just reads until it gets all the stuff it wants
|
|
279 |
"""
|
|
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
280 |
offsets = list(offsets) |
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
281 |
sorted_offsets = sorted(offsets) |
282 |
||
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
283 |
# The algorithm works as follows:
|
284 |
# 1) Coalesce nearby reads into a single chunk
|
|
285 |
# This generates a list of combined regions, the total size
|
|
286 |
# and the size of the sub regions. This coalescing step is limited
|
|
287 |
# in the number of nearby chunks to combine, and is allowed to
|
|
288 |
# skip small breaks in the requests. Limiting it makes sure that
|
|
289 |
# we can start yielding some data earlier, and skipping means we
|
|
290 |
# make fewer requests. (Beneficial even when using async)
|
|
291 |
# 2) Break up this combined regions into chunks that are smaller
|
|
292 |
# than 64KiB. Technically the limit is 65536, but we are a
|
|
293 |
# little bit conservative. This is because sftp has a maximum
|
|
294 |
# return chunk size of 64KiB (max size of an unsigned short)
|
|
295 |
# 3) Issue a readv() to paramiko to create an async request for
|
|
296 |
# all of this data
|
|
297 |
# 4) Read in the data as it comes back, until we've read one
|
|
298 |
# continuous section as determined in step 1
|
|
299 |
# 5) Break up the full sections into hunks for the original requested
|
|
300 |
# offsets. And put them in a cache
|
|
301 |
# 6) Check if the next request is in the cache, and if it is, remove
|
|
302 |
# it from the cache, and yield its data. Continue until no more
|
|
303 |
# entries are in the cache.
|
|
304 |
# 7) loop back to step 4 until all data has been read
|
|
1864.5.16
by John Arbash Meinel
instrument to enable/disable readv support for testing |
305 |
#
|
306 |
# TODO: jam 20060725 This could be optimized one step further, by
|
|
307 |
# attempting to yield whatever data we have read, even before
|
|
1864.5.20
by John Arbash Meinel
Some small cleanups. |
308 |
# the first coallesced section has been fully processed.
|
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
309 |
|
1864.5.17
by John Arbash Meinel
A little cleanup, and restrict to 32K to conform better to sftp spec |
310 |
# When coalescing for use with readv(), we don't really need to
|
311 |
# use any fudge factor, because the requests are made asynchronously
|
|
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
312 |
coalesced = list(self._coalesce_offsets(sorted_offsets, |
313 |
limit=self._max_readv_combine, |
|
1864.5.17
by John Arbash Meinel
A little cleanup, and restrict to 32K to conform better to sftp spec |
314 |
fudge_factor=0, |
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
315 |
))
|
316 |
requests = [] |
|
317 |
for c_offset in coalesced: |
|
318 |
start = c_offset.start |
|
1864.5.17
by John Arbash Meinel
A little cleanup, and restrict to 32K to conform better to sftp spec |
319 |
size = c_offset.length |
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
320 |
|
321 |
# We need to break this up into multiple requests
|
|
1864.5.17
by John Arbash Meinel
A little cleanup, and restrict to 32K to conform better to sftp spec |
322 |
while size > 0: |
1864.5.20
by John Arbash Meinel
Some small cleanups. |
323 |
next_size = min(size, self._max_request_size) |
1864.5.17
by John Arbash Meinel
A little cleanup, and restrict to 32K to conform better to sftp spec |
324 |
requests.append((start, next_size)) |
325 |
size -= next_size |
|
326 |
start += next_size |
|
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
327 |
|
1864.5.16
by John Arbash Meinel
instrument to enable/disable readv support for testing |
328 |
mutter('SFTP.readv() %s offsets => %s coalesced => %s requests', |
329 |
len(offsets), len(coalesced), len(requests)) |
|
330 |
||
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
331 |
# Queue the current read until we have read the full coalesced section
|
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
332 |
cur_data = [] |
333 |
cur_data_len = 0 |
|
334 |
cur_coalesced_stack = iter(coalesced) |
|
335 |
cur_coalesced = cur_coalesced_stack.next() |
|
336 |
||
1864.5.15
by John Arbash Meinel
Cleanup and document the new sftp readv implementation |
337 |
# Cache the results, but only until they have been fulfilled
|
338 |
data_map = {} |
|
339 |
# turn the list of offsets into a stack
|
|
340 |
offset_stack = iter(offsets) |
|
341 |
cur_offset_and_size = offset_stack.next() |
|
342 |
||
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
343 |
for data in fp.readv(requests): |
344 |
cur_data += data |
|
345 |
cur_data_len += len(data) |
|
346 |
||
347 |
if cur_data_len < cur_coalesced.length: |
|
348 |
continue
|
|
349 |
assert cur_data_len == cur_coalesced.length, \ |
|
350 |
"Somehow we read too much: %s != %s" % (cur_data_len, |
|
351 |
cur_coalesced.length) |
|
1864.5.20
by John Arbash Meinel
Some small cleanups. |
352 |
all_data = ''.join(cur_data) |
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
353 |
cur_data = [] |
354 |
cur_data_len = 0 |
|
355 |
||
356 |
for suboffset, subsize in cur_coalesced.ranges: |
|
357 |
key = (cur_coalesced.start+suboffset, subsize) |
|
1864.5.20
by John Arbash Meinel
Some small cleanups. |
358 |
data_map[key] = all_data[suboffset:suboffset+subsize] |
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
359 |
|
360 |
# Now that we've read some data, see if we can yield anything back
|
|
361 |
while cur_offset_and_size in data_map: |
|
362 |
this_data = data_map.pop(cur_offset_and_size) |
|
363 |
yield cur_offset_and_size[0], this_data |
|
364 |
cur_offset_and_size = offset_stack.next() |
|
365 |
||
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
366 |
# We read a coalesced entry, so mark it as done
|
367 |
cur_coalesced = None |
|
1864.5.13
by John Arbash Meinel
Try using paramiko.readv() |
368 |
# Now that we've read all of the data for this coalesced section
|
369 |
# on to the next
|
|
370 |
cur_coalesced = cur_coalesced_stack.next() |
|
1864.5.12
by John Arbash Meinel
Using a simple sftp_file.readv() request if possible. |
371 |
|
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
372 |
if cur_coalesced is not None: |
373 |
raise errors.ShortReadvError(relpath, cur_coalesced.start, |
|
2001.3.3
by John Arbash Meinel
review feedback: add the actual count written to ShortReadvError |
374 |
cur_coalesced.length, len(data)) |
2001.3.2
by John Arbash Meinel
Force all transports to raise ShortReadvError if they can |
375 |
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
376 |
def put_file(self, relpath, f, mode=None): |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
377 |
"""
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
378 |
Copy the file-like object into the location.
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
379 |
|
380 |
:param relpath: Location to put the contents, relative to base.
|
|
1955.3.6
by John Arbash Meinel
Lots of deprecation warnings, but no errors |
381 |
:param f: File-like object.
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
382 |
:param mode: The final mode for the file
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
383 |
"""
|
1524.1.1
by Robert Collins
Test sftp with relative, absolute-in-homedir and absolute-not-in-homedir |
384 |
final_path = self._remote_path(relpath) |
2745.5.2
by Robert Collins
* ``bzrlib.transport.Transport.put_file`` now returns the number of bytes |
385 |
return self._put(final_path, f, mode=mode) |
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
386 |
|
387 |
def _put(self, abspath, f, mode=None): |
|
388 |
"""Helper function so both put() and copy_abspaths can reuse the code"""
|
|
389 |
tmp_abspath = '%s.tmp.%.9f.%d.%d' % (abspath, time.time(), |
|
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
390 |
os.getpid(), random.randint(0,0x7FFFFFFF)) |
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
391 |
fout = self._sftp_open_exclusive(tmp_abspath, mode=mode) |
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
392 |
closed = False |
1185.41.6
by Robey Pointer
modified version of john's patch to add atomic put and locking to the sftp transport |
393 |
try: |
1185.49.1
by John Arbash Meinel
Updating SftpTransport.put() so that it is atomic |
394 |
try: |
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
395 |
fout.set_pipelined(True) |
2745.5.2
by Robert Collins
* ``bzrlib.transport.Transport.put_file`` now returns the number of bytes |
396 |
length = self._pump(f, fout) |
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
397 |
except (IOError, paramiko.SSHException), e: |
398 |
self._translate_io_exception(e, tmp_abspath) |
|
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
399 |
# XXX: This doesn't truly help like we would like it to.
|
400 |
# The problem is that openssh strips sticky bits. So while we
|
|
401 |
# can properly set group write permission, we lose the group
|
|
402 |
# sticky bit. So it is probably best to stop chmodding, and
|
|
403 |
# just tell users that they need to set the umask correctly.
|
|
404 |
# The attr.st_mode = mode, in _sftp_open_exclusive
|
|
405 |
# will handle when the user wants the final mode to be more
|
|
406 |
# restrictive. And then we avoid a round trip. Unless
|
|
407 |
# paramiko decides to expose an async chmod()
|
|
408 |
||
409 |
# This is designed to chmod() right before we close.
|
|
410 |
# Because we set_pipelined() earlier, theoretically we might
|
|
411 |
# avoid the round trip for fout.close()
|
|
1185.50.20
by John Arbash Meinel
merge permissions branch, also fixup tests so they are lined up with bzr.dev to help prevent conflicts. |
412 |
if mode is not None: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
413 |
self._get_sftp().chmod(tmp_abspath, mode) |
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
414 |
fout.close() |
415 |
closed = True |
|
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
416 |
self._rename_and_overwrite(tmp_abspath, abspath) |
2745.5.2
by Robert Collins
* ``bzrlib.transport.Transport.put_file`` now returns the number of bytes |
417 |
return length |
1185.49.1
by John Arbash Meinel
Updating SftpTransport.put() so that it is atomic |
418 |
except Exception, e: |
419 |
# If we fail, try to clean up the temporary file
|
|
420 |
# before we throw the exception
|
|
421 |
# but don't let another exception mess things up
|
|
1185.50.20
by John Arbash Meinel
merge permissions branch, also fixup tests so they are lined up with bzr.dev to help prevent conflicts. |
422 |
# Write out the traceback, because otherwise
|
423 |
# the catch and throw destroys it
|
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
424 |
import traceback |
425 |
mutter(traceback.format_exc()) |
|
1185.49.1
by John Arbash Meinel
Updating SftpTransport.put() so that it is atomic |
426 |
try: |
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
427 |
if not closed: |
428 |
fout.close() |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
429 |
self._get_sftp().remove(tmp_abspath) |
1185.49.1
by John Arbash Meinel
Updating SftpTransport.put() so that it is atomic |
430 |
except: |
1532
by Robert Collins
Merge in John Meinels integration branch. |
431 |
# raise the saved except
|
432 |
raise e |
|
433 |
# raise the original with its traceback if we can.
|
|
434 |
raise
|
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
435 |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
436 |
def _put_non_atomic_helper(self, relpath, writer, mode=None, |
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
437 |
create_parent_dir=False, |
438 |
dir_mode=None): |
|
1946.1.6
by John Arbash Meinel
Basic implementation of non_atomic_put for sftp |
439 |
abspath = self._remote_path(relpath) |
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
440 |
|
441 |
# TODO: jam 20060816 paramiko doesn't publicly expose a way to
|
|
442 |
# set the file mode at create time. If it does, use it.
|
|
443 |
# But for now, we just chmod later anyway.
|
|
1946.1.6
by John Arbash Meinel
Basic implementation of non_atomic_put for sftp |
444 |
|
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
445 |
def _open_and_write_file(): |
446 |
"""Try to open the target file, raise error on failure"""
|
|
1946.1.10
by John Arbash Meinel
make sure to always close the file object. |
447 |
fout = None |
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
448 |
try: |
1946.1.10
by John Arbash Meinel
make sure to always close the file object. |
449 |
try: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
450 |
fout = self._get_sftp().file(abspath, mode='wb') |
1946.1.10
by John Arbash Meinel
make sure to always close the file object. |
451 |
fout.set_pipelined(True) |
1955.3.21
by John Arbash Meinel
Update the LocalTransport and SftpTransport to implement non_atomic_* |
452 |
writer(fout) |
1946.1.10
by John Arbash Meinel
make sure to always close the file object. |
453 |
except (paramiko.SSHException, IOError), e: |
1955.3.21
by John Arbash Meinel
Update the LocalTransport and SftpTransport to implement non_atomic_* |
454 |
self._translate_io_exception(e, abspath, |
455 |
': unable to open') |
|
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
456 |
|
1946.1.10
by John Arbash Meinel
make sure to always close the file object. |
457 |
# This is designed to chmod() right before we close.
|
458 |
# Because we set_pipelined() earlier, theoretically we might
|
|
459 |
# avoid the round trip for fout.close()
|
|
460 |
if mode is not None: |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
461 |
self._get_sftp().chmod(abspath, mode) |
1946.1.10
by John Arbash Meinel
make sure to always close the file object. |
462 |
finally: |
463 |
if fout is not None: |
|
464 |
fout.close() |
|
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
465 |
|
466 |
if not create_parent_dir: |
|
467 |
_open_and_write_file() |
|
468 |
return
|
|
469 |
||
470 |
# Try error handling to create the parent directory if we need to
|
|
471 |
try: |
|
472 |
_open_and_write_file() |
|
473 |
except NoSuchFile: |
|
474 |
# Try to create the parent directory, and then go back to
|
|
475 |
# writing the file
|
|
476 |
parent_dir = os.path.dirname(abspath) |
|
1946.2.15
by John Arbash Meinel
Fix the sftp mkdir mode code |
477 |
self._mkdir(parent_dir, dir_mode) |
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
478 |
_open_and_write_file() |
1946.1.6
by John Arbash Meinel
Basic implementation of non_atomic_put for sftp |
479 |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
480 |
def put_file_non_atomic(self, relpath, f, mode=None, |
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
481 |
create_parent_dir=False, |
482 |
dir_mode=None): |
|
1955.3.21
by John Arbash Meinel
Update the LocalTransport and SftpTransport to implement non_atomic_* |
483 |
"""Copy the file-like object into the target location.
|
484 |
||
485 |
This function is not strictly safe to use. It is only meant to
|
|
486 |
be used when you already know that the target does not exist.
|
|
487 |
It is not safe, because it will open and truncate the remote
|
|
488 |
file. So there may be a time when the file has invalid contents.
|
|
489 |
||
490 |
:param relpath: The remote location to put the contents.
|
|
491 |
:param f: File-like object.
|
|
492 |
:param mode: Possible access permissions for new file.
|
|
493 |
None means do not set remote permissions.
|
|
494 |
:param create_parent_dir: If we cannot create the target file because
|
|
495 |
the parent directory does not exist, go ahead and
|
|
496 |
create it, and then try again.
|
|
497 |
"""
|
|
498 |
def writer(fout): |
|
499 |
self._pump(f, fout) |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
500 |
self._put_non_atomic_helper(relpath, writer, mode=mode, |
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
501 |
create_parent_dir=create_parent_dir, |
502 |
dir_mode=dir_mode) |
|
1955.3.21
by John Arbash Meinel
Update the LocalTransport and SftpTransport to implement non_atomic_* |
503 |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
504 |
def put_bytes_non_atomic(self, relpath, bytes, mode=None, |
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
505 |
create_parent_dir=False, |
506 |
dir_mode=None): |
|
1955.3.21
by John Arbash Meinel
Update the LocalTransport and SftpTransport to implement non_atomic_* |
507 |
def writer(fout): |
508 |
fout.write(bytes) |
|
1955.3.27
by John Arbash Meinel
rename non_atomic_put_* to put_*non_atomic, and re-order the functions |
509 |
self._put_non_atomic_helper(relpath, writer, mode=mode, |
1946.2.12
by John Arbash Meinel
Add ability to pass a directory mode to non_atomic_put |
510 |
create_parent_dir=create_parent_dir, |
511 |
dir_mode=dir_mode) |
|
1955.3.21
by John Arbash Meinel
Update the LocalTransport and SftpTransport to implement non_atomic_* |
512 |
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
513 |
def iter_files_recursive(self): |
514 |
"""Walk the relative paths of all files in this transport."""
|
|
515 |
queue = list(self.list_dir('.')) |
|
516 |
while queue: |
|
1959.2.1
by John Arbash Meinel
David Allouche: Make transports return escaped paths |
517 |
relpath = queue.pop(0) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
518 |
st = self.stat(relpath) |
519 |
if stat.S_ISDIR(st.st_mode): |
|
520 |
for i, basename in enumerate(self.list_dir(relpath)): |
|
521 |
queue.insert(i, relpath+'/'+basename) |
|
522 |
else: |
|
523 |
yield relpath |
|
524 |
||
1988.1.1
by John Arbash Meinel
Restore mode bit tests for sftp, and track down bugs |
525 |
def _mkdir(self, abspath, mode=None): |
526 |
if mode is None: |
|
527 |
local_mode = 0777 |
|
528 |
else: |
|
529 |
local_mode = mode |
|
530 |
try: |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
531 |
self._get_sftp().mkdir(abspath, local_mode) |
1988.1.1
by John Arbash Meinel
Restore mode bit tests for sftp, and track down bugs |
532 |
if mode is not None: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
533 |
self._get_sftp().chmod(abspath, mode=mode) |
1988.1.1
by John Arbash Meinel
Restore mode bit tests for sftp, and track down bugs |
534 |
except (paramiko.SSHException, IOError), e: |
535 |
self._translate_io_exception(e, abspath, ': unable to mkdir', |
|
536 |
failure_exc=FileExists) |
|
537 |
||
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
538 |
def mkdir(self, relpath, mode=None): |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
539 |
"""Create a directory at the given path."""
|
1988.1.1
by John Arbash Meinel
Restore mode bit tests for sftp, and track down bugs |
540 |
self._mkdir(self._remote_path(relpath), mode=mode) |
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
541 |
|
2671.3.9
by Robert Collins
Review feedback and fix VFat emulated transports to not claim to have unix permissions. |
542 |
def open_write_stream(self, relpath, mode=None): |
543 |
"""See Transport.open_write_stream."""
|
|
2671.3.2
by Robert Collins
Start open_file_stream logic. |
544 |
# initialise the file to zero-length
|
545 |
# this is three round trips, but we don't use this
|
|
546 |
# api more than once per write_group at the moment so
|
|
547 |
# it is a tolerable overhead. Better would be to truncate
|
|
548 |
# the file after opening. RBC 20070805
|
|
2671.3.3
by Robert Collins
Add mode parameter to Transport.open_file_stream. |
549 |
self.put_bytes_non_atomic(relpath, "", mode) |
2671.3.2
by Robert Collins
Start open_file_stream logic. |
550 |
abspath = self._remote_path(relpath) |
551 |
# TODO: jam 20060816 paramiko doesn't publicly expose a way to
|
|
552 |
# set the file mode at create time. If it does, use it.
|
|
553 |
# But for now, we just chmod later anyway.
|
|
554 |
handle = None |
|
555 |
try: |
|
556 |
handle = self._get_sftp().file(abspath, mode='wb') |
|
557 |
handle.set_pipelined(True) |
|
558 |
except (paramiko.SSHException, IOError), e: |
|
559 |
self._translate_io_exception(e, abspath, |
|
560 |
': unable to open') |
|
561 |
_file_streams[self.abspath(relpath)] = handle |
|
2671.3.6
by Robert Collins
Review feedback. |
562 |
return FileFileStream(self, relpath, handle) |
2671.3.2
by Robert Collins
Start open_file_stream logic. |
563 |
|
2052.6.4
by Robert Collins
Tidier SFTP code (feedback from Aaron). |
564 |
def _translate_io_exception(self, e, path, more_info='', |
565 |
failure_exc=PathError): |
|
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
566 |
"""Translate a paramiko or IOError into a friendlier exception.
|
567 |
||
568 |
:param e: The original exception
|
|
569 |
:param path: The path in question when the error is raised
|
|
570 |
:param more_info: Extra information that can be included,
|
|
571 |
such as what was going on
|
|
572 |
:param failure_exc: Paramiko has the super fun ability to raise completely
|
|
573 |
opaque errors that just set "e.args = ('Failure',)" with
|
|
574 |
no more information.
|
|
1662.1.12
by Martin Pool
Translate unknown sftp errors to PathError, no NoSuchFile |
575 |
If this parameter is set, it defines the exception
|
576 |
to raise in these cases.
|
|
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
577 |
"""
|
1489
by Robert Collins
Make the paramiko tests pass. Nice huh. |
578 |
# paramiko seems to generate detailless errors.
|
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
579 |
self._translate_error(e, path, raise_generic=False) |
1963.2.6
by Robey Pointer
pychecker is on crack; go back to using 'is None'. |
580 |
if getattr(e, 'args', None) is not None: |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
581 |
if (e.args == ('No such file or directory',) or |
582 |
e.args == ('No such file',)): |
|
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
583 |
raise NoSuchFile(path, str(e) + more_info) |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
584 |
if (e.args == ('mkdir failed',)): |
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
585 |
raise FileExists(path, str(e) + more_info) |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
586 |
# strange but true, for the paramiko server.
|
587 |
if (e.args == ('Failure',)): |
|
2052.6.4
by Robert Collins
Tidier SFTP code (feedback from Aaron). |
588 |
raise failure_exc(path, str(e) + more_info) |
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
589 |
mutter('Raising exception with args %s', e.args) |
1963.2.6
by Robey Pointer
pychecker is on crack; go back to using 'is None'. |
590 |
if getattr(e, 'errno', None) is not None: |
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
591 |
mutter('Raising exception with errno %s', e.errno) |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
592 |
raise e |
1489
by Robert Collins
Make the paramiko tests pass. Nice huh. |
593 |
|
1955.3.15
by John Arbash Meinel
Deprecate 'Transport.append' in favor of Transport.append_file or Transport.append_bytes |
594 |
def append_file(self, relpath, f, mode=None): |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
595 |
"""
|
596 |
Append the text in the file-like object into the final
|
|
597 |
location.
|
|
598 |
"""
|
|
599 |
try: |
|
1524.1.1
by Robert Collins
Test sftp with relative, absolute-in-homedir and absolute-not-in-homedir |
600 |
path = self._remote_path(relpath) |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
601 |
fout = self._get_sftp().file(path, 'ab') |
1666.1.6
by Robert Collins
Make knit the default format. |
602 |
if mode is not None: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
603 |
self._get_sftp().chmod(path, mode) |
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
604 |
result = fout.tell() |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
605 |
self._pump(f, fout) |
1563.2.3
by Robert Collins
Change the return signature of transport.append and append_multi to return the length of the pre-append content. |
606 |
return result |
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
607 |
except (IOError, paramiko.SSHException), e: |
608 |
self._translate_io_exception(e, relpath, ': unable to append') |
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
609 |
|
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
610 |
def rename(self, rel_from, rel_to): |
611 |
"""Rename without special overwriting"""
|
|
612 |
try: |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
613 |
self._get_sftp().rename(self._remote_path(rel_from), |
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
614 |
self._remote_path(rel_to)) |
615 |
except (IOError, paramiko.SSHException), e: |
|
616 |
self._translate_io_exception(e, rel_from, |
|
617 |
': unable to rename to %r' % (rel_to)) |
|
618 |
||
619 |
def _rename_and_overwrite(self, abs_from, abs_to): |
|
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
620 |
"""Do a fancy rename on the remote server.
|
621 |
|
|
622 |
Using the implementation provided by osutils.
|
|
623 |
"""
|
|
624 |
try: |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
625 |
sftp = self._get_sftp() |
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
626 |
fancy_rename(abs_from, abs_to, |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
627 |
rename_func=sftp.rename, |
628 |
unlink_func=sftp.remove) |
|
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
629 |
except (IOError, paramiko.SSHException), e: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
630 |
self._translate_io_exception(e, abs_from, |
631 |
': unable to rename to %r' % (abs_to)) |
|
1185.31.47
by John Arbash Meinel
Added a fancy footwork rename to osutils, made SftpTransport use it. |
632 |
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
633 |
def move(self, rel_from, rel_to): |
634 |
"""Move the item at rel_from to the location at rel_to"""
|
|
1524.1.1
by Robert Collins
Test sftp with relative, absolute-in-homedir and absolute-not-in-homedir |
635 |
path_from = self._remote_path(rel_from) |
636 |
path_to = self._remote_path(rel_to) |
|
1553.5.13
by Martin Pool
New Transport.rename that mustn't overwrite |
637 |
self._rename_and_overwrite(path_from, path_to) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
638 |
|
639 |
def delete(self, relpath): |
|
640 |
"""Delete the item at relpath"""
|
|
1524.1.1
by Robert Collins
Test sftp with relative, absolute-in-homedir and absolute-not-in-homedir |
641 |
path = self._remote_path(relpath) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
642 |
try: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
643 |
self._get_sftp().remove(path) |
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
644 |
except (IOError, paramiko.SSHException), e: |
645 |
self._translate_io_exception(e, path, ': unable to delete') |
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
646 |
|
2634.1.1
by Robert Collins
(robertc) Reinstate the accidentally backed out external_url patch. |
647 |
def external_url(self): |
648 |
"""See bzrlib.transport.Transport.external_url."""
|
|
649 |
# the external path for SFTP is the base
|
|
650 |
return self.base |
|
651 |
||
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
652 |
def listable(self): |
653 |
"""Return True if this store supports listing."""
|
|
654 |
return True |
|
655 |
||
656 |
def list_dir(self, relpath): |
|
657 |
"""
|
|
658 |
Return a list of all files at the given location.
|
|
659 |
"""
|
|
660 |
# does anything actually use this?
|
|
1959.2.1
by John Arbash Meinel
David Allouche: Make transports return escaped paths |
661 |
# -- Unknown
|
662 |
# This is at least used by copy_tree for remote upgrades.
|
|
663 |
# -- David Allouche 2006-08-11
|
|
1524.1.1
by Robert Collins
Test sftp with relative, absolute-in-homedir and absolute-not-in-homedir |
664 |
path = self._remote_path(relpath) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
665 |
try: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
666 |
entries = self._get_sftp().listdir(path) |
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
667 |
except (IOError, paramiko.SSHException), e: |
668 |
self._translate_io_exception(e, path, ': failed to list_dir') |
|
1959.2.2
by John Arbash Meinel
Apply Andrew Bennetts list_dir fixes |
669 |
return [urlutils.escape(entry) for entry in entries] |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
670 |
|
1534.4.15
by Robert Collins
Remove shutil dependency in upgrade - create a delete_tree method for transports. |
671 |
def rmdir(self, relpath): |
672 |
"""See Transport.rmdir."""
|
|
673 |
path = self._remote_path(relpath) |
|
674 |
try: |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
675 |
return self._get_sftp().rmdir(path) |
1534.4.15
by Robert Collins
Remove shutil dependency in upgrade - create a delete_tree method for transports. |
676 |
except (IOError, paramiko.SSHException), e: |
677 |
self._translate_io_exception(e, path, ': failed to rmdir') |
|
678 |
||
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
679 |
def stat(self, relpath): |
680 |
"""Return the stat information for a file."""
|
|
1524.1.1
by Robert Collins
Test sftp with relative, absolute-in-homedir and absolute-not-in-homedir |
681 |
path = self._remote_path(relpath) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
682 |
try: |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
683 |
return self._get_sftp().stat(path) |
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
684 |
except (IOError, paramiko.SSHException), e: |
685 |
self._translate_io_exception(e, path, ': unable to stat') |
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
686 |
|
687 |
def lock_read(self, relpath): |
|
688 |
"""
|
|
689 |
Lock the given file for shared (read) access.
|
|
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
690 |
:return: A lock object, which has an unlock() member function
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
691 |
"""
|
692 |
# FIXME: there should be something clever i can do here...
|
|
693 |
class BogusLock(object): |
|
694 |
def __init__(self, path): |
|
695 |
self.path = path |
|
696 |
def unlock(self): |
|
697 |
pass
|
|
698 |
return BogusLock(relpath) |
|
699 |
||
700 |
def lock_write(self, relpath): |
|
701 |
"""
|
|
702 |
Lock the given file for exclusive (write) access.
|
|
703 |
WARNING: many transports do not support this, so trying avoid using it
|
|
704 |
||
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
705 |
:return: A lock object, which has an unlock() member function
|
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
706 |
"""
|
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
707 |
# This is a little bit bogus, but basically, we create a file
|
708 |
# which should not already exist, and if it does, we assume
|
|
709 |
# that there is a lock, and if it doesn't, the we assume
|
|
710 |
# that we have taken the lock.
|
|
1185.49.3
by John Arbash Meinel
Added a form of locking to sftp branches. Refactored _sftp_open_exclusive to take a relative path |
711 |
return SFTPLock(relpath, self) |
1185.16.87
by mbp at sourcefrog
- fix line endings in sftp transport |
712 |
|
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
713 |
def _sftp_open_exclusive(self, abspath, mode=None): |
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
714 |
"""Open a remote path exclusively.
|
715 |
||
716 |
SFTP supports O_EXCL (SFTP_FLAG_EXCL), which fails if
|
|
717 |
the file already exists. However it does not expose this
|
|
718 |
at the higher level of SFTPClient.open(), so we have to
|
|
719 |
sneak away with it.
|
|
720 |
||
721 |
WARNING: This breaks the SFTPClient abstraction, so it
|
|
722 |
could easily break against an updated version of paramiko.
|
|
723 |
||
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
724 |
:param abspath: The remote absolute path where the file should be opened
|
725 |
:param mode: The mode permissions bits for the new file
|
|
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
726 |
"""
|
1946.1.8
by John Arbash Meinel
Update non_atomic_put to have a create_parent_dir flag |
727 |
# TODO: jam 20060816 Paramiko >= 1.6.2 (probably earlier) supports
|
728 |
# using the 'x' flag to indicate SFTP_FLAG_EXCL.
|
|
729 |
# However, there is no way to set the permission mode at open
|
|
730 |
# time using the sftp_client.file() functionality.
|
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
731 |
path = self._get_sftp()._adjust_cwd(abspath) |
1711.5.1
by John Arbash Meinel
Get most SFTP tests to pass. StubSFTPServer now talks the same path protocol that SFTPTransport talks. on win32 |
732 |
# mutter('sftp abspath %s => %s', abspath, path)
|
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
733 |
attr = SFTPAttributes() |
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
734 |
if mode is not None: |
735 |
attr.st_mode = mode |
|
736 |
omode = (SFTP_FLAG_WRITE | SFTP_FLAG_CREATE |
|
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
737 |
| SFTP_FLAG_TRUNC | SFTP_FLAG_EXCL) |
738 |
try: |
|
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
739 |
t, msg = self._get_sftp()._request(CMD_OPEN, path, omode, attr) |
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
740 |
if t != CMD_HANDLE: |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
741 |
raise TransportError('Expected an SFTP handle') |
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
742 |
handle = msg.get_string() |
2485.8.38
by Vincent Ladeuil
Finish sftp refactoring. Test suite passing. |
743 |
return SFTPFile(self._get_sftp(), handle, 'wb', -1) |
1185.31.44
by John Arbash Meinel
Cleaned up Exceptions for all transports. |
744 |
except (paramiko.SSHException, IOError), e: |
1185.58.2
by John Arbash Meinel
Added mode to the appropriate transport functions, and tests to make sure they work. |
745 |
self._translate_io_exception(e, abspath, ': unable to open', |
1185.50.13
by John Arbash Meinel
Expanded the Transport test suite. Including delete, copy, move, etc. Updated SftpTransport to conform. |
746 |
failure_exc=FileExists) |
1185.49.2
by John Arbash Meinel
Adding a open_exclusive function since paramiko supports it, but doesn't expose it |
747 |
|
1988.1.1
by John Arbash Meinel
Restore mode bit tests for sftp, and track down bugs |
748 |
def _can_roundtrip_unix_modebits(self): |
749 |
if sys.platform == 'win32': |
|
750 |
# anyone else?
|
|
751 |
return False |
|
752 |
else: |
|
753 |
return True |
|
1530.1.1
by Robert Collins
Minimal infrastructure to test TransportTestProviderAdapter. |
754 |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
755 |
# ------------- server test implementation --------------
|
756 |
import threading |
|
757 |
||
758 |
from bzrlib.tests.stub_sftp import StubServer, StubSFTPServer |
|
759 |
||
760 |
STUB_SERVER_KEY = """ |
|
761 |
-----BEGIN RSA PRIVATE KEY-----
|
|
762 |
MIICWgIBAAKBgQDTj1bqB4WmayWNPB+8jVSYpZYk80Ujvj680pOTh2bORBjbIAyz
|
|
763 |
oWGW+GUjzKxTiiPvVmxFgx5wdsFvF03v34lEVVhMpouqPAYQ15N37K/ir5XY+9m/
|
|
764 |
d8ufMCkjeXsQkKqFbAlQcnWMCRnOoPHS3I4vi6hmnDDeeYTSRvfLbW0fhwIBIwKB
|
|
765 |
gBIiOqZYaoqbeD9OS9z2K9KR2atlTxGxOJPXiP4ESqP3NVScWNwyZ3NXHpyrJLa0
|
|
766 |
EbVtzsQhLn6rF+TzXnOlcipFvjsem3iYzCpuChfGQ6SovTcOjHV9z+hnpXvQ/fon
|
|
767 |
soVRZY65wKnF7IAoUwTmJS9opqgrN6kRgCd3DASAMd1bAkEA96SBVWFt/fJBNJ9H
|
|
768 |
tYnBKZGw0VeHOYmVYbvMSstssn8un+pQpUm9vlG/bp7Oxd/m+b9KWEh2xPfv6zqU
|
|
769 |
avNwHwJBANqzGZa/EpzF4J8pGti7oIAPUIDGMtfIcmqNXVMckrmzQ2vTfqtkEZsA
|
|
770 |
4rE1IERRyiJQx6EJsz21wJmGV9WJQ5kCQQDwkS0uXqVdFzgHO6S++tjmjYcxwr3g
|
|
771 |
H0CoFYSgbddOT6miqRskOQF3DZVkJT3kyuBgU2zKygz52ukQZMqxCb1fAkASvuTv
|
|
772 |
qfpH87Qq5kQhNKdbbwbmd2NxlNabazPijWuphGTdW0VfJdWfklyS2Kr+iqrs/5wV
|
|
773 |
HhathJt636Eg7oIjAkA8ht3MQ+XSl9yIJIS8gVpbPxSw5OMfw0PjVE7tBdQruiSc
|
|
774 |
nvuQES5C9BMHjF39LZiGH1iLQy7FgdHyoP+eodI7
|
|
775 |
-----END RSA PRIVATE KEY-----
|
|
776 |
"""
|
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
777 |
|
778 |
||
779 |
class SocketListener(threading.Thread): |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
780 |
|
781 |
def __init__(self, callback): |
|
782 |
threading.Thread.__init__(self) |
|
783 |
self._callback = callback |
|
784 |
self._socket = socket.socket() |
|
785 |
self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1) |
|
786 |
self._socket.bind(('localhost', 0)) |
|
787 |
self._socket.listen(1) |
|
788 |
self.port = self._socket.getsockname()[1] |
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
789 |
self._stop_event = threading.Event() |
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
790 |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
791 |
def stop(self): |
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
792 |
# called from outside this thread
|
793 |
self._stop_event.set() |
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
794 |
# use a timeout here, because if the test fails, the server thread may
|
795 |
# never notice the stop_event.
|
|
796 |
self.join(5.0) |
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
797 |
self._socket.close() |
798 |
||
799 |
def run(self): |
|
800 |
while True: |
|
1551.6.40
by Aaron Bentley
Wrapped long lines in Robey's patch |
801 |
readable, writable_unused, exception_unused = \ |
802 |
select.select([self._socket], [], [], 0.1) |
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
803 |
if self._stop_event.isSet(): |
804 |
return
|
|
805 |
if len(readable) == 0: |
|
806 |
continue
|
|
807 |
try: |
|
808 |
s, addr_unused = self._socket.accept() |
|
809 |
# because the loopback socket is inline, and transports are
|
|
810 |
# never explicitly closed, best to launch a new thread.
|
|
811 |
threading.Thread(target=self._callback, args=(s,)).start() |
|
812 |
except socket.error, x: |
|
813 |
sys.excepthook(*sys.exc_info()) |
|
1551.6.40
by Aaron Bentley
Wrapped long lines in Robey's patch |
814 |
warning('Socket error during accept() within unit test server' |
815 |
' thread: %r' % x) |
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
816 |
except Exception, x: |
1551.6.40
by Aaron Bentley
Wrapped long lines in Robey's patch |
817 |
# probably a failed test; unit test thread will log the
|
818 |
# failure/error
|
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
819 |
sys.excepthook(*sys.exc_info()) |
1551.6.40
by Aaron Bentley
Wrapped long lines in Robey's patch |
820 |
warning('Exception from within unit test server thread: %r' % |
821 |
x) |
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
822 |
|
823 |
||
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
824 |
class SocketDelay(object): |
825 |
"""A socket decorator to make TCP appear slower.
|
|
826 |
||
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
827 |
This changes recv, send, and sendall to add a fixed latency to each python
|
828 |
call if a new roundtrip is detected. That is, when a recv is called and the
|
|
829 |
flag new_roundtrip is set, latency is charged. Every send and send_all
|
|
830 |
sets this flag.
|
|
831 |
||
832 |
In addition every send, sendall and recv sleeps a bit per character send to
|
|
833 |
simulate bandwidth.
|
|
834 |
||
835 |
Not all methods are implemented, this is deliberate as this class is not a
|
|
836 |
replacement for the builtin sockets layer. fileno is not implemented to
|
|
837 |
prevent the proxy being bypassed.
|
|
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
838 |
"""
|
839 |
||
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
840 |
simulated_time = 0 |
1874.1.9
by Carl Friedrich Bolz
Try to fix all the issues outline by john and Robert. |
841 |
_proxied_arguments = dict.fromkeys([ |
842 |
"close", "getpeername", "getsockname", "getsockopt", "gettimeout", |
|
843 |
"setblocking", "setsockopt", "settimeout", "shutdown"]) |
|
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
844 |
|
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
845 |
def __init__(self, sock, latency, bandwidth=1.0, |
1874.1.9
by Carl Friedrich Bolz
Try to fix all the issues outline by john and Robert. |
846 |
really_sleep=True): |
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
847 |
"""
|
848 |
:param bandwith: simulated bandwith (MegaBit)
|
|
1874.1.9
by Carl Friedrich Bolz
Try to fix all the issues outline by john and Robert. |
849 |
:param really_sleep: If set to false, the SocketDelay will just
|
850 |
increase a counter, instead of calling time.sleep. This is useful for
|
|
851 |
unittesting the SocketDelay.
|
|
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
852 |
"""
|
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
853 |
self.sock = sock |
854 |
self.latency = latency |
|
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
855 |
self.really_sleep = really_sleep |
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
856 |
self.time_per_byte = 1 / (bandwidth / 8.0 * 1024 * 1024) |
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
857 |
self.new_roundtrip = False |
858 |
||
859 |
def sleep(self, s): |
|
860 |
if self.really_sleep: |
|
861 |
time.sleep(s) |
|
1874.1.9
by Carl Friedrich Bolz
Try to fix all the issues outline by john and Robert. |
862 |
else: |
863 |
SocketDelay.simulated_time += s |
|
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
864 |
|
1874.1.9
by Carl Friedrich Bolz
Try to fix all the issues outline by john and Robert. |
865 |
def __getattr__(self, attr): |
866 |
if attr in SocketDelay._proxied_arguments: |
|
867 |
return getattr(self.sock, attr) |
|
868 |
raise AttributeError("'SocketDelay' object has no attribute %r" % |
|
869 |
attr) |
|
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
870 |
|
871 |
def dup(self): |
|
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
872 |
return SocketDelay(self.sock.dup(), self.latency, self.time_per_byte, |
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
873 |
self._sleep) |
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
874 |
|
875 |
def recv(self, *args): |
|
876 |
data = self.sock.recv(*args) |
|
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
877 |
if data and self.new_roundtrip: |
878 |
self.new_roundtrip = False |
|
879 |
self.sleep(self.latency) |
|
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
880 |
self.sleep(len(data) * self.time_per_byte) |
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
881 |
return data |
882 |
||
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
883 |
def sendall(self, data, flags=0): |
884 |
if not self.new_roundtrip: |
|
885 |
self.new_roundtrip = True |
|
886 |
self.sleep(self.latency) |
|
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
887 |
self.sleep(len(data) * self.time_per_byte) |
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
888 |
return self.sock.sendall(data, flags) |
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
889 |
|
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
890 |
def send(self, data, flags=0): |
891 |
if not self.new_roundtrip: |
|
892 |
self.new_roundtrip = True |
|
893 |
self.sleep(self.latency) |
|
894 |
bytes_sent = self.sock.send(data, flags) |
|
1874.1.5
by holger krekel
(cfbolz, hpk) Refactored bandwidth parameter. Fixed indentation bug in 'send'. |
895 |
self.sleep(bytes_sent * self.time_per_byte) |
1874.1.2
by Carl Friedrich Bolz
Refined the SocketDelay to charge latency only once per round-trip and to |
896 |
return bytes_sent |
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
897 |
|
898 |
||
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
899 |
class SFTPServer(Server): |
900 |
"""Common code for SFTP server facilities."""
|
|
901 |
||
2018.1.1
by Andrew Bennetts
Make bzr+ssh:// actually work (at least with absolute paths). |
902 |
def __init__(self, server_interface=StubServer): |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
903 |
self._original_vendor = None |
904 |
self._homedir = None |
|
1530.1.8
by Robert Collins
More NEWS, move sibling sftp tests into new framework, nuke legacy local transport tests. |
905 |
self._server_homedir = None |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
906 |
self._listener = None |
907 |
self._root = None |
|
1951.1.8
by Andrew Bennetts
Make _get_ssh_vendor return the vendor object, rather than just a string. |
908 |
self._vendor = ssh.ParamikoVendor() |
2018.1.1
by Andrew Bennetts
Make bzr+ssh:// actually work (at least with absolute paths). |
909 |
self._server_interface = server_interface |
1530.1.6
by Robert Collins
Trim duplicate sftp tests. |
910 |
# sftp server logs
|
911 |
self.logs = [] |
|
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
912 |
self.add_latency = 0 |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
913 |
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
914 |
def _get_sftp_url(self, path): |
915 |
"""Calculate an sftp url to this server for path."""
|
|
916 |
return 'sftp://foo:bar@localhost:%d/%s' % (self._listener.port, path) |
|
917 |
||
1530.1.6
by Robert Collins
Trim duplicate sftp tests. |
918 |
def log(self, message): |
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
919 |
"""StubServer uses this to log when a new server is created."""
|
1530.1.6
by Robert Collins
Trim duplicate sftp tests. |
920 |
self.logs.append(message) |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
921 |
|
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
922 |
def _run_server_entry(self, sock): |
923 |
"""Entry point for all implementations of _run_server.
|
|
924 |
|
|
925 |
If self.add_latency is > 0.000001 then sock is given a latency adding
|
|
926 |
decorator.
|
|
927 |
"""
|
|
928 |
if self.add_latency > 0.000001: |
|
929 |
sock = SocketDelay(sock, self.add_latency) |
|
930 |
return self._run_server(sock) |
|
931 |
||
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
932 |
def _run_server(self, s): |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
933 |
ssh_server = paramiko.Transport(s) |
1711.5.1
by John Arbash Meinel
Get most SFTP tests to pass. StubSFTPServer now talks the same path protocol that SFTPTransport talks. on win32 |
934 |
key_file = pathjoin(self._homedir, 'test_rsa.key') |
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
935 |
f = open(key_file, 'w') |
936 |
f.write(STUB_SERVER_KEY) |
|
937 |
f.close() |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
938 |
host_key = paramiko.RSAKey.from_private_key_file(key_file) |
939 |
ssh_server.add_server_key(host_key) |
|
2018.1.1
by Andrew Bennetts
Make bzr+ssh:// actually work (at least with absolute paths). |
940 |
server = self._server_interface(self) |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
941 |
ssh_server.set_subsystem_handler('sftp', paramiko.SFTPServer, |
942 |
StubSFTPServer, root=self._root, |
|
1530.1.8
by Robert Collins
More NEWS, move sibling sftp tests into new framework, nuke legacy local transport tests. |
943 |
home=self._server_homedir) |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
944 |
event = threading.Event() |
945 |
ssh_server.start_server(event, server) |
|
946 |
event.wait(5.0) |
|
2221.5.21
by Dmitry Vasiliev
Reverted trailing whitespace removal |
947 |
|
2018.5.114
by Robert Collins
Commit current test pass improvements. |
948 |
def setUp(self, backing_server=None): |
949 |
# XXX: TODO: make sftpserver back onto backing_server rather than local
|
|
950 |
# disk.
|
|
951 |
assert (backing_server is None or |
|
952 |
isinstance(backing_server, local.LocalURLServer)), ( |
|
953 |
"backing_server should not be %r, because this can only serve the " |
|
954 |
"local current working directory." % (backing_server,)) |
|
2221.5.18
by Dmitry Vasiliev
Fixed variable name |
955 |
self._original_vendor = ssh._ssh_vendor_manager._cached_ssh_vendor |
956 |
ssh._ssh_vendor_manager._cached_ssh_vendor = self._vendor |
|
1711.6.3
by John Arbash Meinel
Fix regression on unicode tests for SFTP under Linux |
957 |
if sys.platform == 'win32': |
958 |
# Win32 needs to use the UNICODE api
|
|
959 |
self._homedir = getcwd() |
|
960 |
else: |
|
961 |
# But Linux SFTP servers should just deal in bytestreams
|
|
962 |
self._homedir = os.getcwd() |
|
1530.1.8
by Robert Collins
More NEWS, move sibling sftp tests into new framework, nuke legacy local transport tests. |
963 |
if self._server_homedir is None: |
964 |
self._server_homedir = self._homedir |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
965 |
self._root = '/' |
1711.5.1
by John Arbash Meinel
Get most SFTP tests to pass. StubSFTPServer now talks the same path protocol that SFTPTransport talks. on win32 |
966 |
if sys.platform == 'win32': |
967 |
self._root = '' |
|
1871.1.3
by Robert Collins
proof of concept slowsocket wrapper. |
968 |
self._listener = SocketListener(self._run_server_entry) |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
969 |
self._listener.setDaemon(True) |
970 |
self._listener.start() |
|
971 |
||
972 |
def tearDown(self): |
|
973 |
"""See bzrlib.transport.Server.tearDown."""
|
|
974 |
self._listener.stop() |
|
2221.5.18
by Dmitry Vasiliev
Fixed variable name |
975 |
ssh._ssh_vendor_manager._cached_ssh_vendor = self._original_vendor |
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
976 |
|
1711.2.42
by John Arbash Meinel
enable bogus_url support for SFTP tests |
977 |
def get_bogus_url(self): |
978 |
"""See bzrlib.transport.Server.get_bogus_url."""
|
|
1185.49.35
by John Arbash Meinel
Update tests to use a truly unused port |
979 |
# this is chosen to try to prevent trouble with proxies, wierd dns, etc
|
980 |
# we bind a random socket, so that we get a guaranteed unused port
|
|
981 |
# we just never listen on that port
|
|
982 |
s = socket.socket() |
|
983 |
s.bind(('localhost', 0)) |
|
984 |
return 'sftp://%s:%s/' % s.getsockname() |
|
1711.2.42
by John Arbash Meinel
enable bogus_url support for SFTP tests |
985 |
|
986 |
||
1534.4.50
by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running. |
987 |
class SFTPFullAbsoluteServer(SFTPServer): |
988 |
"""A test server for sftp transports, using absolute urls and ssh."""
|
|
989 |
||
990 |
def get_url(self): |
|
991 |
"""See bzrlib.transport.Server.get_url."""
|
|
2321.3.7
by Alexander Belchenko
fixes for passing test_sftp_transport on win32 (thankyou John) |
992 |
homedir = self._homedir |
993 |
if sys.platform != 'win32': |
|
994 |
# Remove the initial '/' on all platforms but win32
|
|
995 |
homedir = homedir[1:] |
|
996 |
return self._get_sftp_url(urlutils.escape(homedir)) |
|
1534.4.50
by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running. |
997 |
|
998 |
||
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
999 |
class SFTPServerWithoutSSH(SFTPServer): |
1534.4.50
by Robert Collins
Got the bzrdir api straightened out, plenty of refactoring to use it pending, but the api is up and running. |
1000 |
"""An SFTP server that uses a simple TCP socket pair rather than SSH."""
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
1001 |
|
1002 |
def __init__(self): |
|
1003 |
super(SFTPServerWithoutSSH, self).__init__() |
|
1951.1.8
by Andrew Bennetts
Make _get_ssh_vendor return the vendor object, rather than just a string. |
1004 |
self._vendor = ssh.LoopbackVendor() |
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
1005 |
|
1551.6.39
by Aaron Bentley
Apply Robey Pointer's listener patch |
1006 |
def _run_server(self, sock): |
1951.1.10
by Andrew Bennetts
Move register_ssh_vendor, _ssh_vendor and _get_ssh_vendor into ssh.py |
1007 |
# Re-import these as locals, so that they're still accessible during
|
1008 |
# interpreter shutdown (when all module globals get set to None, leading
|
|
1009 |
# to confusing errors like "'NoneType' object has no attribute 'error'".
|
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
1010 |
class FakeChannel(object): |
1011 |
def get_transport(self): |
|
1012 |
return self |
|
1013 |
def get_log_channel(self): |
|
1014 |
return 'paramiko' |
|
1015 |
def get_name(self): |
|
1016 |
return '1' |
|
1017 |
def get_hexdump(self): |
|
1018 |
return False |
|
1711.2.1
by John Arbash Meinel
[patch] Marien Zwart: paramiko-1.6 compatibility fix |
1019 |
def close(self): |
1020 |
pass
|
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
1021 |
|
1022 |
server = paramiko.SFTPServer(FakeChannel(), 'sftp', StubServer(self), StubSFTPServer, |
|
1023 |
root=self._root, home=self._server_homedir) |
|
1806.1.1
by Robey Pointer
absorb a broken-pipe exception from paramiko when running the unit tests. it's okay for the client to vanish abruptly and paramiko really ought to mask the exception itself. |
1024 |
try: |
1025 |
server.start_subsystem('sftp', None, sock) |
|
1026 |
except socket.error, e: |
|
1027 |
if (len(e.args) > 0) and (e.args[0] == errno.EPIPE): |
|
1028 |
# it's okay for the client to disconnect abruptly
|
|
1029 |
# (bug in paramiko 1.6: it should absorb this exception)
|
|
1030 |
pass
|
|
1031 |
else: |
|
1032 |
raise
|
|
1033 |
except Exception, e: |
|
2018.11.2
by Andrew Bennetts
Hookup RemoteBranch for testing by bzrlib/tests/branch_implementations tests. |
1034 |
# This typically seems to happen during interpreter shutdown, so
|
1035 |
# most of the useful ways to report this error are won't work.
|
|
1036 |
# Writing the exception type, and then the text of the exception,
|
|
1037 |
# seems to be the best we can do.
|
|
1038 |
import sys |
|
1039 |
sys.stderr.write('\nEXCEPTION %r: ' % (e.__class__,)) |
|
1040 |
sys.stderr.write('%s\n\n' % (e,)) |
|
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
1041 |
server.finish_subsystem() |
1042 |
||
1043 |
||
1044 |
class SFTPAbsoluteServer(SFTPServerWithoutSSH): |
|
1530.1.1
by Robert Collins
Minimal infrastructure to test TransportTestProviderAdapter. |
1045 |
"""A test server for sftp transports, using absolute urls."""
|
1046 |
||
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1047 |
def get_url(self): |
1048 |
"""See bzrlib.transport.Server.get_url."""
|
|
2321.3.7
by Alexander Belchenko
fixes for passing test_sftp_transport on win32 (thankyou John) |
1049 |
homedir = self._homedir |
1050 |
if sys.platform != 'win32': |
|
1051 |
# Remove the initial '/' on all platforms but win32
|
|
1052 |
homedir = homedir[1:] |
|
1053 |
return self._get_sftp_url(urlutils.escape(homedir)) |
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1054 |
|
1055 |
||
1547.1.1
by Robey Pointer
modify the sftp unit tests to perform sftp over a direct localhost socket instead of over an actual ssh2 transport |
1056 |
class SFTPHomeDirServer(SFTPServerWithoutSSH): |
1530.1.1
by Robert Collins
Minimal infrastructure to test TransportTestProviderAdapter. |
1057 |
"""A test server for sftp transports, using homedir relative urls."""
|
1530.1.3
by Robert Collins
transport implementations now tested consistently. |
1058 |
|
1059 |
def get_url(self): |
|
1060 |
"""See bzrlib.transport.Server.get_url."""
|
|
1534.1.8
by Robert Collins
Update SFTP Urls as per mailing list thread. |
1061 |
return self._get_sftp_url("~/") |
1530.1.8
by Robert Collins
More NEWS, move sibling sftp tests into new framework, nuke legacy local transport tests. |
1062 |
|
1063 |
||
1064 |
class SFTPSiblingAbsoluteServer(SFTPAbsoluteServer): |
|
2018.5.114
by Robert Collins
Commit current test pass improvements. |
1065 |
"""A test server for sftp transports where only absolute paths will work.
|
1066 |
||
1067 |
It does this by serving from a deeply-nested directory that doesn't exist.
|
|
1068 |
"""
|
|
1069 |
||
1070 |
def setUp(self, backing_server=None): |
|
1530.1.8
by Robert Collins
More NEWS, move sibling sftp tests into new framework, nuke legacy local transport tests. |
1071 |
self._server_homedir = '/dev/noone/runs/tests/here' |
2018.5.114
by Robert Collins
Commit current test pass improvements. |
1072 |
super(SFTPSiblingAbsoluteServer, self).setUp(backing_server) |
1530.1.11
by Robert Collins
Push the transport permutations list into each transport module allowing for automatic testing of new modules that are registered as transports. |
1073 |
|
1074 |
||
1075 |
def get_test_permutations(): |
|
1076 |
"""Return the permutations to be used in testing."""
|
|
1077 |
return [(SFTPTransport, SFTPAbsoluteServer), |
|
1078 |
(SFTPTransport, SFTPHomeDirServer), |
|
1079 |
(SFTPTransport, SFTPSiblingAbsoluteServer), |
|
1080 |
]
|