~bzr-pqm/bzr/bzr.dev

6619.1.1 by Vincent Ladeuil
Merge 2.7, resolving conflicts
1
# Copyright (C) 2006-2013, 2016, 2017 Canonical Ltd
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
2
#
3
# This program is free software; you can redistribute it and/or modify
4
# it under the terms of the GNU General Public License as published by
5
# the Free Software Foundation; either version 2 of the License, or
6
# (at your option) any later version.
7
#
8
# This program is distributed in the hope that it will be useful,
9
# but WITHOUT ANY WARRANTY; without even the implied warranty of
10
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
11
# GNU General Public License for more details.
12
#
13
# You should have received a copy of the GNU General Public License
14
# along with this program; if not, write to the Free Software
4183.7.1 by Sabin Iacob
update FSF mailing address
15
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
16
6238.2.9 by Jelmer Vernooij
Add some tests for ssl.ca_certs config option.
17
"""Implementation of urllib2 tailored to bzr needs
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
18
2363.4.7 by Vincent Ladeuil
Deeper tests, prepare the auth setting that will avoid the
19
This file complements the urllib2 class hierarchy with custom classes.
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
20
21
For instance, we create a new HTTPConnection and HTTPSConnection that inherit
22
from the original urllib2.HTTP(s)Connection objects, but also have a new base
3059.2.2 by Vincent Ladeuil
Read http responses on demand without buffering the whole body
23
which implements a custom getresponse and cleanup_pipe handlers.
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
24
25
And then we implement custom HTTPHandler and HTTPSHandler classes, that use
26
the custom HTTPConnection classes.
27
28
We have a custom Response class, which lets us maintain a keep-alive
29
connection even for requests that urllib2 doesn't expect to contain body data.
30
2363.4.10 by Vincent Ladeuil
Complete tests.
31
And a custom Request class that lets us track redirections, and
2363.4.12 by Vincent Ladeuil
Take jam's review comments into account. Fix typos, give better
32
handle authentication schemes.
3430.1.1 by Vincent Ladeuil
Fix bug #229076 by fixing header names before sending the request.
33
34
For coherency with python libraries, we use capitalized header names throughout
35
the code, even if the header names will be titled just before sending the
36
request (see AbstractHTTPHandler.do_open).
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
37
"""
38
6379.6.7 by Jelmer Vernooij
Move importing from future until after doc string, otherwise the doc string will disappear.
39
from __future__ import absolute_import
40
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
41
DEBUG = 0
42
43
# FIXME: Oversimplifying, two kind of exceptions should be
44
# raised, once a request is issued: URLError before we have been
45
# able to process the response, HTTPError after that. Process the
46
# response means we are able to leave the socket clean, so if we
47
# are not able to do that, we should close the connection. The
48
# actual code more or less do that, tests should be written to
2004.1.16 by v.ladeuil+lp at free
Add tests against erroneous http status lines.
49
# ensure that.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
50
6615.4.1 by Vincent Ladeuil
Fix long user and password header generation for Basic http authentification
51
import base64
4628.1.2 by Vincent Ladeuil
More complete fix.
52
import errno
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
53
import httplib
6238.2.9 by Jelmer Vernooij
Add some tests for ssl.ca_certs config option.
54
import os
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
55
import socket
56
import urllib
57
import urllib2
58
import urlparse
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
59
import re
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
60
import ssl
2004.1.16 by v.ladeuil+lp at free
Add tests against erroneous http status lines.
61
import sys
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
62
import time
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
63
2004.1.25 by v.ladeuil+lp at free
Shuffle http related test code. Hopefully it ends up at the right place :)
64
from bzrlib import __version__ as bzrlib_version
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
65
from bzrlib import (
2900.2.6 by Vincent Ladeuil
Make http aware of authentication config.
66
    config,
3052.3.3 by Vincent Ladeuil
Add -Dhttp support.
67
    debug,
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
68
    errors,
6238.2.7 by Jelmer Vernooij
Add ssl.cert_reqs.
69
    lazy_import,
2929.3.1 by Vincent Ladeuil
Fix python2.6 deprecation warnings (still 4 failures 5 errors in test suite).
70
    osutils,
3052.3.3 by Vincent Ladeuil
Add -Dhttp support.
71
    trace,
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
72
    transport,
6524.4.2 by Marius Kruger
use ui.ui_factory.show_user_warning to warn about not checking ssl certs, so that we only do it once
73
    ui,
4795.4.5 by Vincent Ladeuil
Make sure all redirection code paths can handle authentication.
74
    urlutils,
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
75
)
76
77
try:
78
    _ = (ssl.match_hostname, ssl.CertificateError)
79
except AttributeError:
80
    # Provide fallbacks for python < 2.7.9
81
    def match_hostname(cert, host):
82
        trace.warning(
83
            '%s cannot be verified, https certificates verification is only'
84
            ' available for python versions >= 2.7.9' % (host,))
85
    ssl.match_hostname = match_hostname
86
    ssl.CertificateError = ValueError
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
87
6238.2.9 by Jelmer Vernooij
Add some tests for ssl.ca_certs config option.
88
6437.25.1 by Vincent Ladeuil
Provide default paths for ca certs for supported platforms
89
# Note for packagers: if there is no package providing certs for your platform,
90
# the curl project produces http://curl.haxx.se/ca/cacert.pem weekly.
6437.25.4 by Vincent Ladeuil
Get rid of 'optional' for ssl.ca_reqs to simplify the model.
91
_ssl_ca_certs_known_locations = [
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
92
    u'/etc/ssl/certs/ca-certificates.crt',  # Ubuntu/debian/gentoo
93
    u'/etc/pki/tls/certs/ca-bundle.crt',  # Fedora/CentOS/RH
94
    u'/etc/ssl/ca-bundle.pem',  # OpenSuse
95
    u'/etc/ssl/cert.pem',  # OpenSuse
96
    u"/usr/local/share/certs/ca-root-nss.crt",  # FreeBSD
6437.25.3 by Vincent Ladeuil
Simplify: all posix systems will try know locations and select the first
97
    # XXX: Needs checking, can't trust the interweb ;) -- vila 2012-01-25
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
98
    u'/etc/openssl/certs/ca-certificates.crt',  # Solaris
99
]
100
101
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
102
def default_ca_certs():
6437.25.3 by Vincent Ladeuil
Simplify: all posix systems will try know locations and select the first
103
    if sys.platform == 'win32':
6437.34.1 by Martin Packman
Expect name 'cacert.pem' rather than 'ca_bundle.crt' for curl extracted certs on win32
104
        return os.path.join(os.path.dirname(sys.executable), u"cacert.pem")
6437.25.1 by Vincent Ladeuil
Provide default paths for ca certs for supported platforms
105
    elif sys.platform == 'darwin':
106
        # FIXME: Needs some default value for osx, waiting for osx installers
107
        # guys feedback -- vila 2012-01-25
108
        pass
6437.25.3 by Vincent Ladeuil
Simplify: all posix systems will try know locations and select the first
109
    else:
110
        # Try known locations for friendly OSes providing the root certificates
111
        # without making them hard to use for any https client.
6437.25.4 by Vincent Ladeuil
Get rid of 'optional' for ssl.ca_reqs to simplify the model.
112
        for path in _ssl_ca_certs_known_locations:
6437.25.3 by Vincent Ladeuil
Simplify: all posix systems will try know locations and select the first
113
            if os.path.exists(path):
114
                # First found wins
115
                return path
116
    # A default path that makes sense and will be mentioned in the error
117
    # presented to the user, even if not correct for all platforms
6437.25.4 by Vincent Ladeuil
Get rid of 'optional' for ssl.ca_reqs to simplify the model.
118
    return _ssl_ca_certs_known_locations[0]
6238.2.9 by Jelmer Vernooij
Add some tests for ssl.ca_certs config option.
119
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
120
6238.2.10 by Jelmer Vernooij
Add more tests for ssl.ca_certs option.
121
def ca_certs_from_store(path):
122
    if not os.path.exists(path):
123
        raise ValueError("ca certs path %s does not exist" % path)
124
    return path
125
126
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
127
def cert_reqs_from_store(unicode_str):
128
    import ssl
129
    try:
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
130
        return {"required": ssl.CERT_REQUIRED,
131
                "none": ssl.CERT_NONE}[unicode_str]
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
132
    except KeyError:
133
        raise ValueError("invalid value %s" % unicode_str)
134
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
135
6437.31.1 by Vincent Ladeuil
Default to no ssl cert verification on osx and windows
136
def default_ca_reqs():
137
    if sys.platform in ('win32', 'darwin'):
138
        # FIXME: Once we get a native access to root certificates there, this
139
        # won't needed anymore. See http://pad.lv/920455 -- vila 2012-02-15
140
        return u'none'
141
    else:
142
        return u'required'
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
143
144
opt_ssl_ca_certs = config.Option('ssl.ca_certs',
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
145
                                 from_unicode=ca_certs_from_store,
146
                                 default=default_ca_certs,
147
                                 invalid='warning',
148
                                 help="""\
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
149
Path to certification authority certificates to trust.
6437.25.6 by Vincent Ladeuil
Feedback from mgz.
150
151
This should be a valid path to a bundle containing all root Certificate
152
Authorities used to verify an https server certificate.
153
154
Use ssl.cert_reqs=none to disable certificate verification.
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
155
""")
156
157
opt_ssl_cert_reqs = config.Option('ssl.cert_reqs',
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
158
                                  default=default_ca_reqs,
159
                                  from_unicode=cert_reqs_from_store,
160
                                  invalid='error',
161
                                  help="""\
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
162
Whether to require a certificate from the remote side. (default:required)
163
164
Possible values:
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
165
 * none: Certificates ignored
6437.25.4 by Vincent Ladeuil
Get rid of 'optional' for ssl.ca_reqs to simplify the model.
166
 * required: Certificates required and validated
6238.2.8 by Jelmer Vernooij
Move options to bzrlib/transport/http/_urllib2_wrappers.py
167
""")
168
6238.1.2 by John Arbash Meinel, Jelmer Vernooij
Only attempt to import kerberos module once.
169
checked_kerberos = False
170
kerberos = None
171
172
5393.6.1 by Toshio Kuratomi
Fix branching from lp: repositories under python-2.7
173
class addinfourl(urllib2.addinfourl):
5393.6.3 by Toshio Kuratomi
Fix typo in implementation of python-2.7 xmlrpclib fix and add comment + entry in NEWS
174
    '''Replacement addinfourl class compatible with python-2.7's xmlrpclib
175
176
    In python-2.7, xmlrpclib expects that the response object that it receives
177
    has a getheader method.  httplib.HTTPResponse provides this but
178
    urllib2.addinfourl does not.  Add the necessary functions here, ported to
179
    use the internal data structures of addinfourl.
180
    '''
5393.6.1 by Toshio Kuratomi
Fix branching from lp: repositories under python-2.7
181
182
    def getheader(self, name, default=None):
183
        if self.headers is None:
184
            raise httplib.ResponseNotReady()
5393.6.3 by Toshio Kuratomi
Fix typo in implementation of python-2.7 xmlrpclib fix and add comment + entry in NEWS
185
        return self.headers.getheader(name, default)
5393.6.1 by Toshio Kuratomi
Fix branching from lp: repositories under python-2.7
186
187
    def getheaders(self):
188
        if self.headers is None:
189
            raise httplib.ResponseNotReady()
190
        return self.headers.items()
191
192
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
193
class _ReportingFileSocket(object):
194
195
    def __init__(self, filesock, report_activity=None):
196
        self.filesock = filesock
197
        self._report_activity = report_activity
198
4776.2.1 by Vincent Ladeuil
Support no activity report on http sockets.
199
    def report_activity(self, size, direction):
200
        if self._report_activity:
201
            self._report_activity(size, direction)
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
202
203
    def read(self, size=1):
204
        s = self.filesock.read(size)
4776.2.1 by Vincent Ladeuil
Support no activity report on http sockets.
205
        self.report_activity(len(s), 'read')
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
206
        return s
207
5848.2.1 by John Arbash Meinel
Break compatibility with python <2.6.
208
    def readline(self, size=-1):
209
        s = self.filesock.readline(size)
210
        self.report_activity(len(s), 'read')
211
        return s
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
212
213
    def __getattr__(self, name):
214
        return getattr(self.filesock, name)
215
216
217
class _ReportingSocket(object):
218
219
    def __init__(self, sock, report_activity=None):
3287.3.3 by Andrew Bennetts
A slightly neater hack for forcing buffering, thanks to John.
220
        self.sock = sock
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
221
        self._report_activity = report_activity
222
4776.2.1 by Vincent Ladeuil
Support no activity report on http sockets.
223
    def report_activity(self, size, direction):
224
        if self._report_activity:
225
            self._report_activity(size, direction)
226
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
227
    def sendall(self, s, *args):
4105.1.1 by Andrew Bennetts
Clean-up _ReportingSocket.send/sendall slightly.
228
        self.sock.sendall(s, *args)
4776.2.1 by Vincent Ladeuil
Support no activity report on http sockets.
229
        self.report_activity(len(s), 'write')
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
230
231
    def recv(self, *args):
232
        s = self.sock.recv(*args)
4776.2.1 by Vincent Ladeuil
Support no activity report on http sockets.
233
        self.report_activity(len(s), 'read')
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
234
        return s
3287.3.3 by Andrew Bennetts
A slightly neater hack for forcing buffering, thanks to John.
235
236
    def makefile(self, mode='r', bufsize=-1):
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
237
        # httplib creates a fileobject that doesn't do buffering, which
238
        # makes fp.readline() very expensive because it only reads one byte
239
        # at a time.  So we wrap the socket in an object that forces
240
        # sock.makefile to make a buffered file.
241
        fsock = self.sock.makefile(mode, 65536)
242
        # And wrap that into a reporting kind of fileobject
243
        return _ReportingFileSocket(fsock, self._report_activity)
3287.3.3 by Andrew Bennetts
A slightly neater hack for forcing buffering, thanks to John.
244
245
    def __getattr__(self, name):
246
        return getattr(self.sock, name)
247
248
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
249
# We define our own Response class to keep our httplib pipe clean
250
class Response(httplib.HTTPResponse):
2004.1.16 by v.ladeuil+lp at free
Add tests against erroneous http status lines.
251
    """Custom HTTPResponse, to avoid the need to decorate.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
252
253
    httplib prefers to decorate the returned objects, rather
254
    than using a custom object.
255
    """
256
2004.1.7 by vila
Better handling of passwords (user should be queried only once).
257
    # Some responses have bodies in which we have no interest
5504.4.1 by Vincent Ladeuil
Fix http test spurious failures and get rid of some useless messages in log.
258
    _body_ignored_responses = [301,302, 303, 307, 400, 401, 403, 404, 501]
2004.1.7 by vila
Better handling of passwords (user should be queried only once).
259
3146.3.4 by Vincent Ladeuil
Review feedback, simpler loops.
260
    # in finish() below, we may have to discard several MB in the worst
261
    # case. To avoid buffering that much, we read and discard by chunks
262
    # instead. The underlying file is either a socket or a StringIO, so reading
263
    # 8k chunks should be fine.
264
    _discarded_buf_size = 8192
265
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
266
    def begin(self):
267
        """Begin to read the response from the server.
268
269
        httplib assumes that some responses get no content and do
270
        not even attempt to read the body in that case, leaving
271
        the body in the socket, blocking the next request. Let's
272
        try to workaround that.
273
        """
2004.1.2 by vila
Implements a BasicAuthManager.
274
        httplib.HTTPResponse.begin(self)
2004.1.7 by vila
Better handling of passwords (user should be queried only once).
275
        if self.status in self._body_ignored_responses:
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
276
            if self.debuglevel >= 2:
2004.1.16 by v.ladeuil+lp at free
Add tests against erroneous http status lines.
277
                print "For status: [%s]," % self.status,
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
278
                print "will ready body, length: %s" % self.length
2004.1.16 by v.ladeuil+lp at free
Add tests against erroneous http status lines.
279
            if not (self.length is None or self.will_close):
280
                # In some cases, we just can't read the body not
281
                # even try or we may encounter a 104, 'Connection
282
                # reset by peer' error if there is indeed no body
283
                # and the server closed the connection just after
284
                # having issued the response headers (even if the
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
285
                # headers indicate a Content-Type...)
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
286
                body = self.read(self.length)
287
                if self.debuglevel >= 9:
3024.2.3 by Vincent Ladeuil
Rewrite http_readv to allow several GET requests. Smoke tested against branch reported in the bug.
288
                    # This one can be huge and is generally not interesting
2004.1.16 by v.ladeuil+lp at free
Add tests against erroneous http status lines.
289
                    print "Consumed body: [%s]" % body
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
290
            self.close()
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
291
        elif self.status == 200:
292
            # Whatever the request is, it went ok, so we surely don't want to
293
            # close the connection. Some cases are not correctly detected by
294
            # httplib.HTTPConnection.getresponse (called by
295
            # httplib.HTTPResponse.begin). The CONNECT response for the https
2955.2.1 by Vincent Ladeuil
Fix #160012 by leaving the http pipeline related exceptions raise.
296
            # through proxy case is one.  Note: the 'will_close' below refers
297
            # to the "true" socket between us and the server, whereas the
298
            # 'close()' above refers to the copy of that socket created by
299
            # httplib for the response itself. So, in the if above we close the
300
            # socket to indicate that we are done with the response whereas
301
            # below we keep the socket with the server opened.
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
302
            self.will_close = False
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
303
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
304
    def finish(self):
3059.2.11 by Vincent Ladeuil
Fix typos mentioned by spiv.
305
        """Finish reading the body.
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
306
3059.2.11 by Vincent Ladeuil
Fix typos mentioned by spiv.
307
        In some cases, the client may have left some bytes to read in the
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
308
        body. That will block the next request to succeed if we use a
3059.2.11 by Vincent Ladeuil
Fix typos mentioned by spiv.
309
        persistent connection. If we don't use a persistent connection, well,
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
310
        nothing will block the next request since a new connection will be
311
        issued anyway.
3104.3.3 by Vincent Ladeuil
Jam's and Aaron feedback about bug #175886.
312
313
        :return: the number of bytes left on the socket (may be None)
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
314
        """
3104.3.3 by Vincent Ladeuil
Jam's and Aaron feedback about bug #175886.
315
        pending = None
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
316
        if not self.isclosed():
317
            # Make sure nothing was left to be read on the socket
3104.3.1 by Vincent Ladeuil
Fix #175886 by reading remaining bytes by chunks.
318
            pending = 0
3146.3.2 by Vincent Ladeuil
Fix #179368 by keeping the current range hint on ShortReadvErrors.
319
            data = True
3146.3.4 by Vincent Ladeuil
Review feedback, simpler loops.
320
            while data and self.length:
321
                # read() will update self.length
322
                data = self.read(min(self.length, self._discarded_buf_size))
3104.3.1 by Vincent Ladeuil
Fix #175886 by reading remaining bytes by chunks.
323
                pending += len(data)
324
            if pending:
3146.3.2 by Vincent Ladeuil
Fix #179368 by keeping the current range hint on ShortReadvErrors.
325
                trace.mutter("%s bytes left on the HTTP socket", pending)
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
326
            self.close()
3104.3.3 by Vincent Ladeuil
Jam's and Aaron feedback about bug #175886.
327
        return pending
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
328
2004.1.2 by vila
Implements a BasicAuthManager.
329
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
330
# Not inheriting from 'object' because httplib.HTTPConnection doesn't.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
331
class AbstractHTTPConnection:
332
    """A custom HTTP(S) Connection, which can reset itself on a bad response"""
333
334
    response_class = Response
335
3104.3.3 by Vincent Ladeuil
Jam's and Aaron feedback about bug #175886.
336
    # When we detect a server responding with the whole file to range requests,
337
    # we want to warn. But not below a given thresold.
338
    _range_warning_thresold = 1024 * 1024
339
4776.2.1 by Vincent Ladeuil
Support no activity report on http sockets.
340
    def __init__(self, report_activity=None):
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
341
        self._response = None
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
342
        self._report_activity = report_activity
3104.3.3 by Vincent Ladeuil
Jam's and Aaron feedback about bug #175886.
343
        self._ranges_received_whole_file = None
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
344
345
    def _mutter_connect(self):
3104.3.4 by Vincent Ladeuil
Add test.
346
        netloc = '%s:%s' % (self.host, self.port)
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
347
        if self.proxied_host is not None:
348
            netloc += '(proxy for %s)' % self.proxied_host
349
        trace.mutter('* About to connect() to %s' % netloc)
350
351
    def getresponse(self):
352
        """Capture the response to be able to cleanup"""
353
        self._response = httplib.HTTPConnection.getresponse(self)
354
        return self._response
355
3059.2.2 by Vincent Ladeuil
Read http responses on demand without buffering the whole body
356
    def cleanup_pipe(self):
3111.1.24 by Vincent Ladeuil
Cleanups.
357
        """Read the remaining bytes of the last response if any."""
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
358
        if self._response is not None:
5504.4.1 by Vincent Ladeuil
Fix http test spurious failures and get rid of some useless messages in log.
359
            try:
360
                pending = self._response.finish()
361
                # Warn the user (once)
362
                if (self._ranges_received_whole_file is None
363
                    and self._response.status == 200
364
                    and pending and pending > self._range_warning_thresold
365
                    ):
366
                    self._ranges_received_whole_file = True
367
                    trace.warning(
368
                        'Got a 200 response when asking for multiple ranges,'
369
                        ' does your server at %s:%s support range requests?',
370
                        self.host, self.port)
371
            except socket.error, e:
372
                # It's conceivable that the socket is in a bad state here
373
                # (including some test cases) and in this case, it doesn't need
374
                # cleaning anymore, so no need to fail, we just get rid of the
375
                # socket and let callers reconnect
376
                if (len(e.args) == 0
377
                    or e.args[0] not in (errno.ECONNRESET, errno.ECONNABORTED)):
378
                    raise
379
                self.close()
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
380
            self._response = None
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
381
        # Preserve our preciousss
382
        sock = self.sock
383
        self.sock = None
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
384
        # Let httplib.HTTPConnection do its housekeeping
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
385
        self.close()
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
386
        # Restore our preciousss
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
387
        self.sock = sock
388
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
389
    def _wrap_socket_for_reporting(self, sock):
390
        """Wrap the socket before anybody use it."""
391
        self.sock = _ReportingSocket(sock, self._report_activity)
392
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
393
394
class HTTPConnection(AbstractHTTPConnection, httplib.HTTPConnection):
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
395
396
    # XXX: Needs refactoring at the caller level.
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
397
    def __init__(self, host, port=None, proxied_host=None,
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
398
                 report_activity=None, ca_certs=None):
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
399
        AbstractHTTPConnection.__init__(self, report_activity=report_activity)
2929.3.9 by Vincent Ladeuil
Don't pretend we support HTTP/0.9 since we don't and do that correctly.
400
        # Use strict=True since we don't support HTTP/0.9
401
        httplib.HTTPConnection.__init__(self, host, port, strict=True)
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
402
        self.proxied_host = proxied_host
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
403
        # ca_certs is ignored, it's only relevant for https
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
404
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
405
    def connect(self):
406
        if 'http' in debug.debug_flags:
407
            self._mutter_connect()
408
        httplib.HTTPConnection.connect(self)
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
409
        self._wrap_socket_for_reporting(self.sock)
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
410
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
411
412
class HTTPSConnection(AbstractHTTPConnection, httplib.HTTPSConnection):
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
413
414
    def __init__(self, host, port=None, key_file=None, cert_file=None,
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
415
                 proxied_host=None,
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
416
                 report_activity=None, ca_certs=None):
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
417
        AbstractHTTPConnection.__init__(self, report_activity=report_activity)
2929.3.9 by Vincent Ladeuil
Don't pretend we support HTTP/0.9 since we don't and do that correctly.
418
        # Use strict=True since we don't support HTTP/0.9
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
419
        httplib.HTTPSConnection.__init__(self, host, port,
2929.3.9 by Vincent Ladeuil
Don't pretend we support HTTP/0.9 since we don't and do that correctly.
420
                                         key_file, cert_file, strict=True)
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
421
        self.proxied_host = proxied_host
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
422
        self.ca_certs = ca_certs
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
423
424
    def connect(self):
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
425
        if 'http' in debug.debug_flags:
426
            self._mutter_connect()
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
427
        httplib.HTTPConnection.connect(self)
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
428
        self._wrap_socket_for_reporting(self.sock)
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
429
        if self.proxied_host is None:
430
            self.connect_to_origin()
431
432
    def connect_to_origin(self):
6238.2.7 by Jelmer Vernooij
Add ssl.cert_reqs.
433
        # FIXME JRV 2011-12-18: Use location config here?
434
        config_stack = config.GlobalStack()
435
        cert_reqs = config_stack.get('ssl.cert_reqs')
6437.50.1 by Martin Packman
Match certs against proxied hostname when using HTTPS proxy
436
        if self.proxied_host is not None:
437
            host = self.proxied_host.split(":", 1)[0]
438
        else:
439
            host = self.host
6238.2.13 by Jelmer Vernooij
add notes on using -Ossl.cert_reqs=none.
440
        if cert_reqs == ssl.CERT_NONE:
6524.4.2 by Marius Kruger
use ui.ui_factory.show_user_warning to warn about not checking ssl certs, so that we only do it once
441
            ui.ui_factory.show_user_warning('not_checking_ssl_cert', host=host)
442
            ui.ui_factory.suppressed_warnings.add('not_checking_ssl_cert')
6437.25.1 by Vincent Ladeuil
Provide default paths for ca certs for supported platforms
443
            ca_certs = None
6238.2.13 by Jelmer Vernooij
add notes on using -Ossl.cert_reqs=none.
444
        else:
6437.25.1 by Vincent Ladeuil
Provide default paths for ca certs for supported platforms
445
            if self.ca_certs is None:
446
                ca_certs = config_stack.get('ssl.ca_certs')
447
            else:
448
                ca_certs = self.ca_certs
6238.2.14 by Jelmer Vernooij
Urgh - only warn if there actually isn't a ca certs file set.
449
            if ca_certs is None:
450
                trace.warning(
6437.25.6 by Vincent Ladeuil
Feedback from mgz.
451
                    "No valid trusted SSL CA certificates file set. See "
6238.2.14 by Jelmer Vernooij
Urgh - only warn if there actually isn't a ca certs file set.
452
                    "'bzr help ssl.ca_certs' for more information on setting "
6437.25.6 by Vincent Ladeuil
Feedback from mgz.
453
                    "trusted CAs.")
6238.2.13 by Jelmer Vernooij
add notes on using -Ossl.cert_reqs=none.
454
        try:
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
455
            ssl_sock = ssl.wrap_socket(
456
                self.sock, self.key_file, self.cert_file,
6238.2.13 by Jelmer Vernooij
add notes on using -Ossl.cert_reqs=none.
457
                cert_reqs=cert_reqs, ca_certs=ca_certs)
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
458
        except ssl.SSLError:
6437.25.7 by Vincent Ladeuil
We need a clearer understanding of ssl errors.
459
            trace.note(
460
                "\n"
461
                "See `bzr help ssl.ca_certs` for how to specify trusted CA"
462
                "certificates.\n"
463
                "Pass -Ossl.cert_reqs=none to disable certificate "
464
                "verification entirely.\n")
6238.2.13 by Jelmer Vernooij
add notes on using -Ossl.cert_reqs=none.
465
            raise
6437.25.4 by Vincent Ladeuil
Get rid of 'optional' for ssl.ca_reqs to simplify the model.
466
        if cert_reqs == ssl.CERT_REQUIRED:
6613.1.5 by Vincent Ladeuil
Help python2.6 compatibility.
467
            peer_cert = ssl_sock.getpeercert()
468
            ssl.match_hostname(peer_cert, host)
6238.2.4 by Jelmer Vernooij
Verify ssl certificates when using urllib.
469
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
470
        # Wrap the ssl socket before anybody use it
471
        self._wrap_socket_for_reporting(ssl_sock)
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
472
473
474
class Request(urllib2.Request):
475
    """A custom Request object.
476
477
    urllib2 determines the request method heuristically (based on
478
    the presence or absence of data). We set the method
479
    statically.
480
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
481
    The Request object tracks:
482
    - the connection the request will be made on.
483
    - the authentication parameters needed to preventively set
484
      the authentication header once a first authentication have
485
       been made.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
486
    """
487
488
    def __init__(self, method, url, data=None, headers={},
489
                 origin_req_host=None, unverifiable=False,
2520.2.2 by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request
490
                 connection=None, parent=None,
4795.4.2 by Vincent Ladeuil
Revert auth reuse.
491
                 accepted_errors=None):
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
492
        urllib2.Request.__init__(self, url, data, headers,
493
                                 origin_req_host, unverifiable)
494
        self.method = method
495
        self.connection = connection
2520.2.2 by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request
496
        self.accepted_errors = accepted_errors
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
497
        # To handle redirections
498
        self.parent = parent
499
        self.redirected_to = None
2164.2.15 by Vincent Ladeuil
Http redirections are not followed by default. Do not use hints
500
        # Unless told otherwise, redirections are not followed
501
        self.follow_redirections = False
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
502
        # auth and proxy_auth are dicts containing, at least
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
503
        # (scheme, host, port, realm, user, password, protocol, path).
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
504
        # The dict entries are mostly handled by the AuthHandler.
505
        # Some authentication schemes may add more entries.
4795.4.2 by Vincent Ladeuil
Revert auth reuse.
506
        self.auth = {}
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
507
        self.proxy_auth = {}
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
508
        self.proxied_host = None
2420.1.3 by Vincent Ladeuil
Implement http proxy basic authentication.
509
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
510
    def get_method(self):
511
        return self.method
512
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
513
    def set_proxy(self, proxy, type):
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
514
        """Set the proxy and remember the proxied host."""
4776.2.7 by Vincent Ladeuil
Fix proxy CONNECT for non-default ports.
515
        host, port = urllib.splitport(self.get_host())
516
        if port is None:
517
            # We need to set the default port ourselves way before it gets set
518
            # in the HTTP[S]Connection object at build time.
519
            if self.type == 'https':
520
                conn_class = HTTPSConnection
521
            else:
522
                conn_class = HTTPConnection
523
            port = conn_class.default_port
524
        self.proxied_host = '%s:%s' % (host, port)
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
525
        urllib2.Request.set_proxy(self, proxy, type)
4797.75.1 by Andrew Bennetts
Backport fix for bug 558343 from lp:bzr r5220.
526
        # When urllib2 makes a https request with our wrapper code and a proxy,
527
        # it sets Host to the https proxy, not the host we want to talk to.
528
        # I'm fairly sure this is our fault, but what is the cause is an open
529
        # question. -- Robert Collins May 8 2010.
530
        self.add_unredirected_header('Host', self.proxied_host)
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
531
532
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
533
class _ConnectRequest(Request):
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
534
535
    def __init__(self, request):
536
        """Constructor
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
537
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
538
        :param request: the first request sent to the proxied host, already
539
            processed by the opener (i.e. proxied_host is already set).
540
        """
541
        # We give a fake url and redefine get_selector or urllib2 will be
542
        # confused
543
        Request.__init__(self, 'CONNECT', request.get_full_url(),
544
                         connection=request.connection)
3376.2.4 by Martin Pool
Remove every assert statement from bzrlib!
545
        if request.proxied_host is None:
546
            raise AssertionError()
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
547
        self.proxied_host = request.proxied_host
548
549
    def get_selector(self):
550
        return self.proxied_host
551
552
    def set_proxy(self, proxy, type):
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
553
        """Set the proxy without remembering the proxied host.
554
555
        We already know the proxied host by definition, the CONNECT request
556
        occurs only when the connection goes through a proxy. The usual
557
        processing (masquerade the request so that the connection is done to
558
        the proxy while the request is targeted at another host) does not apply
559
        here. In fact, the connection is already established with proxy and we
560
        just want to enable the SSL tunneling.
561
        """
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
562
        urllib2.Request.set_proxy(self, proxy, type)
563
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
564
565
class ConnectionHandler(urllib2.BaseHandler):
566
    """Provides connection-sharing by pre-processing requests.
567
568
    urllib2 provides no way to access the HTTPConnection object
569
    internally used. But we need it in order to achieve
570
    connection sharing. So, we add it to the request just before
571
    it is processed, and then we override the do_open method for
2363.4.7 by Vincent Ladeuil
Deeper tests, prepare the auth setting that will avoid the
572
    http[s] requests in AbstractHTTPHandler.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
573
    """
574
575
    handler_order = 1000 # after all pre-processings
576
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
577
    def __init__(self, report_activity=None, ca_certs=None):
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
578
        self._report_activity = report_activity
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
579
        self.ca_certs = ca_certs
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
580
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
581
    def create_connection(self, request, http_connection_class):
582
        host = request.get_host()
583
        if not host:
2004.1.15 by v.ladeuil+lp at free
Better design for bogus servers. Both urllib and pycurl pass tests.
584
            # Just a bit of paranoia here, this should have been
585
            # handled in the higher levels
2004.1.27 by v.ladeuil+lp at free
Fix bug #57644 by issuing an explicit error message.
586
            raise errors.InvalidURL(request.get_full_url(), 'no host given.')
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
587
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
588
        # We create a connection (but it will not connect until the first
589
        # request is made)
2004.1.42 by v.ladeuil+lp at free
Fix #70803 by catching the httplib exception.
590
        try:
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
591
            connection = http_connection_class(
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
592
                host, proxied_host=request.proxied_host,
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
593
                report_activity=self._report_activity,
594
                ca_certs=self.ca_certs)
2004.1.42 by v.ladeuil+lp at free
Fix #70803 by catching the httplib exception.
595
        except httplib.InvalidURL, exception:
596
            # There is only one occurrence of InvalidURL in httplib
597
            raise errors.InvalidURL(request.get_full_url(),
598
                                    extra='nonnumeric port')
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
599
600
        return connection
601
602
    def capture_connection(self, request, http_connection_class):
603
        """Capture or inject the request connection.
604
605
        Two cases:
606
        - the request have no connection: create a new one,
607
608
        - the request have a connection: this one have been used
609
          already, let's capture it, so that we can give it to
610
          another transport to be reused. We don't do that
611
          ourselves: the Transport object get the connection from
612
          a first request and then propagate it, from request to
613
          request or to cloned transports.
614
        """
615
        connection = request.connection
616
        if connection is None:
617
            # Create a new one
618
            connection = self.create_connection(request, http_connection_class)
619
            request.connection = connection
620
621
        # All connections will pass here, propagate debug level
622
        connection.set_debuglevel(DEBUG)
623
        return request
624
625
    def http_request(self, request):
626
        return self.capture_connection(request, HTTPConnection)
627
628
    def https_request(self, request):
629
        return self.capture_connection(request, HTTPSConnection)
630
631
632
class AbstractHTTPHandler(urllib2.AbstractHTTPHandler):
633
    """A custom handler for HTTP(S) requests.
634
635
    We overrive urllib2.AbstractHTTPHandler to get a better
636
    control of the connection, the ability to implement new
637
    request types and return a response able to cope with
638
    persistent connections.
639
    """
640
641
    # We change our order to be before urllib2 HTTP[S]Handlers
2004.3.1 by vila
Test ConnectionError exceptions.
642
    # and be chosen instead of them (the first http_open called
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
643
    # wins).
644
    handler_order = 400
645
646
    _default_headers = {'Pragma': 'no-cache',
647
                        'Cache-control': 'max-age=0',
648
                        'Connection': 'Keep-Alive',
2004.1.15 by v.ladeuil+lp at free
Better design for bogus servers. Both urllib and pycurl pass tests.
649
                        'User-agent': 'bzr/%s (urllib)' % bzrlib_version,
2004.3.3 by vila
Better (but still incomplete) design for bogus servers.
650
                        'Accept': '*/*',
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
651
                        }
652
653
    def __init__(self):
2004.1.16 by v.ladeuil+lp at free
Add tests against erroneous http status lines.
654
        urllib2.AbstractHTTPHandler.__init__(self, debuglevel=DEBUG)
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
655
2004.1.15 by v.ladeuil+lp at free
Better design for bogus servers. Both urllib and pycurl pass tests.
656
    def http_request(self, request):
657
        """Common headers setting"""
658
659
        request.headers.update(self._default_headers.copy())
660
        # FIXME: We may have to add the Content-Length header if
661
        # we have data to send.
662
        return request
663
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
664
    def retry_or_raise(self, http_class, request, first_try):
665
        """Retry the request (once) or raise the exception.
2004.3.1 by vila
Test ConnectionError exceptions.
666
667
        urllib2 raises exception of application level kind, we
668
        just have to translate them.
669
670
        httplib can raise exceptions of transport level (badly
671
        formatted dialog, loss of connexion or socket level
672
        problems). In that case we should issue the request again
673
        (httplib will close and reopen a new connection if
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
674
        needed).
675
        """
676
        # When an exception occurs, we give back the original
677
        # Traceback or the bugs are hard to diagnose.
678
        exc_type, exc_val, exc_tb = sys.exc_info()
679
        if exc_type == socket.gaierror:
680
            # No need to retry, that will not help
681
            raise errors.ConnectionError("Couldn't resolve host '%s'"
682
                                         % request.get_origin_req_host(),
683
                                         orig_error=exc_val)
2955.2.1 by Vincent Ladeuil
Fix #160012 by leaving the http pipeline related exceptions raise.
684
        elif isinstance(exc_val, httplib.ImproperConnectionState):
685
            # The httplib pipeline is in incorrect state, it's a bug in our
686
            # implementation.
687
            raise exc_type, exc_val, exc_tb
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
688
        else:
689
            if first_try:
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
690
                if self._debuglevel >= 2:
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
691
                    print 'Received exception: [%r]' % exc_val
692
                    print '  On connection: [%r]' % request.connection
693
                    method = request.get_method()
694
                    url = request.get_full_url()
695
                    print '  Will retry, %s %r' % (method, url)
696
                request.connection.close()
697
                response = self.do_open(http_class, request, False)
698
            else:
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
699
                if self._debuglevel >= 2:
2004.1.39 by v.ladeuil+lp at free
Fix a race condition that make selftest fail once in a while.
700
                    print 'Received second exception: [%r]' % exc_val
701
                    print '  On connection: [%r]' % request.connection
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
702
                if exc_type in (httplib.BadStatusLine, httplib.UnknownProtocol):
703
                    # httplib.BadStatusLine and
704
                    # httplib.UnknownProtocol indicates that a
705
                    # bogus server was encountered or a bad
706
                    # connection (i.e. transient errors) is
707
                    # experimented, we have already retried once
708
                    # for that request so we raise the exception.
709
                    my_exception = errors.InvalidHttpResponse(
710
                        request.get_full_url(),
711
                        'Bad status line received',
712
                        orig_error=exc_val)
4628.1.2 by Vincent Ladeuil
More complete fix.
713
                elif (isinstance(exc_val, socket.error) and len(exc_val.args)
5599.3.3 by John Arbash Meinel
Treate WSAECONNABORTED the same as WSAECONNRESET in the http _urllib2 code. Bug #686587
714
                      and exc_val.args[0] in (errno.ECONNRESET, 10053, 10054)):
715
                      # 10053 == WSAECONNABORTED
716
                      # 10054 == WSAECONNRESET
4628.1.2 by Vincent Ladeuil
More complete fix.
717
                    raise errors.ConnectionReset(
718
                        "Connection lost while sending request.")
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
719
                else:
720
                    # All other exception are considered connection related.
721
722
                    # socket errors generally occurs for reasons
723
                    # far outside our scope, so closing the
724
                    # connection and retrying is the best we can
725
                    # do.
726
727
                    my_exception = errors.ConnectionError(
728
                        msg= 'while sending %s %s:' % (request.get_method(),
729
                                                       request.get_selector()),
730
                        orig_error=exc_val)
731
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
732
                if self._debuglevel >= 2:
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
733
                    print 'On connection: [%r]' % request.connection
734
                    method = request.get_method()
735
                    url = request.get_full_url()
736
                    print '  Failed again, %s %r' % (method, url)
737
                    print '  Will raise: [%r]' % my_exception
738
                raise my_exception, None, exc_tb
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
739
        return response
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
740
741
    def do_open(self, http_class, request, first_try=True):
742
        """See urllib2.AbstractHTTPHandler.do_open for the general idea.
743
744
        The request will be retried once if it fails.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
745
        """
746
        connection = request.connection
3376.2.4 by Martin Pool
Remove every assert statement from bzrlib!
747
        if connection is None:
748
            raise AssertionError(
749
                'Cannot process a request without a connection')
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
750
2004.1.19 by v.ladeuil+lp at free
Test protocol version in http responses.
751
        # Get all the headers
2004.1.15 by v.ladeuil+lp at free
Better design for bogus servers. Both urllib and pycurl pass tests.
752
        headers = {}
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
753
        headers.update(request.header_items())
754
        headers.update(request.unredirected_hdrs)
3430.1.1 by Vincent Ladeuil
Fix bug #229076 by fixing header names before sending the request.
755
        # Some servers or proxies will choke on headers not properly
756
        # cased. httplib/urllib/urllib2 all use capitalize to get canonical
757
        # header names, but only python2.5 urllib2 use title() to fix them just
758
        # before sending the request. And not all versions of python 2.5 do
759
        # that. Since we replace urllib2.AbstractHTTPHandler.do_open we do it
760
        # ourself below.
3430.1.2 by Vincent Ladeuil
Fixed as per Matt Nordhoff review.
761
        headers = dict((name.title(), val) for name, val in headers.iteritems())
2004.3.1 by vila
Test ConnectionError exceptions.
762
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
763
        try:
3052.3.3 by Vincent Ladeuil
Add -Dhttp support.
764
            method = request.get_method()
765
            url = request.get_selector()
766
            connection._send_request(method, url,
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
767
                                     # FIXME: implements 100-continue
768
                                     #None, # We don't send the body yet
769
                                     request.get_data(),
770
                                     headers)
3052.3.3 by Vincent Ladeuil
Add -Dhttp support.
771
            if 'http' in debug.debug_flags:
772
                trace.mutter('> %s %s' % (method, url))
5957.2.3 by Vincent Ladeuil
Mask credentials in the -Dhttp logging
773
                hdrs = []
774
                for k,v in headers.iteritems():
775
                    # People are often told to paste -Dhttp output to help
776
                    # debug. Don't compromise credentials.
777
                    if k in ('Authorization', 'Proxy-Authorization'):
778
                        v = '<masked>'
779
                    hdrs.append('%s: %s' % (k, v))
3052.3.3 by Vincent Ladeuil
Add -Dhttp support.
780
                trace.mutter('> ' + '\n> '.join(hdrs) + '\n')
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
781
            if self._debuglevel >= 1:
782
                print 'Request sent: [%r] from (%s)' \
783
                    % (request, request.connection.sock.getsockname())
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
784
            response = connection.getresponse()
785
            convert_to_addinfourl = True
6613.1.1 by Vincent Ladeuil
Use ssl module for the match_hostname function
786
        except (ssl.SSLError, ssl.CertificateError):
6238.2.21 by Vincent Ladeuil
The https test server will now refuse connections if an ssl error occurs during the handshake. Ssl errors and certificate errors aborts requests without re-trying them.
787
            # Something is wrong with either the certificate or the hostname,
788
            # re-trying won't help
789
            raise
2004.1.37 by v.ladeuil+lp at free
Small refactoring.
790
        except (socket.gaierror, httplib.BadStatusLine, httplib.UnknownProtocol,
791
                socket.error, httplib.HTTPException):
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
792
            response = self.retry_or_raise(http_class, request, first_try)
793
            convert_to_addinfourl = False
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
794
795
# FIXME: HTTPConnection does not fully support 100-continue (the
796
# server responses are just ignored)
797
798
#        if code == 100:
799
#            mutter('Will send the body')
800
#            # We can send the body now
801
#            body = request.get_data()
802
#            if body is None:
803
#                raise URLError("No data given")
804
#            connection.send(body)
805
#            response = connection.getresponse()
806
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
807
        if self._debuglevel >= 2:
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
808
            print 'Receives response: %r' % response
809
            print '  For: %r(%r)' % (request.get_method(),
810
                                     request.get_full_url())
811
812
        if convert_to_addinfourl:
813
            # Shamelessly copied from urllib2
814
            req = request
815
            r = response
816
            r.recv = r.read
3287.3.2 by Andrew Bennetts
Buffer 64k, rather than just 8k.
817
            fp = socket._fileobject(r, bufsize=65536)
5393.6.1 by Toshio Kuratomi
Fix branching from lp: repositories under python-2.7
818
            resp = addinfourl(fp, r.msg, req.get_full_url())
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
819
            resp.code = r.status
820
            resp.msg = r.reason
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
821
            resp.version = r.version
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
822
            if self._debuglevel >= 2:
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
823
                print 'Create addinfourl: %r' % resp
824
                print '  For: %r(%r)' % (request.get_method(),
825
                                         request.get_full_url())
3059.2.5 by Vincent Ladeuil
DAMN^64, the http test server is 1.0 not 1.1 :( Better pipe cleaning and less readv caching (since that's the point of the whole fix).
826
            if 'http' in debug.debug_flags:
827
                version = 'HTTP/%d.%d'
828
                try:
829
                    version = version % (resp.version / 10,
830
                                         resp.version % 10)
831
                except:
832
                    version = 'HTTP/%r' % resp.version
833
                trace.mutter('< %s %s %s' % (version, resp.code,
834
                                             resp.msg))
835
                # Use the raw header lines instead of treating resp.info() as a
836
                # dict since we may miss duplicated headers otherwise.
837
                hdrs = [h.rstrip('\r\n') for h in resp.info().headers]
838
                trace.mutter('< ' + '\n< '.join(hdrs) + '\n')
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
839
        else:
840
            resp = response
841
        return resp
842
843
844
class HTTPHandler(AbstractHTTPHandler):
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
845
    """A custom handler that just thunks into HTTPConnection"""
846
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
847
    def http_open(self, request):
848
        return self.do_open(HTTPConnection, request)
849
850
851
class HTTPSHandler(AbstractHTTPHandler):
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
852
    """A custom handler that just thunks into HTTPSConnection"""
853
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
854
    https_request = AbstractHTTPHandler.http_request
855
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
856
    def https_open(self, request):
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
857
        connection = request.connection
858
        if connection.sock is None and \
859
                connection.proxied_host is not None and \
860
                request.get_method() != 'CONNECT' : # Don't loop
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
861
            # FIXME: We need a gazillion connection tests here, but we still
862
            # miss a https server :-( :
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
863
            # - with and without proxy
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
864
            # - with and without certificate
865
            # - with self-signed certificate
866
            # - with and without authentication
2929.3.9 by Vincent Ladeuil
Don't pretend we support HTTP/0.9 since we don't and do that correctly.
867
            # - with good and bad credentials (especially the proxy auth around
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
868
            #   CONNECT)
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
869
            # - with basic and digest schemes
870
            # - reconnection on errors
871
            # - connection persistence behaviour (including reconnection)
872
873
            # We are about to connect for the first time via a proxy, we must
874
            # issue a CONNECT request first to establish the encrypted link
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
875
            connect = _ConnectRequest(request)
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
876
            response = self.parent.open(connect)
877
            if response.code != 200:
4797.75.1 by Andrew Bennetts
Backport fix for bug 558343 from lp:bzr r5220.
878
                raise errors.ConnectionError("Can't connect to %s via proxy %s" % (
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
879
                        connect.proxied_host, self.host))
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
880
            # Housekeeping
3059.2.2 by Vincent Ladeuil
Read http responses on demand without buffering the whole body
881
            connection.cleanup_pipe()
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
882
            # Establish the connection encryption
2540.2.1 by Vincent Ladeuil
Rough, working, tested against squid+apache in basic auth fix for #120678
883
            connection.connect_to_origin()
884
            # Propagate the connection to the original request
885
            request.connection = connection
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
886
        return self.do_open(HTTPSConnection, request)
887
888
class HTTPRedirectHandler(urllib2.HTTPRedirectHandler):
889
    """Handles redirect requests.
890
891
    We have to implement our own scheme because we use a specific
892
    Request object and because we want to implement a specific
893
    policy.
894
    """
895
    _debuglevel = DEBUG
896
    # RFC2616 says that only read requests should be redirected
897
    # without interacting with the user. But bzr use some
898
    # shortcuts to optimize against roundtrips which can leads to
899
    # write requests being issued before read requests of
900
    # containing dirs can be redirected. So we redirect write
901
    # requests in the same way which seems to respect the spirit
902
    # of the RFC if not its letter.
903
904
    def redirect_request(self, req, fp, code, msg, headers, newurl):
905
        """See urllib2.HTTPRedirectHandler.redirect_request"""
906
        # We would have preferred to update the request instead
907
        # of creating a new one, but the urllib2.Request object
908
        # has a too complicated creation process to provide a
909
        # simple enough equivalent update process. Instead, when
2164.2.29 by Vincent Ladeuil
Test the http redirection at the request level even if it's not
910
        # redirecting, we only update the following request in
911
        # the redirect chain with a reference to the parent
912
        # request .
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
913
2164.2.1 by v.ladeuil+lp at free
First rough http branch redirection implementation.
914
        # Some codes make no sense in our context and are treated
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
915
        # as errors:
916
917
        # 300: Multiple choices for different representations of
918
        #      the URI. Using that mechanisn with bzr will violate the
919
        #      protocol neutrality of Transport.
920
921
        # 304: Not modified (SHOULD only occurs with conditional
922
        #      GETs which are not used by our implementation)
923
924
        # 305: Use proxy. I can't imagine this one occurring in
925
        #      our context-- vila/20060909
926
927
        # 306: Unused (if the RFC says so...)
928
2164.2.1 by v.ladeuil+lp at free
First rough http branch redirection implementation.
929
        # If the code is 302 and the request is HEAD, some may
930
        # think that it is a sufficent hint that the file exists
931
        # and that we MAY avoid following the redirections. But
932
        # if we want to be sure, we MUST follow them.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
933
934
        if code in (301, 302, 303, 307):
935
            return Request(req.get_method(),newurl,
936
                           headers = req.headers,
937
                           origin_req_host = req.get_origin_req_host(),
938
                           unverifiable = True,
939
                           # TODO: It will be nice to be able to
940
                           # detect virtual hosts sharing the same
941
                           # IP address, that will allow us to
942
                           # share the same connection...
943
                           connection = None,
944
                           parent = req,
945
                           )
946
        else:
947
            raise urllib2.HTTPError(req.get_full_url(), code, msg, headers, fp)
948
2164.2.29 by Vincent Ladeuil
Test the http redirection at the request level even if it's not
949
    def http_error_302(self, req, fp, code, msg, headers):
2004.3.1 by vila
Test ConnectionError exceptions.
950
        """Requests the redirected to URI.
951
3059.2.2 by Vincent Ladeuil
Read http responses on demand without buffering the whole body
952
        Copied from urllib2 to be able to clean the pipe of the associated
953
        connection, *before* issuing the redirected request but *after* having
954
        eventually raised an error.
2004.3.1 by vila
Test ConnectionError exceptions.
955
        """
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
956
        # Some servers (incorrectly) return multiple Location headers
957
        # (so probably same goes for URI).  Use first header.
958
959
        # TODO: Once we get rid of addinfourl objects, the
960
        # following will need to be updated to use correct case
961
        # for headers.
962
        if 'location' in headers:
963
            newurl = headers.getheaders('location')[0]
964
        elif 'uri' in headers:
965
            newurl = headers.getheaders('uri')[0]
966
        else:
967
            return
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
968
        if self._debuglevel >= 1:
2164.2.1 by v.ladeuil+lp at free
First rough http branch redirection implementation.
969
            print 'Redirected to: %s (followed: %r)' % (newurl,
970
                                                        req.follow_redirections)
971
        if req.follow_redirections is False:
972
            req.redirected_to = newurl
973
            return fp
974
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
975
        newurl = urlparse.urljoin(req.get_full_url(), newurl)
976
977
        # This call succeeds or raise an error. urllib2 returns
978
        # if redirect_request returns None, but our
979
        # redirect_request never returns None.
980
        redirected_req = self.redirect_request(req, fp, code, msg, headers,
981
                                               newurl)
982
983
        # loop detection
984
        # .redirect_dict has a key url if url was previously visited.
985
        if hasattr(req, 'redirect_dict'):
986
            visited = redirected_req.redirect_dict = req.redirect_dict
987
            if (visited.get(newurl, 0) >= self.max_repeats or
988
                len(visited) >= self.max_redirections):
989
                raise urllib2.HTTPError(req.get_full_url(), code,
990
                                        self.inf_msg + msg, headers, fp)
991
        else:
992
            visited = redirected_req.redirect_dict = req.redirect_dict = {}
993
        visited[newurl] = visited.get(newurl, 0) + 1
994
995
        # We can close the fp now that we are sure that we won't
996
        # use it with HTTPError.
997
        fp.close()
998
        # We have all we need already in the response
3059.2.2 by Vincent Ladeuil
Read http responses on demand without buffering the whole body
999
        req.connection.cleanup_pipe()
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1000
1001
        return self.parent.open(redirected_req)
1002
2164.2.29 by Vincent Ladeuil
Test the http redirection at the request level even if it's not
1003
    http_error_301 = http_error_303 = http_error_307 = http_error_302
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1004
1005
2167.3.1 by v.ladeuil+lp at free
Fix bug #74759.
1006
class ProxyHandler(urllib2.ProxyHandler):
1007
    """Handles proxy setting.
1008
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
1009
    Copied and modified from urllib2 to be able to modify the request during
1010
    the request pre-processing instead of modifying it at _open time. As we
1011
    capture (or create) the connection object during request processing, _open
1012
    time was too late.
1013
1014
    The main task is to modify the request so that the connection is done to
1015
    the proxy while the request still refers to the destination host.
1016
1017
    Note: the proxy handling *may* modify the protocol used; the request may be
1018
    against an https server proxied through an http proxy. So, https_request
1019
    will be called, but later it's really http_open that will be called. This
2540.2.3 by Vincent Ladeuil
Take Aaron's comments into account.
1020
    explains why we don't have to call self.parent.open as the urllib2 did.
2167.3.1 by v.ladeuil+lp at free
Fix bug #74759.
1021
    """
1022
1023
    # Proxies must be in front
1024
    handler_order = 100
2167.3.3 by v.ladeuil+lp at free
* bzrlib/transport/http/_urllib2_wrappers.py:
1025
    _debuglevel = DEBUG
2167.3.1 by v.ladeuil+lp at free
Fix bug #74759.
1026
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1027
    def __init__(self, proxies=None):
2167.3.1 by v.ladeuil+lp at free
Fix bug #74759.
1028
        urllib2.ProxyHandler.__init__(self, proxies)
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1029
        # First, let's get rid of urllib2 implementation
2167.3.1 by v.ladeuil+lp at free
Fix bug #74759.
1030
        for type, proxy in self.proxies.items():
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
1031
            if self._debuglevel >= 3:
2167.3.3 by v.ladeuil+lp at free
* bzrlib/transport/http/_urllib2_wrappers.py:
1032
                print 'Will unbind %s_open for %r' % (type, proxy)
2167.3.1 by v.ladeuil+lp at free
Fix bug #74759.
1033
            delattr(self, '%s_open' % type)
1034
4797.75.1 by Andrew Bennetts
Backport fix for bug 558343 from lp:bzr r5220.
1035
        def bind_scheme_request(proxy, scheme):
1036
            if proxy is None:
1037
                return
1038
            scheme_request = scheme + '_request'
1039
            if self._debuglevel >= 3:
1040
                print 'Will bind %s for %r' % (scheme_request, proxy)
1041
            setattr(self, scheme_request,
1042
                lambda request: self.set_proxy(request, scheme))
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1043
        # We are interested only by the http[s] proxies
2167.3.6 by v.ladeuil+lp at free
Take John's comments into account and add more tests.
1044
        http_proxy = self.get_proxy_env_var('http')
4797.75.1 by Andrew Bennetts
Backport fix for bug 558343 from lp:bzr r5220.
1045
        bind_scheme_request(http_proxy, 'http')
2167.3.6 by v.ladeuil+lp at free
Take John's comments into account and add more tests.
1046
        https_proxy = self.get_proxy_env_var('https')
4797.75.1 by Andrew Bennetts
Backport fix for bug 558343 from lp:bzr r5220.
1047
        bind_scheme_request(https_proxy, 'https')
2167.3.3 by v.ladeuil+lp at free
* bzrlib/transport/http/_urllib2_wrappers.py:
1048
2167.3.6 by v.ladeuil+lp at free
Take John's comments into account and add more tests.
1049
    def get_proxy_env_var(self, name, default_to='all'):
1050
        """Get a proxy env var.
1051
2182.1.1 by Aaron Bentley
Respect proxy environment settings (Vincent Ladeuil, #74759)
1052
        Note that we indirectly rely on
2167.3.6 by v.ladeuil+lp at free
Take John's comments into account and add more tests.
1053
        urllib.getproxies_environment taking into account the
1054
        uppercased values for proxy variables.
1055
        """
2167.3.3 by v.ladeuil+lp at free
* bzrlib/transport/http/_urllib2_wrappers.py:
1056
        try:
1057
            return self.proxies[name.lower()]
1058
        except KeyError:
2167.3.6 by v.ladeuil+lp at free
Take John's comments into account and add more tests.
1059
            if default_to is not None:
1060
                # Try to get the alternate environment variable
1061
                try:
1062
                    return self.proxies[default_to]
1063
                except KeyError:
1064
                    pass
1065
        return None
2167.3.3 by v.ladeuil+lp at free
* bzrlib/transport/http/_urllib2_wrappers.py:
1066
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1067
    def proxy_bypass(self, host):
5639.2.2 by Vincent Ladeuil
Add tests and comments to clarify the feature.
1068
        """Check if host should be proxied or not.
1069
1070
        :returns: True to skip the proxy, False otherwise.
1071
        """
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1072
        no_proxy = self.get_proxy_env_var('no', default_to=None)
5639.2.2 by Vincent Ladeuil
Add tests and comments to clarify the feature.
1073
        bypass = self.evaluate_proxy_bypass(host, no_proxy)
1074
        if bypass is None:
5639.2.1 by Martin Pool
Empty entries in the ``NO_PROXY`` variable are no longer treated as matching every host.
1075
            # Nevertheless, there are platform-specific ways to
1076
            # ignore proxies...
1077
            return urllib.proxy_bypass(host)
1078
        else:
5639.2.2 by Vincent Ladeuil
Add tests and comments to clarify the feature.
1079
            return bypass
5639.2.1 by Martin Pool
Empty entries in the ``NO_PROXY`` variable are no longer treated as matching every host.
1080
1081
    def evaluate_proxy_bypass(self, host, no_proxy):
5639.2.2 by Vincent Ladeuil
Add tests and comments to clarify the feature.
1082
        """Check the host against a comma-separated no_proxy list as a string.
1083
1084
        :param host: ``host:port`` being requested
1085
5639.2.1 by Martin Pool
Empty entries in the ``NO_PROXY`` variable are no longer treated as matching every host.
1086
        :param no_proxy: comma-separated list of hosts to access directly.
5639.2.2 by Vincent Ladeuil
Add tests and comments to clarify the feature.
1087
5639.2.1 by Martin Pool
Empty entries in the ``NO_PROXY`` variable are no longer treated as matching every host.
1088
        :returns: True to skip the proxy, False not to, or None to
1089
            leave it to urllib.
1090
        """
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1091
        if no_proxy is None:
5639.2.2 by Vincent Ladeuil
Add tests and comments to clarify the feature.
1092
            # All hosts are proxied
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1093
            return False
1094
        hhost, hport = urllib.splitport(host)
2182.1.1 by Aaron Bentley
Respect proxy environment settings (Vincent Ladeuil, #74759)
1095
        # Does host match any of the domains mentioned in
1096
        # no_proxy ? The rules about what is authorized in no_proxy
1097
        # are fuzzy (to say the least). We try to allow most
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1098
        # commonly seen values.
1099
        for domain in no_proxy.split(','):
5639.2.1 by Martin Pool
Empty entries in the ``NO_PROXY`` variable are no longer treated as matching every host.
1100
            domain = domain.strip()
1101
            if domain == '':
1102
                continue
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1103
            dhost, dport = urllib.splitport(domain)
2167.3.5 by v.ladeuil+lp at free
Tests for proxies, covering #74759.
1104
            if hport == dport or dport is None:
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1105
                # Protect glob chars
1106
                dhost = dhost.replace(".", r"\.")
1107
                dhost = dhost.replace("*", r".*")
1108
                dhost = dhost.replace("?", r".")
2167.3.5 by v.ladeuil+lp at free
Tests for proxies, covering #74759.
1109
                if re.match(dhost, hhost, re.IGNORECASE):
5639.2.2 by Vincent Ladeuil
Add tests and comments to clarify the feature.
1110
                    return True
1111
        # Nothing explicitly avoid the host
1112
        return None
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1113
2167.3.3 by v.ladeuil+lp at free
* bzrlib/transport/http/_urllib2_wrappers.py:
1114
    def set_proxy(self, request, type):
2167.3.4 by v.ladeuil+lp at free
Better fix for #74759, but still not tests.
1115
        if self.proxy_bypass(request.get_host()):
1116
            return request
1117
2167.3.6 by v.ladeuil+lp at free
Take John's comments into account and add more tests.
1118
        proxy = self.get_proxy_env_var(type)
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
1119
        if self._debuglevel >= 3:
2167.3.3 by v.ladeuil+lp at free
* bzrlib/transport/http/_urllib2_wrappers.py:
1120
            print 'set_proxy %s_request for %r' % (type, proxy)
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
1121
        # FIXME: python 2.5 urlparse provides a better _parse_proxy which can
1122
        # grok user:password@host:port as well as
1123
        # http://user:password@host:port
1124
6055.2.1 by Jelmer Vernooij
Add UnparsedUrl.
1125
        parsed_url = transport.ConnectedTransport._split_url(proxy)
1126
        if not parsed_url.host:
4294.2.9 by Robert Collins
Fixup tests broken by cleaning up the layering.
1127
            raise errors.InvalidURL(proxy, 'No host component')
2900.2.15 by Vincent Ladeuil
AuthenticationConfig can be queried for logins too (first step).
1128
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1129
        if request.proxy_auth == {}:
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1130
            # No proxy auth parameter are available, we are handling the first
1131
            # proxied request, intialize.  scheme (the authentication scheme)
1132
            # and realm will be set by the AuthHandler
1133
            request.proxy_auth = {
6055.2.1 by Jelmer Vernooij
Add UnparsedUrl.
1134
                                  'host': parsed_url.host,
1135
                                  'port': parsed_url.port,
1136
                                  'user': parsed_url.user,
1137
                                  'password': parsed_url.password,
1138
                                  'protocol': parsed_url.scheme,
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1139
                                   # We ignore path since we connect to a proxy
1140
                                  'path': None}
6055.2.1 by Jelmer Vernooij
Add UnparsedUrl.
1141
        if parsed_url.port is None:
1142
            phost = parsed_url.host
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1143
        else:
6055.2.1 by Jelmer Vernooij
Add UnparsedUrl.
1144
            phost = parsed_url.host + ':%d' % parsed_url.port
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1145
        request.set_proxy(phost, type)
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
1146
        if self._debuglevel >= 3:
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1147
            print 'set_proxy: proxy set to %s://%s' % (type, phost)
2167.3.1 by v.ladeuil+lp at free
Fix bug #74759.
1148
        return request
1149
1150
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1151
class AbstractAuthHandler(urllib2.BaseHandler):
1152
    """A custom abstract authentication handler for all http authentications.
1153
1154
    Provides the meat to handle authentication errors and
1155
    preventively set authentication headers after the first
1156
    successful authentication.
1157
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1158
    This can be used for http and proxy, as well as for basic, negotiate and
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1159
    digest authentications.
1160
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1161
    This provides an unified interface for all authentication handlers
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1162
    (urllib2 provides far too many with different policies).
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1163
1164
    The interaction between this handler and the urllib2
1165
    framework is not obvious, it works as follow:
1166
1167
    opener.open(request) is called:
1168
1169
    - that may trigger http_request which will add an authentication header
1170
      (self.build_header) if enough info is available.
1171
1172
    - the request is sent to the server,
1173
1174
    - if an authentication error is received self.auth_required is called,
1175
      we acquire the authentication info in the error headers and call
1176
      self.auth_match to check that we are able to try the
1177
      authentication and complete the authentication parameters,
1178
1179
    - we call parent.open(request), that may trigger http_request
1180
      and will add a header (self.build_header), but here we have
1181
      all the required info (keep in mind that the request and
1182
      authentication used in the recursive calls are really (and must be)
1183
      the *same* objects).
1184
1185
    - if the call returns a response, the authentication have been
1186
      successful and the request authentication parameters have been updated.
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1187
    """
1188
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1189
    scheme = None
1190
    """The scheme as it appears in the server header (lower cased)"""
1191
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1192
    _max_retry = 3
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
1193
    """We don't want to retry authenticating endlessly"""
1194
4050.2.3 by Vincent Ladeuil
Slight cosmetic tweaks.
1195
    requires_username = True
1196
    """Whether the auth mechanism requires a username."""
1197
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1198
    # The following attributes should be defined by daughter
1199
    # classes:
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1200
    # - auth_required_header:  the header received from the server
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1201
    # - auth_header: the header sent in the request
1202
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1203
    def __init__(self):
1204
        # We want to know when we enter into an try/fail cycle of
1205
        # authentications so we initialize to None to indicate that we aren't
1206
        # in such a cycle by default.
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
1207
        self._retry_count = None
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1208
4050.2.2 by Vincent Ladeuil
Ensures all auth handlers correctly parse all auth headers.
1209
    def _parse_auth_header(self, server_header):
1210
        """Parse the authentication header.
1211
1212
        :param server_header: The value of the header sent by the server
1213
            describing the authenticaion request.
1214
1215
        :return: A tuple (scheme, remainder) scheme being the first word in the
1216
            given header (lower cased), remainder may be None.
1217
        """
1218
        try:
1219
            scheme, remainder = server_header.split(None, 1)
1220
        except ValueError:
1221
            scheme = server_header
1222
            remainder = None
1223
        return (scheme.lower(), remainder)
1224
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1225
    def update_auth(self, auth, key, value):
1226
        """Update a value in auth marking the auth as modified if needed"""
1227
        old_value = auth.get(key, None)
1228
        if old_value != value:
1229
            auth[key] = value
1230
            auth['modified'] = True
1231
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1232
    def auth_required(self, request, headers):
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1233
        """Retry the request if the auth scheme is ours.
1234
1235
        :param request: The request needing authentication.
1236
        :param headers: The headers for the authentication error response.
1237
        :return: None or the response for the authenticated request.
1238
        """
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
1239
        # Don't try  to authenticate endlessly
1240
        if self._retry_count is None:
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1241
            # The retry being recusrsive calls, None identify the first retry
1242
            self._retry_count = 1
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
1243
        else:
1244
            self._retry_count += 1
1245
            if self._retry_count > self._max_retry:
1246
                # Let's be ready for next round
1247
                self._retry_count = None
1248
                return None
4307.4.2 by Vincent Ladeuil
Handle servers proposing several authentication schemes.
1249
        server_headers = headers.getheaders(self.auth_required_header)
1250
        if not server_headers:
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1251
            # The http error MUST have the associated
1252
            # header. This must never happen in production code.
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1253
            raise KeyError('%s not found' % self.auth_required_header)
1254
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1255
        auth = self.get_auth(request)
1256
        auth['modified'] = False
4795.4.5 by Vincent Ladeuil
Make sure all redirection code paths can handle authentication.
1257
        # Put some common info in auth if the caller didn't
1258
        if auth.get('path', None) is None:
6055.2.7 by Jelmer Vernooij
Change parse_url to URL.from_string.
1259
            parsed_url = urlutils.URL.from_string(request.get_full_url())
6055.2.1 by Jelmer Vernooij
Add UnparsedUrl.
1260
            self.update_auth(auth, 'protocol', parsed_url.scheme)
1261
            self.update_auth(auth, 'host', parsed_url.host)
1262
            self.update_auth(auth, 'port', parsed_url.port)
1263
            self.update_auth(auth, 'path', parsed_url.path)
4307.4.2 by Vincent Ladeuil
Handle servers proposing several authentication schemes.
1264
        # FIXME: the auth handler should be selected at a single place instead
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1265
        # of letting all handlers try to match all headers, but the current
1266
        # design doesn't allow a simple implementation.
4307.4.2 by Vincent Ladeuil
Handle servers proposing several authentication schemes.
1267
        for server_header in server_headers:
1268
            # Several schemes can be proposed by the server, try to match each
1269
            # one in turn
1270
            matching_handler = self.auth_match(server_header, auth)
1271
            if matching_handler:
1272
                # auth_match may have modified auth (by adding the
1273
                # password or changing the realm, for example)
1274
                if (request.get_header(self.auth_header, None) is not None
1275
                    and not auth['modified']):
1276
                    # We already tried that, give up
1277
                    return None
1278
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1279
                # Only the most secure scheme proposed by the server should be
1280
                # used, since the handlers use 'handler_order' to describe that
1281
                # property, the first handler tried takes precedence, the
1282
                # others should not attempt to authenticate if the best one
1283
                # failed.
1284
                best_scheme = auth.get('best_scheme', None)
1285
                if best_scheme is None:
1286
                    # At that point, if current handler should doesn't succeed
1287
                    # the credentials are wrong (or incomplete), but we know
1288
                    # that the associated scheme should be used.
1289
                    best_scheme = auth['best_scheme'] = self.scheme
1290
                if  best_scheme != self.scheme:
1291
                    continue
1292
4307.4.2 by Vincent Ladeuil
Handle servers proposing several authentication schemes.
1293
                if self.requires_username and auth.get('user', None) is None:
1294
                    # Without a known user, we can't authenticate
1295
                    return None
1296
1297
                # Housekeeping
1298
                request.connection.cleanup_pipe()
1299
                # Retry the request with an authentication header added
1300
                response = self.parent.open(request)
1301
                if response:
1302
                    self.auth_successful(request, response)
1303
                return response
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1304
        # We are not qualified to handle the authentication.
1305
        # Note: the authentication error handling will try all
1306
        # available handlers. If one of them authenticates
1307
        # successfully, a response will be returned. If none of
1308
        # them succeeds, None will be returned and the error
1309
        # handler will raise the 401 'Unauthorized' or the 407
1310
        # 'Proxy Authentication Required' error.
1311
        return None
1312
1313
    def add_auth_header(self, request, header):
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1314
        """Add the authentication header to the request"""
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1315
        request.add_unredirected_header(self.auth_header, header)
1316
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1317
    def auth_match(self, header, auth):
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1318
        """Check that we are able to handle that authentication scheme.
1319
1320
        The request authentication parameters may need to be
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1321
        updated with info from the server. Some of these
1322
        parameters, when combined, are considered to be the
1323
        authentication key, if one of them change the
1324
        authentication result may change. 'user' and 'password'
1325
        are exampls, but some auth schemes may have others
1326
        (digest's nonce is an example, digest's nonce_count is a
1327
        *counter-example*). Such parameters must be updated by
1328
        using the update_auth() method.
3943.8.1 by Marius Kruger
remove all trailing whitespace from bzr source
1329
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1330
        :param header: The authentication header sent by the server.
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1331
        :param auth: The auth parameters already known. They may be
1332
             updated.
1333
        :returns: True if we can try to handle the authentication.
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1334
        """
1335
        raise NotImplementedError(self.auth_match)
1336
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1337
    def build_auth_header(self, auth, request):
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1338
        """Build the value of the header used to authenticate.
1339
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1340
        :param auth: The auth parameters needed to build the header.
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1341
        :param request: The request needing authentication.
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1342
1343
        :return: None or header.
1344
        """
1345
        raise NotImplementedError(self.build_auth_header)
1346
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1347
    def auth_successful(self, request, response):
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1348
        """The authentification was successful for the request.
1349
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1350
        Additional infos may be available in the response.
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1351
1352
        :param request: The succesfully authenticated request.
2420.1.9 by Vincent Ladeuil
Refactor proxy and auth test classes. Tests failing for digest auth.
1353
        :param response: The server response (may contain auth info).
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1354
        """
2540.2.2 by Vincent Ladeuil
Fix #120678 by issuing a CONNECT request when https is used via a proxy.
1355
        # It may happen that we need to reconnect later, let's be ready
1356
        self._retry_count = None
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1357
2900.2.20 by Vincent Ladeuil
http can query AuthenticationConfig for logins too.
1358
    def get_user_password(self, auth):
3910.2.3 by Ben Jansen
Made tweaks requested by John Arbash Meinel.
1359
        """Ask user for a password if none is already available.
1360
3910.2.4 by Vincent Ladeuil
Fixed as per John's review.
1361
        :param auth: authentication info gathered so far (from the initial url
1362
            and then during dialog with the server).
3910.2.3 by Ben Jansen
Made tweaks requested by John Arbash Meinel.
1363
        """
2900.2.20 by Vincent Ladeuil
http can query AuthenticationConfig for logins too.
1364
        auth_conf = config.AuthenticationConfig()
4795.4.4 by Vincent Ladeuil
Protect more access to 'user' and 'password' auth attributes.
1365
        user = auth.get('user', None)
1366
        password = auth.get('password', None)
2900.2.20 by Vincent Ladeuil
http can query AuthenticationConfig for logins too.
1367
        realm = auth['realm']
5484.2.2 by Martin Pool
Cope gracefully if urllib2 doesn't tell us the port number in the authentication callback
1368
        port = auth.get('port', None)
2900.2.20 by Vincent Ladeuil
http can query AuthenticationConfig for logins too.
1369
1370
        if user is None:
3910.2.1 by Ben Jansen
Changed HTTP transport auth so that URLs no longer need to include the username for HTTP Auth to work. Now, if bzr gets a 401 HTTP response, it looks in the authentication config for an appropriate username and password. If it doesn't find a username, it defaults to the local user. If it doesn't find a password, it prompts.
1371
            user = auth_conf.get_user(auth['protocol'], auth['host'],
5484.2.2 by Martin Pool
Cope gracefully if urllib2 doesn't tell us the port number in the authentication callback
1372
                                      port=port, path=auth['path'],
4222.3.12 by Jelmer Vernooij
Check that the HTTP transport prompts for usernames.
1373
                                      realm=realm, ask=True,
1374
                                      prompt=self.build_username_prompt(auth))
3910.2.2 by Vincent Ladeuil
Fix bug #300347 by allowing querying authentication.conf if no
1375
        if user is not None and password is None:
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1376
            password = auth_conf.get_password(
5484.2.2 by Martin Pool
Cope gracefully if urllib2 doesn't tell us the port number in the authentication callback
1377
                auth['protocol'], auth['host'], user,
1378
                port=port,
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1379
                path=auth['path'], realm=realm,
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1380
                prompt=self.build_password_prompt(auth))
2900.2.20 by Vincent Ladeuil
http can query AuthenticationConfig for logins too.
1381
1382
        return user, password
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1383
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1384
    def _build_password_prompt(self, auth):
1385
        """Build a prompt taking the protocol used into account.
1386
1387
        The AuthHandler is used by http and https, we want that information in
1388
        the prompt, so we build the prompt from the authentication dict which
1389
        contains all the needed parts.
1390
3133.1.3 by Vincent Ladeuil
Fix typo (hi John ;).
1391
        Also, http and proxy AuthHandlers present different prompts to the
3133.1.2 by Vincent Ladeuil
Fix #177643 by making pycurl handle url-embedded credentials again.
1392
        user. The daughter classes should implements a public
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1393
        build_password_prompt using this method.
1394
        """
5923.1.2 by Vincent Ladeuil
Fix some more prompts to be unicode.
1395
        prompt = u'%s' % auth['protocol'].upper() + u' %(user)s@%(host)s'
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1396
        realm = auth['realm']
1397
        if realm is not None:
5923.1.2 by Vincent Ladeuil
Fix some more prompts to be unicode.
1398
            prompt += u", Realm: '%s'" % realm.decode('utf8')
1399
        prompt += u' password'
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1400
        return prompt
1401
4222.3.12 by Jelmer Vernooij
Check that the HTTP transport prompts for usernames.
1402
    def _build_username_prompt(self, auth):
1403
        """Build a prompt taking the protocol used into account.
1404
1405
        The AuthHandler is used by http and https, we want that information in
1406
        the prompt, so we build the prompt from the authentication dict which
1407
        contains all the needed parts.
1408
1409
        Also, http and proxy AuthHandlers present different prompts to the
1410
        user. The daughter classes should implements a public
1411
        build_username_prompt using this method.
1412
        """
5923.1.2 by Vincent Ladeuil
Fix some more prompts to be unicode.
1413
        prompt = u'%s' % auth['protocol'].upper() + u' %(host)s'
4222.3.12 by Jelmer Vernooij
Check that the HTTP transport prompts for usernames.
1414
        realm = auth['realm']
1415
        if realm is not None:
5923.1.2 by Vincent Ladeuil
Fix some more prompts to be unicode.
1416
            prompt += u", Realm: '%s'" % realm.decode('utf8')
1417
        prompt += u' username'
4222.3.12 by Jelmer Vernooij
Check that the HTTP transport prompts for usernames.
1418
        return prompt
1419
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1420
    def http_request(self, request):
1421
        """Insert an authentication header if information is available"""
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1422
        auth = self.get_auth(request)
1423
        if self.auth_params_reusable(auth):
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1424
            self.add_auth_header(request, self.build_auth_header(auth, request))
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1425
        return request
1426
1427
    https_request = http_request # FIXME: Need test
1428
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1429
1430
class NegotiateAuthHandler(AbstractAuthHandler):
1431
    """A authentication handler that handles WWW-Authenticate: Negotiate.
1432
4032.1.4 by John Arbash Meinel
Found 2 more files with trailing whitespace.
1433
    At the moment this handler supports just Kerberos. In the future,
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1434
    NTLM support may also be added.
1435
    """
1436
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1437
    scheme = 'negotiate'
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1438
    handler_order = 480
4017.5.1 by Jelmer Vernooij
Allow HTTP authentication handlers (such as the NegotiateAuthHandler) to
1439
    requires_username = False
1440
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1441
    def auth_match(self, header, auth):
4050.2.2 by Vincent Ladeuil
Ensures all auth handlers correctly parse all auth headers.
1442
        scheme, raw_auth = self._parse_auth_header(header)
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1443
        if scheme != self.scheme:
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1444
            return False
1445
        self.update_auth(auth, 'scheme', scheme)
4011.3.2 by Jelmer Vernooij
Only attempt GSSAPI authentication when the kerberos module is present.
1446
        resp = self._auth_match_kerberos(auth)
1447
        if resp is None:
1448
            return False
1449
        # Optionally should try to authenticate using NTLM here
1450
        self.update_auth(auth, 'negotiate_response', resp)
1451
        return True
1452
1453
    def _auth_match_kerberos(self, auth):
1454
        """Try to create a GSSAPI response for authenticating against a host."""
6238.1.2 by John Arbash Meinel, Jelmer Vernooij
Only attempt to import kerberos module once.
1455
        global kerberos, checked_kerberos
1456
        if kerberos is None and not checked_kerberos:
1457
            try:
1458
                import kerberos
1459
            except ImportError:
1460
                kerberos = None
1461
            checked_kerberos = True
1462
        if kerberos is None:
4011.3.2 by Jelmer Vernooij
Only attempt GSSAPI authentication when the kerberos module is present.
1463
            return None
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1464
        ret, vc = kerberos.authGSSClientInit("HTTP@%(host)s" % auth)
1465
        if ret < 1:
4011.3.5 by Jelmer Vernooij
Move import next to other system libs, fix format.
1466
            trace.warning('Unable to create GSSAPI context for %s: %d',
1467
                auth['host'], ret)
4011.3.2 by Jelmer Vernooij
Only attempt GSSAPI authentication when the kerberos module is present.
1468
            return None
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1469
        ret = kerberos.authGSSClientStep(vc, "")
1470
        if ret < 0:
1471
            trace.mutter('authGSSClientStep failed: %d', ret)
4011.3.2 by Jelmer Vernooij
Only attempt GSSAPI authentication when the kerberos module is present.
1472
            return None
1473
        return kerberos.authGSSClientResponse(vc)
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1474
1475
    def build_auth_header(self, auth, request):
4011.3.2 by Jelmer Vernooij
Only attempt GSSAPI authentication when the kerberos module is present.
1476
        return "Negotiate %s" % auth['negotiate_response']
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1477
1478
    def auth_params_reusable(self, auth):
1479
        # If the auth scheme is known, it means a previous
1480
        # authentication was successful, all information is
1481
        # available, no further checks are needed.
4032.1.4 by John Arbash Meinel
Found 2 more files with trailing whitespace.
1482
        return (auth.get('scheme', None) == 'negotiate' and
4011.3.4 by Jelmer Vernooij
review from vila: mention HTTPS, clarify error a bit, move import to top-level.
1483
                auth.get('negotiate_response', None) is not None)
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1484
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1485
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1486
class BasicAuthHandler(AbstractAuthHandler):
1487
    """A custom basic authentication handler."""
1488
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1489
    scheme = 'basic'
2545.2.1 by Vincent Ladeuil
Fix 121889 by working around urllib2 bug.
1490
    handler_order = 500
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1491
    auth_regexp = re.compile('realm="([^"]*)"', re.I)
1492
1493
    def build_auth_header(self, auth, request):
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1494
        raw = '%s:%s' % (auth['user'], auth['password'])
6615.4.1 by Vincent Ladeuil
Fix long user and password header generation for Basic http authentification
1495
        auth_header = 'Basic ' + base64.b64encode(raw)
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1496
        return auth_header
1497
4284.1.1 by Vincent Ladeuil
Fix wrong realm extraction in http basic authentication (reported
1498
    def extract_realm(self, header_value):
1499
        match = self.auth_regexp.search(header_value)
1500
        realm = None
1501
        if match:
1502
            realm = match.group(1)
1503
        return match, realm
1504
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1505
    def auth_match(self, header, auth):
4050.2.2 by Vincent Ladeuil
Ensures all auth handlers correctly parse all auth headers.
1506
        scheme, raw_auth = self._parse_auth_header(header)
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1507
        if scheme != self.scheme:
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1508
            return False
1509
4284.1.1 by Vincent Ladeuil
Fix wrong realm extraction in http basic authentication (reported
1510
        match, realm = self.extract_realm(raw_auth)
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1511
        if match:
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1512
            # Put useful info into auth
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1513
            self.update_auth(auth, 'scheme', scheme)
1514
            self.update_auth(auth, 'realm', realm)
4795.4.3 by Vincent Ladeuil
Protect access to 'user' and 'password' auth attributes.
1515
            if (auth.get('user', None) is None
1516
                or auth.get('password', None) is None):
2900.2.20 by Vincent Ladeuil
http can query AuthenticationConfig for logins too.
1517
                user, password = self.get_user_password(auth)
1518
                self.update_auth(auth, 'user', user)
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1519
                self.update_auth(auth, 'password', password)
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1520
        return match is not None
1521
1522
    def auth_params_reusable(self, auth):
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1523
        # If the auth scheme is known, it means a previous
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1524
        # authentication was successful, all information is
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1525
        # available, no further checks are needed.
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1526
        return auth.get('scheme', None) == 'basic'
1527
1528
1529
def get_digest_algorithm_impls(algorithm):
1530
    H = None
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1531
    KD = None
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1532
    if algorithm == 'MD5':
2929.3.1 by Vincent Ladeuil
Fix python2.6 deprecation warnings (still 4 failures 5 errors in test suite).
1533
        H = lambda x: osutils.md5(x).hexdigest()
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1534
    elif algorithm == 'SHA':
5849.1.3 by Jelmer Vernooij
Simplify call to sha_string.
1535
        H = osutils.sha_string
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1536
    if H is not None:
1537
        KD = lambda secret, data: H("%s:%s" % (secret, data))
1538
    return H, KD
1539
1540
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1541
def get_new_cnonce(nonce, nonce_count):
1542
    raw = '%s:%d:%s:%s' % (nonce, nonce_count, time.ctime(),
1543
                           urllib2.randombytes(8))
5849.1.1 by Jelmer Vernooij
Use osutils.sha_string() when possible.
1544
    return osutils.sha_string(raw)[:16]
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1545
1546
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1547
class DigestAuthHandler(AbstractAuthHandler):
1548
    """A custom digest authentication handler."""
1549
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1550
    scheme = 'digest'
4050.2.3 by Vincent Ladeuil
Slight cosmetic tweaks.
1551
    # Before basic as digest is a bit more secure and should be preferred
2545.2.1 by Vincent Ladeuil
Fix 121889 by working around urllib2 bug.
1552
    handler_order = 490
1553
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1554
    def auth_params_reusable(self, auth):
1555
        # If the auth scheme is known, it means a previous
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1556
        # authentication was successful, all information is
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1557
        # available, no further checks are needed.
1558
        return auth.get('scheme', None) == 'digest'
1559
1560
    def auth_match(self, header, auth):
4050.2.2 by Vincent Ladeuil
Ensures all auth handlers correctly parse all auth headers.
1561
        scheme, raw_auth = self._parse_auth_header(header)
4307.4.3 by Vincent Ladeuil
Tighten multiple auth schemes handling.
1562
        if scheme != self.scheme:
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1563
            return False
1564
1565
        # Put the requested authentication info into a dict
1566
        req_auth = urllib2.parse_keqv_list(urllib2.parse_http_list(raw_auth))
1567
1568
        # Check that we can handle that authentication
1569
        qop = req_auth.get('qop', None)
1570
        if qop != 'auth': # No auth-int so far
1571
            return False
1572
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1573
        H, KD = get_digest_algorithm_impls(req_auth.get('algorithm', 'MD5'))
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1574
        if H is None:
1575
            return False
1576
1577
        realm = req_auth.get('realm', None)
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1578
        # Put useful info into auth
1579
        self.update_auth(auth, 'scheme', scheme)
1580
        self.update_auth(auth, 'realm', realm)
4795.4.3 by Vincent Ladeuil
Protect access to 'user' and 'password' auth attributes.
1581
        if auth.get('user', None) is None or auth.get('password', None) is None:
2900.2.20 by Vincent Ladeuil
http can query AuthenticationConfig for logins too.
1582
            user, password = self.get_user_password(auth)
1583
            self.update_auth(auth, 'user', user)
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1584
            self.update_auth(auth, 'password', password)
1585
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1586
        try:
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1587
            if req_auth.get('algorithm', None) is not None:
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1588
                self.update_auth(auth, 'algorithm', req_auth.get('algorithm'))
1589
            nonce = req_auth['nonce']
1590
            if auth.get('nonce', None) != nonce:
1591
                # A new nonce, never used
1592
                self.update_auth(auth, 'nonce_count', 0)
1593
            self.update_auth(auth, 'nonce', nonce)
1594
            self.update_auth(auth, 'qop', qop)
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1595
            auth['opaque'] = req_auth.get('opaque', None)
1596
        except KeyError:
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1597
            # Some required field is not there
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1598
            return False
1599
1600
        return True
1601
1602
    def build_auth_header(self, auth, request):
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1603
        url_scheme, url_selector = urllib.splittype(request.get_selector())
1604
        sel_host, uri = urllib.splithost(url_selector)
1605
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1606
        A1 = '%s:%s:%s' % (auth['user'], auth['realm'], auth['password'])
1607
        A2 = '%s:%s' % (request.get_method(), uri)
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1608
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1609
        nonce = auth['nonce']
1610
        qop = auth['qop']
1611
2420.1.16 by Vincent Ladeuil
Handle nonce changes. Fix a nasty bug breaking the auth parameters sharing.
1612
        nonce_count = auth['nonce_count'] + 1
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1613
        ncvalue = '%08x' % nonce_count
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1614
        cnonce = get_new_cnonce(nonce, nonce_count)
1615
1616
        H, KD = get_digest_algorithm_impls(auth.get('algorithm', 'MD5'))
1617
        nonce_data = '%s:%s:%s:%s:%s' % (nonce, ncvalue, cnonce, qop, H(A2))
1618
        request_digest = KD(H(A1), nonce_data)
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1619
1620
        header = 'Digest '
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1621
        header += 'username="%s", realm="%s", nonce="%s"' % (auth['user'],
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1622
                                                             auth['realm'],
1623
                                                             nonce)
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1624
        header += ', uri="%s"' % uri
1625
        header += ', cnonce="%s", nc=%s' % (cnonce, ncvalue)
1626
        header += ', qop="%s"' % qop
1627
        header += ', response="%s"' % request_digest
1628
        # Append the optional fields
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1629
        opaque = auth.get('opaque', None)
1630
        if opaque:
1631
            header += ', opaque="%s"' % opaque
2420.1.14 by Vincent Ladeuil
Tested against squid-2.6.5 with digest authentication.
1632
        if auth.get('algorithm', None):
1633
            header += ', algorithm="%s"' % auth.get('algorithm')
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1634
1635
        # We have used the nonce once more, update the count
1636
        auth['nonce_count'] = nonce_count
1637
1638
        return header
1639
1640
1641
class HTTPAuthHandler(AbstractAuthHandler):
1642
    """Custom http authentication handler.
2004.3.1 by vila
Test ConnectionError exceptions.
1643
2363.4.12 by Vincent Ladeuil
Take jam's review comments into account. Fix typos, give better
1644
    Send the authentication preventively to avoid the roundtrip
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1645
    associated with the 401 error and keep the revelant info in
1646
    the auth request attribute.
2004.3.1 by vila
Test ConnectionError exceptions.
1647
    """
1648
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1649
    auth_required_header = 'www-authenticate'
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1650
    auth_header = 'Authorization'
1651
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1652
    def get_auth(self, request):
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1653
        """Get the auth params from the request"""
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1654
        return request.auth
1655
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1656
    def set_auth(self, request, auth):
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1657
        """Set the auth params for the request"""
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1658
        request.auth = auth
2004.3.1 by vila
Test ConnectionError exceptions.
1659
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1660
    def build_password_prompt(self, auth):
1661
        return self._build_password_prompt(auth)
1662
4222.3.12 by Jelmer Vernooij
Check that the HTTP transport prompts for usernames.
1663
    def build_username_prompt(self, auth):
1664
        return self._build_username_prompt(auth)
1665
2363.4.9 by Vincent Ladeuil
Catch first succesful authentification to avoid further 401
1666
    def http_error_401(self, req, fp, code, msg, headers):
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1667
        return self.auth_required(req, headers)
1668
1669
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1670
class ProxyAuthHandler(AbstractAuthHandler):
1671
    """Custom proxy authentication handler.
2420.1.3 by Vincent Ladeuil
Implement http proxy basic authentication.
1672
1673
    Send the authentication preventively to avoid the roundtrip
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1674
    associated with the 407 error and keep the revelant info in
1675
    the proxy_auth request attribute..
2420.1.3 by Vincent Ladeuil
Implement http proxy basic authentication.
1676
    """
1677
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1678
    auth_required_header = 'proxy-authenticate'
2420.1.7 by Vincent Ladeuil
Tested against squid-2.6.5 with basic authentication.
1679
    # FIXME: the correct capitalization is Proxy-Authorization,
2420.1.8 by Vincent Ladeuil
Interesting typo :-) A mix between capitalize, title and fuzzy may be...
1680
    # but python-2.4 urllib2.Request insist on using capitalize()
2420.1.7 by Vincent Ladeuil
Tested against squid-2.6.5 with basic authentication.
1681
    # instead of title().
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1682
    auth_header = 'Proxy-authorization'
1683
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1684
    def get_auth(self, request):
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1685
        """Get the auth params from the request"""
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1686
        return request.proxy_auth
1687
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1688
    def set_auth(self, request, auth):
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1689
        """Set the auth params for the request"""
2420.1.6 by Vincent Ladeuil
Update NEWS to explain the intent of the modification. Also, use dicts
1690
        request.proxy_auth = auth
2420.1.3 by Vincent Ladeuil
Implement http proxy basic authentication.
1691
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1692
    def build_password_prompt(self, auth):
1693
        prompt = self._build_password_prompt(auth)
5923.1.2 by Vincent Ladeuil
Fix some more prompts to be unicode.
1694
        prompt = u'Proxy ' + prompt
2900.2.19 by Vincent Ladeuil
Mention proxy and https in the password prompts, with tests.
1695
        return prompt
1696
4222.3.12 by Jelmer Vernooij
Check that the HTTP transport prompts for usernames.
1697
    def build_username_prompt(self, auth):
1698
        prompt = self._build_username_prompt(auth)
5923.1.2 by Vincent Ladeuil
Fix some more prompts to be unicode.
1699
        prompt = u'Proxy ' + prompt
4222.3.12 by Jelmer Vernooij
Check that the HTTP transport prompts for usernames.
1700
        return prompt
1701
2420.1.3 by Vincent Ladeuil
Implement http proxy basic authentication.
1702
    def http_error_407(self, req, fp, code, msg, headers):
2420.1.5 by Vincent Ladeuil
Refactor http and proxy authentication. Tests passing. proxy password can be prompted too.
1703
        return self.auth_required(req, headers)
2420.1.3 by Vincent Ladeuil
Implement http proxy basic authentication.
1704
1705
2420.1.11 by Vincent Ladeuil
Implement digest authentication. Test suite passes. Tested against apache-2.x.
1706
class HTTPBasicAuthHandler(BasicAuthHandler, HTTPAuthHandler):
1707
    """Custom http basic authentication handler"""
1708
1709
1710
class ProxyBasicAuthHandler(BasicAuthHandler, ProxyAuthHandler):
1711
    """Custom proxy basic authentication handler"""
1712
1713
1714
class HTTPDigestAuthHandler(DigestAuthHandler, HTTPAuthHandler):
1715
    """Custom http basic authentication handler"""
1716
1717
1718
class ProxyDigestAuthHandler(DigestAuthHandler, ProxyAuthHandler):
1719
    """Custom proxy basic authentication handler"""
1720
2420.1.3 by Vincent Ladeuil
Implement http proxy basic authentication.
1721
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1722
class HTTPNegotiateAuthHandler(NegotiateAuthHandler, HTTPAuthHandler):
1723
    """Custom http negotiate authentication handler"""
1724
1725
1726
class ProxyNegotiateAuthHandler(NegotiateAuthHandler, ProxyAuthHandler):
1727
    """Custom proxy negotiate authentication handler"""
1728
1729
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1730
class HTTPErrorProcessor(urllib2.HTTPErrorProcessor):
1731
    """Process HTTP error responses.
1732
1733
    We don't really process the errors, quite the contrary
1734
    instead, we leave our Transport handle them.
1735
    """
1736
2520.2.2 by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request
1737
    accepted_errors = [200, # Ok
1738
                       206, # Partial content
1739
                       404, # Not found
1740
                       ]
1741
    """The error codes the caller will handle.
1742
1743
    This can be specialized in the request on a case-by case basis, but the
1744
    common cases are covered here.
1745
    """
1746
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1747
    def http_response(self, request, response):
1748
        code, msg, hdrs = response.code, response.msg, response.info()
1749
2520.2.2 by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request
1750
        accepted_errors = request.accepted_errors
1751
        if accepted_errors is None:
1752
            accepted_errors = self.accepted_errors
1753
1754
        if code not in accepted_errors:
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1755
            response = self.parent.error('http', request, response,
1756
                                         code, msg, hdrs)
1757
        return response
1758
1759
    https_response = http_response
1760
1761
1762
class HTTPDefaultErrorHandler(urllib2.HTTPDefaultErrorHandler):
1763
    """Translate common errors into bzr Exceptions"""
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
1764
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1765
    def http_error_default(self, req, fp, code, msg, hdrs):
2520.2.2 by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request
1766
        if code == 403:
3430.3.1 by Vincent Ladeuil
Fix #230223 by making both http implementations raise appropriate exceptions.
1767
            raise errors.TransportError(
1768
                'Server refuses to fulfill the request (403 Forbidden)'
1769
                ' for %s' % req.get_full_url())
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1770
        else:
2004.1.27 by v.ladeuil+lp at free
Fix bug #57644 by issuing an explicit error message.
1771
            raise errors.InvalidHttpResponse(req.get_full_url(),
1772
                                             'Unable to handle http code %d: %s'
1773
                                             % (code, msg))
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
1774
2520.2.2 by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request
1775
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1776
class Opener(object):
1777
    """A wrapper around urllib2.build_opener
1778
1779
    Daughter classes can override to build their own specific opener
1780
    """
2145.1.1 by mbp at sourcefrog
merge urllib keepalive etc
1781
    # TODO: Provides hooks for daughter classes.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1782
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
1783
    def __init__(self,
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1784
                 connection=ConnectionHandler,
1785
                 redirect=HTTPRedirectHandler,
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
1786
                 error=HTTPErrorProcessor,
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
1787
                 report_activity=None,
1788
                 ca_certs=None):
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
1789
        self._opener = urllib2.build_opener(
6238.2.22 by Vincent Ladeuil
Create a specific test permutation for urllib https so we can inject our test ca certs. The wiring in _urllib2_wrappers is a bit hackish and it will need to use auth instead so different certs can be used for proxies and real servers but this could wait until authentication.conf is migrated to the config stacks. With this change in place, all https tests pass without the need to create a dedicated GlobalStore.
1790
            connection(report_activity=report_activity, ca_certs=ca_certs),
3945.1.5 by Vincent Ladeuil
Start implementing http activity reporting at socket level.
1791
            redirect, error,
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1792
            ProxyHandler(),
1793
            HTTPBasicAuthHandler(),
1794
            HTTPDigestAuthHandler(),
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1795
            HTTPNegotiateAuthHandler(),
2900.2.16 by Vincent Ladeuil
Make hhtp proxy aware of AuthenticationConfig (for password).
1796
            ProxyBasicAuthHandler(),
1797
            ProxyDigestAuthHandler(),
4011.3.1 by Jelmer Vernooij
Add simple support for GSSAPI authentication over HTTP.
1798
            ProxyNegotiateAuthHandler(),
2004.1.2 by vila
Implements a BasicAuthManager.
1799
            HTTPHandler,
1800
            HTTPSHandler,
1801
            HTTPDefaultErrorHandler,
2004.2.1 by John Arbash Meinel
Cleanup of urllib functions
1802
            )
2520.2.2 by Vincent Ladeuil
Fix #115209 by issuing a single range request on 400: Bad Request
1803
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1804
        self.open = self._opener.open
3111.1.20 by Vincent Ladeuil
Make all the test pass. Looks like we are HTTP/1.1 compliant.
1805
        if DEBUG >= 9:
2004.1.9 by vila
Takes jam's remarks into account when possible, add TODOs for the rest.
1806
            # When dealing with handler order, it's easy to mess
1807
            # things up, the following will help understand which
1808
            # handler is used, when and for what.
2004.1.1 by vila
Connection sharing, with redirection. without authentification.
1809
            import pprint
1810
            pprint.pprint(self._opener.__dict__)