14
The smart protocol provides a way to send a requests and corresponding
15
responses to communicate with a remote bzr process.
23
At the bottom level there is either a socket, pipes, or an HTTP
24
request/response. We call this layer the *medium*. It is responsible for
25
carrying bytes between a client and server. For sockets, we have the idea
26
that you have multiple requests and get a read error because the other
27
side did shutdown. For pipes we have read pipe which will have a zero
28
read which marks end-of-file. For HTTP server environment there is no
29
end-of-stream because each request coming into the server is independent.
31
So we need a wrapper around pipes and sockets to separate out requests
32
from substrate and this will give us a single model which is consistent
33
for HTTP, sockets and pipes.
38
On top of the medium is the *protocol*. This is the layer that
39
deserialises bytes into the structured data that requests and responses
42
Request/Response processing
43
---------------------------
45
On top of the protocol is the logic for processing requests (on the
46
server) or responses (on the client).
53
MEDIUM (factory for protocol, reads bytes & pushes to protocol,
54
uses protocol to detect end-of-request, sends written
55
bytes to client) e.g. socket, pipe, HTTP request handler.
60
PROTOCOL(serialization, deserialization) accepts bytes for one
61
request, decodes according to internal state, pushes
62
structured data to handler. accepts structured data from
63
handler and encodes and writes to the medium. factory for
69
HANDLER (domain logic) accepts structured data, operates state
70
machine until the request can be satisfied,
71
sends structured data to the protocol.
73
Request handlers are registered in the `bzrlib.smart.request` module.
81
CLIENT domain logic, accepts domain requests, generated structured
82
data, reads structured data from responses and turns into
83
domain data. Sends structured data to the protocol.
84
Operates state machines until the request can be delivered
85
(e.g. reading from a bundle generated in bzrlib to deliver a
88
This is RemoteBzrDir, RemoteRepository, etc.
93
PROTOCOL (serialization, deserialization) accepts structured data for one
94
request, encodes and writes to the medium. Reads bytes from the
95
medium, decodes and allows the client to read structured data.
100
MEDIUM accepts bytes from the protocol & delivers to the remote server.
101
Allows the protocol to read bytes e.g. socket, pipe, HTTP request.
103
The domain logic is in `bzrlib.remote`: `RemoteBzrDir`, `RemoteBranch`,
106
There is also an plain file-level transport that calls remote methods to
107
manipulate files on the server in `bzrlib.transport.remote`.
115
Version one of the protocol was introduced in Bazaar 0.11.
117
The protocol (for both requests and responses) is described by::
119
REQUEST := MESSAGE_V1
120
RESPONSE := MESSAGE_V1
121
MESSAGE_V1 := ARGS [BODY]
123
ARGS := ARG [MORE_ARGS] NEWLINE
124
MORE_ARGS := SEP ARG [MORE_ARGS]
127
BODY := LENGTH NEWLINE BODY_BYTES TRAILER
128
LENGTH := decimal integer
129
TRAILER := "done" NEWLINE
131
That is, a tuple of arguments separated by Ctrl-A and terminated with a
132
newline, followed by length prefixed body with a constant trailer. Note
133
that although arguments are not 8-bit safe (they cannot include 0x01 or
134
0x0a bytes without breaking the protocol encoding), the body is.
139
Version two was introduced in Bazaar 0.16.
141
The request protocol is::
143
REQUEST_V2 := "bzr request 2" NEWLINE MESSAGE_V2
145
The response protocol is::
147
RESPONSE_V2 := "bzr response 2" NEWLINE RESPONSE_STATUS NEWLINE MESSAGE_V2
148
RESPONSE_STATUS := "success" | "failed"
150
Future versions should follow this structure, like version two does::
152
FUTURE_MESSAGE := VERSION_STRING NEWLINE REST_OF_MESSAGE
154
This is so that clients and servers can read bytes up to the first newline
155
byte to determine what version a message is.
157
For compatibility will all versions (past and future) of bzr clients,
158
servers that receive a request in an unknown protocol version should
159
respond with a single-line error terminated with 0x0a (NEWLINE), rather
160
than structured response prefixed with a version string.
162
Version two of the message protocol is::
164
MESSAGE_V2 := ARGS [BODY_V2]
165
BODY_V2 := BODY | STREAMED_BODY
167
That is, a version one length-prefixed body, or a version two streamed
170
Version two with streamed bodies
171
--------------------------------
173
An extension to version two allows streamed bodies. A streamed body looks
174
a lot like HTTP's chunked encoding::
176
STREAMED_BODY := "chunked" NEWLINE CHUNKS TERMINATOR
177
CHUNKS := CHUNK [CHUNKS]
178
CHUNK := HEX_LENGTH CHUNK_CONTENT
179
HEX_LENGTH := HEX_DIGITS NEWLINE
180
CHUNK_CONTENT := bytes
182
TERMINATOR := SUCCESS_TERMINATOR | ERROR_TERMINATOR
183
SUCCESS_TERMINATOR := 'END' NEWLINE
184
ERROR_TERMINATOR := 'ERR' NEWLINE CHUNKS SUCCESS_TERMINATOR
186
That is, the body consists of a series of chunks. Each chunk starts with
187
a length prefix in hexadecimal digits, followed by an ASCII newline byte.
188
The end of the body is signaled by '``END\\n``', or by '``ERR\\n``'
189
followed by error args, one per chunk. Note that these args are 8-bit
190
safe, unlike request args.
192
A streamed body starts with the string "chunked" so that legacy clients
193
and servers will not mistake the first chunk as the start of a version one
196
The type of body (length-prefixed or chunked) in a response is always the
197
same for a given request method. Only new request methods introduced in
198
Bazaar 0.91 and later use streamed bodies.
205
For some discussion of the requirements that led to this new protocol
206
version, see `bug #83935`_.
208
.. _bug #83935: https://bugs.launchpad.net/bzr/+bug/83935
210
Version three has bencoding of most protocol structures, to make parsing
211
simpler. For extra parsing convenience, these structures are length
214
LENGTH_PREFIX := 32-bit unsigned integer in network byte order
216
Unlike earlier versions, clients and servers are no longer required to
217
know which request verbs and responses will have bodies attached. Because
218
of length-prefixing and other changes, it is always possible to know when
219
a complete request or response has been read, even if the server
222
The underlying message format is::
224
MESSAGE := MAGIC NEWLINE HEADERS CONTENTS END_MESSAGE
225
MAGIC := "bzr message 3 (bzr 1.6)"
226
HEADERS := LENGTH_PREFIX bencoded_dict
229
BODY := MESSAGE_PART+
230
MESSAGE_PART := ONE_BYTE | STRUCTURE | BYTES
232
STRUCTURE := "s" LENGTH_PREFIX bencoded_structure
233
BYTES := "b" LENGTH_PREFIX bytes
235
(Where ``+`` indicates one or more.)
237
This format allows an arbitrary sequence of message parts to be encoded
238
in a single message. The contents of a MESSAGE have a higher-level
239
message, but knowing just this amount of data it's possible to
240
deserialize and consume a message, so that implementations can respond to
241
messages sent by later versions.
246
Each request and response will have “headers”, a dictionary of key-value pairs.
247
The keys must be strings, not any other type of value.
249
Currently, the only defined header is “Software version”. Both the client and
250
the server should include a “Software version” header, with a value of a
251
free-form string such as “bzrlib 1.5”, to aid debugging and logging. Clients
252
and servers **should not** vary behaviour based on this string.
254
Conventional requests and responses
255
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
257
By convention, most requests and responses have a simple “arguments plus
258
optional body” structure, as in earlier protocol versions. This section
259
describes how such messages are encoded. All requests and responses
260
defined by earlier protocol versions must be encoded in this way.
262
Conventional requests will send a CONTENTS of ::
264
CONV_REQ := ARGS SINGLE_OR_STREAMED_BODY?
265
SINGLE_OR_STREAMED_BODY := BYTES
268
ARGS := STRUCTURE(argument_tuple)
269
TRAILER := SUCCESS_STATUS | ERROR
270
SUCCESS_STATUS := ONE_BYTE("S")
271
ERROR := ONE_BYTE("E") STRUCTURE(argument_tuple)
273
Conventional responses will send CONTENTS of ::
275
CONV_RESP := RESP_STATUS ARGS SINGLE_OR_STREAMED_BODY?
276
RESP_STATUS := ONE_BYTE("S") | ONE_BYTE("E")
278
If the RESP_STATUS is success ("S"), the arguments are the
279
method-dependent result.
281
For errors (where the Status byte of a response or a streamed body is
282
"E"), the situation is analagous to requests. The first item in the
283
encoded sequence must be a string of the error name. The other arguments
284
supply details about the error, and their number and types will depend on
285
the type of error (as identified by the error name).
287
Note that the streamed body from version two is now just multiple
290
The end of the request or response is indicated by the lower-level
291
END_MESSAGE. If there's only one BYTES element in the body, the TRAILER
292
may or may not be present, depending on whether it was sent as a single
293
chunk or as a stream that happens to have one element.
295
*(Discussion)* The success marker at the end of a streamed body seems
296
redundant; it doesn't have space for any arguments, and the end of the
297
body is marked anyhow by the end of the message. Recipients shouldn't
298
take any action on it, though they should map an error into raising an
301
1.10 clients don't assert that they get a status byte at the end of the
302
message. They will complain (in
303
``ConventionalResponseHandler.byte_part_received``) if they get an
304
initial success and then another byte part with no intervening bytes.
305
If we stop sending the final success message and only flag errors
306
they'll only get one if the error is detected after streaming starts but
307
before any bytes are actually sent. Possibly we should wait until at
308
least the first chunk is ready before declaring success.
310
For new methods, these sequences are just a convention and may be varied
311
if appropriate for a particular request or response. However, each
312
request should at least start with a STRUCTURE encoding the arguments
313
tuple. The first element of that tuple must be a string that names the
314
request method. (Note that arguments in this protocol version are
315
bencoded. As a result, unlike previous protocol versions, arguments in
316
this version are 8-bit clean.)
318
(Discussion) We're discussing having the byte segments be not just a
319
method for sending a stream across the network, but actually having them
320
be preserved in the rpc from end to end. This may be useful when
321
there's an iterator on one side feeding in to an iterator on the other,
322
if it avoids doing chunking and byte-counting at two levels, and if
323
those iterators are a natural place to get good granularity. Also, for
324
cases like ``insert_record_stream`` the server can't do much with the
325
data until it gets a whole chunk, and so it'll be natural and efficient
326
for it to be called with one chunk at a time.
328
On the other hand, there may be times when we've got some bytes from the
329
network but not a full chunk, and it might be worthwhile to pass it up.
330
If we promise to preserve chunks, then to do this we'd need two separate
331
streaming interfaces: "we got a chunk" and "we got some bytes but not
332
yet a full chunk". For ``insert_record_stream`` the second might not be
333
useful, but it might be good when writing to a file where any number of
334
bytes can be processed.
336
If we promise to preserve chunks, it'll tend to make some RPCs work only
337
in chunks, and others just on whole blocks, and we can't so easily
338
migrate RPCs from one to the other transparently to older
341
The data inside those chunks will be serialized anyhow, and possibly the
342
data inside them will already be able to be serialized apart without
343
understanding the chunks. Also, we might want to use these formats e.g.
344
for pack files or in bundles, and so they don't particularly need
345
lower-level chunking. So the current (unmerged, unstable) record stream
346
serialization turns each record into a bencoded tuple and it'd be
347
feasible to parse one tuple at a time from a byte stream that contains a
350
So we've decided that the chunks won't be semantic, and code should not
351
count on them being preserved from client to server.
356
*(Discussion)* It would be nice if the server could notify the client of
357
errors even before a streaming request has finished. This could cover
358
situtaions such as the server not understanding the request, it being
359
unable to open the requested location, or it finding that some of the
360
revisions being sent are not actually needed.
362
Especially in the last case, we'd like to be able to gracefully notice
363
the condition while the client is writing, and then have it adapt its
364
behaviour. In any case, we don't want to have drop and restart the
367
It should be possible for the client to finish its current chunk and
368
then its message, possibly with an error to cancel what's already been
371
This relies on the client being able to read back from the server while
372
it's writing. This is technically difficult for http but feasible over
375
We'd need a clean way to pass this back to the request method, even
376
though it's presumably in the middle of doing its body iterator.
377
Possibly the body iterator could be manually given a reference to the
378
request object, and it can poll it to see if there's a response.
380
Perhaps we need to distinguish error conditions, which should turn into
381
a client-side error regardless of the request code, from early success,
382
which should be handled only if the request code specifically wants to
385
Full-duplex operation
386
~~~~~~~~~~~~~~~~~~~~~
388
Code not geared to do pipelined requests, and this might require doing
389
asynchrony within bzrlib. We might want to either go fully pipelined
390
and asynchronous, but there might be a profitable middle ground.
392
The particular case where duplex communication would be good is in
393
working towards the common points in the graphs between the client and
394
server: we want to send speculatively, but detect as soon as they've
397
So we could for instance have a synchronous core, but rely on the OS
398
network buffering to allow us to work on batches of say 64kB. We can
399
also pipeline requests and responses, without allowing for them
400
happening out of order, or mixed requests happening at the same time.
402
Wonder how our network performance would have turned out now if we'd
403
done full-duplex from the start, and ignored hpss over http. We have
404
pretty good (readonly) http support just over dumb http, and that may be
405
better for many users.
412
On the client, the bzrlib code is "in charge": when it makes a request, or
413
asks from data from the network, that causes network IO. The server is
414
event driven: the network code tells the response handler when data has
415
been received, and it takes back a Response object from the request
416
handler that is then polled for body stream data.
421
Paths are passed across the network. The client needs to see a namespace
422
that includes any repository that might need to be referenced, and the
423
client needs to know about a root directory beyond which it cannot ascend.
425
Servers run over ssh will typically want to be able to access any path the
426
user can access. Public servers on the other hand (which might be over
427
http, ssh or tcp) will typically want to restrict access to only a
428
particular directory and its children, so will want to do a software
429
virtual root at that level. In other words they'll want to rewrite
430
incoming paths to be under that level (and prevent escaping using ../
431
tricks). The default implementation in bzrlib does this using the
432
`bzrlib.transport.chroot` module.
434
URLs that include ~ should probably be passed across to the server
435
verbatim and the server can expand them. This will proably not be
436
meaningful when limited to a directory? See `bug 109143`_.
438
.. _bug 109143: https://bugs.launchpad.net/bzr/+bug/109143
444
The first argument of a request specifies the request method.
446
The available request methods are registered in `bzrlib.smart.request`.
448
**XXX**: ideally the request methods should be documented here.
449
Contributions welcome!
455
The first argument of an error response specifies the error type.
457
One possible error name is ``UnknownMethod``, which means the server does
458
not recognise the verb used by the client's request. This error was
459
introduced in version three.
461
**XXX**: ideally the error types should be documented here. Contributions
added
removed
doc/developers/network-protocol.txt
