4634.39.32
by Ian Clatworthy
proper Contents panel in bzr-developers.chm |
1 |
==================== |
2 |
Bazaar Testing Guide |
|
3 |
==================== |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
4 |
|
5 |
||
6 |
The Importance of Testing |
|
7 |
========================= |
|
8 |
||
9 |
Reliability is a critical success factor for any Version Control System. |
|
10 |
We want Bazaar to be highly reliable across multiple platforms while |
|
11 |
evolving over time to meet the needs of its community. |
|
12 |
||
13 |
In a nutshell, this is what we expect and encourage: |
|
14 |
||
15 |
* New functionality should have test cases. Preferably write the |
|
16 |
test before writing the code. |
|
17 |
||
18 |
In general, you can test at either the command-line level or the |
|
19 |
internal API level. See `Writing tests`_ below for more detail. |
|
20 |
||
21 |
* Try to practice Test-Driven Development: before fixing a bug, write a |
|
22 |
test case so that it does not regress. Similarly for adding a new |
|
23 |
feature: write a test case for a small version of the new feature before |
|
24 |
starting on the code itself. Check the test fails on the old code, then |
|
25 |
add the feature or fix and check it passes. |
|
26 |
||
27 |
By doing these things, the Bazaar team gets increased confidence that |
|
28 |
changes do what they claim to do, whether provided by the core team or |
|
29 |
by community members. Equally importantly, we can be surer that changes |
|
30 |
down the track do not break new features or bug fixes that you are |
|
31 |
contributing today. |
|
32 |
||
4665.2.2
by Martin Pool
Doc update that there are actually many more tests now |
33 |
As of September 2009, Bazaar ships with a test suite containing over |
34 |
23,000 tests and growing. We are proud of it and want to remain so. As |
|
35 |
community members, we all benefit from it. Would you trust version control |
|
36 |
on your project to a product *without* a test suite like Bazaar has? |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
37 |
|
38 |
||
39 |
Running the Test Suite |
|
40 |
====================== |
|
41 |
||
5004.2.5
by Martin Pool
More docs on testing |
42 |
As of Bazaar 2.1, you must have the testtools_ library installed to run |
43 |
the bzr test suite. |
|
44 |
||
45 |
.. _testtools: https://launchpad.net/testtools/ |
|
46 |
||
5004.2.4
by Martin Pool
More tips on running tests |
47 |
To test all of Bazaar, just run:: |
48 |
||
49 |
bzr selftest |
|
50 |
||
5004.2.5
by Martin Pool
More docs on testing |
51 |
With ``--verbose`` bzr will print the name of every test as it is run. |
52 |
||
5004.2.4
by Martin Pool
More tips on running tests |
53 |
This should always pass, whether run from a source tree or an installed |
54 |
copy of Bazaar. Please investigate and/or report any failures. |
|
55 |
||
56 |
||
57 |
Running particular tests |
|
58 |
------------------------ |
|
59 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
60 |
Currently, bzr selftest is used to invoke tests. |
61 |
You can provide a pattern argument to run a subset. For example, |
|
62 |
to run just the blackbox tests, run:: |
|
63 |
||
64 |
./bzr selftest -v blackbox |
|
65 |
||
66 |
To skip a particular test (or set of tests), use the --exclude option |
|
67 |
(shorthand -x) like so:: |
|
68 |
||
69 |
./bzr selftest -v -x blackbox |
|
70 |
||
71 |
To ensure that all tests are being run and succeeding, you can use the |
|
72 |
--strict option which will fail if there are any missing features or known |
|
73 |
failures, like so:: |
|
74 |
||
75 |
./bzr selftest --strict |
|
76 |
||
77 |
To list tests without running them, use the --list-only option like so:: |
|
78 |
||
79 |
./bzr selftest --list-only |
|
80 |
||
81 |
This option can be combined with other selftest options (like -x) and |
|
82 |
filter patterns to understand their effect. |
|
83 |
||
84 |
Once you understand how to create a list of tests, you can use the --load-list |
|
85 |
option to run only a restricted set of tests that you kept in a file, one test |
|
86 |
id by line. Keep in mind that this will never be sufficient to validate your |
|
87 |
modifications, you still need to run the full test suite for that, but using it |
|
88 |
can help in some cases (like running only the failed tests for some time):: |
|
89 |
||
90 |
./bzr selftest -- load-list my_failing_tests |
|
91 |
||
92 |
This option can also be combined with other selftest options, including |
|
93 |
patterns. It has some drawbacks though, the list can become out of date pretty |
|
94 |
quick when doing Test Driven Development. |
|
95 |
||
96 |
To address this concern, there is another way to run a restricted set of tests: |
|
97 |
the --starting-with option will run only the tests whose name starts with the |
|
98 |
specified string. It will also avoid loading the other tests and as a |
|
99 |
consequence starts running your tests quicker:: |
|
100 |
||
101 |
./bzr selftest --starting-with bzrlib.blackbox |
|
102 |
||
103 |
This option can be combined with all the other selftest options including |
|
104 |
--load-list. The later is rarely used but allows to run a subset of a list of |
|
105 |
failing tests for example. |
|
106 |
||
5004.2.4
by Martin Pool
More tips on running tests |
107 |
Disabling plugins |
108 |
----------------- |
|
109 |
||
110 |
To test only the bzr core, ignoring any plugins you may have installed, |
|
111 |
use:: |
|
112 |
||
113 |
./bzr --no-plugins selftest |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
114 |
|
5004.2.2
by Martin Pool
Recommend using -Dno_apport for development |
115 |
Disabling crash reporting |
116 |
------------------------- |
|
117 |
||
118 |
By default Bazaar uses apport_ to report program crashes. In developing |
|
119 |
Bazaar it's normal and expected to have it crash from time to time, at |
|
120 |
least because a test failed if for no other reason. |
|
121 |
||
122 |
Therefore you should probably add ``debug_flags = no_apport`` to your |
|
123 |
``bazaar.conf`` file (in ``~/.bazaar/`` on Unix), so that failures just |
|
124 |
print a traceback rather than writing a crash file. |
|
125 |
||
126 |
.. _apport: https://launchpad.net/apport/ |
|
127 |
||
128 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
129 |
Test suite debug flags |
130 |
---------------------- |
|
131 |
||
132 |
Similar to the global ``-Dfoo`` debug options, bzr selftest accepts |
|
133 |
``-E=foo`` debug flags. These flags are: |
|
134 |
||
135 |
:allow_debug: do *not* clear the global debug flags when running a test. |
|
136 |
This can provide useful logging to help debug test failures when used |
|
137 |
with e.g. ``bzr -Dhpss selftest -E=allow_debug`` |
|
138 |
||
5004.2.3
by Martin Pool
Caveat on -Eallow_debug |
139 |
Note that this will probably cause some tests to fail, because they |
140 |
don't expect to run with any debug flags on. |
|
141 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
142 |
|
5004.2.5
by Martin Pool
More docs on testing |
143 |
Using subunit |
144 |
------------- |
|
145 |
||
146 |
Bazaar can optionally produce output in the machine-readable subunit_ |
|
5060.2.1
by Robert Collins
* bzr now has a ``.testr.conf`` file in its source tree configured |
147 |
format, so that test output can be post-processed by various tools. To |
148 |
generate a subunit test stream:: |
|
149 |
||
150 |
$ ./bzr selftest --subunit |
|
151 |
||
152 |
Processing such a stream can be done using a variety of tools including: |
|
153 |
||
154 |
* The builtin ``subunit2pyunit``, ``subunit-filter``, ``subunit-ls``, |
|
155 |
``subunit2junitxml`` from the subunit project. |
|
156 |
||
157 |
* tribunal_, a GUI for showing test results. |
|
158 |
||
159 |
* testrepository_, a tool for gathering and managing test runs. |
|
5004.2.5
by Martin Pool
More docs on testing |
160 |
|
161 |
.. _subunit: https://launchpad.net/subunit/ |
|
5060.2.1
by Robert Collins
* bzr now has a ``.testr.conf`` file in its source tree configured |
162 |
.. _tribunal: https://launchpad.net/tribunal/ |
163 |
||
164 |
||
165 |
Using testrepository |
|
166 |
-------------------- |
|
167 |
||
168 |
Bazaar ships with a config file for testrepository_. This can be very |
|
169 |
useful for keeping track of failing tests and doing general workflow |
|
170 |
support. To run tests using testrepository:: |
|
171 |
||
172 |
$ testr run |
|
173 |
||
174 |
To run only failing tests:: |
|
175 |
||
176 |
$ testr run --failing |
|
177 |
||
178 |
To run only some tests, without plugins:: |
|
179 |
||
180 |
$ test run test_selftest -- --no-plugins |
|
181 |
||
182 |
See the testrepository documentation for more details. |
|
183 |
||
184 |
.. _testrepository: https://launchpad.net/testrepository |
|
5004.2.5
by Martin Pool
More docs on testing |
185 |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
186 |
Writing Tests |
187 |
============= |
|
188 |
||
5004.2.5
by Martin Pool
More docs on testing |
189 |
Normally you should add or update a test for all bug fixes or new features |
190 |
in Bazaar. |
|
191 |
||
192 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
193 |
Where should I put a new test? |
194 |
------------------------------ |
|
195 |
||
196 |
Bzrlib's tests are organised by the type of test. Most of the tests in |
|
197 |
bzr's test suite belong to one of these categories: |
|
198 |
||
199 |
- Unit tests |
|
200 |
- Blackbox (UI) tests |
|
201 |
- Per-implementation tests |
|
202 |
- Doctests |
|
203 |
||
204 |
A quick description of these test types and where they belong in bzrlib's |
|
205 |
source follows. Not all tests fall neatly into one of these categories; |
|
206 |
in those cases use your judgement. |
|
207 |
||
208 |
||
209 |
Unit tests |
|
210 |
~~~~~~~~~~ |
|
211 |
||
212 |
Unit tests make up the bulk of our test suite. These are tests that are |
|
213 |
focused on exercising a single, specific unit of the code as directly |
|
214 |
as possible. Each unit test is generally fairly short and runs very |
|
215 |
quickly. |
|
216 |
||
217 |
They are found in ``bzrlib/tests/test_*.py``. So in general tests should |
|
218 |
be placed in a file named test_FOO.py where FOO is the logical thing under |
|
219 |
test. |
|
220 |
||
221 |
For example, tests for merge3 in bzrlib belong in bzrlib/tests/test_merge3.py. |
|
222 |
See bzrlib/tests/test_sampler.py for a template test script. |
|
223 |
||
224 |
||
225 |
Blackbox (UI) tests |
|
226 |
~~~~~~~~~~~~~~~~~~~ |
|
227 |
||
228 |
Tests can be written for the UI or for individual areas of the library. |
|
229 |
Choose whichever is appropriate: if adding a new command, or a new command |
|
230 |
option, then you should be writing a UI test. If you are both adding UI |
|
231 |
functionality and library functionality, you will want to write tests for |
|
232 |
both the UI and the core behaviours. We call UI tests 'blackbox' tests |
|
233 |
and they belong in ``bzrlib/tests/blackbox/*.py``. |
|
234 |
||
235 |
When writing blackbox tests please honour the following conventions: |
|
236 |
||
237 |
1. Place the tests for the command 'name' in |
|
238 |
bzrlib/tests/blackbox/test_name.py. This makes it easy for developers |
|
239 |
to locate the test script for a faulty command. |
|
240 |
||
241 |
2. Use the 'self.run_bzr("name")' utility function to invoke the command |
|
242 |
rather than running bzr in a subprocess or invoking the |
|
243 |
cmd_object.run() method directly. This is a lot faster than |
|
244 |
subprocesses and generates the same logging output as running it in a |
|
245 |
subprocess (which invoking the method directly does not). |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
246 |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
247 |
3. Only test the one command in a single test script. Use the bzrlib |
248 |
library when setting up tests and when evaluating the side-effects of |
|
249 |
the command. We do this so that the library api has continual pressure |
|
250 |
on it to be as functional as the command line in a simple manner, and |
|
251 |
to isolate knock-on effects throughout the blackbox test suite when a |
|
252 |
command changes its name or signature. Ideally only the tests for a |
|
253 |
given command are affected when a given command is changed. |
|
254 |
||
255 |
4. If you have a test which does actually require running bzr in a |
|
256 |
subprocess you can use ``run_bzr_subprocess``. By default the spawned |
|
257 |
process will not load plugins unless ``--allow-plugins`` is supplied. |
|
258 |
||
259 |
||
260 |
Per-implementation tests |
|
261 |
~~~~~~~~~~~~~~~~~~~~~~~~ |
|
262 |
||
263 |
Per-implementation tests are tests that are defined once and then run |
|
264 |
against multiple implementations of an interface. For example, |
|
4913.3.7
by John Arbash Meinel
Doc updates for permute_for_extension |
265 |
``per_transport.py`` defines tests that all Transport implementations |
266 |
(local filesystem, HTTP, and so on) must pass. They are found in |
|
267 |
``bzrlib/tests/per_*/*.py``, and ``bzrlib/tests/per_*.py``. |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
268 |
|
269 |
These are really a sub-category of unit tests, but an important one. |
|
270 |
||
4913.3.7
by John Arbash Meinel
Doc updates for permute_for_extension |
271 |
Along the same lines are tests for extension modules. We generally have |
272 |
both a pure-python and a compiled implementation for each module. As such, |
|
273 |
we want to run the same tests against both implementations. These can |
|
274 |
generally be found in ``bzrlib/tests/*__*.py`` since extension modules are |
|
275 |
usually prefixed with an underscore. Since there are only two |
|
276 |
implementations, we have a helper function |
|
277 |
``bzrlib.tests.permute_for_extension``, which can simplify the |
|
278 |
``load_tests`` implementation. |
|
279 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
280 |
|
281 |
Doctests |
|
282 |
~~~~~~~~ |
|
283 |
||
284 |
We make selective use of doctests__. In general they should provide |
|
285 |
*examples* within the API documentation which can incidentally be tested. We |
|
286 |
don't try to test every important case using doctests |--| regular Python |
|
287 |
tests are generally a better solution. That is, we just use doctests to |
|
288 |
make our documentation testable, rather than as a way to make tests. |
|
289 |
||
290 |
Most of these are in ``bzrlib/doc/api``. More additions are welcome. |
|
291 |
||
292 |
__ http://docs.python.org/lib/module-doctest.html |
|
293 |
||
294 |
||
4665.5.20
by Vincent Ladeuil
Fixed as per Martin's review. |
295 |
Shell-like tests |
4917.2.1
by Martin Pool
Add better example for ScriptRunner and tweak its place in the document hierarchy |
296 |
---------------- |
4665.5.20
by Vincent Ladeuil
Fixed as per Martin's review. |
297 |
|
298 |
``bzrlib/tests/script.py`` allows users to write tests in a syntax very close to a shell session, |
|
299 |
using a restricted and limited set of commands that should be enough to mimic |
|
300 |
most of the behaviours. |
|
301 |
||
302 |
A script is a set of commands, each command is composed of: |
|
303 |
||
304 |
* one mandatory command line, |
|
305 |
* one optional set of input lines to feed the command, |
|
306 |
* one optional set of output expected lines, |
|
307 |
* one optional set of error expected lines. |
|
308 |
||
309 |
Input, output and error lines can be specified in any order. |
|
310 |
||
311 |
Except for the expected output, all lines start with a special |
|
312 |
string (based on their origin when used under a Unix shell): |
|
313 |
||
314 |
* '$ ' for the command, |
|
315 |
* '<' for input, |
|
316 |
* nothing for output, |
|
317 |
* '2>' for errors, |
|
318 |
||
319 |
Comments can be added anywhere, they start with '#' and end with |
|
320 |
the line. |
|
321 |
||
322 |
The execution stops as soon as an expected output or an expected error is not |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
323 |
matched. |
4665.5.20
by Vincent Ladeuil
Fixed as per Martin's review. |
324 |
|
325 |
When no output is specified, any ouput from the command is accepted |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
326 |
and execution continue. |
4665.5.20
by Vincent Ladeuil
Fixed as per Martin's review. |
327 |
|
328 |
If an error occurs and no expected error is specified, the execution stops. |
|
329 |
||
330 |
An error is defined by a returned status different from zero, not by the |
|
331 |
presence of text on the error stream. |
|
332 |
||
333 |
The matching is done on a full string comparison basis unless '...' is used, in |
|
334 |
which case expected output/errors can be less precise. |
|
335 |
||
336 |
Examples: |
|
337 |
||
338 |
The following will succeeds only if 'bzr add' outputs 'adding file':: |
|
339 |
||
340 |
$ bzr add file |
|
341 |
>adding file |
|
342 |
||
343 |
If you want the command to succeed for any output, just use:: |
|
344 |
||
345 |
$ bzr add file |
|
346 |
||
347 |
The following will stop with an error:: |
|
348 |
||
349 |
$ bzr not-a-command |
|
350 |
||
351 |
If you want it to succeed, use:: |
|
352 |
||
353 |
$ bzr not-a-command |
|
354 |
2> bzr: ERROR: unknown command "not-a-command" |
|
355 |
||
356 |
You can use ellipsis (...) to replace any piece of text you don't want to be |
|
357 |
matched exactly:: |
|
358 |
||
359 |
$ bzr branch not-a-branch |
|
360 |
2>bzr: ERROR: Not a branch...not-a-branch/". |
|
361 |
||
362 |
This can be used to ignore entire lines too:: |
|
363 |
||
364 |
$ cat |
|
365 |
<first line |
|
366 |
<second line |
|
367 |
<third line |
|
368 |
# And here we explain that surprising fourth line |
|
369 |
<fourth line |
|
370 |
<last line |
|
371 |
>first line |
|
372 |
>... |
|
373 |
>last line |
|
374 |
||
375 |
You can check the content of a file with cat:: |
|
376 |
||
377 |
$ cat <file |
|
378 |
>expected content |
|
379 |
||
380 |
You can also check the existence of a file with cat, the following will fail if |
|
381 |
the file doesn't exist:: |
|
382 |
||
383 |
$ cat file |
|
384 |
||
4917.2.1
by Martin Pool
Add better example for ScriptRunner and tweak its place in the document hierarchy |
385 |
The actual use of ScriptRunner within a TestCase looks something like |
386 |
this:: |
|
387 |
||
388 |
def test_unshelve_keep(self): |
|
389 |
# some setup here |
|
390 |
sr = ScriptRunner() |
|
391 |
sr.run_script(self, ''' |
|
392 |
$ bzr add file |
|
393 |
$ bzr shelve --all -m Foo |
|
394 |
$ bzr shelve --list |
|
395 |
1: Foo |
|
396 |
$ bzr unshelve --keep |
|
397 |
$ bzr shelve --list |
|
398 |
1: Foo |
|
399 |
$ cat file |
|
400 |
contents of file |
|
401 |
''') |
|
402 |
||
4665.5.20
by Vincent Ladeuil
Fixed as per Martin's review. |
403 |
|
5017.2.2
by Martin Pool
Add import tariff tests |
404 |
Import tariff tests |
405 |
------------------- |
|
406 |
||
407 |
`bzrlib.tests.test_import_tariff` has some tests that measure how many |
|
408 |
Python modules are loaded to run some representative commands. |
|
409 |
||
410 |
We want to avoid loading code unnecessarily, for reasons including: |
|
411 |
||
412 |
* Python modules are interpreted when they're loaded, either to define |
|
413 |
classes or modules or perhaps to initialize some structures. |
|
414 |
||
415 |
* With a cold cache we may incur blocking real disk IO for each module. |
|
416 |
||
417 |
* Some modules depend on many others. |
|
418 |
||
419 |
* Some optional modules such as `testtools` are meant to be soft |
|
420 |
dependencies and only needed for particular cases. If they're loaded in |
|
421 |
other cases then bzr may break for people who don't have those modules. |
|
422 |
||
423 |
`test_import_tarrif` allows us to check that removal of imports doesn't |
|
424 |
regress. |
|
425 |
||
426 |
This is done by running the command in a subprocess with |
|
427 |
``--profile-imports``. Starting a whole Python interpreter is pretty |
|
428 |
slow, so we don't want exhaustive testing here, but just enough to guard |
|
429 |
against distinct fixed problems. |
|
430 |
||
431 |
Assertions about precisely what is loaded tend to be brittle so we instead |
|
432 |
make assertions that particular things aren't loaded. |
|
433 |
||
434 |
Unless selftest is run with ``--no-plugins``, modules will be loaded in |
|
435 |
the usual way and checks made on what they cause to be loaded. This is |
|
436 |
probably worth checking into, because many bzr users have at least some |
|
437 |
plugins installed (and they're included in binary installers). |
|
438 |
||
439 |
In theory, plugins might have a good reason to load almost anything: |
|
440 |
someone might write a plugin that opens a network connection or pops up a |
|
441 |
gui window every time you run 'bzr status'. However, it's more likely |
|
442 |
that the code to do these things is just being loaded accidentally. We |
|
443 |
might eventually need to have a way to make exceptions for particular |
|
444 |
plugins. |
|
445 |
||
446 |
Some things to check: |
|
447 |
||
448 |
* non-GUI commands shouldn't load GUI libraries |
|
449 |
||
450 |
* operations on bzr native formats sholudn't load foreign branch libraries |
|
451 |
||
452 |
* network code shouldn't be loaded for purely local operations |
|
453 |
||
454 |
* particularly expensive Python built-in modules shouldn't be loaded |
|
455 |
unless there is a good reason |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
456 |
|
457 |
||
5077.3.1
by Martin Pool
Tip on testing locking behaviour |
458 |
Testing locking behaviour |
459 |
------------------------- |
|
460 |
||
461 |
You may want to write tests that particular objects are or aren't locked |
|
462 |
during particular operations: see for example `bug 498409`__. |
|
463 |
||
464 |
__ https://launchpad.net/bugs/498409 |
|
465 |
||
466 |
The `TestCase` base class registers hooks that record lock actions into |
|
467 |
``._lock_actions`` in this format:: |
|
468 |
||
469 |
[ |
|
470 |
('acquired', LockResult(file:///tmp/testbzr-J2pcy2.tmp/.bzr/branch-lockc4au55ppz8wdym11z1aq)), |
|
471 |
('released', LockResult(file:///tmp/testbzr-J2pcy2.tmp/.bzr/branch-lockc4au55ppz8wdym11z1aq)), |
|
472 |
('acquired', LockResult(file:///tmp/testbzr-J2pcy2.tmp/.bzr/repository/lockyxb3rn4sw1oyx1jzkt45)), |
|
473 |
('released', LockResult(file:///tmp/testbzr-J2pcy2.tmp/.bzr/repository/lockyxb3rn4sw1oyx1jzkt45)), |
|
474 |
('acquired', LockResult(file:///tmp/testbzr-J2pcy2.tmp/.bzr/branch/lockh8c6t28rcjdkgxtndbje)), |
|
475 |
('released', LockResult(file:///tmp/testbzr-J2pcy2.tmp/.bzr/branch/lockh8c6t28rcjdkgxtndbje)), |
|
476 |
... |
|
477 |
||
478 |
Alternatively you can register your own hooks to make custom assertions: |
|
479 |
see `TestCase._check_locks` for an example. |
|
480 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
481 |
Skipping tests |
482 |
-------------- |
|
483 |
||
484 |
In our enhancements to unittest we allow for some addition results beyond |
|
485 |
just success or failure. |
|
486 |
||
487 |
If a test can't be run, it can say that it's skipped by raising a special |
|
488 |
exception. This is typically used in parameterized tests |--| for example |
|
489 |
if a transport doesn't support setting permissions, we'll skip the tests |
|
490 |
that relating to that. :: |
|
491 |
||
492 |
try: |
|
493 |
return self.branch_format.initialize(repo.bzrdir) |
|
494 |
except errors.UninitializableFormat: |
|
495 |
raise tests.TestSkipped('Uninitializable branch format') |
|
496 |
||
497 |
Raising TestSkipped is a good idea when you want to make it clear that the |
|
498 |
test was not run, rather than just returning which makes it look as if it |
|
499 |
was run and passed. |
|
500 |
||
501 |
Several different cases are distinguished: |
|
502 |
||
503 |
TestSkipped |
|
504 |
Generic skip; the only type that was present up to bzr 0.18. |
|
505 |
||
506 |
TestNotApplicable |
|
507 |
The test doesn't apply to the parameters with which it was run. |
|
508 |
This is typically used when the test is being applied to all |
|
509 |
implementations of an interface, but some aspects of the interface |
|
510 |
are optional and not present in particular concrete |
|
511 |
implementations. (Some tests that should raise this currently |
|
512 |
either silently return or raise TestSkipped.) Another option is |
|
513 |
to use more precise parameterization to avoid generating the test |
|
514 |
at all. |
|
515 |
||
516 |
UnavailableFeature |
|
517 |
The test can't be run because a dependency (typically a Python |
|
518 |
library) is not available in the test environment. These |
|
519 |
are in general things that the person running the test could fix |
|
520 |
by installing the library. It's OK if some of these occur when |
|
521 |
an end user runs the tests or if we're specifically testing in a |
|
522 |
limited environment, but a full test should never see them. |
|
523 |
||
524 |
See `Test feature dependencies`_ below. |
|
525 |
||
526 |
KnownFailure |
|
527 |
The test exists but is known to fail, for example this might be |
|
528 |
appropriate to raise if you've committed a test for a bug but not |
|
529 |
the fix for it, or if something works on Unix but not on Windows. |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
530 |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
531 |
Raising this allows you to distinguish these failures from the |
532 |
ones that are not expected to fail. If the test would fail |
|
533 |
because of something we don't expect or intend to fix, |
|
534 |
KnownFailure is not appropriate, and TestNotApplicable might be |
|
535 |
better. |
|
536 |
||
537 |
KnownFailure should be used with care as we don't want a |
|
538 |
proliferation of quietly broken tests. |
|
539 |
||
4873.2.4
by John Arbash Meinel
Add a NEWS entry and an entry in the testing docs about ModuleAvailableFeature |
540 |
|
541 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
542 |
We plan to support three modes for running the test suite to control the |
543 |
interpretation of these results. Strict mode is for use in situations |
|
544 |
like merges to the mainline and releases where we want to make sure that |
|
545 |
everything that can be tested has been tested. Lax mode is for use by |
|
546 |
developers who want to temporarily tolerate some known failures. The |
|
547 |
default behaviour is obtained by ``bzr selftest`` with no options, and |
|
548 |
also (if possible) by running under another unittest harness. |
|
549 |
||
550 |
======================= ======= ======= ======== |
|
551 |
result strict default lax |
|
552 |
======================= ======= ======= ======== |
|
553 |
TestSkipped pass pass pass |
|
554 |
TestNotApplicable pass pass pass |
|
3619.3.2
by Andrew Bennetts
Remove references to unimplemented TestPlatformLimit, remove some redundant (and misplaced) text from 'Test feature dependencies'. |
555 |
UnavailableFeature fail pass pass |
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
556 |
KnownFailure fail pass pass |
557 |
======================= ======= ======= ======== |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
558 |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
559 |
|
560 |
Test feature dependencies |
|
561 |
------------------------- |
|
562 |
||
563 |
Writing tests that require a feature |
|
564 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
565 |
||
566 |
Rather than manually checking the environment in each test, a test class |
|
567 |
can declare its dependence on some test features. The feature objects are |
|
568 |
checked only once for each run of the whole test suite. |
|
569 |
||
570 |
(For historical reasons, as of May 2007 many cases that should depend on |
|
571 |
features currently raise TestSkipped.) |
|
572 |
||
573 |
For example:: |
|
574 |
||
575 |
class TestStrace(TestCaseWithTransport): |
|
576 |
||
577 |
_test_needs_features = [StraceFeature] |
|
578 |
||
3619.3.2
by Andrew Bennetts
Remove references to unimplemented TestPlatformLimit, remove some redundant (and misplaced) text from 'Test feature dependencies'. |
579 |
This means all tests in this class need the feature. If the feature is |
580 |
not available the test will be skipped using UnavailableFeature. |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
581 |
|
582 |
Individual tests can also require a feature using the ``requireFeature`` |
|
583 |
method:: |
|
584 |
||
585 |
self.requireFeature(StraceFeature) |
|
586 |
||
5004.2.1
by Martin Pool
Better documentation of ModuleAvailableFeature |
587 |
The old naming style for features is CamelCase, but because they're |
588 |
actually instances not classses they're now given instance-style names |
|
589 |
like ``apport``. |
|
590 |
||
591 |
Features already defined in ``bzrlib.tests`` and ``bzrlib.tests.features`` |
|
592 |
include: |
|
593 |
||
594 |
- apport |
|
595 |
- paramiko |
|
596 |
- SymlinkFeature |
|
597 |
- HardlinkFeature |
|
598 |
- OsFifoFeature |
|
599 |
- UnicodeFilenameFeature |
|
600 |
- FTPServerFeature |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
601 |
- CaseInsensitiveFilesystemFeature. |
5094.3.1
by Martin Pool
``.bazaar``, ``.bazaar/bazaar.conf`` and ``.bzr.log`` inherit user and group ownership from the containing directory. This allow bzr to work better with sudo. |
602 |
- chown_feature: The test can rely on OS being POSIX and python |
5051.4.6
by Parth Malwankar
documented ChownFeature in testing.txt |
603 |
supporting os.chown. |
5094.3.1
by Martin Pool
``.bazaar``, ``.bazaar/bazaar.conf`` and ``.bzr.log`` inherit user and group ownership from the containing directory. This allow bzr to work better with sudo. |
604 |
- posix_permissions_feature: The test can use POSIX-style |
605 |
user/group/other permission bits. |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
606 |
|
607 |
||
608 |
Defining a new feature that tests can require |
|
609 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
610 |
||
611 |
New features for use with ``_test_needs_features`` or ``requireFeature`` |
|
612 |
are defined by subclassing ``bzrlib.tests.Feature`` and overriding the |
|
613 |
``_probe`` and ``feature_name`` methods. For example:: |
|
614 |
||
615 |
class _SymlinkFeature(Feature): |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
616 |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
617 |
def _probe(self): |
618 |
return osutils.has_symlinks() |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
619 |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
620 |
def feature_name(self): |
621 |
return 'symlinks' |
|
4853.1.1
by Patrick Regan
Removed trailing whitespace from files in doc directory |
622 |
|
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
623 |
SymlinkFeature = _SymlinkFeature() |
624 |
||
5004.2.1
by Martin Pool
Better documentation of ModuleAvailableFeature |
625 |
A helper for handling running tests based on whether a python |
626 |
module is available. This can handle 3rd-party dependencies (is |
|
627 |
``paramiko`` available?) as well as stdlib (``termios``) or |
|
628 |
extension modules (``bzrlib._groupcompress_pyx``). You create a |
|
629 |
new feature instance with:: |
|
630 |
||
631 |
# in bzrlib/tests/features.py |
|
632 |
apport = tests.ModuleAvailableFeature('apport') |
|
633 |
||
634 |
||
635 |
# then in bzrlib/tests/test_apport.py |
|
636 |
class TestApportReporting(TestCaseInTempDir): |
|
637 |
||
638 |
_test_needs_features = [features.apport] |
|
639 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
640 |
|
641 |
Testing exceptions and errors |
|
642 |
----------------------------- |
|
643 |
||
644 |
It's important to test handling of errors and exceptions. Because this |
|
645 |
code is often not hit in ad-hoc testing it can often have hidden bugs -- |
|
646 |
it's particularly common to get NameError because the exception code |
|
647 |
references a variable that has since been renamed. |
|
648 |
||
649 |
.. TODO: Something about how to provoke errors in the right way? |
|
650 |
||
651 |
In general we want to test errors at two levels: |
|
652 |
||
653 |
1. A test in ``test_errors.py`` checking that when the exception object is |
|
654 |
constructed with known parameters it produces an expected string form. |
|
655 |
This guards against mistakes in writing the format string, or in the |
|
656 |
``str`` representations of its parameters. There should be one for |
|
657 |
each exception class. |
|
658 |
||
659 |
2. Tests that when an api is called in a particular situation, it raises |
|
660 |
an error of the expected class. You should typically use |
|
661 |
``assertRaises``, which in the Bazaar test suite returns the exception |
|
662 |
object to allow you to examine its parameters. |
|
663 |
||
664 |
In some cases blackbox tests will also want to check error reporting. But |
|
665 |
it can be difficult to provoke every error through the commandline |
|
666 |
interface, so those tests are only done as needed |--| eg in response to a |
|
667 |
particular bug or if the error is reported in an unusual way(?) Blackbox |
|
668 |
tests should mostly be testing how the command-line interface works, so |
|
669 |
should only test errors if there is something particular to the cli in how |
|
670 |
they're displayed or handled. |
|
671 |
||
672 |
||
673 |
Testing warnings |
|
674 |
---------------- |
|
675 |
||
676 |
The Python ``warnings`` module is used to indicate a non-fatal code |
|
677 |
problem. Code that's expected to raise a warning can be tested through |
|
678 |
callCatchWarnings. |
|
679 |
||
680 |
The test suite can be run with ``-Werror`` to check no unexpected errors |
|
681 |
occur. |
|
682 |
||
683 |
However, warnings should be used with discretion. It's not an appropriate |
|
684 |
way to give messages to the user, because the warning is normally shown |
|
685 |
only once per source line that causes the problem. You should also think |
|
686 |
about whether the warning is serious enought that it should be visible to |
|
687 |
users who may not be able to fix it. |
|
688 |
||
689 |
||
690 |
Interface implementation testing and test scenarios |
|
691 |
--------------------------------------------------- |
|
692 |
||
693 |
There are several cases in Bazaar of multiple implementations of a common |
|
694 |
conceptual interface. ("Conceptual" because it's not necessary for all |
|
695 |
the implementations to share a base class, though they often do.) |
|
696 |
Examples include transports and the working tree, branch and repository |
|
697 |
classes. |
|
698 |
||
699 |
In these cases we want to make sure that every implementation correctly |
|
700 |
fulfils the interface requirements. For example, every Transport should |
|
701 |
support the ``has()`` and ``get()`` and ``clone()`` methods. We have a |
|
702 |
sub-suite of tests in ``test_transport_implementations``. (Most |
|
703 |
per-implementation tests are in submodules of ``bzrlib.tests``, but not |
|
704 |
the transport tests at the moment.) |
|
705 |
||
706 |
These tests are repeated for each registered Transport, by generating a |
|
707 |
new TestCase instance for the cross product of test methods and transport |
|
708 |
implementations. As each test runs, it has ``transport_class`` and |
|
709 |
``transport_server`` set to the class it should test. Most tests don't |
|
710 |
access these directly, but rather use ``self.get_transport`` which returns |
|
711 |
a transport of the appropriate type. |
|
712 |
||
713 |
The goal is to run per-implementation only the tests that relate to that |
|
714 |
particular interface. Sometimes we discover a bug elsewhere that happens |
|
715 |
with only one particular transport. Once it's isolated, we can consider |
|
716 |
whether a test should be added for that particular implementation, |
|
717 |
or for all implementations of the interface. |
|
718 |
||
719 |
The multiplication of tests for different implementations is normally |
|
720 |
accomplished by overriding the ``load_tests`` function used to load tests |
|
721 |
from a module. This function typically loads all the tests, then applies |
|
722 |
a TestProviderAdapter to them, which generates a longer suite containing |
|
723 |
all the test variations. |
|
724 |
||
725 |
See also `Per-implementation tests`_ (above). |
|
726 |
||
727 |
||
728 |
Test scenarios |
|
729 |
-------------- |
|
730 |
||
731 |
Some utilities are provided for generating variations of tests. This can |
|
732 |
be used for per-implementation tests, or other cases where the same test |
|
733 |
code needs to run several times on different scenarios. |
|
734 |
||
735 |
The general approach is to define a class that provides test methods, |
|
736 |
which depend on attributes of the test object being pre-set with the |
|
737 |
values to which the test should be applied. The test suite should then |
|
738 |
also provide a list of scenarios in which to run the tests. |
|
739 |
||
740 |
Typically ``multiply_tests_from_modules`` should be called from the test |
|
741 |
module's ``load_tests`` function. |
|
742 |
||
743 |
||
744 |
Test support |
|
745 |
------------ |
|
746 |
||
747 |
We have a rich collection of tools to support writing tests. Please use |
|
748 |
them in preference to ad-hoc solutions as they provide portability and |
|
749 |
performance benefits. |
|
750 |
||
751 |
||
752 |
TestCase and its subclasses |
|
753 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
|
754 |
||
755 |
The ``bzrlib.tests`` module defines many TestCase classes to help you |
|
756 |
write your tests. |
|
757 |
||
758 |
TestCase |
|
759 |
A base TestCase that extends the Python standard library's |
|
760 |
TestCase in several ways. It adds more assertion methods (e.g. |
|
761 |
``assertContainsRe``), ``addCleanup``, and other features (see its API |
|
762 |
docs for details). It also has a ``setUp`` that makes sure that |
|
763 |
global state like registered hooks and loggers won't interfere with |
|
764 |
your test. All tests should use this base class (whether directly or |
|
765 |
via a subclass). |
|
766 |
||
767 |
TestCaseWithMemoryTransport |
|
768 |
Extends TestCase and adds methods like ``get_transport``, |
|
769 |
``make_branch`` and ``make_branch_builder``. The files created are |
|
770 |
stored in a MemoryTransport that is discarded at the end of the test. |
|
771 |
This class is good for tests that need to make branches or use |
|
772 |
transports, but that don't require storing things on disk. All tests |
|
773 |
that create bzrdirs should use this base class (either directly or via |
|
774 |
a subclass) as it ensures that the test won't accidentally operate on |
|
775 |
real branches in your filesystem. |
|
776 |
||
777 |
TestCaseInTempDir |
|
778 |
Extends TestCaseWithMemoryTransport. For tests that really do need |
|
779 |
files to be stored on disk, e.g. because a subprocess uses a file, or |
|
780 |
for testing functionality that accesses the filesystem directly rather |
|
781 |
than via the Transport layer (such as dirstate). |
|
782 |
||
783 |
TestCaseWithTransport |
|
784 |
Extends TestCaseInTempDir. Provides ``get_url`` and |
|
785 |
``get_readonly_url`` facilities. Subclasses can control the |
|
786 |
transports used by setting ``vfs_transport_factory``, |
|
787 |
``transport_server`` and/or ``transport_readonly_server``. |
|
788 |
||
789 |
||
790 |
See the API docs for more details. |
|
791 |
||
792 |
||
793 |
BranchBuilder |
|
794 |
~~~~~~~~~~~~~ |
|
795 |
||
796 |
When writing a test for a feature, it is often necessary to set up a |
|
797 |
branch with a certain history. The ``BranchBuilder`` interface allows the |
|
798 |
creation of test branches in a quick and easy manner. Here's a sample |
|
799 |
session:: |
|
800 |
||
801 |
builder = self.make_branch_builder('relpath') |
|
802 |
builder.build_commit() |
|
803 |
builder.build_commit() |
|
804 |
builder.build_commit() |
|
805 |
branch = builder.get_branch() |
|
806 |
||
807 |
``make_branch_builder`` is a method of ``TestCaseWithMemoryTransport``. |
|
808 |
||
809 |
Note that many current tests create test branches by inheriting from |
|
810 |
``TestCaseWithTransport`` and using the ``make_branch_and_tree`` helper to |
|
811 |
give them a ``WorkingTree`` that they can commit to. However, using the |
|
812 |
newer ``make_branch_builder`` helper is preferred, because it can build |
|
813 |
the changes in memory, rather than on disk. Tests that are explictly |
|
814 |
testing how we work with disk objects should, of course, use a real |
|
815 |
``WorkingTree``. |
|
816 |
||
817 |
Please see bzrlib.branchbuilder for more details. |
|
818 |
||
4070.5.2
by Martin Pool
Recommend setting timestamp in BranchBuilder |
819 |
If you're going to examine the commit timestamps e.g. in a test for log |
820 |
output, you should set the timestamp on the tree, rather than using fuzzy |
|
821 |
matches in the test. |
|
822 |
||
3619.3.1
by Andrew Bennetts
Move the notes on writing tests out of HACKING into a new file, and improve |
823 |
|
824 |
TreeBuilder |
|
825 |
~~~~~~~~~~~ |
|
826 |
||
827 |
The ``TreeBuilder`` interface allows the construction of arbitrary trees |
|
828 |
with a declarative interface. A sample session might look like:: |
|
829 |
||
830 |
tree = self.make_branch_and_tree('path') |
|
831 |
builder = TreeBuilder() |
|
832 |
builder.start_tree(tree) |
|
833 |
builder.build(['foo', "bar/", "bar/file"]) |
|
834 |
tree.commit('commit the tree') |
|
835 |
builder.finish_tree() |
|
836 |
||
837 |
Usually a test will create a tree using ``make_branch_and_memory_tree`` (a |
|
838 |
method of ``TestCaseWithMemoryTransport``) or ``make_branch_and_tree`` (a |
|
839 |
method of ``TestCaseWithTransport``). |
|
840 |
||
841 |
Please see bzrlib.treebuilder for more details. |
|
842 |
||
843 |
||
844 |
.. |--| unicode:: U+2014 |
|
845 |
||
846 |
.. |
|
847 |
vim: ft=rst tw=74 ai |