« back to all changes in this revision
Viewing changes to doc/developers/HACKING.txt
- browse files at revision 3619.3.1
- compare with another revision
- compare with revision 1185.16.85
- download diff
- download tarball
- view history from revision 3619.3.1
- Committer: Andrew Bennetts
- Date: 2008-08-12 14:53:26 UTC
- mto: This revision was merged to the branch mainline in revision 3624.
- Revision ID: andrew.bennetts@canonical.com-20080812145326-yx693x2jc4rcovb7
Move the notes on writing tests out of HACKING into a new file, and improve
them.
Many of the testing notes in the HACKING file were in duplicated in two places
in that file! This change removes that duplication. It also adds new sections
on “Where should I put a new test?” and “TestCase and its subclasses”, and
others like “Test feature dependencies” have been expanded. The whole document
has generally been edited to be a bit more coherent.
them.
Many of the testing notes in the HACKING file were in duplicated in two places
in that file! This change removes that duplication. It also adds new sections
on “Where should I put a new test?” and “TestCase and its subclasses”, and
others like “Test feature dependencies” have been expanded. The whole document
has generally been edited to be a bit more coherent.
- files added:
- doc/en/developer-guide/testing.txt
- files modified:
- Makefile
added
removed
doc/developers/HACKING.txt
268
268
section of this guide.
269
269
270
270
271
Testing Bazaar
272
##############
273
274
The Importance of Testing
275
=========================
276
277
Reliability is a critical success factor for any Version Control System.
278
We want Bazaar to be highly reliable across multiple platforms while
279
evolving over time to meet the needs of its community.
280
281
In a nutshell, this is want we expect and encourage:
282
283
* New functionality should have test cases. Preferably write the
284
test before writing the code.
285
286
In general, you can test at either the command-line level or the
287
internal API level. See Writing tests below for more detail.
288
289
* Try to practice Test-Driven Development: before fixing a bug, write a
290
test case so that it does not regress. Similarly for adding a new
291
feature: write a test case for a small version of the new feature before
292
starting on the code itself. Check the test fails on the old code, then
293
add the feature or fix and check it passes.
294
295
By doing these things, the Bazaar team gets increased confidence that
296
changes do what they claim to do, whether provided by the core team or
297
by community members. Equally importantly, we can be surer that changes
298
down the track do not break new features or bug fixes that you are
299
contributing today.
300
301
As of May 2008, Bazaar ships with a test suite containing over 12000 tests
302
and growing. We are proud of it and want to remain so. As community
303
members, we all benefit from it. Would you trust version control on
304
your project to a product *without* a test suite like Bazaar has?
305
306
307
Running the Test Suite
308
======================
309
310
Currently, bzr selftest is used to invoke tests.
311
You can provide a pattern argument to run a subset. For example,
312
to run just the blackbox tests, run::
313
314
./bzr selftest -v blackbox
315
316
To skip a particular test (or set of tests), use the --exclude option
317
(shorthand -x) like so::
318
319
./bzr selftest -v -x blackbox
320
321
To ensure that all tests are being run and succeeding, you can use the
322
--strict option which will fail if there are any missing features or known
323
failures, like so::
324
325
./bzr selftest --strict
326
327
To list tests without running them, use the --list-only option like so::
328
329
./bzr selftest --list-only
330
331
This option can be combined with other selftest options (like -x) and
332
filter patterns to understand their effect.
333
334
Once you understand how to create a list of tests, you can use the --load-list
335
option to run only a restricted set of tests that you kept in a file, one test
336
id by line. Keep in mind that this will never be sufficient to validate your
337
modifications, you still need to run the full test suite for that, but using it
338
can help in some cases (like running only the failed tests for some time)::
339
340
./bzr selftest -- load-list my_failing_tests
341
342
This option can also be combined with other selftest options, including
343
patterns. It has some drawbacks though, the list can become out of date pretty
344
quick when doing Test Driven Development.
345
346
To address this concern, there is another way to run a restricted set of tests:
347
the --starting-with option will run only the tests whose name starts with the
348
specified string. It will also avoid loading the other tests and as a
349
consequence starts running your tests quicker::
350
351
./bzr selftest --starting-with bzrlib.blackbox
352
353
This option can be combined with all the other selftest options including
354
--load-list. The later is rarely used but allows to run a subset of a list of
355
failing tests for example.
356
357
Test suite debug flags
358
----------------------
359
360
Similar to the global ``-Dfoo`` debug options, bzr selftest accepts
361
``-E=foo`` debug flags. These flags are:
362
363
:allow_debug: do *not* clear the global debug flags when running a test.
364
This can provide useful logging to help debug test failures when used
365
with e.g. ``bzr -Dhpss selftest -E=allow_debug``
366
367
368
Writing Tests
369
=============
370
371
In general tests should be placed in a file named test_FOO.py where
372
FOO is the logical thing under test. That file should be placed in the
373
tests subdirectory under the package being tested.
374
375
For example, tests for merge3 in bzrlib belong in bzrlib/tests/test_merge3.py.
376
See bzrlib/tests/test_sampler.py for a template test script.
377
378
Tests can be written for the UI or for individual areas of the library.
379
Choose whichever is appropriate: if adding a new command, or a new command
380
option, then you should be writing a UI test. If you are both adding UI
381
functionality and library functionality, you will want to write tests for
382
both the UI and the core behaviours. We call UI tests 'blackbox' tests
383
and they are found in ``bzrlib/tests/blackbox/*.py``.
384
385
When writing blackbox tests please honour the following conventions:
386
387
1. Place the tests for the command 'name' in
388
bzrlib/tests/blackbox/test_name.py. This makes it easy for developers
389
to locate the test script for a faulty command.
390
391
2. Use the 'self.run_bzr("name")' utility function to invoke the command
392
rather than running bzr in a subprocess or invoking the
393
cmd_object.run() method directly. This is a lot faster than
394
subprocesses and generates the same logging output as running it in a
395
subprocess (which invoking the method directly does not).
396
397
3. Only test the one command in a single test script. Use the bzrlib
398
library when setting up tests and when evaluating the side-effects of
399
the command. We do this so that the library api has continual pressure
400
on it to be as functional as the command line in a simple manner, and
401
to isolate knock-on effects throughout the blackbox test suite when a
402
command changes its name or signature. Ideally only the tests for a
403
given command are affected when a given command is changed.
404
405
4. If you have a test which does actually require running bzr in a
406
subprocess you can use ``run_bzr_subprocess``. By default the spawned
407
process will not load plugins unless ``--allow-plugins`` is supplied.
408
409
When writing library functionality, it is often necessary to set up a
410
branch with a certain history. Most current tests do this by inheriting
411
from ``TestCaseWithTransport`` and using the ``make_branch_and_tree``
412
helper to give them a ``WorkingTree`` that they can commit to. However,
413
there is a newer api available from ``TestCaseWithMemoryTransport`` using
414
the ``make_branch_builder`` helper. This helper is preferred, because it
415
can build the changes in memory, rather than on disk. Tests that are
416
explictly testing how we work with disk objects should, of course, use a
417
real ``WorkingTree``. See ``bzrlib/branch_builder.py`` for how to use the
418
class.
419
420
421
Doctests
422
--------
423
424
We make selective use of doctests__. In general they should provide
425
*examples* within the API documentation which can incidentally be tested. We
426
don't try to test every important case using doctests -- regular Python
427
tests are generally a better solution.
428
429
Most of these are in ``bzrlib/doc/api``. More additions are welcome.
430
431
__ http://docs.python.org/lib/module-doctest.html
432
433
434
Skipping tests and test requirements
435
------------------------------------
436
437
In our enhancements to unittest we allow for some addition results beyond
438
just success or failure.
439
440
If a test can't be run, it can say that it's skipped. This is typically
441
used in parameterized tests - for example if a transport doesn't support
442
setting permissions, we'll skip the tests that relating to that. ::
443
444
try:
445
return self.branch_format.initialize(repo.bzrdir)
446
except errors.UninitializableFormat:
447
raise tests.TestSkipped('Uninitializable branch format')
448
449
Raising TestSkipped is a good idea when you want to make it clear that the
450
test was not run, rather than just returning which makes it look as if it
451
was run and passed.
452
453
Several different cases are distinguished:
454
455
TestSkipped
456
Generic skip; the only type that was present up to bzr 0.18.
457
458
TestNotApplicable
459
The test doesn't apply to the parameters with which it was run.
460
This is typically used when the test is being applied to all
461
implementations of an interface, but some aspects of the interface
462
are optional and not present in particular concrete
463
implementations. (Some tests that should raise this currently
464
either silently return or raise TestSkipped.) Another option is
465
to use more precise parameterization to avoid generating the test
466
at all.
467
468
TestPlatformLimit
469
**(Not implemented yet)**
470
The test can't be run because of an inherent limitation of the
471
environment, such as not having symlinks or not supporting
472
unicode.
473
474
UnavailableFeature
475
The test can't be run because a dependency (typically a Python
476
library) is not available in the test environment. These
477
are in general things that the person running the test could fix
478
by installing the library. It's OK if some of these occur when
479
an end user runs the tests or if we're specifically testing in a
480
limited environment, but a full test should never see them.
481
482
KnownFailure
483
The test exists but is known to fail, for example because the
484
code to fix it hasn't been run yet. Raising this allows
485
you to distinguish these failures from the ones that are not
486
expected to fail. This could be conditionally raised if something
487
is broken on some platforms but not on others.
488
489
If the test would fail because of something we don't expect or
490
intend to fix, KnownFailure is not appropriate, and
491
TestNotApplicable might be better.
492
493
We plan to support three modes for running the test suite to control the
494
interpretation of these results. Strict mode is for use in situations
495
like merges to the mainline and releases where we want to make sure that
496
everything that can be tested has been tested. Lax mode is for use by
497
developers who want to temporarily tolerate some known failures. The
498
default behaviour is obtained by ``bzr selftest`` with no options, and
499
also (if possible) by running under another unittest harness.
500
501
======================= ======= ======= ========
502
result strict default lax
503
======================= ======= ======= ========
504
TestSkipped pass pass pass
505
TestNotApplicable pass pass pass
506
TestPlatformLimit pass pass pass
507
TestDependencyMissing fail pass pass
508
KnownFailure fail pass pass
509
======================= ======= ======= ========
510
511
512
Test feature dependencies
513
-------------------------
514
515
Rather than manually checking the environment in each test, a test class
516
can declare its dependence on some test features. The feature objects are
517
checked only once for each run of the whole test suite.
518
519
For historical reasons, as of May 2007 many cases that should depend on
520
features currently raise TestSkipped.)
521
522
::
523
524
class TestStrace(TestCaseWithTransport):
525
526
_test_needs_features = [StraceFeature]
527
528
This means all tests in this class need the feature. The feature itself
529
should provide a ``_probe`` method which is called once to determine if
530
it's available.
531
532
These should generally be equivalent to either TestDependencyMissing or
533
sometimes TestPlatformLimit.
534
535
536
Known failures
537
--------------
538
539
Known failures are when a test exists but we know it currently doesn't
540
work, allowing the test suite to still pass. These should be used with
541
care, we don't want a proliferation of quietly broken tests. It might be
542
appropriate to use them if you've committed a test for a bug but not the
543
fix for it, or if something works on Unix but not on Windows.
544
545
546
Testing exceptions and errors
547
-----------------------------
548
549
It's important to test handling of errors and exceptions. Because this
550
code is often not hit in ad-hoc testing it can often have hidden bugs --
551
it's particularly common to get NameError because the exception code
552
references a variable that has since been renamed.
553
554
.. TODO: Something about how to provoke errors in the right way?
555
556
In general we want to test errors at two levels:
557
558
1. A test in ``test_errors.py`` checking that when the exception object is
559
constructed with known parameters it produces an expected string form.
560
This guards against mistakes in writing the format string, or in the
561
``str`` representations of its parameters. There should be one for
562
each exception class.
563
564
2. Tests that when an api is called in a particular situation, it raises
565
an error of the expected class. You should typically use
566
``assertRaises``, which in the Bazaar test suite returns the exception
567
object to allow you to examine its parameters.
568
569
In some cases blackbox tests will also want to check error reporting. But
570
it can be difficult to provoke every error through the commandline
571
interface, so those tests are only done as needed -- eg in response to a
572
particular bug or if the error is reported in an unusual way(?) Blackbox
573
tests should mostly be testing how the command-line interface works, so
574
should only test errors if there is something particular to the cli in how
575
they're displayed or handled.
576
577
578
Testing warnings
579
----------------
580
581
The Python ``warnings`` module is used to indicate a non-fatal code
582
problem. Code that's expected to raise a warning can be tested through
583
callCatchWarnings.
584
585
The test suite can be run with ``-Werror`` to check no unexpected errors
586
occur.
587
588
However, warnings should be used with discretion. It's not an appropriate
589
way to give messages to the user, because the warning is normally shown
590
only once per source line that causes the problem. You should also think
591
about whether the warning is serious enought that it should be visible to
592
users who may not be able to fix it.
593
594
595
Interface implementation testing and test scenarios
596
---------------------------------------------------
597
598
There are several cases in Bazaar of multiple implementations of a common
599
conceptual interface. ("Conceptual" because
600
it's not necessary for all the implementations to share a base class,
601
though they often do.) Examples include transports and the working tree,
602
branch and repository classes.
603
604
In these cases we want to make sure that every implementation correctly
605
fulfils the interface requirements. For example, every Transport should
606
support the ``has()`` and ``get()`` and ``clone()`` methods. We have a
607
sub-suite of tests in ``test_transport_implementations``. (Most
608
per-implementation tests are in submodules of ``bzrlib.tests``, but not
609
the transport tests at the moment.)
610
611
These tests are repeated for each registered Transport, by generating a
612
new TestCase instance for the cross product of test methods and transport
613
implementations. As each test runs, it has ``transport_class`` and
614
``transport_server`` set to the class it should test. Most tests don't
615
access these directly, but rather use ``self.get_transport`` which returns
616
a transport of the appropriate type.
617
618
The goal is to run per-implementation only tests that relate to that
619
particular interface. Sometimes we discover a bug elsewhere that happens
620
with only one particular transport. Once it's isolated, we can consider
621
whether a test should be added for that particular implementation,
622
or for all implementations of the interface.
623
624
The multiplication of tests for different implementations is normally
625
accomplished by overriding the ``test_suite`` function used to load
626
tests from a module. This function typically loads all the tests,
627
then applies a TestProviderAdapter to them, which generates a longer
628
suite containing all the test variations.
629
630
631
Test scenarios
632
--------------
633
634
Some utilities are provided for generating variations of tests. This can
635
be used for per-implementation tests, or other cases where the same test
636
code needs to run several times on different scenarios.
637
638
The general approach is to define a class that provides test methods,
639
which depend on attributes of the test object being pre-set with the
640
values to which the test should be applied. The test suite should then
641
also provide a list of scenarios in which to run the tests.
642
643
Typically ``multiply_tests_from_modules`` should be called from the test
644
module's ``test_suite`` function.
645
646
647
271
Essential Domain Classes
648
272
########################
649
273
1042
666
be better.
1043
667
1044
668
669
Test coverage
670
=============
671
672
All code should be exercised by the test suite. See `Guide to Testing
673
Bazaar <testing.html>`_ for detailed information about writing tests.
674
1045
675
1046
676
Core Topics
1047
677
###########
1183
813
sentences.
1184
814
1185
815
1186
Writing tests
1187
=============
1188
1189
In general tests should be placed in a file named test_FOO.py where
1190
FOO is the logical thing under test. That file should be placed in the
1191
tests subdirectory under the package being tested.
1192
1193
For example, tests for merge3 in bzrlib belong in bzrlib/tests/test_merge3.py.
1194
See bzrlib/tests/test_sampler.py for a template test script.
1195
1196
Tests can be written for the UI or for individual areas of the library.
1197
Choose whichever is appropriate: if adding a new command, or a new command
1198
option, then you should be writing a UI test. If you are both adding UI
1199
functionality and library functionality, you will want to write tests for
1200
both the UI and the core behaviours. We call UI tests 'blackbox' tests
1201
and they are found in ``bzrlib/tests/blackbox/*.py``.
1202
1203
When writing blackbox tests please honour the following conventions:
1204
1205
1. Place the tests for the command 'name' in
1206
bzrlib/tests/blackbox/test_name.py. This makes it easy for developers
1207
to locate the test script for a faulty command.
1208
1209
2. Use the 'self.run_bzr("name")' utility function to invoke the command
1210
rather than running bzr in a subprocess or invoking the
1211
cmd_object.run() method directly. This is a lot faster than
1212
subprocesses and generates the same logging output as running it in a
1213
subprocess (which invoking the method directly does not).
1214
1215
3. Only test the one command in a single test script. Use the bzrlib
1216
library when setting up tests and when evaluating the side-effects of
1217
the command. We do this so that the library api has continual pressure
1218
on it to be as functional as the command line in a simple manner, and
1219
to isolate knock-on effects throughout the blackbox test suite when a
1220
command changes its name or signature. Ideally only the tests for a
1221
given command are affected when a given command is changed.
1222
1223
4. If you have a test which does actually require running bzr in a
1224
subprocess you can use ``run_bzr_subprocess``. By default the spawned
1225
process will not load plugins unless ``--allow-plugins`` is supplied.
1226
1227
1228
Test support
1229
------------
1230
1231
We have a rich collection of tools to support writing tests. Please use
1232
them in preference to ad-hoc solutions as they provide portability and
1233
performance benefits.
1234
1235
TreeBuilder
1236
~~~~~~~~~~~
1237
1238
The ``TreeBuilder`` interface allows the construction of arbitrary trees
1239
with a declarative interface. A sample session might look like::
1240
1241
tree = self.make_branch_and_tree('path')
1242
builder = TreeBuilder()
1243
builder.start_tree(tree)
1244
builder.build(['foo', "bar/", "bar/file"])
1245
tree.commit('commit the tree')
1246
builder.finish_tree()
1247
1248
Please see bzrlib.treebuilder for more details.
1249
1250
BranchBuilder
1251
~~~~~~~~~~~~~
1252
1253
The ``BranchBuilder`` interface allows the creation of test branches in a
1254
quick and easy manner. A sample session::
1255
1256
builder = BranchBuilder(self.get_transport().clone('relpath'))
1257
builder.build_commit()
1258
builder.build_commit()
1259
builder.build_commit()
1260
branch = builder.get_branch()
1261
1262
Please see bzrlib.branchbuilder for more details.
1263
1264
Doctests
1265
--------
1266
1267
We make selective use of doctests__. In general they should provide
1268
*examples* within the API documentation which can incidentally be tested. We
1269
don't try to test every important case using doctests -- regular Python
1270
tests are generally a better solution.
1271
1272
Most of these are in ``bzrlib/doc/api``. More additions are welcome.
1273
1274
__ http://docs.python.org/lib/module-doctest.html
1275
1276
1277
Running tests
1278
=============
1279
Currently, bzr selftest is used to invoke tests.
1280
You can provide a pattern argument to run a subset. For example,
1281
to run just the blackbox tests, run::
1282
1283
./bzr selftest -v blackbox
1284
1285
To skip a particular test (or set of tests), use the --exclude option
1286
(shorthand -x) like so::
1287
1288
./bzr selftest -v -x blackbox
1289
1290
To list tests without running them, use the --list-only option like so::
1291
1292
./bzr selftest --list-only
1293
1294
This option can be combined with other selftest options (like -x) and
1295
filter patterns to understand their effect.
1296
1297
1298
816
Handling Errors and Exceptions
1299
817
==============================
1300
818
Loggerhead is a web-based interface for Breezy
Version: 2.0.1