« back to all changes in this revision
Viewing changes to doc/developers/nested-trees.txt
- browse files at revision 6445.4.1
- compare with another revision
- compare with revision 5401
- download diff
- download tarball
- view history from revision 6445.4.1
- Committer: Jelmer Vernooij
- Date: 2012-01-24 13:14:06 UTC
- mto: (6445.4.5 nested-trees-spec)
- mto: This revision was merged to the branch mainline in revision 6518.
- Revision ID: jelmer@samba.org-20120124131406-wedftkorbpv37bm0
Import nested tree doc from devnotes.
- files added:
- doc/developers/nested-trees.txt
- files modified:
- doc/developers/plans.txt
added
removed
doc/developers/nested-trees.txt
1
************
2
Nested Trees
3
************
4
5
:status: 2009-12-23: Draft spec; needs consideration of foreign branches
6
7
.. sectnum::
8
9
.. contents::
10
11
Principles
12
**********
13
14
- Never store a location in versioned data.
15
16
- Implementation of nested trees shall not make operations observably slower
17
for those not using nested trees. (Using nested trees will impact
18
performance in the initial implementation, but the design will allow a
19
performant implementation.)
20
21
- A repository that holds a revision R should be able to reconstruct the
22
whole contents of that revision, including any nested trees. Corolary:
23
if I fetch that revision, even into a branch that has no working tree, it
24
should bring across any referenced revisions.
25
26
Core Concepts
27
*************
28
29
**Subtree** A tree which is inside another tree, which bzr has been asked to
30
treat as part of the outer tree.
31
32
**Subbranch** The branch associated with a subtree.
33
34
**Containing tree** A tree which has another tree inside it
35
36
**Tree reference** A directory in a containing tree which contains a subtree.
37
38
**Composite tree** A tree which behaves as a single tree, but is actually made
39
up of several trees.
40
41
42
Basic design approach
43
*********************
44
(see "Design decisions" for extended rationale)
45
By default, commands in containing trees should behave as though the subtrees
46
were plain directories. By default, commands in subtrees should not affect the
47
containing trees.
48
49
Downwards recursion
50
~~~~~~~~~~~~~~~~~~~
51
One of the objectives of nested trees is to provide ways of reproducing
52
historical combinations of different codebases. The dependency chain points
53
downwards, such that trees are affected by the revision of their subtrees, but
54
subtrees are oblivious to their containing trees. Just as bazaar doesn't
55
entice people to commit inconsistent trees, it should not entice people to
56
commit inconsistent combinations of containing tree and subtree. Therefore,
57
commit should recurse downwards by default.
58
59
Status and diff should reflect what will happen when commit is used, so they
60
should also recurse downward by default. Add almost does this already. With
61
status, diff, commit and add recursing downwards, it would be confusing to
62
users if other operations did not. Therefore, all operations should recurse
63
downwards by default.
64
65
No upwards recursion by default
66
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
67
One of the reasons for using nested trees is to gain performance by only
68
committing in a subtree. Therefore, operations should not recurse upwards by
69
default. However, some users do want to have upwards recursion, so it should
70
be provided as an option.
71
72
Modelling nested trees as a composite tree.
73
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
74
The idea that a set of nested trees behaves like a single, larger tree seems
75
relatively easy to grasp. Both for users and for developers, it provides a
76
clear expectation for the behaviour of nested trees. There are no obvious
77
drawbacks in terms of code clarity or performance. Therefore, it seems like a
78
good model to start with.
79
80
There are cases where users place multiple copies of a source tree inside
81
another tree, and treating a set of nested trees like a single large tree is in
82
conflict with that. It is essentially the same as supporting file copies, and
83
file copies themselves are a big and complicated feature to implement. It
84
seems likely that Bazaar will one day support file copies, and when it does,
85
this conflict will disappear. Implementing limited support for copies via
86
nesting would have similar complexity to implementing general support for file
87
copies, but would have fewer fruits. It would lead to complication when true
88
support for file copies is implemented. It does not seem appropriate for
89
initial nested-tree support. On the other hand, it could be implemented later;
90
none of the data model changes are in conflict with that.
91
92
Using root file-ids for tree-references
93
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
94
The idea that tree-reference file-ids are the same as the file-ids of the
95
corresponding root directories has a nice symmetry. It is one way of ensuring
96
that "bzr split" is deterministic, and "bzr join" is deterministic. When
97
performing operations involving a tree and a split version of that tree, using
98
the same file-id makes it easy to ensure that operations such as moves and
99
renames are applied appropriately to the tree-reference. Providing mechanisms
100
whereby a tree-reference can be treated as it would if it had its old file-id
101
encroaches on the territory of path tokens or file-id aliases. Having "split"
102
cause file-id changes means that in comparing these revisions, it would be seen
103
as a deleting a directory and creating a new tree-reference with the same name.
104
Handling this correctly in operations such as merge and revert would be more
105
complicated than if it were treated as a kind change, especially when
106
unversioned files are present in the subtree.
107
108
Sub-branches
109
~~~~~~~~~~~~
110
The branches associated with subtrees shall be called "subbranches".
111
112
The branch for the top tree will be in a special format, whose last_revision
113
file lists all the last_revision info for all of the branches associated with
114
the nested tree. The .bzr directories of subtrees will have a "branch" that
115
simply indicates that the top tree's branch should be used.
116
117
In the top tree's last_revision file, the revision id and revno will be
118
provided, indexed by the tree-reference file-id.
119
120
The repository used by the top-tree's branch must be a shared repository, and
121
will be used by the sub-branches.
122
123
Only the top branch will have a branch.conf. When an operation on a subbranch
124
would normally use values from branch.conf it will look them up in the top
125
branch's branch.conf and adjust for the sub-location if appropriate. e.g. "bzr
126
push" in a subtree will push just that subbranch to the corresponding subbranch
127
in the configured push location of the top branch.
128
129
Rationale
130
.........
131
If the branches were not local, the local subtrees might not be committable,
132
and commits to the remote branch would make the local subtree out of date.
133
They should not be in a separate location from the containing branch, because
134
they might share history with the tree-reference's branch. However, those
135
local branches should not be at the same location as their tree, because the
136
tree might be deleted or moved. Indeed, they should not be anywhere within a
137
working tree.
138
139
subtree branches should not be above or beside their containing branch, because
140
it could cause terrible confusion if subtrees from two different trees were
141
updating the same branches with every push, pull, commit and uncommit.
142
143
Subtree branches could be plain branches stored somewhere in the top tree's
144
branch, but then a lookup mechanism would be needed to translate from file_id
145
to location, and performance with large numbers of subbranches would be poor.
146
147
Pull and non-initial push
148
~~~~~~~~~~~~~~~~~~~~~~~~~
149
When a pull involves updates to tree references, pull will always pull into the
150
reference branch. For all new revisions in the upper branch, it will determine
151
the revision values of tree references, and fetch them into the repository.
152
153
When new tree references are encountered, pull should create a corresponding
154
subbranch in the top branch.
155
156
Pulls will update the subtrees whose tree-references change, including creating
157
trees for new sub-branches.
158
159
Implementation strategies
160
*************************
161
162
Start from the top
163
~~~~~~~~~~~~~~~~~~
164
In the initial implementation, recursion into subtrees should be implemented at
165
the highest level possible. This will make experimentation easier, reduce the
166
chance for regressions in core code, and reduce the number of public API
167
changes. This will entail some redundant post-operation determination of
168
whether subtrees are present. Speed of operations involving subtrees is not a
169
major concern, but operations that do not use subtrees must not be observably
170
slower.
171
172
Aaron Bentley believes that recursion at the top level is also easier to
173
understand and debug, but Robert Collins and Vincent Ladeuil disagree.
174
175
As time goes on and the design is proven, operations can be rewritten as more
176
intrusive changes, in order to provide better performance or better handling of
177
edge cases. It may make sense to implement them on NestedTrees, so that they
178
can take advantage of its subtree caching.
179
180
However, some low-level operations would be dubious to implement as recursive.
181
For example, a recursive Tree.id2path might open several subtrees in order to
182
determine a single path. It would be better to support a batch operation in
183
NestedTrees.
184
185
Fast subtree detection
186
~~~~~~~~~~~~~~~~~~~~~~
187
The obvious way to prevent slowness when there are no subtrees is to provide a
188
fast way of determining whether there are subtrees. This could be provided by
189
additional on-disk indices, or potentially by providing a flag indicating
190
whether there are subtrees. Such a flag could be updated by Tree.iter_entries,
191
to avoid re-running Tree.iter_entries. A parallel flag for iter_changes is
192
also conceivable, but would be more complicated, since iter_changes is relative
193
to another Tree. Further, during ``iter_changes`` it is possible to have
194
subtrees that were not encountered because they were unchanged between the
195
trees (as ``iter_changes`` does not always walk the entire tree).
196
197
Composite Locking
198
~~~~~~~~~~~~~~~~~
199
Locking a NestedTrees should lock all subtrees, to reduce the potential for new
200
failure modes. This suggests the need for an on-disk index of subtrees.
201
202
CompositeTree
203
~~~~~~~~~~~~~
204
For some operations such as diff, status and export, knowing which directories
205
are tree references is not essential. They can be satisfied by a Tree
206
implementation that papers over tree-references, making the set of nested trees
207
look like a single big tree with directories instead of tree-references. This
208
implementation strategy has a limited scope, and probably a limited shelf life,
209
but provides a fast way of getting coverage of several important commands.
210
211
Scope
212
*****
213
Ultimately, all commands should support nested trees. Initial work is focused
214
on:
215
216
- checkout
217
- branch
218
- push
219
- pull
220
- merge
221
- revert
222
- mv
223
- status
224
- diff
225
- commit
226
- export
227
228
229
Data storage
230
************
231
232
Trees
233
~~~~~
234
The root-ids of trees must be unique, so that the same file-id can be used in
235
both the containing tree and the subtree, to simplify access within trees.
236
Tree references are an inventory type that is distinct from a directory, and
237
has a revision-id associated with it.
238
All modern working trees support tree references. Indices may be provided to
239
ensure fast access to the list of subtrees.
240
241
Because of the need to use branches for recursion, it would be desirable to
242
associate RevisionTrees with branches. The alternative is for many operations
243
to accept Tree + Branch as input, and return Tree, Branch pairs.
244
245
Branches
246
~~~~~~~~
247
A new branch format, "subbranches", is introduced which provides multiple
248
sub-branches, with their data referenced by file-id. A new branch refrerence
249
format, "subbranch-reference", is introduced which refers to sub-branches in a
250
"subbranches" branch.
251
252
Repositories
253
~~~~~~~~~~~~
254
Some repository formats have 'subtree' variants, e.g. pack-0.92-subtree,
255
development-subtree. These are hidden, experimental formats that support
256
storing tree-references in their inventory formats. The most recent format
257
which supports subtrees is RepositoryFormatKnitPack3.
258
259
Repository indexing might be extended to provide fast access to
260
tree-references.
261
262
Commands
263
********
264
265
The following new commands are introduced:
266
267
``split`` Convert a single working tree into two trees, one inside the other.
268
269
``join --nested`` Cause an inner tree to be treated as a subtree. The outer
270
tree's branch must be in the new "subbranches" format. The inner tree's branch
271
will be cloned into the "subbranches" branch, and the local branch will be
272
replaced with a "subbranch-reference". Finally, a tree-reference will be
273
added to the containing tree.
274
275
API Changes
276
***********
277
278
Introduce NestedTrees
279
~~~~~~~~~~~~~~~~~~~~~
280
New class to provide services for sets of nested trees, including coordinating
281
locking, caching Tree instances, and mapping file-ids and paths to the
282
appropriate Tree and Branch. Includes::
283
284
NestedTrees.lock_read
285
286
NestedTrees.lock_write
287
288
NestedTrees.unlock
289
290
NestedTrees.paths_info(self, paths)
291
- return a list of tuples of Tree, relpaths, where relpaths is a list of
292
relative paths, and Tree is the subtree that contains them.
293
294
NestedTrees.get_tree_and_treepath
295
- return the tree and relative path within that tree of a file-id
296
297
NestedTrees.apply_inventory_delta
298
- splits an inventory delta into the affected subtrees and applies it to them
299
all
300
301
NestedTrees.iter_changes
302
- provide iter_changes over a set of nested subtrees. Emits tree-references
303
in preference to tree roots.
304
305
306
Introduce NestedTreeTransform
307
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
308
New class to permit smooth recursion of TreeTransform operations like merge and
309
revert even when the subtree boundaries differ. API essentially the same as
310
TreeTransform, except that tree-references can have children, as directories
311
do.
312
313
Introduce CompositeTree
314
~~~~~~~~~~~~~~~~~~~~~~~
315
New class to provide easy transition of superficial commands like diff, status,
316
export. API is a subset of Tree API. Additional methods::
317
318
CompositeTree.maybe_composite
319
320
- return a CompositeTree if the input tree has subtrees, otherwise return the
321
input Tree. This restricts the scope of any regressions to Trees that
322
are actually using the features.
323
324
New Tree methods
325
~~~~~~~~~~~~~~~~
326
327
get_nested_tree_and_branch(file_id, branch)
328
329
- Return a Tree and Branch for the tree-reference with the specified file_id
330
331
Tree changes
332
~~~~~~~~~~~~
333
334
WorkingTree will cache the list of tree-references, to accelerate
335
WorkingTree.iter_references. This may be an in-memory cache, or it may
336
require a new format.
337
338
WorkingTree.pull recurses downwards by default. Subtrees are pulled from the
339
reference location of the specified branch.
340
341
MutableTree.commit recurses downwards by default.
342
343
WorkingTree.revert recurses downwards by default.
344
345
Merger recurses downwards by default. This is not equivalent to running merge
346
in each of the subtrees-- it uses the tree-reference revision-id to determine
347
which revision to merge into each subtree, recursively. Merger should not
348
care whether subtree boundaries have changed. Like WorkingTree.pull, it
349
fetches from the reference location of the branch associated with the tree.
350
351
New Branch methods
352
~~~~~~~~~~~~~~~~~~
353
get_subbranch
354
355
- Return branch for tree reference
356
357
Branch changes
358
~~~~~~~~~~~~~~
359
360
pull recurses into reference branches, and pulls *from* the source's reference
361
branches.
362
363
Repository changes
364
~~~~~~~~~~~~~~~~~~
365
366
fetch provides a list of tree-reference revision ids/file-id pairs for the
367
revisions that were fetched. Fetch automatically fetches all revisions
368
associted with tree-references that were fetched.
369
370
Use Cases
371
*********
372
373
Case 1
374
~~~~~~
375
Barry works on a project with three libraries. He wants to keep up to date
376
with the tip of those libraries, but he doesn't want them to be part of his
377
source tree.
378
379
Example commands::
380
381
Set up the tree:
382
$ bzr branch --nested http://library1 project
383
$ bzr branch --nested http://library2 project
384
$ bzr branch --nested http://library3 project
385
$ bzr commit project -m "Added three libraries"
386
387
Update a library to tip:
388
$ bzr pull -d project/library1 http://library1
389
390
Case 2
391
~~~~~~
392
Now, Barry wants to add a fourth library.
393
394
Example commands::
395
396
$ bzr branch --nested http://library4 project
397
398
Case 3
399
~~~~~~
400
Barry wants to publish his project.
401
402
Example commands::
403
404
$ bzr push -d project bzr+ssh://project/trunk
405
406
Case 4
407
~~~~~~
408
Barry decides to make part of his project into another library
409
410
Example commands::
411
412
$ bzr split --nested project/newlibrary
413
414
Case 5
415
~~~~~~
416
Curtis wants to hack on Barry's project
417
418
Example commands::
419
420
$ bzr branch http://project/trunk
421
422
Case 6
423
~~~~~~
424
Barry wants to drop one of the libraries he was using
425
426
Example commands::
427
428
$ rm project/library1
429
$ bzr commit project -m "Removed library1"
430
431
Case 7
432
~~~~~~
433
Curtis has made changes to one of the libraries. Barry wants to merge Curtis'
434
changes into his copy.
435
436
Example commands::
437
438
$ bzr merge -d project http://curtis.org/trunk#library2
439
440
Or alternatively:
441
$ bzr merge -d project/library2 http://curtis.org/trunk#library2
442
443
Case 8
444
~~~~~~
445
Curtis has made changes to Barry's main project. Barry wants to merge Curtis'
446
changes into his copy.
447
448
Example commands::
449
450
$ bzr merge -d project http://curtis.org/trunk
451
452
453
Case 9
454
~~~~~~
455
Barry makes changes in his project and in a library, and he runs status
456
457
Example commands::
458
459
$ echo bar > project/foo
460
$ echo qux > project/library2/baz
461
$ bzr status project
462
M foo
463
M library2/baz
464
465
Case 10
466
~~~~~~~
467
Barry wants to upgrade the bazaar format of his project
468
469
Example commands::
470
471
$ bzr upgrade project
472
473
Case 11
474
~~~~~~~
475
Curtis wants to apply Barry's latest changes.
476
477
Example commands::
478
479
$ bzr merge -d project http://project/trunk
480
481
Case 12
482
~~~~~~~
483
Danilo wants to start a project with two libraries using nested trees from
484
scratch.
485
486
Example commands::
487
488
$ bzr init project
489
$ bzr branch --nested http://library4 project
490
$ bzr branch --nested http://library5 project
491
$ bzr commit project -m "Created new project."
492
493
Case 13
494
~~~~~~~
495
Edwin has a project that doesn't use nested trees and he wants to start using
496
nested trees.
497
498
Example commands::
499
$ bzr split --nested project/subdir
500
501
Case 14
502
~~~~~~~
503
Françis has a project with nested trees where the containing tree uses one
504
Bazaar format and the subtree uses a different Bazaar format.
505
506
Not supported.
507
508
Case 15
509
~~~~~~~
510
Barry commits some changes to a library and to the main project, and then
511
discovers the changes are not appropriate. He has not yet pushed his changes
512
anywhere.
513
514
Example commands::
515
516
$ bzr merge -d project http://library2
517
$ bzr commit project -m "Updated library2"
518
$ bzr uncommit project --force
519
520
Case 16
521
~~~~~~~
522
Barry commits some changes to a library and to the main project, publishes his
523
branch, and then discovers the changes are not appropriate.
524
525
Example commands::
526
527
$ bzr merge -d project http://library2
528
$ bzr commit project -m "Updated library2"
529
$ bzr push -d project
530
$ bzr revert project
531
$ bzr commit project -m "Reverted inappropriate changes."
532
$ bzr push -d project
533
534
Case 17
535
~~~~~~~
536
Gary is writing a project. Henninge wants to split a library out of it.
537
538
Example commands::
539
540
$ bzr branch project
541
$ bzr split project/library6
542
$ mv project/library6 .
543
$ rm project
544
$ bzr commit -m "split library6 into its own library."
545
546
Case 18
547
~~~~~~~
548
Henning wants to update to receive Gary's latest changes.
549
550
Example commands::
551
552
$ bzr merge -d library6
553
554
Case 19
555
~~~~~~~
556
Gary wants to update to receive Henninge's changes, including splitting a
557
library out.
558
559
Example commands::
560
561
$ bzr split --nested project/library6
562
$ bzr commit project -m "Turned library6 into a library"
563
$ bzr merge -d project/library6 http://library6
564
$ bzr commit project -m "Merge Henninge's changes."
565
566
567
Case 20
568
~~~~~~~
569
Gary wants to update to receive Henninge's changes, without splitting a library
570
out.
571
572
$ bzr split --nested project/library6
573
$ bzr commit project -m "Turned library6 into a library"
574
# i.e. a cherrypick that skips the revision where library6 became a library.
575
$ bzr merge -d project/library6 http://library6 -r 5..-1
576
$ bzr commit project -m "Merge Henninge's changes."
577
578
Case 21
579
~~~~~~~
580
581
John works on a project called FooBar, but has decided that it would be better
582
structured as two projects, where Bar is a library that may be of general use
583
outside of Foo. As it happens, bar already has its own subdirectory.
584
585
He runs:
586
::
587
588
# Convert into two trees: foobar and foobar/bar.
589
# In each tree, files will be removed and deleted. In foobar/bar, "bar"
590
# will have been moved to become the tree root.
591
# These changes will be committed later.
592
$ bzr upgrade foobar --format=subbranches
593
$ bzr split foobar/bar
594
595
# Add a tree-reference from foobar to foobar/bar, change bar's branch
596
# to a reference to subbranch in foobar's branch.
597
$ bzr join --nested foobar/bar
598
599
# This recurses into foobar/bar and commits the deletion of the containing
600
# tree. In foobar, it commits a kind change for 'bar' from directory to
601
# tree-reference, and the deletion of the contents of bar.
602
$ bzr commit foobar
603
604
This commits new revisions to foobar and bar, and foobar's tree-reference bar
605
refers to the revision-id of bar.
606
607
Next, he adds two new files: foobar/baz and foobar/bar/qux::
608
609
$ vi foobar/baz
610
$ vi foobar/bar/qux
611
# This adds qux to foobar/bar and adds baz to foobar.
612
$ bzr add foobar
613
614
Since foobar/bar/qux is in a commitable state and foobar/baz is not, he invokes
615
::
616
617
$ bzr commit foobar/bar
618
619
This commits foobar/bar/baz/qux to the subtree and commits foobar/bar to the
620
containing tree.
621
622
(Had he wanted to commit to just the subtree, or just the containing tree, he
623
could have specified an option.)
624
625
626
Case 21
627
~~~~~~~
628
629
Robert wants to hack on a project, Baz, that is structured as a nested tree,
630
which uses the library "quxlib", from quxlib.org.
631
632
He runs:
633
::
634
635
$ bzr branch http://baz.org/dev baz
636
637
This creates a "subbranches" branch and working tree for baz, as normal. Since
638
tree-references were encountered, it adds subbranches for them to the baz
639
branch. All data is retrieved from baz.org, not quxlib.org.
640
641
It creates a working tree for quxlib with a subbranch-reference. It uses the
642
revision-id from the tree-reference in the containing tree, not the head
643
revision at baz.org. This allows Robert to get a known-good nested tree.
644
645
Later, Robert decides to update the version of quxlib being used to the latest
646
from quxlib.org. He runs::
647
648
$ bzr pull -d http://quxlib.org
649
650
This updates the version of quxlib in the working tree, which mean that baz is
651
now out-of-date with its last-committed tree. Unfortunately, the new rev on
652
quxlib is not completely compatible with the old one, and Robert must tweak a
653
few files before Baz runs properly. Once he has done so, he runs::
654
655
$ bzr commit baz
656
657
Now he has committed a known-good nested tree, and the baz working tree is once
658
again up-to-date.
659
660
661
User documentation
662
******************
663
664
For many large projects, it is often useful to incorporate libraries
665
maintained elsewhere or to construct them from multiple subprojects.
666
While it is easy for a single user to set up a particular layout of
667
multiple branches by hand, the different branches really need to be
668
linked together if others are to reproduce the desired layout, and
669
if the relationships are going to be managed over time.
670
671
Bazaar has good support for building and managing external libraries
672
and subprojects via a feature known as *nested trees*. In particular,
673
nearly all of Bazaar's commonly used commands understand nested trees
674
and Do The Right Thing as explained below. The relationship is hierarchical:
675
the containing tree knows about its nested trees, but nested trees are unaware
676
of the tree (or trees) containing them.
677
678
At the moment, *nested trees* are the only type of nested item
679
supported though *nested files* may be supported in the future.
680
Nested trees may contain other nested trees as required.
681
682
Note: This feature requires a recent branch format such as ``2.0``
683
or later.
684
685
686
Nesting an external project
687
~~~~~~~~~~~~~~~~~~~~~~~~~~~
688
689
To link an external project into a branch, use the ``branch`` command
690
with the ``--nested`` option like this::
691
692
bzr branch --nested SOURCE-URL TARGET-DIR
693
694
For example, assuming you already have a ``src/lib`` directory where
695
libraries are kept::
696
697
bzr branch --nested http://example.com/xmlsaxlib src/lib/sax
698
699
This will create a nested branch in the ``src/lib/sax`` directory,
700
join it into the containing branch and save the source location.
701
702
If you now run ``bzr status``, it will show the nested branch as
703
uncommitted changes like this::
704
705
+ src/lib/sax
706
+ src/lib/sax/README
707
+ src/lib/sax/parser.py
708
...
709
710
To record this change, use the ``commit`` command as you normally would::
711
712
bzr commit -m "added SAX parsing library"
713
714
Note that Bazaar stores the tip revision of each nested branch. This
715
is an important feature in that it's then easy to reproduce the exact
716
combination of libraries used for historical revisions. It also means
717
that other developers pulling or merging your changes will get nested
718
branches created for them at the right revisions of each.
719
720
721
Refreshing a nested branch
722
~~~~~~~~~~~~~~~~~~~~~~~~~~
723
724
As bugs are fixed and enhancements are made to nested projects, you
725
will want to update the version being used. To do this, ``pull`` the
726
latest version of the nested branch. For example::
727
728
bzr pull -d src/lib/sax
729
730
If the latest revision is too unstable, you can always use the ``-r``
731
option on the ``pull`` command to nominate a particular revision or tag.
732
733
Now that you have the required version of the code, you can make
734
any required adjustments (e.g. API changes), run your automated tests
735
and commit something like this::
736
737
view src/lib/sax/README
738
(hack, hack, hack)
739
make test
740
bzr commit -m "upgraded SAX library to version 2.1.3"
741
742
743
Changing a nested tree
744
~~~~~~~~~~~~~~~~~~~~~~
745
746
As well as keeping track of which revisions of external libraries
747
are used over time, one of the reasons for nesting projects is to
748
make minor changes. You may want to do this in order to fix and
749
track particular bugs you need addressed. In other cases, you may want
750
to make various local enhancements that aren't valuable outside
751
the context of your project.
752
753
As support for nested branches is integrated into most commonly
754
used commands, this is actually quite easy to do: simply make
755
the change to the required files as you normally would! For example::
756
757
edit src/lib/sax/parser.py
758
bzr commit -m "fix bug #42 in sax parser"
759
760
Note that Bazaar is smart enough to recurse by default into nested
761
branches, commit changes there, and commit the new nested branch tips
762
in the current branch. Both commits get the same commit message.
763
764
If you want to only commit the change to a nested branch for now, you
765
can change into the nested branch before running commit like this::
766
767
cd src/lib/sax
768
bzr commit -m "fix bug #42 in sax parser"
769
770
Alternatively, you can use a selective commit like this::
771
772
bzr commit -m "fix bug #42 in sax parser" src/lib/sax
773
774
775
Reviewing nested tree changes
776
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
777
778
Just like ``commit``, the ``status`` and ``diff`` commands implicitly
779
recurse into nested trees. In the case of ``status``, it shows both the
780
nested tree as having a pending change as well as the items within it that have
781
changed. For example::
782
783
M src/lib/sax
784
M src/lib/sax/parser.py
785
786
Once again, if you change into a nested tree though, ``status`` and
787
``diff`` will operate just on that tree and not recurse upwards by
788
default.
789
790
791
Browsing nested tree history
792
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
793
794
As the branches of nested trees have their own history, the ``log`` command
795
shows just the history of the containing branch. To see the history for
796
a nested branch, nominate the branch explicitly like this::
797
798
bzr log src/lib/sax
799
800
Note however that ``log -v`` and ``log -p`` on the containing branch
801
will show what files in nested branches were changed in each revision.
802
803
804
Splitting out a project
805
~~~~~~~~~~~~~~~~~~~~~~~
806
807
If you already have a large project and wish to partition it into
808
reusable subprojects, use the ``split`` command. This takes an existing
809
directory and makes it a separate branch. For example, imagine you have
810
a directory holding UI widgets that another project would like to
811
leverage. You can make it a separate branch like this::
812
813
bzr split src/uiwidgets
814
815
To make the new project available to others, push it to a shared location
816
like this::
817
818
cd src/uiwidgets
819
bzr push bzr://example.com/uiwidgets
820
821
You also need to link it back into the original project as a nested branch
822
using the ``join`` command like this (assuming the current directory is
823
``src/uiwidgets``)::
824
825
bzr join --nested .
826
bzr commit -m "uiwidgets is now a nested project"
827
828
Similar to ``branch --nested``, ``join --nested`` joins the nominated directory
829
(which must hold a branch) into the containing tree. In order to make sure
830
that all versions of a tree can be reproduced, the branches of nested trees
831
share a repository with their containing tree.
832
833
834
Virtual projects
835
~~~~~~~~~~~~~~~~
836
837
By design, Bazaar is strict about tracking the actual revisions used of
838
nested branches over time. Without this, projects cannot accurately
839
reproduce exactly what was used to make a given build. There are
840
isolated use cases though where is advantageous to say "give me the
841
latest tip of these loosely coupled branches". To do this, create a
842
small 'virtual project' which is just a bunch of *unpegged* nested
843
branches. To mark nested branches as unpegged, use the ``--no-pegged``
844
option of the ``join`` command like this::
845
846
bzr join --nested --no-pegged [DIR]
847
848
To stop the nested branch tips from floating and to begin recording
849
the tip revisions again, use the ``pegged`` option::
850
851
bzr join --nested --pegged [DIR]
852
853
After changing whether one or more nested branches are pegged or not, you
854
need to ``commit`` the branch to record that metadata. (The pegged state
855
is recorded over time.)
856
857
For example, you may be managing a company intranet site as a project
858
which is nothing more than a list of unrelated departmental websites
859
bundled together. You can set this up like this::
860
861
bzr init intranet-site
862
cd intranet-site
863
bzr branch bzr://ourserver/websites/research
864
bzr branch bzr://ourserver/websites/development
865
bzr branch bzr://ourserver/websites/support
866
bzr branch bzr://ourserver/websites/hr
867
bzr join --nested --no-pegged research
868
bzr join --nested --no-pegged development
869
bzr join --nested --no-pegged support
870
bzr join --nested --no-pegged hr
871
bzr commit -m "initial configuration of intranet-site"
872
873
Publishing the overall site is then as easy as going to the server
874
hosting your intranet and running something like::
875
876
bzr branch http://mymachine//projects/intranet-site
877
878
Refreshing the overall site is as easy as::
879
880
bzr pull
881
882
Virtual projects are also useful for providing a partial 'view' over
883
a large project containing a large number of subprojects. For example,
884
you may be working on an office suite and have a bunch of developers
885
that only care about the word processor. You can create a virtual
886
project for them like this::
887
888
bzr init wp-modules
889
cd wp-modules
890
bzr branch ../common
891
bzr branch ../printing
892
bzr branch ../spellchecker
893
bzr branch ../wordprocessor
894
bzr join --nested --no-pegged common
895
bzr join --nested --no-pegged printing
896
bzr join --nested --no-pegged spellchecker
897
bzr join --nested --no-pegged wordprocessor
898
bzr commit -m "initial configuration of wp-modules"
899
900
Those developers can then get bootstrapped faster and have *just* the
901
subprojects they care about by branching from ``wp-modules``.
902
903
904
Nested branch tips & tricks
905
~~~~~~~~~~~~~~~~~~~~~~~~~~~
906
907
As explained above, most of Bazaar's commonly used commands recurse
908
downwards into nested branches by default. To prevent this recursion,
909
use the ``--no-recurse-nested`` option on various commands (including
910
``commit``, ``status`` and ``diff``) that support it.
911
912
Thanks to plugins like bzr-svn and bzr-git, Bazaar has strong support
913
for transparently accessing branches managed by foreign VCS tools. This
914
means that Bazaar can support projects where nested branches are hosted
915
in supported foreign systems. For example, to nest a library maintained
916
in Subversion::
917
918
bzr branch --nested svn://example.com/xmlhelpers src/lib/xmlhelpers
919
920
If you want revisions to be committed both to a remote location and a
921
local location, make the top-level branch a bound branch. (Nested branches
922
have no configuration of their own.)
923
924
Most likely, you will have some branches that are identical to their upstream
925
version and can be pulled, and some that have local changes and must be merged.
926
You can update all of them at once using ``merge --pull``. This will pull
927
into the trees with no local changes, and merge into the ones with local
928
changes. Afterward, you should commit, which will commit only into the
929
trees that were merged into.
930
931
As you'd expect, a nested branch can be moved or deleted using the
932
normal commands. For example, after splitting out a subproject, you
933
may want to change its location like this::
934
935
bzr mv src/uiwidgets src/lib/uiwidgets
936
bzr commit -m "move uiwidgets into src/lib"
937
938
Things to be aware of
939
~~~~~~~~~~~~~~~~~~~~~
940
941
Commands like ``commit`` and ``push`` need online access to the locations
942
for nested branches which have updated their tip. In particular, ``commit``
943
will update any changed nested branches first and only commit to the
944
containing branch if all nested branch commits succeed. If you are working
945
offline, you may want to ensure you have a local mirror location defined
946
for nested branches you are likely to tweak. Alternatively, the
947
``no-recurse-nested`` option to the ``commit`` command might to useful to
948
commit some changes, leaving the nested branch commits until you are back
949
online.
950
951
A given top level tree cannot contain multiple copies of a nested tree.
952
As a consequence, you cannot nest two projects if they both nest
953
the same project somewhere within them. This limitation may be removed
954
in the future. In the meantime, consider restructuring things so that each
955
project is only nested once (and leverage symbolic links as appropriate).
956
In most programming environments, having different parts of the project
957
using different versions of a library is an integration no-no anyhow,
958
so enforcing *one* common revision is the right way to prevent this from
959
happening.
960
961
At the moment, nested trees need to be incorporated as a whole.
962
Filtered views can to used to restrict the set of files and directories
963
logically seen. Currently though, filtered views are a lens onto a tree:
964
they do not delete other files and the exposed files/directories must
965
have the same paths as they do in the original branch. In the future,
966
we may add support for nesting and moving selected files from a
967
(read-only) nested branch something like this::
968
969
bzr nested DIR --file LICENSE --file doc/README::README
970
bzr commit -m "change which files are nested from project DIR"
971
972
If you require this feature, please contact us with your needs.
973
974
Design decisions
975
****************
976
977
The branches of subtrees shall share a repository with the containing tree.
978
979
The branches of subtrees shall be in a special format that shares a single
980
last_revision file that is stored in the containing branch.
981
982
The subtree branches shall be referenced in the last_revision file by file-id.
983
984
Subtree branches shall not support individual configuration.
985
986
Fetch shall automatically fetch the revisions mentioned by tree-references,
987
recursively.
988
989
The reserved revision-id "head:" shall be used in tree-references to refer to
990
the tip revision of a branch.
991
992
bzr-svn repositories with externals shall behave as though the multiple
993
repositories were a single Bazaar repository with multiple branches.
994
995
Shall commands recurse downwards by default?
996
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
997
998
Yes.
999
1000
Pros:
1001
1002
- It is hard to accidentally produce inconsistent trees
1003
- Inconsistent trees are hard for remote users to handle
1004
- Accidentally committing too many things at once is easy to resolve
1005
- It is hard to accidentally commit too many things at once
1006
1007
Cons:
1008
1009
- Accidentally committing nuclear launch codes is easier to do
1010
- More risk of exposing users to bugs
1011
- Makes incompleteness more glaring
1012
- A commit message that makes sense for the top may not make sense lower down.
1013
1014
Shall commands recurse upwards by default?
1015
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1016
1017
No.
1018
1019
1020
Shall subtree branches be addressable?
1021
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1022
1023
Yes. The problems to be solved include:
1024
1025
- the "name" of subbranches can change from revision to revision. In fact,
1026
subbranches may have no name in a given revision.
1027
- a user may want to get revision X of a subbranch named "foo" in revision Y
1028
of the top branch.
1029
1030
The syntax will be ``BRANCH#SUBBRANCH``, e.g. ``lp:bigproject#subproject``.
1031
``BRANCH`` is a regular URL and ``SUBBRANCH`` is a path within BRANCH's tree
1032
identifying a subbranch. Simultaneously addressing a specific revision in both
1033
BRANCH and SUBBRANCH is not currently defined.
1034
1035
1036
Shall we model nested trees as a composite tree?
1037
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1038
1039
Yes. Users will see recurse-downwards behaviour that allows operations that
1040
cross subtree boundaries, e.g. a merge in the top tree can move a file between
1041
subtrees.
1042
1043
The downside is that we can't have cheap support for subtrees that are copies
1044
of one another, because we wouldn't know which copy to apply sets of changes
1045
to.
1046
1047
1048
Shall we use root-ids for tree references?
1049
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1050
1051
Yes. This fits well with our current lack of support of file copies. If we do
1052
support file copies in future it will be possible to change this in a future
1053
format, and perform deterministic upgrades to that format.
1054
1055
1056
Shall we recurse our low-level operations from the beginning?
1057
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1058
1059
No. It will be necessary eventually for good performance, but the initial
1060
implementation is conservative to minimise risk to the rest of bzr.
1061
1062
Shall we lock recursively?
1063
~~~~~~~~~~~~~~~~~~~~~~~~~~
1064
1065
Yes. It matches existing behaviour by failing earlier, and the extra cost does
1066
not seem onerous. (To be fully efficient this requires an index of the
1067
subtrees, otherwise we need to scan the fully inventory/dirstate.)
1068
1069
(Also, this decision can be changed later with no compatibility concerns.)
1070
1071
1072
How do we handle merge when the subtree hasn't diverged?
1073
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1074
1075
"bzr merge --pull" will be changed so that it will merge (not pull) when the
1076
local last revision's revno would change (i.e. is a non-lhs parent in the merge
1077
source). This is expected to be the most common way to update nested trees.
1078
1079
The existing "bzr merge --pull" behaviour will be renamed to "bzr merge
1080
--pull-renumber".
1081
1082
"bzr merge" (with no "--pull") will do a merge in all trees. "bzr pull" will
1083
do a pull in all trees.
1084
1085
The rationale is that a very common use-case is that the top tree is a project
1086
the user is actively committing to, and the subtrees are mainly libraries that
1087
are being mirrored. So a behaviour that forced every update to be a merge
1088
would be undesirable for the mirrored subtrees, but an update that is a pull
1089
wouldn't suit the changing top tree. And the existing "merge --pull" (that can
1090
renumber revisions) isn't desireable for either the top tree or subtrees in
1091
this case.
1092
1093
1094
What should uncommit do?
1095
~~~~~~~~~~~~~~~~~~~~~~~~
1096
1097
It will recurse, and subtrees will be uncommitted back to the revision recorded
1098
by the revision the top tree is uncommitting to.
1099
1100
This means that operations like::
1101
1102
$ echo foo > versioned-file-in-top-tree.txt
1103
$ bzr ci -m "Change file"
1104
$ bzr uncommit
1105
1106
will not cause a change in subtrees, since the top-level commit did not affect
1107
them. But on the other hand:
1108
1109
$ echo foo > subtree/versioned-file-in-subtree.txt
1110
$ bzr ci -m "Change file"
1111
$ bzr uncommit
1112
1113
will first commit to the subtree, then to the top tree. The uncommit will
1114
restore both trees to their previous state.
1115
1116
1117
Shall we use a CompositeTree object as a shim to make existing commands work?
1118
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1119
1120
Yes. The affected commands are diff, status, and export. These are all
1121
read-only commands so there's no danger of harming user data. The only risks
1122
are giving inaccurate results and poor performance.
1123
1124
1125
Some subtrees should have commits and some should not. How?
1126
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1127
1128
We will not provide special support for this initially. We might later support
1129
flagging some sub-trees as mirror-only something similar, but this seems like
1130
it could be a general feture not specific to nesting. (and it may only require
1131
a working tree format bump to add).
1132
1133
How do we handle files that are moved into subtrees (from another tree in the same top tree)?
1134
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1135
1136
Add a revision property:
1137
"Files-from-other-trees: (rev-id: [f-id, ...]), ..."
1138
1139
(Once we can record cherry-pick merges, we might use those instead, but it
1140
depends on their history-fetching requirements.)
1141
1142
1143
1144
1145
Comparison with other systems
1146
*****************************
1147
1148
Git submodules
1149
~~~~~~~~~~~~~~
1150
1151
This allows separate repositories to be used for submodules. At the UI layer,
1152
they are quite different, because submodules does not attempt to integrate
1153
subtree support into the core commands.
1154
1155
Mercurial Forests
1156
~~~~~~~~~~~~~~~~~
1157
1158
The wiki page does not give confidence that this is a well-maintained project.
1159
It seems similar to config-manager-- its 'snapshot' files are like
1160
config-manager's config files, describing what branches to get and where to put
1161
them, optionally specifying a revision. No metadata about nesting is stored in
1162
the tree. Optionally, a 'snapshot.txt' file may be stored in the containing
1163
tree, but it can also be stored somewhere else.
1164
1165
There is no attempt to integrate subtree support into the core commands.
1166
1167
Mercurial Nested Repositories
1168
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1169
This design was for an integrated feature, but there are apparently 4
1170
implementations as extensions. While it does integrate subtree support into
1171
core commands, this may be off by default: "The alternative that I lean towards
1172
is to not recurse unless explicitly instructed to. Most probably, only a few
1173
commands should arguably even be aware of modules."
1174
1175
Command comparison:
1176
1177
- hg module add ~= bzr join --nested
1178
- hg module remove ~= bzr remove --keep
1179
- hg module record's functionality is part of nested commits in nested trees.
1180
1181
Like submodules, this stores location information in versioned data: a
1182
.hgmodules directory.
1183
Like submodules and nested trees, particular revisions are recorded.
1184
1185
Subversion "svn:externals"
1186
~~~~~~~~~~~~~~~~~~~~~~~~~~
1187
Like bzr, uses per-file metadata. Like submodules and nested repositories,
1188
locations are versioned data. Like Forests, revisions are optional. Like
1189
nested repositories, there is limited integration into core commands; checkout
1190
and update support externals, and commit may support them in the future.
1191
However, there is no UI specific to creating and updating svn:externals
1192
references.
1193
1194
Unlike all other alternatives, supports partial checkouts. This is because svn
1195
natively supports partial checkouts. Also, supports checkouts of tags, because
1196
tags are merely a convention in svn.
1197
1198
Supports pulling in "head" subtrees too, not just specific ("known-good")
1199
revisions. Nested trees supports this in order to provide high-fidelity
1200
imports.
1201
1202
Some support for single-file svn:externals (see
1203
http://subversion.tigris.org/svn_1.6_releasenotes.html#externals), whereas bzr
1204
subtrees must be directories.
1205
1206
Has a --ignore-externals option for checking out without pulling in the
1207
svn:externals items (svn checkout is more or less equivalent to bzr branch).
1208
You can also use that option on update (more or less like bzr merge).
1209
1210
1211
Comments on differences
1212
~~~~~~~~~~~~~~~~~~~~~~~
1213
externals support partial checkouts. Nested trees could gain support for this
1214
once Bazaar itself supports partial checkouts. Supporting a single file as a
1215
"subtree" would also depend on native bzr support. On platforms that support
1216
symlinks, using symlinks to portions of a subtree can be an effective
1217
substitute.
1218
1219
1220
Future work
1221
***********
1222
It would be nice to support multiple copies of subtree in a master tree.
1223
Currently, this would lead to having multiple copies of a given file-id in the
1224
(composite) tree, which is forbidden. Allowing this would involve one of:
1225
1226
- permitting multiple copies of a file-id within a tree
1227
- not treating nested trees as composite trees
1228
- using fake file-ids within the composite tree
1229
- replacing the file-id concept with something else (e.g. path tokens)
1230
1231
1232
.. vim: ft=rst
Loggerhead is a web-based interface for Breezy
Version: 2.0.1