« back to all changes in this revision
Viewing changes to doc/developers/commit.txt
- browse files at revision 2541.2.2
- compare with another revision
- compare with revision 692
- download diff
- download tarball
- view history from revision 2541.2.2
- Committer: Robert Collins
- Date: 2007-06-28 02:43:50 UTC
- mfrom: (2553 +trunk)
- mto: This revision was merged to the branch mainline in revision 2558.
- Revision ID: robertc@robertcollins.net-20070628024350-z8bdm0y6yz2uyf4o
Merge bzr.dev.
- files added:
- bzrlib/smtp_connection.py
- files renamed:
- doc/developers/performance-commit.txt => doc/developers/commit.txt
- files modified:
- NEWS
added
removed
doc/developers/commit.txt
3
3
4
4
.. contents:: :local:
5
5
6
Changes to commit
7
-----------------
8
9
We want to improve the commit code in two phases.
10
11
Phase one is to have a better separation from the format-specific logic,
12
the user interface, and the general process of committing.
13
14
Phase two is to have better interfaces by which a good workingtree format
15
can efficiently pass data to a good storage format. If we get phase one
16
right, it will be relatively easy and non-disruptive to bring this in.
17
18
6
19
Commit: The Minimum Work Required
7
20
---------------------------------
8
21
9
This is Martin Pool's email to the mailing list on the minimum work that
10
commit needs to do. Be sure to check the mailing list archives
11
(https://lists.ubuntu.com/archives/bazaar/2007q2/025791.html)
12
for follow-up comments from Robert and others ...
13
14
22
Here is a description of the minimum work that commit must do. We
15
23
want to make sure that our design doesn't cost too much more than this
16
24
minimum. I am trying to do this without making too many assumptions
57
65
the current scheme we get the file id, last-revision from the
58
66
dirstate, look into the knit for that text, extract that text in
59
67
total, generate a delta, then store that into the knit. Most delta
60
operations are O(n2) to O(n3) in the size of the modified files.
68
operations are O(n**2) to O(n**3) in the size of the modified files.
61
69
62
70
1.4 - Cache annotation information for the changes: at the moment this
63
71
is done as part of the delta storage. There are some flaws in that
260
268
sooner you can stop evaluating the 99% that you don't care about, the
261
269
less work you do.
262
270
271
272
Avoiding work: avoiding reading parent data
273
-------------------------------------------
274
275
We would like to avoid the work of reading any data about the parent
276
revisions. We should at least try to avoid reading anything from the
277
repository; we can also consider whether it is possible or useful to hold
278
less parent information in the working tree.
279
280
When a commit of selected files is requested, the committed snapshot is a
281
composite of some directories from the parent revision and some from the
282
working tree. In this case it is logically necessary to have the parent
283
inventory information.
284
285
If file last-change information or per-file graph information is stored
286
then it must be available from the parent trees.
287
288
If the Branch's storage method does delta compression at commit time it
289
may need to retrieve file or inventory texts from the repository.
290
291
It is desirable to avoid roundtrips to the Repository during commit,
292
particularly because it may be remote. If the WorkingTree can determine
293
by itself that a text was in the parent and therefore should be in the
294
Repository that avoids one roundtrip per file.
295
296
There is a possibility here that the parent revision is not stored, or not
297
correctly stored, in the repository the tree is being committed into, and
298
so the committed tree would not be reconstructable. We could check that
299
the parent revision is present in the inventory and rely on the invariant
300
that if a revision is present, everything to reconstruct it will be
301
present too.
302
303
304
Code structure
305
--------------
306
307
Caller starts a commit
308
309
>>> Branch.commit(from_tree, options)
310
311
This creates a CommitBuilder object matched to the Branch, Repository and
312
Tree. It can vary depending on model differences or by knowledge of what
313
is efficient with the Repository and Tree. Model differences might
314
include whether no-text-change merges need to be reported, and whether the
315
316
The basic CommitBuilder.commit structure can be
317
318
1. Ask the branch if it is ready to commit (up to date with master if
319
any.)
320
321
2. Ask the tree if it is ready to commit to the branch (up to date with
322
branch?), no conflicts, etc
323
324
3. Commit changed files; prototype implementation:
325
326
a. Ask the working tree for all committable files; for each it should
327
return the per-file parents, stat information, kind, etc.
328
329
b. Ask the repository to store the new file text; the repository should
330
return the stored sha1 and new revision id.
331
332
4. Commit changed inventory
333
334
5. Commit revision object
335
336
337
338
339
340
341
342
343
344
Complications of commit
345
-----------------------
346
347
Bazaar (as of 0.17) does not support selective-file commit of a merge;
348
this could be done if we decide how it should be recorded - is this to be
349
stored as an overall merge revision; as a preliminary non-merge revisions;
350
or will the per-file graph diverge from the revision graph.
351
352
There are several checks that may cause the commit to be refused, which
353
may be activated or deactivated by options.
354
355
* presence of conflicts in the tree
356
357
* presence of unknown files
358
359
* the working tree basis is up to date with the branch tip
360
361
* the local branch is up to date with the master branch, if there
362
is one and --local is not specified
363
364
* an empty commit message is given,
365
366
* a hook flags an error
367
368
* a "pointless" commit, with no inventory changes
369
370
Most of these require walking the tree and can be easily done while
371
recording the tree shape. This does require that it be possible to abort
372
the commit after the tree changes have been recorded. It could be ok to
373
either leave the unreachable partly-committed records in the repository,
374
or to roll back.
375
376
Other complications:
377
378
* when automatically adding new files or deleting missing files during
379
commit, they must be noted during commit and written into the working
380
tree at some point
381
382
* refuse "pointless" commits with no file changes - should be easy by
383
just refusing to do the final step of storing a new overall inventory
384
and revision object
385
386
* heuristic detection of renames between add and delete (out of scope for
387
this change)
388
389
* pushing changes to a master branch if any
390
391
* running hooks, pre and post commit
392
393
* prompting for a commit message if necessary, including a list of the
394
changes that have already been observed
395
396
* if there are tree references and recursing into them is enabled, then
397
do so
398
399
Commit needs to protect against duplicated file ids
400
401
402
Updates that need to be made in the working tree, either on conclusion
403
of commit or during the scan, include
404
405
* Changes made to the tree shape, including automatic adds, renames or
406
deletes
407
408
* For trees (eg dirstate) that cache parent inventories, the old parent
409
information must be removed and the new one inserted
410
411
* The tree hashcache information should be updated to reflect the stat
412
value at which the file was the same as the committed version, and the
413
content hash it was observed to have. This needs to be done carefully to
414
prevent inconsistencies if the file is modified during or shortly after
415
the commit. Perhaps it would work to read the mtime of the file before we
416
read its text to commit.
417
418
419
Interface stack
420
---------------
421
422
The commit api is invoked by the command interface, and copies information
423
from the tree into the branch and its repository, possibly updating the
424
WorkingTree afterwards.
425
426
The command interface passes:
427
428
* a commit message (from an option, if any),
429
* or an indication that it should be read interactively from the ui object;
430
* a list of files to commit
431
* an option for a dry-run commit
432
* verbose option, or callback to indicate
433
* timestamp, timezone, committer, chosen revision id
434
* config (for what?)
435
* option for local-only commit on a bound branch
436
* option for strict commits (fail if there are unknown or missing files)
437
* option to allow "pointless" commits (with no tree changes)
438
439
(This is rather a lot of options to pass individually and just for code tidyness maybe some of them should be combine into objects.)
440
441
>>> Branch.commit(from_tree, message, files_to_commit, ...)
442
443
There will be different implementations of this for different Branch
444
classes, whether for foreign branches or Bazaar repositories using
445
different storage methods.
446
447
Most of the commit should occur during a single lockstep iteration across
448
the workingtree and parent trees. The WorkingTree interface needs to
449
provide methods that give commit all it needs. Some of these methods
450
(such as answering the file's last change revision) may be deprecated in
451
newer working trees and there we have a choice of either calculating the
452
value from the data that is present, or refusing to support commit to
453
newer repositories.
454
455
For a dirstate tree the iteration of changes from the parent can easily be
456
done within its own iter_changes.
457
458
Dirstate inventories may be most easily updated in a single operation at
459
the end; however it may be best to accumulate data as we proceed through
460
the tree rather than revisiting it at the end.
461
462
Showing a progress bar for commit may not be necessary if we report files
463
as they are committed. Alternatively we could transiently show a progress
464
bar for each directory that's scanned, even if no changes are observed.
465
466
This needs to collect a list of added/changed/removed files, each of which
467
must have its text stored (if any) and containing directory updated. This
468
can be done by calling Tree._iter_changes on the source tree, asking for
469
changes
470
471
In the 0.17 model the commit operation needs to know the per-file parents
472
and per-file last-changed revision.
473
474
(In this and other operations we must avoid having multiple layers walk
475
over the tree separately. For example, it is no good to have the Command
476
layer walk the tree to generate a list of all file ids to commit, because
477
the tree will also be walked later. The layers that do need to operate
478
per-file should probably be bound together in a per-dirblock iterator,
479
rather than each iterating independently.)
480
481
Branch->Tree interface
482
----------------------
483
484
The Branch commit code needs to ask the Tree what should be committed, in
485
terms of changes from the parent revisions. If the Tree holds all the
486
necessary parent tree information itself it can do it single handed;
487
otherwise it may need to ask the Repository for parent information.
488
489
This should be a streaming interface, probably like iter_changes returning
490
information per directory block.
491
492
The interface should not return a block for directories that are
493
recursively unchanged.
494
495
The tree's idea of what is possibly changed may be more conservative than
496
that of the branch. For example the tree may report on merges of files
497
where the text is identical to the parents: this must be recorded for
498
Bazaar branches that record per-file ancestry but is not necessary for all
499
branches. If the tree is responsible for determining when directories
500
have been recursively modified then it will report on all the parents of
501
such files. There are several implementation options:
502
503
1. Return all files and directories the branch might want to commit, even
504
if the branch ends up taking no action on them.
505
506
2. When starting the iteration, the branch can specify what type of change
507
is considered interesting.
508
509
Since these types of changes are probably (??) rare compared to files that
510
are either completely unmodified or substantially modified, the first may
511
be the best and simplest option.
512
513
The branch needs to build an inventory to commit, which must include
514
unchanged files within changed directories. This should be returned from
515
the working tree too. Repositories that store per-directory inventories
516
will want to build and store these from the lowest directories up.
517
For 0.17 format repositories with an all-in-one inventory it may be
518
easiest to accumulate inventory entries in arbitrary order into an
519
in-memory Inventory and then serialize it.
520
521
It ought to be possible to commit any Tree into a Branch, without
522
requiring a WorkingTree; the commit code should cope if the tree is not
523
interested in updating hashcache information or does not have a
524
``last_revision``.
525
526
527
Information from the tree to repository
528
---------------------------------------
529
530
The main things the tree needs to tell the Branch about are:
531
532
* A file is modified from its parent revision (in text, permissions,
533
other), and so its text may need to be stored.
534
535
Files should also be reported if they have more than one unique parent
536
revision, for repositories that store per-file graphs or last-change
537
revisions. Perhaps this behaviour should be optional.
538
539
**XXX:** are renames/deletions reported here too?
540
541
* The complete contents of a modified directory, so that its inventory
542
text may be stored. This should be done after all the contained files
543
and directories have been reported. If there are unmodified files,
544
or unselected files carried through from
545
546
XXX: Actually perhaps not grouped by directory, but rather grouped
547
appropriately for the shape of inventory storage in the repository.
548
549
In a zoomed-in checkout the workingtree may not have all the shape data
550
for the entire tree.
551
552
* A file is missing -- could cause either automatic removal or an aborted
553
commit.
554
555
* Any unknown files -- can cause automatic addition, abortion of a strict
556
commit, or just reporting.
557
558
559
Information from the repository to the tree
560
-------------------------------------------
561
562
After the commit the tree needs to be updated to the new revision. Some
563
information which was accumulated during the commit must be made available
564
to the workingtree. It's probably reasonable to hold it all in memory and
565
allow the workingtree to get it in whatever order it wants.
566
567
* A list of modified entries, and for each one:
568
569
* The stat values observed when the file was first read.
570
571
* The hash of the committed file text.
572
573
* The file's last-change revision, if appropriate.
574
575
This should include any entries automatically added or removed.
576
577
This might be construed as an enhanced version of ``set_parent_trees``.
578
We can avoid a stat on each file by using the value that was observed when
579
it was first read.
580
581
582
583
Selective commit
584
----------------
585
586
For a partial commit the directory contents may need to contain a mix of
587
entries from the working tree and parent trees. This code probably
588
shouldn't live in a specific tree implementation; maybe there should be a
589
general filter that selects paths from one tree into another?
590
591
However, the tree walking code does probably need to know about selected
592
paths to avoid examining unselected files or directories.
593
594
We never refuse selective file commits (except of merges).
595
596
597
598
Common commit code
599
------------------
600
601
What is common to all commit implementations, regardless of workingtree or
602
repository format?
603
604
* Prompting for a commit message?
605
* Strictness/conflict checks?
606
* Auto add/remove?
607
608
How should this be separated?
609
610
611
612
Order of traversal
613
------------------
614
615
For current and contemplated Bazaar storage formats, we can only finally
616
commit a directory after its contained files and directories have been
617
committed.
618
619
The dirstate workingtree format naturally iterates by directory in order
620
by path, yielding directories before their contents. This may also be the
621
most efficient order in which to stat and read the files.
622
623
One option would be to construe the interface as a visitor which reports
624
when files are detected to be changed, and also when directories are
625
finished.
626
627
628
Open question: per-file graphs
629
------------------------------
630
631
**XXX:** If we want to retain explicitly stored per-file graphs, it would
632
seem that we do need to record per-file parents. We have not yet finally
633
settled that we do want to remove them or treat them as a cache. This api
634
stack is still ok whether we do or not, but the internals of it may
635
change.
Loggerhead is a web-based interface for Breezy
Version: 2.0.1