2484.1.1
by John Arbash Meinel
Add an initial function to read knit indexes in pyrex. |
1 |
# Copyright (C) 2005, 2006, 2007 Canonical Ltd
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
2 |
#
|
3 |
# This program is free software; you can redistribute it and/or modify
|
|
4 |
# it under the terms of the GNU General Public License as published by
|
|
5 |
# the Free Software Foundation; either version 2 of the License, or
|
|
6 |
# (at your option) any later version.
|
|
7 |
#
|
|
8 |
# This program is distributed in the hope that it will be useful,
|
|
9 |
# but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
10 |
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
11 |
# GNU General Public License for more details.
|
|
12 |
#
|
|
13 |
# You should have received a copy of the GNU General Public License
|
|
14 |
# along with this program; if not, write to the Free Software
|
|
15 |
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
|
16 |
||
17 |
"""Knit versionedfile implementation.
|
|
18 |
||
19 |
A knit is a versioned file implementation that supports efficient append only
|
|
20 |
updates.
|
|
1563.2.6
by Robert Collins
Start check tests for knits (pending), and remove dead code. |
21 |
|
22 |
Knit file layout:
|
|
23 |
lifeless: the data file is made up of "delta records". each delta record has a delta header
|
|
24 |
that contains; (1) a version id, (2) the size of the delta (in lines), and (3) the digest of
|
|
25 |
the -expanded data- (ie, the delta applied to the parent). the delta also ends with a
|
|
26 |
end-marker; simply "end VERSION"
|
|
27 |
||
28 |
delta can be line or full contents.a
|
|
29 |
... the 8's there are the index number of the annotation.
|
|
30 |
version robertc@robertcollins.net-20051003014215-ee2990904cc4c7ad 7 c7d23b2a5bd6ca00e8e266cec0ec228158ee9f9e
|
|
31 |
59,59,3
|
|
32 |
8
|
|
33 |
8 if ie.executable:
|
|
34 |
8 e.set('executable', 'yes')
|
|
35 |
130,130,2
|
|
36 |
8 if elt.get('executable') == 'yes':
|
|
37 |
8 ie.executable = True
|
|
38 |
end robertc@robertcollins.net-20051003014215-ee2990904cc4c7ad
|
|
39 |
||
40 |
||
41 |
whats in an index:
|
|
42 |
09:33 < jrydberg> lifeless: each index is made up of a tuple of; version id, options, position, size, parents
|
|
43 |
09:33 < jrydberg> lifeless: the parents are currently dictionary compressed
|
|
44 |
09:33 < jrydberg> lifeless: (meaning it currently does not support ghosts)
|
|
45 |
09:33 < lifeless> right
|
|
46 |
09:33 < jrydberg> lifeless: the position and size is the range in the data file
|
|
47 |
||
48 |
||
49 |
so the index sequence is the dictionary compressed sequence number used
|
|
50 |
in the deltas to provide line annotation
|
|
51 |
||
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
52 |
"""
|
53 |
||
1563.2.6
by Robert Collins
Start check tests for knits (pending), and remove dead code. |
54 |
# TODOS:
|
55 |
# 10:16 < lifeless> make partial index writes safe
|
|
56 |
# 10:16 < lifeless> implement 'knit.check()' like weave.check()
|
|
57 |
# 10:17 < lifeless> record known ghosts so we can detect when they are filled in rather than the current 'reweave
|
|
58 |
# always' approach.
|
|
1563.2.11
by Robert Collins
Consolidate reweave and join as we have no separate usage, make reweave tests apply to all versionedfile implementations and deprecate the old reweave apis. |
59 |
# move sha1 out of the content so that join is faster at verifying parents
|
60 |
# record content length ?
|
|
1563.2.6
by Robert Collins
Start check tests for knits (pending), and remove dead code. |
61 |
|
62 |
||
1563.2.11
by Robert Collins
Consolidate reweave and join as we have no separate usage, make reweave tests apply to all versionedfile implementations and deprecate the old reweave apis. |
63 |
from cStringIO import StringIO |
1596.2.28
by Robert Collins
more knit profile based tuning. |
64 |
from itertools import izip, chain |
1756.2.17
by Aaron Bentley
Fixes suggested by John Meinel |
65 |
import operator |
1563.2.6
by Robert Collins
Start check tests for knits (pending), and remove dead code. |
66 |
import os |
1594.2.19
by Robert Collins
More coalescing tweaks, and knit feedback. |
67 |
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
68 |
from bzrlib.lazy_import import lazy_import |
69 |
lazy_import(globals(), """ |
|
70 |
from bzrlib import (
|
|
2770.1.1
by Aaron Bentley
Initial implmentation of plain knit annotation |
71 |
annotate,
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
72 |
debug,
|
73 |
diff,
|
|
3224.1.10
by John Arbash Meinel
Introduce the heads_provider for reannotate. |
74 |
graph as _mod_graph,
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
75 |
index as _mod_index,
|
2998.2.2
by John Arbash Meinel
implement a faster path for copying from packs back to knits. |
76 |
lru_cache,
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
77 |
pack,
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
78 |
progress,
|
2745.1.2
by Robert Collins
Ensure mutter_callsite is not directly called on a lazy_load object, to make the stacklevel parameter work correctly. |
79 |
trace,
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
80 |
tsort,
|
81 |
tuned_gzip,
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
82 |
)
|
83 |
""") |
|
1911.2.3
by John Arbash Meinel
Moving everything into a new location so that we can cache more than just revision ids |
84 |
from bzrlib import ( |
85 |
errors, |
|
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
86 |
osutils, |
2104.4.2
by John Arbash Meinel
Small cleanup and NEWS entry about fixing bug #65714 |
87 |
patiencediff, |
2158.3.1
by Dmitry Vasiliev
KnitIndex tests/fixes/optimizations |
88 |
)
|
89 |
from bzrlib.errors import ( |
|
90 |
FileExists, |
|
91 |
NoSuchFile, |
|
92 |
KnitError, |
|
93 |
InvalidRevisionId, |
|
94 |
KnitCorrupt, |
|
95 |
KnitHeaderError, |
|
96 |
RevisionNotPresent, |
|
97 |
RevisionAlreadyPresent, |
|
98 |
)
|
|
99 |
from bzrlib.osutils import ( |
|
100 |
contains_whitespace, |
|
101 |
contains_linebreaks, |
|
2850.1.1
by Robert Collins
* ``KnitVersionedFile.add*`` will no longer cache added records even when |
102 |
sha_string, |
2158.3.1
by Dmitry Vasiliev
KnitIndex tests/fixes/optimizations |
103 |
sha_strings, |
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
104 |
split_lines, |
2158.3.1
by Dmitry Vasiliev
KnitIndex tests/fixes/optimizations |
105 |
)
|
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
106 |
from bzrlib.versionedfile import ( |
3350.3.12
by Robert Collins
Generate streams with absent records. |
107 |
AbsentContentFactory, |
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
108 |
adapter_registry, |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
109 |
ConstantMapper, |
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
110 |
ContentFactory, |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
111 |
FulltextContentFactory, |
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
112 |
VersionedFile, |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
113 |
VersionedFiles, |
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
114 |
)
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
115 |
|
116 |
||
117 |
# TODO: Split out code specific to this format into an associated object.
|
|
118 |
||
119 |
# TODO: Can we put in some kind of value to check that the index and data
|
|
120 |
# files belong together?
|
|
121 |
||
1759.2.1
by Jelmer Vernooij
Fix some types (found using aspell). |
122 |
# TODO: accommodate binaries, perhaps by storing a byte count
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
123 |
|
124 |
# TODO: function to check whole file
|
|
125 |
||
126 |
# TODO: atomically append data, then measure backwards from the cursor
|
|
127 |
# position after writing to work out where it was located. we may need to
|
|
128 |
# bypass python file buffering.
|
|
129 |
||
130 |
DATA_SUFFIX = '.knit' |
|
131 |
INDEX_SUFFIX = '.kndx' |
|
132 |
||
133 |
||
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
134 |
class KnitAdapter(object): |
135 |
"""Base class for knit record adaption."""
|
|
136 |
||
3350.3.7
by Robert Collins
Create a registry of versioned file record adapters. |
137 |
def __init__(self, basis_vf): |
138 |
"""Create an adapter which accesses full texts from basis_vf.
|
|
139 |
|
|
140 |
:param basis_vf: A versioned file to access basis texts of deltas from.
|
|
141 |
May be None for adapters that do not need to access basis texts.
|
|
142 |
"""
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
143 |
self._data = KnitVersionedFiles(None, None) |
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
144 |
self._annotate_factory = KnitAnnotateFactory() |
145 |
self._plain_factory = KnitPlainFactory() |
|
3350.3.7
by Robert Collins
Create a registry of versioned file record adapters. |
146 |
self._basis_vf = basis_vf |
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
147 |
|
148 |
||
149 |
class FTAnnotatedToUnannotated(KnitAdapter): |
|
150 |
"""An adapter from FT annotated knits to unannotated ones."""
|
|
151 |
||
152 |
def get_bytes(self, factory, annotated_compressed_bytes): |
|
153 |
rec, contents = \ |
|
154 |
self._data._parse_record_unchecked(annotated_compressed_bytes) |
|
155 |
content = self._annotate_factory.parse_fulltext(contents, rec[1]) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
156 |
size, bytes = self._data._record_to_data((rec[1],), rec[3], content.text()) |
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
157 |
return bytes |
158 |
||
159 |
||
160 |
class DeltaAnnotatedToUnannotated(KnitAdapter): |
|
161 |
"""An adapter for deltas from annotated to unannotated."""
|
|
162 |
||
163 |
def get_bytes(self, factory, annotated_compressed_bytes): |
|
164 |
rec, contents = \ |
|
165 |
self._data._parse_record_unchecked(annotated_compressed_bytes) |
|
166 |
delta = self._annotate_factory.parse_line_delta(contents, rec[1], |
|
167 |
plain=True) |
|
168 |
contents = self._plain_factory.lower_line_delta(delta) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
169 |
size, bytes = self._data._record_to_data((rec[1],), rec[3], contents) |
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
170 |
return bytes |
171 |
||
172 |
||
173 |
class FTAnnotatedToFullText(KnitAdapter): |
|
174 |
"""An adapter from FT annotated knits to unannotated ones."""
|
|
175 |
||
176 |
def get_bytes(self, factory, annotated_compressed_bytes): |
|
177 |
rec, contents = \ |
|
178 |
self._data._parse_record_unchecked(annotated_compressed_bytes) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
179 |
content, delta = self._annotate_factory.parse_record(factory.key[-1], |
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
180 |
contents, factory._build_details, None) |
181 |
return ''.join(content.text()) |
|
182 |
||
183 |
||
184 |
class DeltaAnnotatedToFullText(KnitAdapter): |
|
185 |
"""An adapter for deltas from annotated to unannotated."""
|
|
186 |
||
187 |
def get_bytes(self, factory, annotated_compressed_bytes): |
|
188 |
rec, contents = \ |
|
189 |
self._data._parse_record_unchecked(annotated_compressed_bytes) |
|
190 |
delta = self._annotate_factory.parse_line_delta(contents, rec[1], |
|
191 |
plain=True) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
192 |
compression_parent = factory.parents[0] |
193 |
basis_entry = self._basis_vf.get_record_stream( |
|
194 |
[compression_parent], 'unordered', True).next() |
|
195 |
if basis_entry.storage_kind == 'absent': |
|
196 |
raise errors.RevisionNotPresent(compression_parent, self._basis_vf) |
|
197 |
basis_lines = split_lines(basis_entry.get_bytes_as('fulltext')) |
|
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
198 |
# Manually apply the delta because we have one annotated content and
|
199 |
# one plain.
|
|
200 |
basis_content = PlainKnitContent(basis_lines, compression_parent) |
|
201 |
basis_content.apply_delta(delta, rec[1]) |
|
202 |
basis_content._should_strip_eol = factory._build_details[1] |
|
203 |
return ''.join(basis_content.text()) |
|
204 |
||
205 |
||
3350.3.5
by Robert Collins
Create adapters from plain compressed knit content. |
206 |
class FTPlainToFullText(KnitAdapter): |
207 |
"""An adapter from FT plain knits to unannotated ones."""
|
|
208 |
||
209 |
def get_bytes(self, factory, compressed_bytes): |
|
210 |
rec, contents = \ |
|
211 |
self._data._parse_record_unchecked(compressed_bytes) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
212 |
content, delta = self._plain_factory.parse_record(factory.key[-1], |
3350.3.5
by Robert Collins
Create adapters from plain compressed knit content. |
213 |
contents, factory._build_details, None) |
214 |
return ''.join(content.text()) |
|
215 |
||
216 |
||
217 |
class DeltaPlainToFullText(KnitAdapter): |
|
218 |
"""An adapter for deltas from annotated to unannotated."""
|
|
219 |
||
220 |
def get_bytes(self, factory, compressed_bytes): |
|
221 |
rec, contents = \ |
|
222 |
self._data._parse_record_unchecked(compressed_bytes) |
|
223 |
delta = self._plain_factory.parse_line_delta(contents, rec[1]) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
224 |
compression_parent = factory.parents[0] |
225 |
# XXX: string splitting overhead.
|
|
226 |
basis_entry = self._basis_vf.get_record_stream( |
|
227 |
[compression_parent], 'unordered', True).next() |
|
228 |
if basis_entry.storage_kind == 'absent': |
|
229 |
raise errors.RevisionNotPresent(compression_parent, self._basis_vf) |
|
230 |
basis_lines = split_lines(basis_entry.get_bytes_as('fulltext')) |
|
3350.3.5
by Robert Collins
Create adapters from plain compressed knit content. |
231 |
basis_content = PlainKnitContent(basis_lines, compression_parent) |
232 |
# Manually apply the delta because we have one annotated content and
|
|
233 |
# one plain.
|
|
234 |
content, _ = self._plain_factory.parse_record(rec[1], contents, |
|
235 |
factory._build_details, basis_content) |
|
236 |
return ''.join(content.text()) |
|
237 |
||
238 |
||
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
239 |
class KnitContentFactory(ContentFactory): |
240 |
"""Content factory for streaming from knits.
|
|
241 |
|
|
242 |
:seealso ContentFactory:
|
|
243 |
"""
|
|
244 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
245 |
def __init__(self, key, parents, build_details, sha1, raw_record, |
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
246 |
annotated, knit=None): |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
247 |
"""Create a KnitContentFactory for key.
|
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
248 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
249 |
:param key: The key.
|
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
250 |
:param parents: The parents.
|
251 |
:param build_details: The build details as returned from
|
|
252 |
get_build_details.
|
|
253 |
:param sha1: The sha1 expected from the full text of this object.
|
|
254 |
:param raw_record: The bytes of the knit data from disk.
|
|
255 |
:param annotated: True if the raw data is annotated.
|
|
256 |
"""
|
|
257 |
ContentFactory.__init__(self) |
|
258 |
self.sha1 = sha1 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
259 |
self.key = key |
260 |
self.parents = parents |
|
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
261 |
if build_details[0] == 'line-delta': |
262 |
kind = 'delta' |
|
263 |
else: |
|
264 |
kind = 'ft' |
|
265 |
if annotated: |
|
266 |
annotated_kind = 'annotated-' |
|
267 |
else: |
|
268 |
annotated_kind = '' |
|
269 |
self.storage_kind = 'knit-%s%s-gz' % (annotated_kind, kind) |
|
270 |
self._raw_record = raw_record |
|
271 |
self._build_details = build_details |
|
272 |
self._knit = knit |
|
273 |
||
274 |
def get_bytes_as(self, storage_kind): |
|
275 |
if storage_kind == self.storage_kind: |
|
276 |
return self._raw_record |
|
277 |
if storage_kind == 'fulltext' and self._knit is not None: |
|
278 |
return self._knit.get_text(self.key[0]) |
|
279 |
else: |
|
280 |
raise errors.UnavailableRepresentation(self.key, storage_kind, |
|
281 |
self.storage_kind) |
|
282 |
||
283 |
||
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
284 |
class KnitContent(object): |
3468.2.4
by Martin Pool
Test and fix #234748 problems in trailing newline diffs |
285 |
"""Content of a knit version to which deltas can be applied.
|
286 |
|
|
3468.2.5
by Martin Pool
Correct comment and remove overbroad except block |
287 |
This is always stored in memory as a list of lines with \n at the end,
|
288 |
plus a flag saying if the final ending is really there or not, because that
|
|
289 |
corresponds to the on-disk knit representation.
|
|
3468.2.4
by Martin Pool
Test and fix #234748 problems in trailing newline diffs |
290 |
"""
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
291 |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
292 |
def __init__(self): |
293 |
self._should_strip_eol = False |
|
294 |
||
2921.2.1
by Robert Collins
* Knit text reconstruction now avoids making copies of the lines list for |
295 |
def apply_delta(self, delta, new_version_id): |
2921.2.2
by Robert Collins
Review feedback. |
296 |
"""Apply delta to this object to become new_version_id."""
|
2921.2.1
by Robert Collins
* Knit text reconstruction now avoids making copies of the lines list for |
297 |
raise NotImplementedError(self.apply_delta) |
298 |
||
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
299 |
def line_delta_iter(self, new_lines): |
1596.2.32
by Robert Collins
Reduce re-extraction of texts during weave to knit joins by providing a memoisation facility. |
300 |
"""Generate line-based delta from this content to new_lines."""
|
2151.1.1
by John Arbash Meinel
(Dmitry Vasiliev) Tune KnitContent and add tests |
301 |
new_texts = new_lines.text() |
302 |
old_texts = self.text() |
|
2781.1.1
by Martin Pool
merge cpatiencediff from Lukas |
303 |
s = patiencediff.PatienceSequenceMatcher(None, old_texts, new_texts) |
2151.1.1
by John Arbash Meinel
(Dmitry Vasiliev) Tune KnitContent and add tests |
304 |
for tag, i1, i2, j1, j2 in s.get_opcodes(): |
305 |
if tag == 'equal': |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
306 |
continue
|
2151.1.1
by John Arbash Meinel
(Dmitry Vasiliev) Tune KnitContent and add tests |
307 |
# ofrom, oto, length, data
|
308 |
yield i1, i2, j2 - j1, new_lines._lines[j1:j2] |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
309 |
|
310 |
def line_delta(self, new_lines): |
|
311 |
return list(self.line_delta_iter(new_lines)) |
|
312 |
||
2520.4.41
by Aaron Bentley
Accelerate mpdiff generation |
313 |
@staticmethod
|
2520.4.48
by Aaron Bentley
Support getting blocks from knit deltas with no final EOL |
314 |
def get_line_delta_blocks(knit_delta, source, target): |
2520.4.41
by Aaron Bentley
Accelerate mpdiff generation |
315 |
"""Extract SequenceMatcher.get_matching_blocks() from a knit delta"""
|
2520.4.48
by Aaron Bentley
Support getting blocks from knit deltas with no final EOL |
316 |
target_len = len(target) |
2520.4.41
by Aaron Bentley
Accelerate mpdiff generation |
317 |
s_pos = 0 |
318 |
t_pos = 0 |
|
319 |
for s_begin, s_end, t_len, new_text in knit_delta: |
|
2520.4.47
by Aaron Bentley
Fix get_line_delta_blocks with eol |
320 |
true_n = s_begin - s_pos |
321 |
n = true_n |
|
2520.4.41
by Aaron Bentley
Accelerate mpdiff generation |
322 |
if n > 0: |
2520.4.48
by Aaron Bentley
Support getting blocks from knit deltas with no final EOL |
323 |
# knit deltas do not provide reliable info about whether the
|
324 |
# last line of a file matches, due to eol handling.
|
|
325 |
if source[s_pos + n -1] != target[t_pos + n -1]: |
|
2520.4.47
by Aaron Bentley
Fix get_line_delta_blocks with eol |
326 |
n-=1 |
327 |
if n > 0: |
|
328 |
yield s_pos, t_pos, n |
|
329 |
t_pos += t_len + true_n |
|
2520.4.41
by Aaron Bentley
Accelerate mpdiff generation |
330 |
s_pos = s_end |
2520.4.48
by Aaron Bentley
Support getting blocks from knit deltas with no final EOL |
331 |
n = target_len - t_pos |
332 |
if n > 0: |
|
333 |
if source[s_pos + n -1] != target[t_pos + n -1]: |
|
334 |
n-=1 |
|
335 |
if n > 0: |
|
336 |
yield s_pos, t_pos, n |
|
2520.4.41
by Aaron Bentley
Accelerate mpdiff generation |
337 |
yield s_pos + (target_len - t_pos), target_len, 0 |
338 |
||
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
339 |
|
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
340 |
class AnnotatedKnitContent(KnitContent): |
341 |
"""Annotated content."""
|
|
342 |
||
343 |
def __init__(self, lines): |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
344 |
KnitContent.__init__(self) |
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
345 |
self._lines = lines |
346 |
||
3316.2.13
by Robert Collins
* ``VersionedFile.annotate_iter`` is deprecated. While in principal this |
347 |
def annotate(self): |
348 |
"""Return a list of (origin, text) for each content line."""
|
|
3468.2.4
by Martin Pool
Test and fix #234748 problems in trailing newline diffs |
349 |
lines = self._lines[:] |
350 |
if self._should_strip_eol: |
|
351 |
origin, last_line = lines[-1] |
|
352 |
lines[-1] = (origin, last_line.rstrip('\n')) |
|
353 |
return lines |
|
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
354 |
|
2921.2.1
by Robert Collins
* Knit text reconstruction now avoids making copies of the lines list for |
355 |
def apply_delta(self, delta, new_version_id): |
2921.2.2
by Robert Collins
Review feedback. |
356 |
"""Apply delta to this object to become new_version_id."""
|
2921.2.1
by Robert Collins
* Knit text reconstruction now avoids making copies of the lines list for |
357 |
offset = 0 |
358 |
lines = self._lines |
|
359 |
for start, end, count, delta_lines in delta: |
|
360 |
lines[offset+start:offset+end] = delta_lines |
|
361 |
offset = offset + (start - end) + count |
|
362 |
||
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
363 |
def text(self): |
2911.1.1
by Martin Pool
Better messages when problems are detected inside a knit |
364 |
try: |
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
365 |
lines = [text for origin, text in self._lines] |
2911.1.1
by Martin Pool
Better messages when problems are detected inside a knit |
366 |
except ValueError, e: |
367 |
# most commonly (only?) caused by the internal form of the knit
|
|
368 |
# missing annotation information because of a bug - see thread
|
|
369 |
# around 20071015
|
|
370 |
raise KnitCorrupt(self, |
|
371 |
"line in annotated knit missing annotation information: %s" |
|
372 |
% (e,)) |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
373 |
if self._should_strip_eol: |
3350.3.4
by Robert Collins
Finish adapters for annotated knits to unannotated knits and full texts. |
374 |
lines[-1] = lines[-1].rstrip('\n') |
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
375 |
return lines |
376 |
||
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
377 |
def copy(self): |
378 |
return AnnotatedKnitContent(self._lines[:]) |
|
379 |
||
380 |
||
381 |
class PlainKnitContent(KnitContent): |
|
2794.1.3
by Robert Collins
Review feedback. |
382 |
"""Unannotated content.
|
383 |
|
|
384 |
When annotate[_iter] is called on this content, the same version is reported
|
|
385 |
for all lines. Generally, annotate[_iter] is not useful on PlainKnitContent
|
|
386 |
objects.
|
|
387 |
"""
|
|
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
388 |
|
389 |
def __init__(self, lines, version_id): |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
390 |
KnitContent.__init__(self) |
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
391 |
self._lines = lines |
392 |
self._version_id = version_id |
|
393 |
||
3316.2.13
by Robert Collins
* ``VersionedFile.annotate_iter`` is deprecated. While in principal this |
394 |
def annotate(self): |
395 |
"""Return a list of (origin, text) for each content line."""
|
|
396 |
return [(self._version_id, line) for line in self._lines] |
|
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
397 |
|
2921.2.1
by Robert Collins
* Knit text reconstruction now avoids making copies of the lines list for |
398 |
def apply_delta(self, delta, new_version_id): |
2921.2.2
by Robert Collins
Review feedback. |
399 |
"""Apply delta to this object to become new_version_id."""
|
2921.2.1
by Robert Collins
* Knit text reconstruction now avoids making copies of the lines list for |
400 |
offset = 0 |
401 |
lines = self._lines |
|
402 |
for start, end, count, delta_lines in delta: |
|
403 |
lines[offset+start:offset+end] = delta_lines |
|
404 |
offset = offset + (start - end) + count |
|
405 |
self._version_id = new_version_id |
|
406 |
||
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
407 |
def copy(self): |
408 |
return PlainKnitContent(self._lines[:], self._version_id) |
|
409 |
||
410 |
def text(self): |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
411 |
lines = self._lines |
412 |
if self._should_strip_eol: |
|
413 |
lines = lines[:] |
|
414 |
lines[-1] = lines[-1].rstrip('\n') |
|
415 |
return lines |
|
416 |
||
417 |
||
418 |
class _KnitFactory(object): |
|
419 |
"""Base class for common Factory functions."""
|
|
420 |
||
421 |
def parse_record(self, version_id, record, record_details, |
|
422 |
base_content, copy_base_content=True): |
|
423 |
"""Parse a record into a full content object.
|
|
424 |
||
425 |
:param version_id: The official version id for this content
|
|
426 |
:param record: The data returned by read_records_iter()
|
|
427 |
:param record_details: Details about the record returned by
|
|
428 |
get_build_details
|
|
429 |
:param base_content: If get_build_details returns a compression_parent,
|
|
430 |
you must return a base_content here, else use None
|
|
431 |
:param copy_base_content: When building from the base_content, decide
|
|
432 |
you can either copy it and return a new object, or modify it in
|
|
433 |
place.
|
|
434 |
:return: (content, delta) A Content object and possibly a line-delta,
|
|
435 |
delta may be None
|
|
436 |
"""
|
|
437 |
method, noeol = record_details |
|
438 |
if method == 'line-delta': |
|
439 |
if copy_base_content: |
|
440 |
content = base_content.copy() |
|
441 |
else: |
|
442 |
content = base_content |
|
443 |
delta = self.parse_line_delta(record, version_id) |
|
444 |
content.apply_delta(delta, version_id) |
|
445 |
else: |
|
446 |
content = self.parse_fulltext(record, version_id) |
|
447 |
delta = None |
|
448 |
content._should_strip_eol = noeol |
|
449 |
return (content, delta) |
|
450 |
||
451 |
||
452 |
class KnitAnnotateFactory(_KnitFactory): |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
453 |
"""Factory for creating annotated Content objects."""
|
454 |
||
455 |
annotated = True |
|
456 |
||
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
457 |
def make(self, lines, version_id): |
458 |
num_lines = len(lines) |
|
459 |
return AnnotatedKnitContent(zip([version_id] * num_lines, lines)) |
|
460 |
||
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
461 |
def parse_fulltext(self, content, version_id): |
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
462 |
"""Convert fulltext to internal representation
|
463 |
||
464 |
fulltext content is of the format
|
|
465 |
revid(utf8) plaintext\n
|
|
466 |
internal representation is of the format:
|
|
467 |
(revid, plaintext)
|
|
468 |
"""
|
|
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
469 |
# TODO: jam 20070209 The tests expect this to be returned as tuples,
|
470 |
# but the code itself doesn't really depend on that.
|
|
471 |
# Figure out a way to not require the overhead of turning the
|
|
472 |
# list back into tuples.
|
|
473 |
lines = [tuple(line.split(' ', 1)) for line in content] |
|
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
474 |
return AnnotatedKnitContent(lines) |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
475 |
|
476 |
def parse_line_delta_iter(self, lines): |
|
2163.1.2
by John Arbash Meinel
Don't modify the list during parse_line_delta |
477 |
return iter(self.parse_line_delta(lines)) |
1628.1.2
by Robert Collins
More knit micro-optimisations. |
478 |
|
2851.4.2
by Ian Clatworthy
use factory methods in annotated-to-plain conversion instead of duplicating format knowledge |
479 |
def parse_line_delta(self, lines, version_id, plain=False): |
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
480 |
"""Convert a line based delta into internal representation.
|
481 |
||
482 |
line delta is in the form of:
|
|
483 |
intstart intend intcount
|
|
484 |
1..count lines:
|
|
485 |
revid(utf8) newline\n
|
|
1759.2.1
by Jelmer Vernooij
Fix some types (found using aspell). |
486 |
internal representation is
|
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
487 |
(start, end, count, [1..count tuples (revid, newline)])
|
2851.4.2
by Ian Clatworthy
use factory methods in annotated-to-plain conversion instead of duplicating format knowledge |
488 |
|
489 |
:param plain: If True, the lines are returned as a plain
|
|
2911.1.1
by Martin Pool
Better messages when problems are detected inside a knit |
490 |
list without annotations, not as a list of (origin, content) tuples, i.e.
|
2851.4.2
by Ian Clatworthy
use factory methods in annotated-to-plain conversion instead of duplicating format knowledge |
491 |
(start, end, count, [1..count newline])
|
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
492 |
"""
|
1628.1.2
by Robert Collins
More knit micro-optimisations. |
493 |
result = [] |
494 |
lines = iter(lines) |
|
495 |
next = lines.next |
|
2249.5.1
by John Arbash Meinel
Leave revision-ids in utf-8 when reading. |
496 |
|
2249.5.15
by John Arbash Meinel
remove get_cached_utf8 checks which were slowing things down. |
497 |
cache = {} |
498 |
def cache_and_return(line): |
|
499 |
origin, text = line.split(' ', 1) |
|
500 |
return cache.setdefault(origin, origin), text |
|
501 |
||
1628.1.2
by Robert Collins
More knit micro-optimisations. |
502 |
# walk through the lines parsing.
|
2851.4.2
by Ian Clatworthy
use factory methods in annotated-to-plain conversion instead of duplicating format knowledge |
503 |
# Note that the plain test is explicitly pulled out of the
|
504 |
# loop to minimise any performance impact
|
|
505 |
if plain: |
|
506 |
for header in lines: |
|
507 |
start, end, count = [int(n) for n in header.split(',')] |
|
508 |
contents = [next().split(' ', 1)[1] for i in xrange(count)] |
|
509 |
result.append((start, end, count, contents)) |
|
510 |
else: |
|
511 |
for header in lines: |
|
512 |
start, end, count = [int(n) for n in header.split(',')] |
|
513 |
contents = [tuple(next().split(' ', 1)) for i in xrange(count)] |
|
514 |
result.append((start, end, count, contents)) |
|
1628.1.2
by Robert Collins
More knit micro-optimisations. |
515 |
return result |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
516 |
|
2163.2.2
by John Arbash Meinel
Don't deal with annotations when we don't care about them. Saves another 300+ms |
517 |
def get_fulltext_content(self, lines): |
518 |
"""Extract just the content lines from a fulltext."""
|
|
519 |
return (line.split(' ', 1)[1] for line in lines) |
|
520 |
||
521 |
def get_linedelta_content(self, lines): |
|
522 |
"""Extract just the content from a line delta.
|
|
523 |
||
524 |
This doesn't return all of the extra information stored in a delta.
|
|
525 |
Only the actual content lines.
|
|
526 |
"""
|
|
527 |
lines = iter(lines) |
|
528 |
next = lines.next |
|
529 |
for header in lines: |
|
530 |
header = header.split(',') |
|
531 |
count = int(header[2]) |
|
532 |
for i in xrange(count): |
|
533 |
origin, text = next().split(' ', 1) |
|
534 |
yield text |
|
535 |
||
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
536 |
def lower_fulltext(self, content): |
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
537 |
"""convert a fulltext content record into a serializable form.
|
538 |
||
539 |
see parse_fulltext which this inverts.
|
|
540 |
"""
|
|
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
541 |
# TODO: jam 20070209 We only do the caching thing to make sure that
|
542 |
# the origin is a valid utf-8 line, eventually we could remove it
|
|
2249.5.15
by John Arbash Meinel
remove get_cached_utf8 checks which were slowing things down. |
543 |
return ['%s %s' % (o, t) for o, t in content._lines] |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
544 |
|
545 |
def lower_line_delta(self, delta): |
|
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
546 |
"""convert a delta into a serializable form.
|
547 |
||
1628.1.2
by Robert Collins
More knit micro-optimisations. |
548 |
See parse_line_delta which this inverts.
|
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
549 |
"""
|
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
550 |
# TODO: jam 20070209 We only do the caching thing to make sure that
|
551 |
# the origin is a valid utf-8 line, eventually we could remove it
|
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
552 |
out = [] |
553 |
for start, end, c, lines in delta: |
|
554 |
out.append('%d,%d,%d\n' % (start, end, c)) |
|
2249.5.15
by John Arbash Meinel
remove get_cached_utf8 checks which were slowing things down. |
555 |
out.extend(origin + ' ' + text |
1911.2.1
by John Arbash Meinel
Cache encode/decode operations, saves memory and time. Especially when committing a new kernel tree with 7.7M new lines to annotate |
556 |
for origin, text in lines) |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
557 |
return out |
558 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
559 |
def annotate(self, knit, key): |
560 |
content = knit._get_content(key) |
|
561 |
# adjust for the fact that serialised annotations are only key suffixes
|
|
562 |
# for this factory.
|
|
563 |
if type(key) == tuple: |
|
564 |
prefix = key[:-1] |
|
565 |
origins = content.annotate() |
|
566 |
result = [] |
|
567 |
for origin, line in origins: |
|
568 |
result.append((prefix + (origin,), line)) |
|
569 |
return result |
|
570 |
else: |
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
571 |
# XXX: This smells a bit. Why would key ever be a non-tuple here?
|
572 |
# Aren't keys defined to be tuples? -- spiv 20080618
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
573 |
return content.annotate() |
2770.1.1
by Aaron Bentley
Initial implmentation of plain knit annotation |
574 |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
575 |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
576 |
class KnitPlainFactory(_KnitFactory): |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
577 |
"""Factory for creating plain Content objects."""
|
578 |
||
579 |
annotated = False |
|
580 |
||
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
581 |
def make(self, lines, version_id): |
582 |
return PlainKnitContent(lines, version_id) |
|
583 |
||
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
584 |
def parse_fulltext(self, content, version_id): |
1596.2.7
by Robert Collins
Remove the requirement for reannotation in knit joins. |
585 |
"""This parses an unannotated fulltext.
|
586 |
||
587 |
Note that this is not a noop - the internal representation
|
|
588 |
has (versionid, line) - its just a constant versionid.
|
|
589 |
"""
|
|
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
590 |
return self.make(content, version_id) |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
591 |
|
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
592 |
def parse_line_delta_iter(self, lines, version_id): |
2163.1.2
by John Arbash Meinel
Don't modify the list during parse_line_delta |
593 |
cur = 0 |
594 |
num_lines = len(lines) |
|
595 |
while cur < num_lines: |
|
596 |
header = lines[cur] |
|
597 |
cur += 1 |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
598 |
start, end, c = [int(n) for n in header.split(',')] |
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
599 |
yield start, end, c, lines[cur:cur+c] |
2163.1.2
by John Arbash Meinel
Don't modify the list during parse_line_delta |
600 |
cur += c |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
601 |
|
2249.5.12
by John Arbash Meinel
Change the APIs for VersionedFile, Store, and some of Repository into utf-8 |
602 |
def parse_line_delta(self, lines, version_id): |
603 |
return list(self.parse_line_delta_iter(lines, version_id)) |
|
2158.3.1
by Dmitry Vasiliev
KnitIndex tests/fixes/optimizations |
604 |
|
2163.2.2
by John Arbash Meinel
Don't deal with annotations when we don't care about them. Saves another 300+ms |
605 |
def get_fulltext_content(self, lines): |
606 |
"""Extract just the content lines from a fulltext."""
|
|
607 |
return iter(lines) |
|
608 |
||
609 |
def get_linedelta_content(self, lines): |
|
610 |
"""Extract just the content from a line delta.
|
|
611 |
||
612 |
This doesn't return all of the extra information stored in a delta.
|
|
613 |
Only the actual content lines.
|
|
614 |
"""
|
|
615 |
lines = iter(lines) |
|
616 |
next = lines.next |
|
617 |
for header in lines: |
|
618 |
header = header.split(',') |
|
619 |
count = int(header[2]) |
|
620 |
for i in xrange(count): |
|
621 |
yield next() |
|
622 |
||
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
623 |
def lower_fulltext(self, content): |
624 |
return content.text() |
|
625 |
||
626 |
def lower_line_delta(self, delta): |
|
627 |
out = [] |
|
628 |
for start, end, c, lines in delta: |
|
629 |
out.append('%d,%d,%d\n' % (start, end, c)) |
|
2794.1.2
by Robert Collins
Nuke versioned file add/get delta support, allowing easy simplification of unannotated Content, reducing memory copies and friction during commit on unannotated texts. |
630 |
out.extend(lines) |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
631 |
return out |
632 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
633 |
def annotate(self, knit, key): |
3224.1.7
by John Arbash Meinel
_StreamIndex also needs to return the proper values for get_build_details. |
634 |
annotator = _KnitAnnotator(knit) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
635 |
return annotator.annotate(key) |
636 |
||
637 |
||
638 |
||
639 |
def make_file_factory(annotated, mapper): |
|
640 |
"""Create a factory for creating a file based KnitVersionedFiles.
|
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
641 |
|
642 |
This is only functional enough to run interface tests, it doesn't try to
|
|
643 |
provide a full pack environment.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
644 |
|
645 |
:param annotated: knit annotations are wanted.
|
|
646 |
:param mapper: The mapper from keys to paths.
|
|
647 |
"""
|
|
648 |
def factory(transport): |
|
649 |
index = _KndxIndex(transport, mapper, lambda:None, lambda:True, lambda:True) |
|
650 |
access = _KnitKeyAccess(transport, mapper) |
|
651 |
return KnitVersionedFiles(index, access, annotated=annotated) |
|
652 |
return factory |
|
653 |
||
654 |
||
655 |
def make_pack_factory(graph, delta, keylength): |
|
656 |
"""Create a factory for creating a pack based VersionedFiles.
|
|
657 |
||
658 |
This is only functional enough to run interface tests, it doesn't try to
|
|
659 |
provide a full pack environment.
|
|
660 |
|
|
661 |
:param graph: Store a graph.
|
|
662 |
:param delta: Delta compress contents.
|
|
663 |
:param keylength: How long should keys be.
|
|
664 |
"""
|
|
665 |
def factory(transport): |
|
666 |
parents = graph or delta |
|
667 |
ref_length = 0 |
|
668 |
if graph: |
|
669 |
ref_length += 1 |
|
670 |
if delta: |
|
671 |
ref_length += 1 |
|
672 |
max_delta_chain = 200 |
|
673 |
else: |
|
674 |
max_delta_chain = 0 |
|
675 |
graph_index = _mod_index.InMemoryGraphIndex(reference_lists=ref_length, |
|
676 |
key_elements=keylength) |
|
677 |
stream = transport.open_write_stream('newpack') |
|
678 |
writer = pack.ContainerWriter(stream.write) |
|
679 |
writer.begin() |
|
680 |
index = _KnitGraphIndex(graph_index, lambda:True, parents=parents, |
|
681 |
deltas=delta, add_callback=graph_index.add_nodes) |
|
682 |
access = _DirectPackAccess({}) |
|
683 |
access.set_writer(writer, graph_index, (transport, 'newpack')) |
|
684 |
result = KnitVersionedFiles(index, access, |
|
685 |
max_delta_chain=max_delta_chain) |
|
686 |
result.stream = stream |
|
687 |
result.writer = writer |
|
688 |
return result |
|
689 |
return factory |
|
690 |
||
691 |
||
692 |
def cleanup_pack_knit(versioned_files): |
|
693 |
versioned_files.stream.close() |
|
694 |
versioned_files.writer.end() |
|
695 |
||
696 |
||
697 |
class KnitVersionedFiles(VersionedFiles): |
|
698 |
"""Storage for many versioned files using knit compression.
|
|
699 |
||
700 |
Backend storage is managed by indices and data objects.
|
|
701 |
"""
|
|
702 |
||
703 |
def __init__(self, index, data_access, max_delta_chain=200, |
|
704 |
annotated=False): |
|
705 |
"""Create a KnitVersionedFiles with index and data_access.
|
|
706 |
||
707 |
:param index: The index for the knit data.
|
|
708 |
:param data_access: The access object to store and retrieve knit
|
|
709 |
records.
|
|
710 |
:param max_delta_chain: The maximum number of deltas to permit during
|
|
711 |
insertion. Set to 0 to prohibit the use of deltas.
|
|
712 |
:param annotated: Set to True to cause annotations to be calculated and
|
|
713 |
stored during insertion.
|
|
1563.2.25
by Robert Collins
Merge in upstream. |
714 |
"""
|
3316.2.3
by Robert Collins
Remove manual notification of transaction finishing on versioned files. |
715 |
self._index = index |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
716 |
self._access = data_access |
717 |
self._max_delta_chain = max_delta_chain |
|
718 |
if annotated: |
|
719 |
self._factory = KnitAnnotateFactory() |
|
720 |
else: |
|
721 |
self._factory = KnitPlainFactory() |
|
3350.8.1
by Robert Collins
KnitVersionedFiles.add_fallback_versioned_files exists. |
722 |
self._fallback_vfs = [] |
723 |
||
724 |
def add_fallback_versioned_files(self, a_versioned_files): |
|
725 |
"""Add a source of texts for texts not present in this knit.
|
|
726 |
||
727 |
:param a_versioned_files: A VersionedFiles object.
|
|
728 |
"""
|
|
729 |
self._fallback_vfs.append(a_versioned_files) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
730 |
|
731 |
def add_lines(self, key, parents, lines, parent_texts=None, |
|
732 |
left_matching_blocks=None, nostore_sha=None, random_id=False, |
|
733 |
check_content=True): |
|
734 |
"""See VersionedFiles.add_lines()."""
|
|
735 |
self._index._check_write_ok() |
|
736 |
self._check_add(key, lines, random_id, check_content) |
|
737 |
if parents is None: |
|
3350.6.11
by Martin Pool
Review cleanups and documentation from Robert's mail on 2080618 |
738 |
# The caller might pass None if there is no graph data, but kndx
|
739 |
# indexes can't directly store that, so we give them
|
|
740 |
# an empty tuple instead.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
741 |
parents = () |
742 |
return self._add(key, lines, parents, |
|
743 |
parent_texts, left_matching_blocks, nostore_sha, random_id) |
|
744 |
||
745 |
def _add(self, key, lines, parents, parent_texts, |
|
746 |
left_matching_blocks, nostore_sha, random_id): |
|
747 |
"""Add a set of lines on top of version specified by parents.
|
|
748 |
||
749 |
Any versions not present will be converted into ghosts.
|
|
750 |
"""
|
|
751 |
# first thing, if the content is something we don't need to store, find
|
|
752 |
# that out.
|
|
753 |
line_bytes = ''.join(lines) |
|
754 |
digest = sha_string(line_bytes) |
|
755 |
if nostore_sha == digest: |
|
756 |
raise errors.ExistingContent |
|
757 |
||
758 |
present_parents = [] |
|
759 |
if parent_texts is None: |
|
760 |
parent_texts = {} |
|
761 |
# Do a single query to ascertain parent presence.
|
|
762 |
present_parent_map = self.get_parent_map(parents) |
|
763 |
for parent in parents: |
|
764 |
if parent in present_parent_map: |
|
765 |
present_parents.append(parent) |
|
766 |
||
767 |
# Currently we can only compress against the left most present parent.
|
|
768 |
if (len(present_parents) == 0 or |
|
769 |
present_parents[0] != parents[0]): |
|
770 |
delta = False |
|
771 |
else: |
|
772 |
# To speed the extract of texts the delta chain is limited
|
|
773 |
# to a fixed number of deltas. This should minimize both
|
|
774 |
# I/O and the time spend applying deltas.
|
|
775 |
delta = self._check_should_delta(present_parents[0]) |
|
776 |
||
777 |
text_length = len(line_bytes) |
|
778 |
options = [] |
|
779 |
if lines: |
|
780 |
if lines[-1][-1] != '\n': |
|
781 |
# copy the contents of lines.
|
|
782 |
lines = lines[:] |
|
783 |
options.append('no-eol') |
|
784 |
lines[-1] = lines[-1] + '\n' |
|
785 |
line_bytes += '\n' |
|
786 |
||
787 |
for element in key: |
|
788 |
if type(element) != str: |
|
789 |
raise TypeError("key contains non-strings: %r" % (key,)) |
|
790 |
# Knit hunks are still last-element only
|
|
791 |
version_id = key[-1] |
|
792 |
content = self._factory.make(lines, version_id) |
|
793 |
if 'no-eol' in options: |
|
794 |
# Hint to the content object that its text() call should strip the
|
|
795 |
# EOL.
|
|
796 |
content._should_strip_eol = True |
|
797 |
if delta or (self._factory.annotated and len(present_parents) > 0): |
|
798 |
# Merge annotations from parent texts if needed.
|
|
799 |
delta_hunks = self._merge_annotations(content, present_parents, |
|
800 |
parent_texts, delta, self._factory.annotated, |
|
801 |
left_matching_blocks) |
|
802 |
||
803 |
if delta: |
|
804 |
options.append('line-delta') |
|
805 |
store_lines = self._factory.lower_line_delta(delta_hunks) |
|
806 |
size, bytes = self._record_to_data(key, digest, |
|
807 |
store_lines) |
|
808 |
else: |
|
809 |
options.append('fulltext') |
|
810 |
# isinstance is slower and we have no hierarchy.
|
|
811 |
if self._factory.__class__ == KnitPlainFactory: |
|
812 |
# Use the already joined bytes saving iteration time in
|
|
813 |
# _record_to_data.
|
|
814 |
size, bytes = self._record_to_data(key, digest, |
|
815 |
lines, [line_bytes]) |
|
816 |
else: |
|
817 |
# get mixed annotation + content and feed it into the
|
|
818 |
# serialiser.
|
|
819 |
store_lines = self._factory.lower_fulltext(content) |
|
820 |
size, bytes = self._record_to_data(key, digest, |
|
821 |
store_lines) |
|
822 |
||
823 |
access_memo = self._access.add_raw_records([(key, size)], bytes)[0] |
|
824 |
self._index.add_records( |
|
825 |
((key, options, access_memo, parents),), |
|
826 |
random_id=random_id) |
|
827 |
return digest, text_length, content |
|
828 |
||
829 |
def annotate(self, key): |
|
830 |
"""See VersionedFiles.annotate."""
|
|
831 |
return self._factory.annotate(self, key) |
|
832 |
||
833 |
def check(self, progress_bar=None): |
|
834 |
"""See VersionedFiles.check()."""
|
|
835 |
# This doesn't actually test extraction of everything, but that will
|
|
836 |
# impact 'bzr check' substantially, and needs to be integrated with
|
|
837 |
# care. However, it does check for the obvious problem of a delta with
|
|
838 |
# no basis.
|
|
3517.4.14
by Martin Pool
KnitVersionedFiles.check should just check its own keys then recurse into fallbacks |
839 |
keys = self._index.keys() |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
840 |
parent_map = self.get_parent_map(keys) |
841 |
for key in keys: |
|
842 |
if self._index.get_method(key) != 'fulltext': |
|
843 |
compression_parent = parent_map[key][0] |
|
844 |
if compression_parent not in parent_map: |
|
845 |
raise errors.KnitCorrupt(self, |
|
846 |
"Missing basis parent %s for %s" % ( |
|
847 |
compression_parent, key)) |
|
3517.4.14
by Martin Pool
KnitVersionedFiles.check should just check its own keys then recurse into fallbacks |
848 |
for fallback_vfs in self._fallback_vfs: |
849 |
fallback_vfs.check() |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
850 |
|
851 |
def _check_add(self, key, lines, random_id, check_content): |
|
852 |
"""check that version_id and lines are safe to add."""
|
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
853 |
version_id = key[-1] |
854 |
if contains_whitespace(version_id): |
|
3517.3.1
by Andrew Bennetts
Fix error in error path. |
855 |
raise InvalidRevisionId(version_id, self) |
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
856 |
self.check_not_reserved_id(version_id) |
3350.6.11
by Martin Pool
Review cleanups and documentation from Robert's mail on 2080618 |
857 |
# TODO: If random_id==False and the key is already present, we should
|
858 |
# probably check that the existing content is identical to what is
|
|
859 |
# being inserted, and otherwise raise an exception. This would make
|
|
860 |
# the bundle code simpler.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
861 |
if check_content: |
862 |
self._check_lines_not_unicode(lines) |
|
863 |
self._check_lines_are_lines(lines) |
|
864 |
||
865 |
def _check_header(self, key, line): |
|
866 |
rec = self._split_header(line) |
|
867 |
self._check_header_version(rec, key[-1]) |
|
868 |
return rec |
|
869 |
||
870 |
def _check_header_version(self, rec, version_id): |
|
871 |
"""Checks the header version on original format knit records.
|
|
872 |
|
|
873 |
These have the last component of the key embedded in the record.
|
|
874 |
"""
|
|
875 |
if rec[1] != version_id: |
|
876 |
raise KnitCorrupt(self, |
|
877 |
'unexpected version, wanted %r, got %r' % (version_id, rec[1])) |
|
878 |
||
879 |
def _check_should_delta(self, parent): |
|
2147.1.1
by John Arbash Meinel
Factor the common knit delta selection into a helper func, and allow the fulltext to be chosen based on cumulative delta size |
880 |
"""Iterate back through the parent listing, looking for a fulltext.
|
881 |
||
882 |
This is used when we want to decide whether to add a delta or a new
|
|
883 |
fulltext. It searches for _max_delta_chain parents. When it finds a
|
|
884 |
fulltext parent, it sees if the total size of the deltas leading up to
|
|
885 |
it is large enough to indicate that we want a new full text anyway.
|
|
886 |
||
887 |
Return True if we should create a new delta, False if we should use a
|
|
888 |
full text.
|
|
889 |
"""
|
|
890 |
delta_size = 0 |
|
891 |
fulltext_size = None |
|
2147.1.2
by John Arbash Meinel
Simplify the knit max-chain detection code. |
892 |
for count in xrange(self._max_delta_chain): |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
893 |
# XXX: Collapse these two queries:
|
3350.8.9
by Robert Collins
define behaviour for add_lines with stacked storage. |
894 |
try: |
895 |
method = self._index.get_method(parent) |
|
896 |
except RevisionNotPresent: |
|
897 |
# Some basis is not locally present: always delta
|
|
898 |
return False |
|
2592.3.71
by Robert Collins
Basic version of knit-based repository operating, many tests failing. |
899 |
index, pos, size = self._index.get_position(parent) |
2147.1.1
by John Arbash Meinel
Factor the common knit delta selection into a helper func, and allow the fulltext to be chosen based on cumulative delta size |
900 |
if method == 'fulltext': |
901 |
fulltext_size = size |
|
902 |
break
|
|
903 |
delta_size += size |
|
3350.6.11
by Martin Pool
Review cleanups and documentation from Robert's mail on 2080618 |
904 |
# We don't explicitly check for presence because this is in an
|
905 |
# inner loop, and if it's missing it'll fail anyhow.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
906 |
# TODO: This should be asking for compression parent, not graph
|
907 |
# parent.
|
|
908 |
parent = self._index.get_parent_map([parent])[parent][0] |
|
2147.1.2
by John Arbash Meinel
Simplify the knit max-chain detection code. |
909 |
else: |
910 |
# We couldn't find a fulltext, so we must create a new one
|
|
2147.1.1
by John Arbash Meinel
Factor the common knit delta selection into a helper func, and allow the fulltext to be chosen based on cumulative delta size |
911 |
return False |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
912 |
# Simple heuristic - if the total I/O wold be greater as a delta than
|
913 |
# the originally installed fulltext, we create a new fulltext.
|
|
2147.1.2
by John Arbash Meinel
Simplify the knit max-chain detection code. |
914 |
return fulltext_size > delta_size |
2147.1.1
by John Arbash Meinel
Factor the common knit delta selection into a helper func, and allow the fulltext to be chosen based on cumulative delta size |
915 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
916 |
def _build_details_to_components(self, build_details): |
917 |
"""Convert a build_details tuple to a position tuple."""
|
|
918 |
# record_details, access_memo, compression_parent
|
|
919 |
return build_details[3], build_details[0], build_details[1] |
|
920 |
||
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
921 |
def _get_components_positions(self, keys, allow_missing=False): |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
922 |
"""Produce a map of position data for the components of keys.
|
923 |
||
924 |
This data is intended to be used for retrieving the knit records.
|
|
925 |
||
926 |
A dict of key to (record_details, index_memo, next, parents) is
|
|
927 |
returned.
|
|
928 |
method is the way referenced data should be applied.
|
|
929 |
index_memo is the handle to pass to the data access to actually get the
|
|
930 |
data
|
|
931 |
next is the build-parent of the version, or None for fulltexts.
|
|
932 |
parents is the version_ids of the parents of this version
|
|
933 |
||
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
934 |
:param allow_missing: If True do not raise an error on a missing component,
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
935 |
just ignore it.
|
936 |
"""
|
|
937 |
component_data = {} |
|
938 |
pending_components = keys |
|
939 |
while pending_components: |
|
940 |
build_details = self._index.get_build_details(pending_components) |
|
941 |
current_components = set(pending_components) |
|
942 |
pending_components = set() |
|
943 |
for key, details in build_details.iteritems(): |
|
944 |
(index_memo, compression_parent, parents, |
|
945 |
record_details) = details |
|
946 |
method = record_details[0] |
|
947 |
if compression_parent is not None: |
|
948 |
pending_components.add(compression_parent) |
|
949 |
component_data[key] = self._build_details_to_components(details) |
|
950 |
missing = current_components.difference(build_details) |
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
951 |
if missing and not allow_missing: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
952 |
raise errors.RevisionNotPresent(missing.pop(), self) |
953 |
return component_data |
|
954 |
||
955 |
def _get_content(self, key, parent_texts={}): |
|
956 |
"""Returns a content object that makes up the specified
|
|
957 |
version."""
|
|
958 |
cached_version = parent_texts.get(key, None) |
|
959 |
if cached_version is not None: |
|
960 |
# Ensure the cache dict is valid.
|
|
961 |
if not self.get_parent_map([key]): |
|
962 |
raise RevisionNotPresent(key, self) |
|
963 |
return cached_version |
|
964 |
text_map, contents_map = self._get_content_maps([key]) |
|
965 |
return contents_map[key] |
|
966 |
||
3350.8.7
by Robert Collins
get_record_stream for fulltexts working (but note extreme memory use!). |
967 |
def _get_content_maps(self, keys, nonlocal_keys=None): |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
968 |
"""Produce maps of text and KnitContents
|
969 |
|
|
3350.8.7
by Robert Collins
get_record_stream for fulltexts working (but note extreme memory use!). |
970 |
:param keys: The keys to produce content maps for.
|
971 |
:param nonlocal_keys: An iterable of keys(possibly intersecting keys)
|
|
972 |
which are known to not be in this knit, but rather in one of the
|
|
973 |
fallback knits.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
974 |
:return: (text_map, content_map) where text_map contains the texts for
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
975 |
the requested versions and content_map contains the KnitContents.
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
976 |
"""
|
977 |
# FUTURE: This function could be improved for the 'extract many' case
|
|
978 |
# by tracking each component and only doing the copy when the number of
|
|
979 |
# children than need to apply delta's to it is > 1 or it is part of the
|
|
980 |
# final output.
|
|
981 |
keys = list(keys) |
|
982 |
multiple_versions = len(keys) != 1 |
|
3350.8.7
by Robert Collins
get_record_stream for fulltexts working (but note extreme memory use!). |
983 |
record_map = self._get_record_map(keys, allow_missing=True) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
984 |
|
985 |
text_map = {} |
|
986 |
content_map = {} |
|
987 |
final_content = {} |
|
3350.8.7
by Robert Collins
get_record_stream for fulltexts working (but note extreme memory use!). |
988 |
if nonlocal_keys is None: |
989 |
nonlocal_keys = set() |
|
990 |
else: |
|
991 |
nonlocal_keys = frozenset(nonlocal_keys) |
|
992 |
missing_keys = set(nonlocal_keys) |
|
993 |
for source in self._fallback_vfs: |
|
994 |
if not missing_keys: |
|
995 |
break
|
|
996 |
for record in source.get_record_stream(missing_keys, |
|
997 |
'unordered', True): |
|
998 |
if record.storage_kind == 'absent': |
|
999 |
continue
|
|
1000 |
missing_keys.remove(record.key) |
|
1001 |
lines = split_lines(record.get_bytes_as('fulltext')) |
|
1002 |
text_map[record.key] = lines |
|
3350.8.10
by Robert Collins
Stacked insert_record_stream. |
1003 |
content_map[record.key] = PlainKnitContent(lines, record.key) |
1004 |
if record.key in keys: |
|
1005 |
final_content[record.key] = content_map[record.key] |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1006 |
for key in keys: |
3350.8.7
by Robert Collins
get_record_stream for fulltexts working (but note extreme memory use!). |
1007 |
if key in nonlocal_keys: |
1008 |
# already handled
|
|
1009 |
continue
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1010 |
components = [] |
1011 |
cursor = key |
|
1012 |
while cursor is not None: |
|
3350.8.10
by Robert Collins
Stacked insert_record_stream. |
1013 |
try: |
1014 |
record, record_details, digest, next = record_map[cursor] |
|
1015 |
except KeyError: |
|
1016 |
raise RevisionNotPresent(cursor, self) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1017 |
components.append((cursor, record, record_details, digest)) |
3350.8.10
by Robert Collins
Stacked insert_record_stream. |
1018 |
cursor = next |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1019 |
if cursor in content_map: |
3350.8.10
by Robert Collins
Stacked insert_record_stream. |
1020 |
# no need to plan further back
|
1021 |
components.append((cursor, None, None, None)) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1022 |
break
|
1023 |
||
1024 |
content = None |
|
1025 |
for (component_id, record, record_details, |
|
1026 |
digest) in reversed(components): |
|
1027 |
if component_id in content_map: |
|
1028 |
content = content_map[component_id] |
|
1029 |
else: |
|
1030 |
content, delta = self._factory.parse_record(key[-1], |
|
1031 |
record, record_details, content, |
|
1032 |
copy_base_content=multiple_versions) |
|
1033 |
if multiple_versions: |
|
1034 |
content_map[component_id] = content |
|
1035 |
||
1036 |
final_content[key] = content |
|
1037 |
||
1038 |
# digest here is the digest from the last applied component.
|
|
1039 |
text = content.text() |
|
1040 |
actual_sha = sha_strings(text) |
|
1041 |
if actual_sha != digest: |
|
1042 |
raise KnitCorrupt(self, |
|
1043 |
'\n sha-1 %s' |
|
1044 |
'\n of reconstructed text does not match' |
|
1045 |
'\n expected %s' |
|
1046 |
'\n for version %s' % |
|
1047 |
(actual_sha, digest, key)) |
|
1048 |
text_map[key] = text |
|
1049 |
return text_map, final_content |
|
1050 |
||
1051 |
def get_parent_map(self, keys): |
|
3517.4.17
by Martin Pool
Redo base Repository.get_parent_map to use .revisions graph |
1052 |
"""Get a map of the graph parents of keys.
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1053 |
|
1054 |
:param keys: The keys to look up parents for.
|
|
1055 |
:return: A mapping from keys to parents. Absent keys are absent from
|
|
1056 |
the mapping.
|
|
1057 |
"""
|
|
3350.8.14
by Robert Collins
Review feedback. |
1058 |
return self._get_parent_map_with_sources(keys)[0] |
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1059 |
|
3350.8.14
by Robert Collins
Review feedback. |
1060 |
def _get_parent_map_with_sources(self, keys): |
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1061 |
"""Get a map of the parents of keys.
|
1062 |
||
1063 |
:param keys: The keys to look up parents for.
|
|
1064 |
:return: A tuple. The first element is a mapping from keys to parents.
|
|
1065 |
Absent keys are absent from the mapping. The second element is a
|
|
1066 |
list with the locations each key was found in. The first element
|
|
1067 |
is the in-this-knit parents, the second the first fallback source,
|
|
1068 |
and so on.
|
|
1069 |
"""
|
|
3350.8.2
by Robert Collins
stacked get_parent_map. |
1070 |
result = {} |
1071 |
sources = [self._index] + self._fallback_vfs |
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1072 |
source_results = [] |
3350.8.2
by Robert Collins
stacked get_parent_map. |
1073 |
missing = set(keys) |
1074 |
for source in sources: |
|
1075 |
if not missing: |
|
1076 |
break
|
|
1077 |
new_result = source.get_parent_map(missing) |
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1078 |
source_results.append(new_result) |
3350.8.2
by Robert Collins
stacked get_parent_map. |
1079 |
result.update(new_result) |
1080 |
missing.difference_update(set(new_result)) |
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1081 |
return result, source_results |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1082 |
|
3350.8.3
by Robert Collins
VF.get_sha1s needed changing to be stackable. |
1083 |
def _get_record_map(self, keys, allow_missing=False): |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1084 |
"""Produce a dictionary of knit records.
|
1085 |
|
|
1086 |
:return: {key:(record, record_details, digest, next)}
|
|
1087 |
record
|
|
1088 |
data returned from read_records
|
|
1089 |
record_details
|
|
1090 |
opaque information to pass to parse_record
|
|
1091 |
digest
|
|
1092 |
SHA1 digest of the full text after all steps are done
|
|
1093 |
next
|
|
1094 |
build-parent of the version, i.e. the leftmost ancestor.
|
|
1095 |
Will be None if the record is not a delta.
|
|
3350.8.3
by Robert Collins
VF.get_sha1s needed changing to be stackable. |
1096 |
:param keys: The keys to build a map for
|
1097 |
:param allow_missing: If some records are missing, rather than
|
|
1098 |
error, just return the data that could be generated.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1099 |
"""
|
3350.8.3
by Robert Collins
VF.get_sha1s needed changing to be stackable. |
1100 |
position_map = self._get_components_positions(keys, |
3350.8.13
by Robert Collins
Merge bzr.dev, fixing minor skew. |
1101 |
allow_missing=allow_missing) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1102 |
# key = component_id, r = record_details, i_m = index_memo, n = next
|
1103 |
records = [(key, i_m) for key, (r, i_m, n) |
|
1104 |
in position_map.iteritems()] |
|
1105 |
record_map = {} |
|
1106 |
for key, record, digest in \ |
|
1107 |
self._read_records_iter(records): |
|
1108 |
(record_details, index_memo, next) = position_map[key] |
|
1109 |
record_map[key] = record, record_details, digest, next |
|
1110 |
return record_map |
|
1111 |
||
1112 |
def get_record_stream(self, keys, ordering, include_delta_closure): |
|
1113 |
"""Get a stream of records for keys.
|
|
1114 |
||
1115 |
:param keys: The keys to include.
|
|
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
1116 |
:param ordering: Either 'unordered' or 'topological'. A topologically
|
1117 |
sorted stream has compression parents strictly before their
|
|
1118 |
children.
|
|
1119 |
:param include_delta_closure: If True then the closure across any
|
|
1120 |
compression parents will be included (in the opaque data).
|
|
1121 |
:return: An iterator of ContentFactory objects, each of which is only
|
|
1122 |
valid until the iterator is advanced.
|
|
1123 |
"""
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1124 |
# keys might be a generator
|
1125 |
keys = set(keys) |
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1126 |
if not keys: |
1127 |
return
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1128 |
if not self._index.has_graph: |
1129 |
# Cannot topological order when no graph has been stored.
|
|
1130 |
ordering = 'unordered' |
|
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
1131 |
if include_delta_closure: |
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1132 |
positions = self._get_components_positions(keys, allow_missing=True) |
3350.3.3
by Robert Collins
Functional get_record_stream interface tests covering full interface. |
1133 |
else: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1134 |
build_details = self._index.get_build_details(keys) |
3350.6.11
by Martin Pool
Review cleanups and documentation from Robert's mail on 2080618 |
1135 |
# map from key to
|
1136 |
# (record_details, access_memo, compression_parent_key)
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1137 |
positions = dict((key, self._build_details_to_components(details)) |
1138 |
for key, details in build_details.iteritems()) |
|
1139 |
absent_keys = keys.difference(set(positions)) |
|
1140 |
# There may be more absent keys : if we're missing the basis component
|
|
1141 |
# and are trying to include the delta closure.
|
|
1142 |
if include_delta_closure: |
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1143 |
needed_from_fallback = set() |
3350.6.11
by Martin Pool
Review cleanups and documentation from Robert's mail on 2080618 |
1144 |
# Build up reconstructable_keys dict. key:True in this dict means
|
1145 |
# the key can be reconstructed.
|
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1146 |
reconstructable_keys = {} |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1147 |
for key in keys: |
1148 |
# the delta chain
|
|
1149 |
try: |
|
1150 |
chain = [key, positions[key][2]] |
|
1151 |
except KeyError: |
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1152 |
needed_from_fallback.add(key) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1153 |
continue
|
1154 |
result = True |
|
1155 |
while chain[-1] is not None: |
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1156 |
if chain[-1] in reconstructable_keys: |
1157 |
result = reconstructable_keys[chain[-1]] |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1158 |
break
|
1159 |
else: |
|
1160 |
try: |
|
1161 |
chain.append(positions[chain[-1]][2]) |
|
1162 |
except KeyError: |
|
1163 |
# missing basis component
|
|
3350.8.10
by Robert Collins
Stacked insert_record_stream. |
1164 |
needed_from_fallback.add(chain[-1]) |
1165 |
result = True |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1166 |
break
|
1167 |
for chain_key in chain[:-1]: |
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1168 |
reconstructable_keys[chain_key] = result |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1169 |
if not result: |
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1170 |
needed_from_fallback.add(key) |
1171 |
# Double index lookups here : need a unified api ?
|
|
3350.8.14
by Robert Collins
Review feedback. |
1172 |
global_map, parent_maps = self._get_parent_map_with_sources(keys) |
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1173 |
if ordering == 'topological': |
1174 |
# Global topological sort
|
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
1175 |
present_keys = tsort.topo_sort(global_map) |
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1176 |
# Now group by source:
|
1177 |
source_keys = [] |
|
1178 |
current_source = None |
|
1179 |
for key in present_keys: |
|
1180 |
for parent_map in parent_maps: |
|
1181 |
if key in parent_map: |
|
1182 |
key_source = parent_map |
|
1183 |
break
|
|
1184 |
if current_source is not key_source: |
|
1185 |
source_keys.append((key_source, [])) |
|
1186 |
current_source = key_source |
|
1187 |
source_keys[-1][1].append(key) |
|
1188 |
else: |
|
1189 |
# Just group by source; remote sources first.
|
|
1190 |
present_keys = [] |
|
1191 |
source_keys = [] |
|
1192 |
for parent_map in reversed(parent_maps): |
|
1193 |
source_keys.append((parent_map, [])) |
|
1194 |
for key in parent_map: |
|
1195 |
present_keys.append(key) |
|
1196 |
source_keys[-1][1].append(key) |
|
1197 |
absent_keys = keys - set(global_map) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1198 |
for key in absent_keys: |
1199 |
yield AbsentContentFactory(key) |
|
1200 |
# restrict our view to the keys we can answer.
|
|
1201 |
# XXX: Memory: TODO: batch data here to cap buffered data at (say) 1MB.
|
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1202 |
# XXX: At that point we need to consider the impact of double reads by
|
1203 |
# utilising components multiple times.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1204 |
if include_delta_closure: |
1205 |
# XXX: get_content_maps performs its own index queries; allow state
|
|
1206 |
# to be passed in.
|
|
3350.8.7
by Robert Collins
get_record_stream for fulltexts working (but note extreme memory use!). |
1207 |
text_map, _ = self._get_content_maps(present_keys, |
1208 |
needed_from_fallback - absent_keys) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1209 |
for key in present_keys: |
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1210 |
yield FulltextContentFactory(key, global_map[key], None, |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1211 |
''.join(text_map[key])) |
1212 |
else: |
|
3350.8.6
by Robert Collins
get_record_stream stacking for delta access. |
1213 |
for source, keys in source_keys: |
1214 |
if source is parent_maps[0]: |
|
1215 |
# this KnitVersionedFiles
|
|
1216 |
records = [(key, positions[key][1]) for key in keys] |
|
1217 |
for key, raw_data, sha1 in self._read_records_iter_raw(records): |
|
1218 |
(record_details, index_memo, _) = positions[key] |
|
1219 |
yield KnitContentFactory(key, global_map[key], |
|
1220 |
record_details, sha1, raw_data, self._factory.annotated, None) |
|
1221 |
else: |
|
1222 |
vf = self._fallback_vfs[parent_maps.index(source) - 1] |
|
1223 |
for record in vf.get_record_stream(keys, ordering, |
|
1224 |
include_delta_closure): |
|
1225 |
yield record |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1226 |
|
1227 |
def get_sha1s(self, keys): |
|
1228 |
"""See VersionedFiles.get_sha1s()."""
|
|
3350.8.3
by Robert Collins
VF.get_sha1s needed changing to be stackable. |
1229 |
missing = set(keys) |
1230 |
record_map = self._get_record_map(missing, allow_missing=True) |
|
1231 |
result = {} |
|
1232 |
for key, details in record_map.iteritems(): |
|
1233 |
if key not in missing: |
|
1234 |
continue
|
|
1235 |
# record entry 2 is the 'digest'.
|
|
1236 |
result[key] = details[2] |
|
1237 |
missing.difference_update(set(result)) |
|
1238 |
for source in self._fallback_vfs: |
|
1239 |
if not missing: |
|
1240 |
break
|
|
1241 |
new_result = source.get_sha1s(missing) |
|
1242 |
result.update(new_result) |
|
1243 |
missing.difference_update(set(new_result)) |
|
1244 |
return result |
|
3052.2.2
by Robert Collins
* Operations pulling data from a smart server where the underlying |
1245 |
|
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1246 |
def insert_record_stream(self, stream): |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1247 |
"""Insert a record stream into this container.
|
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1248 |
|
1249 |
:param stream: A stream of records to insert.
|
|
1250 |
:return: None
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1251 |
:seealso VersionedFiles.get_record_stream:
|
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1252 |
"""
|
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1253 |
def get_adapter(adapter_key): |
1254 |
try: |
|
1255 |
return adapters[adapter_key] |
|
1256 |
except KeyError: |
|
1257 |
adapter_factory = adapter_registry.get(adapter_key) |
|
1258 |
adapter = adapter_factory(self) |
|
1259 |
adapters[adapter_key] = adapter |
|
1260 |
return adapter |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1261 |
if self._factory.annotated: |
3350.3.11
by Robert Collins
Test inserting a stream that overlaps the current content of a knit does not error. |
1262 |
# self is annotated, we need annotated knits to use directly.
|
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1263 |
annotated = "annotated-" |
3350.3.11
by Robert Collins
Test inserting a stream that overlaps the current content of a knit does not error. |
1264 |
convertibles = [] |
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1265 |
else: |
3350.3.11
by Robert Collins
Test inserting a stream that overlaps the current content of a knit does not error. |
1266 |
# self is not annotated, but we can strip annotations cheaply.
|
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1267 |
annotated = "" |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1268 |
convertibles = set(["knit-annotated-ft-gz"]) |
1269 |
if self._max_delta_chain: |
|
1270 |
convertibles.add("knit-annotated-delta-gz") |
|
3350.3.22
by Robert Collins
Review feedback. |
1271 |
# The set of types we can cheaply adapt without needing basis texts.
|
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1272 |
native_types = set() |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1273 |
if self._max_delta_chain: |
1274 |
native_types.add("knit-%sdelta-gz" % annotated) |
|
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1275 |
native_types.add("knit-%sft-gz" % annotated) |
3350.3.11
by Robert Collins
Test inserting a stream that overlaps the current content of a knit does not error. |
1276 |
knit_types = native_types.union(convertibles) |
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1277 |
adapters = {} |
3350.3.22
by Robert Collins
Review feedback. |
1278 |
# Buffer all index entries that we can't add immediately because their
|
3350.3.17
by Robert Collins
Prevent corrupt knits being created when a stream is interrupted with basis parents not present. |
1279 |
# basis parent is missing. We don't buffer all because generating
|
1280 |
# annotations may require access to some of the new records. However we
|
|
1281 |
# can't generate annotations from new deltas until their basis parent
|
|
1282 |
# is present anyway, so we get away with not needing an index that
|
|
3350.3.22
by Robert Collins
Review feedback. |
1283 |
# includes the new keys.
|
3350.3.17
by Robert Collins
Prevent corrupt knits being created when a stream is interrupted with basis parents not present. |
1284 |
# key = basis_parent, value = index entry to add
|
1285 |
buffered_index_entries = {} |
|
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1286 |
for record in stream: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1287 |
parents = record.parents |
3350.3.15
by Robert Collins
Update the insert_record_stream contract to error if an absent record is provided. |
1288 |
# Raise an error when a record is missing.
|
1289 |
if record.storage_kind == 'absent': |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1290 |
raise RevisionNotPresent([record.key], self) |
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1291 |
if record.storage_kind in knit_types: |
1292 |
if record.storage_kind not in native_types: |
|
1293 |
try: |
|
1294 |
adapter_key = (record.storage_kind, "knit-delta-gz") |
|
1295 |
adapter = get_adapter(adapter_key) |
|
1296 |
except KeyError: |
|
1297 |
adapter_key = (record.storage_kind, "knit-ft-gz") |
|
1298 |
adapter = get_adapter(adapter_key) |
|
1299 |
bytes = adapter.get_bytes( |
|
1300 |
record, record.get_bytes_as(record.storage_kind)) |
|
1301 |
else: |
|
1302 |
bytes = record.get_bytes_as(record.storage_kind) |
|
1303 |
options = [record._build_details[0]] |
|
1304 |
if record._build_details[1]: |
|
1305 |
options.append('no-eol') |
|
3350.3.11
by Robert Collins
Test inserting a stream that overlaps the current content of a knit does not error. |
1306 |
# Just blat it across.
|
1307 |
# Note: This does end up adding data on duplicate keys. As
|
|
1308 |
# modern repositories use atomic insertions this should not
|
|
1309 |
# lead to excessive growth in the event of interrupted fetches.
|
|
1310 |
# 'knit' repositories may suffer excessive growth, but as a
|
|
1311 |
# deprecated format this is tolerable. It can be fixed if
|
|
1312 |
# needed by in the kndx index support raising on a duplicate
|
|
1313 |
# add with identical parents and options.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1314 |
access_memo = self._access.add_raw_records( |
1315 |
[(record.key, len(bytes))], bytes)[0] |
|
1316 |
index_entry = (record.key, options, access_memo, parents) |
|
3350.3.17
by Robert Collins
Prevent corrupt knits being created when a stream is interrupted with basis parents not present. |
1317 |
buffered = False |
1318 |
if 'fulltext' not in options: |
|
1319 |
basis_parent = parents[0] |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1320 |
# Note that pack backed knits don't need to buffer here
|
1321 |
# because they buffer all writes to the transaction level,
|
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1322 |
# but we don't expose that difference at the index level. If
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1323 |
# the query here has sufficient cost to show up in
|
1324 |
# profiling we should do that.
|
|
1325 |
if basis_parent not in self.get_parent_map([basis_parent]): |
|
3350.3.17
by Robert Collins
Prevent corrupt knits being created when a stream is interrupted with basis parents not present. |
1326 |
pending = buffered_index_entries.setdefault( |
1327 |
basis_parent, []) |
|
1328 |
pending.append(index_entry) |
|
1329 |
buffered = True |
|
1330 |
if not buffered: |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1331 |
self._index.add_records([index_entry]) |
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1332 |
elif record.storage_kind == 'fulltext': |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1333 |
self.add_lines(record.key, parents, |
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1334 |
split_lines(record.get_bytes_as('fulltext'))) |
1335 |
else: |
|
1336 |
adapter_key = record.storage_kind, 'fulltext' |
|
3350.3.9
by Robert Collins
Avoid full text reconstruction when transferring knit to knit via record streams. |
1337 |
adapter = get_adapter(adapter_key) |
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1338 |
lines = split_lines(adapter.get_bytes( |
1339 |
record, record.get_bytes_as(record.storage_kind))) |
|
3350.3.11
by Robert Collins
Test inserting a stream that overlaps the current content of a knit does not error. |
1340 |
try: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1341 |
self.add_lines(record.key, parents, lines) |
3350.3.11
by Robert Collins
Test inserting a stream that overlaps the current content of a knit does not error. |
1342 |
except errors.RevisionAlreadyPresent: |
1343 |
pass
|
|
3350.3.17
by Robert Collins
Prevent corrupt knits being created when a stream is interrupted with basis parents not present. |
1344 |
# Add any records whose basis parent is now available.
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1345 |
added_keys = [record.key] |
3350.3.17
by Robert Collins
Prevent corrupt knits being created when a stream is interrupted with basis parents not present. |
1346 |
while added_keys: |
1347 |
key = added_keys.pop(0) |
|
1348 |
if key in buffered_index_entries: |
|
1349 |
index_entries = buffered_index_entries[key] |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1350 |
self._index.add_records(index_entries) |
3350.3.17
by Robert Collins
Prevent corrupt knits being created when a stream is interrupted with basis parents not present. |
1351 |
added_keys.extend( |
1352 |
[index_entry[0] for index_entry in index_entries]) |
|
1353 |
del buffered_index_entries[key] |
|
1354 |
# If there were any deltas which had a missing basis parent, error.
|
|
1355 |
if buffered_index_entries: |
|
1356 |
raise errors.RevisionNotPresent(buffered_index_entries.keys()[0], |
|
1357 |
self) |
|
3350.3.8
by Robert Collins
Basic stream insertion, no fast path yet for knit to knit. |
1358 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1359 |
def iter_lines_added_or_present_in_keys(self, keys, pb=None): |
1360 |
"""Iterate over the lines in the versioned files from keys.
|
|
1361 |
||
1362 |
This may return lines from other keys. Each item the returned
|
|
1363 |
iterator yields is a tuple of a line and a text version that that line
|
|
1364 |
is present in (not introduced in).
|
|
1365 |
||
1366 |
Ordering of results is in whatever order is most suitable for the
|
|
1367 |
underlying storage format.
|
|
1368 |
||
1369 |
If a progress bar is supplied, it may be used to indicate progress.
|
|
1370 |
The caller is responsible for cleaning up progress bars (because this
|
|
1371 |
is an iterator).
|
|
1372 |
||
1373 |
NOTES:
|
|
1374 |
* Lines are normalised by the underlying store: they will all have \n
|
|
1375 |
terminators.
|
|
1376 |
* Lines are returned in arbitrary order.
|
|
1377 |
||
1378 |
:return: An iterator over (line, key).
|
|
1379 |
"""
|
|
1380 |
if pb is None: |
|
1381 |
pb = progress.DummyProgress() |
|
1382 |
keys = set(keys) |
|
3350.8.5
by Robert Collins
Iter_lines_added_or_present_in_keys stacks. |
1383 |
total = len(keys) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1384 |
# we don't care about inclusions, the caller cares.
|
1385 |
# but we need to setup a list of records to visit.
|
|
1386 |
# we need key, position, length
|
|
1387 |
key_records = [] |
|
1388 |
build_details = self._index.get_build_details(keys) |
|
3350.8.5
by Robert Collins
Iter_lines_added_or_present_in_keys stacks. |
1389 |
for key, details in build_details.iteritems(): |
1390 |
if key in keys: |
|
1391 |
key_records.append((key, details[0])) |
|
1392 |
keys.remove(key) |
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1393 |
records_iter = enumerate(self._read_records_iter(key_records)) |
1394 |
for (key_idx, (key, data, sha_value)) in records_iter: |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1395 |
pb.update('Walking content.', key_idx, total) |
1396 |
compression_parent = build_details[key][1] |
|
1397 |
if compression_parent is None: |
|
1398 |
# fulltext
|
|
1399 |
line_iterator = self._factory.get_fulltext_content(data) |
|
1400 |
else: |
|
1401 |
# Delta
|
|
1402 |
line_iterator = self._factory.get_linedelta_content(data) |
|
1403 |
# XXX: It might be more efficient to yield (key,
|
|
1404 |
# line_iterator) in the future. However for now, this is a simpler
|
|
1405 |
# change to integrate into the rest of the codebase. RBC 20071110
|
|
1406 |
for line in line_iterator: |
|
1407 |
yield line, key |
|
3350.8.5
by Robert Collins
Iter_lines_added_or_present_in_keys stacks. |
1408 |
for source in self._fallback_vfs: |
1409 |
if not keys: |
|
1410 |
break
|
|
1411 |
source_keys = set() |
|
1412 |
for line, key in source.iter_lines_added_or_present_in_keys(keys): |
|
1413 |
source_keys.add(key) |
|
1414 |
yield line, key |
|
1415 |
keys.difference_update(source_keys) |
|
1416 |
if keys: |
|
1417 |
raise RevisionNotPresent(keys, self.filename) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1418 |
pb.update('Walking content.', total, total) |
1419 |
||
1420 |
def _make_line_delta(self, delta_seq, new_content): |
|
1421 |
"""Generate a line delta from delta_seq and new_content."""
|
|
1422 |
diff_hunks = [] |
|
1423 |
for op in delta_seq.get_opcodes(): |
|
1424 |
if op[0] == 'equal': |
|
1425 |
continue
|
|
1426 |
diff_hunks.append((op[1], op[2], op[4]-op[3], new_content._lines[op[3]:op[4]])) |
|
1427 |
return diff_hunks |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
1428 |
|
1596.2.34
by Robert Collins
Optimise knit add to only diff once per parent, not once per parent + once for the delta generation. |
1429 |
def _merge_annotations(self, content, parents, parent_texts={}, |
2520.4.140
by Aaron Bentley
Use matching blocks from mpdiff for knit delta creation |
1430 |
delta=None, annotated=None, |
1431 |
left_matching_blocks=None): |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1432 |
"""Merge annotations for content and generate deltas.
|
1433 |
|
|
1434 |
This is done by comparing the annotations based on changes to the text
|
|
1435 |
and generating a delta on the resulting full texts. If annotations are
|
|
1436 |
not being created then a simple delta is created.
|
|
1596.2.27
by Robert Collins
Note potential improvements in knit adds. |
1437 |
"""
|
2520.4.146
by Aaron Bentley
Avoid get_matching_blocks for un-annotated text |
1438 |
if left_matching_blocks is not None: |
1439 |
delta_seq = diff._PrematchedMatcher(left_matching_blocks) |
|
1440 |
else: |
|
1441 |
delta_seq = None |
|
1596.2.34
by Robert Collins
Optimise knit add to only diff once per parent, not once per parent + once for the delta generation. |
1442 |
if annotated: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1443 |
for parent_key in parents: |
1444 |
merge_content = self._get_content(parent_key, parent_texts) |
|
1445 |
if (parent_key == parents[0] and delta_seq is not None): |
|
2520.4.146
by Aaron Bentley
Avoid get_matching_blocks for un-annotated text |
1446 |
seq = delta_seq |
2520.4.140
by Aaron Bentley
Use matching blocks from mpdiff for knit delta creation |
1447 |
else: |
1448 |
seq = patiencediff.PatienceSequenceMatcher( |
|
1449 |
None, merge_content.text(), content.text()) |
|
1596.2.34
by Robert Collins
Optimise knit add to only diff once per parent, not once per parent + once for the delta generation. |
1450 |
for i, j, n in seq.get_matching_blocks(): |
1451 |
if n == 0: |
|
1452 |
continue
|
|
3460.2.1
by Robert Collins
* Inserting a bundle which changes the contents of a file with no trailing |
1453 |
# this copies (origin, text) pairs across to the new
|
1454 |
# content for any line that matches the last-checked
|
|
2520.4.146
by Aaron Bentley
Avoid get_matching_blocks for un-annotated text |
1455 |
# parent.
|
1596.2.34
by Robert Collins
Optimise knit add to only diff once per parent, not once per parent + once for the delta generation. |
1456 |
content._lines[j:j+n] = merge_content._lines[i:i+n] |
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1457 |
# XXX: Robert says the following block is a workaround for a
|
1458 |
# now-fixed bug and it can probably be deleted. -- mbp 20080618
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1459 |
if content._lines and content._lines[-1][1][-1] != '\n': |
1460 |
# The copied annotation was from a line without a trailing EOL,
|
|
1461 |
# reinstate one for the content object, to ensure correct
|
|
1462 |
# serialization.
|
|
1463 |
line = content._lines[-1][1] + '\n' |
|
1464 |
content._lines[-1] = (content._lines[-1][0], line) |
|
1596.2.36
by Robert Collins
add a get_delta api to versioned_file. |
1465 |
if delta: |
2520.4.146
by Aaron Bentley
Avoid get_matching_blocks for un-annotated text |
1466 |
if delta_seq is None: |
1596.2.36
by Robert Collins
add a get_delta api to versioned_file. |
1467 |
reference_content = self._get_content(parents[0], parent_texts) |
1468 |
new_texts = content.text() |
|
1469 |
old_texts = reference_content.text() |
|
2104.4.2
by John Arbash Meinel
Small cleanup and NEWS entry about fixing bug #65714 |
1470 |
delta_seq = patiencediff.PatienceSequenceMatcher( |
2100.2.1
by wang
Replace python's difflib by patiencediff because the worst case |
1471 |
None, old_texts, new_texts) |
1596.2.36
by Robert Collins
add a get_delta api to versioned_file. |
1472 |
return self._make_line_delta(delta_seq, content) |
1473 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1474 |
def _parse_record(self, version_id, data): |
1475 |
"""Parse an original format knit record.
|
|
1476 |
||
1477 |
These have the last element of the key only present in the stored data.
|
|
1478 |
"""
|
|
1479 |
rec, record_contents = self._parse_record_unchecked(data) |
|
1480 |
self._check_header_version(rec, version_id) |
|
1481 |
return record_contents, rec[3] |
|
1482 |
||
1483 |
def _parse_record_header(self, key, raw_data): |
|
1484 |
"""Parse a record header for consistency.
|
|
1485 |
||
1486 |
:return: the header and the decompressor stream.
|
|
1487 |
as (stream, header_record)
|
|
1488 |
"""
|
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
1489 |
df = tuned_gzip.GzipFile(mode='rb', fileobj=StringIO(raw_data)) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1490 |
try: |
1491 |
# Current serialise
|
|
1492 |
rec = self._check_header(key, df.readline()) |
|
1493 |
except Exception, e: |
|
1494 |
raise KnitCorrupt(self, |
|
1495 |
"While reading {%s} got %s(%s)" |
|
1496 |
% (key, e.__class__.__name__, str(e))) |
|
1497 |
return df, rec |
|
1498 |
||
1499 |
def _parse_record_unchecked(self, data): |
|
1500 |
# profiling notes:
|
|
1501 |
# 4168 calls in 2880 217 internal
|
|
1502 |
# 4168 calls to _parse_record_header in 2121
|
|
1503 |
# 4168 calls to readlines in 330
|
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
1504 |
df = tuned_gzip.GzipFile(mode='rb', fileobj=StringIO(data)) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1505 |
try: |
1506 |
record_contents = df.readlines() |
|
1507 |
except Exception, e: |
|
1508 |
raise KnitCorrupt(self, "Corrupt compressed record %r, got %s(%s)" % |
|
1509 |
(data, e.__class__.__name__, str(e))) |
|
1510 |
header = record_contents.pop(0) |
|
1511 |
rec = self._split_header(header) |
|
1512 |
last_line = record_contents.pop() |
|
1513 |
if len(record_contents) != int(rec[2]): |
|
1514 |
raise KnitCorrupt(self, |
|
1515 |
'incorrect number of lines %s != %s' |
|
1516 |
' for version {%s} %s' |
|
1517 |
% (len(record_contents), int(rec[2]), |
|
1518 |
rec[1], record_contents)) |
|
1519 |
if last_line != 'end %s\n' % rec[1]: |
|
1520 |
raise KnitCorrupt(self, |
|
1521 |
'unexpected version end line %r, wanted %r' |
|
1522 |
% (last_line, rec[1])) |
|
1523 |
df.close() |
|
1524 |
return rec, record_contents |
|
1525 |
||
1526 |
def _read_records_iter(self, records): |
|
1527 |
"""Read text records from data file and yield result.
|
|
1528 |
||
1529 |
The result will be returned in whatever is the fastest to read.
|
|
1530 |
Not by the order requested. Also, multiple requests for the same
|
|
1531 |
record will only yield 1 response.
|
|
1532 |
:param records: A list of (key, access_memo) entries
|
|
1533 |
:return: Yields (key, contents, digest) in the order
|
|
1534 |
read, not the order requested
|
|
1535 |
"""
|
|
1536 |
if not records: |
|
1537 |
return
|
|
1538 |
||
1539 |
# XXX: This smells wrong, IO may not be getting ordered right.
|
|
1540 |
needed_records = sorted(set(records), key=operator.itemgetter(1)) |
|
1541 |
if not needed_records: |
|
1542 |
return
|
|
1543 |
||
1544 |
# The transport optimizes the fetching as well
|
|
1545 |
# (ie, reads continuous ranges.)
|
|
1546 |
raw_data = self._access.get_raw_records( |
|
1547 |
[index_memo for key, index_memo in needed_records]) |
|
1548 |
||
1549 |
for (key, index_memo), data in \ |
|
1550 |
izip(iter(needed_records), raw_data): |
|
1551 |
content, digest = self._parse_record(key[-1], data) |
|
1552 |
yield key, content, digest |
|
1553 |
||
1554 |
def _read_records_iter_raw(self, records): |
|
1555 |
"""Read text records from data file and yield raw data.
|
|
1556 |
||
1557 |
This unpacks enough of the text record to validate the id is
|
|
1558 |
as expected but thats all.
|
|
1559 |
||
1560 |
Each item the iterator yields is (key, bytes, sha1_of_full_text).
|
|
1561 |
"""
|
|
1562 |
# setup an iterator of the external records:
|
|
1563 |
# uses readv so nice and fast we hope.
|
|
1564 |
if len(records): |
|
1565 |
# grab the disk data needed.
|
|
1566 |
needed_offsets = [index_memo for key, index_memo |
|
1567 |
in records] |
|
1568 |
raw_records = self._access.get_raw_records(needed_offsets) |
|
1569 |
||
1570 |
for key, index_memo in records: |
|
1571 |
data = raw_records.next() |
|
1572 |
# validate the header (note that we can only use the suffix in
|
|
1573 |
# current knit records).
|
|
1574 |
df, rec = self._parse_record_header(key, data) |
|
1575 |
df.close() |
|
1576 |
yield key, data, rec[3] |
|
1577 |
||
1578 |
def _record_to_data(self, key, digest, lines, dense_lines=None): |
|
1579 |
"""Convert key, digest, lines into a raw data block.
|
|
1580 |
|
|
1581 |
:param key: The key of the record. Currently keys are always serialised
|
|
1582 |
using just the trailing component.
|
|
1583 |
:param dense_lines: The bytes of lines but in a denser form. For
|
|
1584 |
instance, if lines is a list of 1000 bytestrings each ending in \n,
|
|
1585 |
dense_lines may be a list with one line in it, containing all the
|
|
1586 |
1000's lines and their \n's. Using dense_lines if it is already
|
|
1587 |
known is a win because the string join to create bytes in this
|
|
1588 |
function spends less time resizing the final string.
|
|
1589 |
:return: (len, a StringIO instance with the raw data ready to read.)
|
|
1590 |
"""
|
|
1591 |
# Note: using a string copy here increases memory pressure with e.g.
|
|
1592 |
# ISO's, but it is about 3 seconds faster on a 1.2Ghz intel machine
|
|
1593 |
# when doing the initial commit of a mozilla tree. RBC 20070921
|
|
1594 |
bytes = ''.join(chain( |
|
1595 |
["version %s %d %s\n" % (key[-1], |
|
1596 |
len(lines), |
|
1597 |
digest)], |
|
1598 |
dense_lines or lines, |
|
1599 |
["end %s\n" % key[-1]])) |
|
1600 |
if type(bytes) != str: |
|
1601 |
raise AssertionError( |
|
1602 |
'data must be plain bytes was %s' % type(bytes)) |
|
1603 |
if lines and lines[-1][-1] != '\n': |
|
1604 |
raise ValueError('corrupt lines value %r' % lines) |
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
1605 |
compressed_bytes = tuned_gzip.bytes_to_gzip(bytes) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1606 |
return len(compressed_bytes), compressed_bytes |
1607 |
||
1608 |
def _split_header(self, line): |
|
1609 |
rec = line.split() |
|
1610 |
if len(rec) != 4: |
|
1611 |
raise KnitCorrupt(self, |
|
1612 |
'unexpected number of elements in record header') |
|
1613 |
return rec |
|
1614 |
||
1615 |
def keys(self): |
|
1616 |
"""See VersionedFiles.keys."""
|
|
1617 |
if 'evil' in debug.debug_flags: |
|
1618 |
trace.mutter_callsite(2, "keys scales with size of history") |
|
3350.8.4
by Robert Collins
Vf.keys() stacking support. |
1619 |
sources = [self._index] + self._fallback_vfs |
1620 |
result = set() |
|
1621 |
for source in sources: |
|
1622 |
result.update(source.keys()) |
|
1623 |
return result |
|
1624 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1625 |
|
1626 |
||
1627 |
class _KndxIndex(object): |
|
1628 |
"""Manages knit index files
|
|
1629 |
||
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1630 |
The index is kept in memory and read on startup, to enable
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
1631 |
fast lookups of revision information. The cursor of the index
|
1632 |
file is always pointing to the end, making it easy to append
|
|
1633 |
entries.
|
|
1634 |
||
1635 |
_cache is a cache for fast mapping from version id to a Index
|
|
1636 |
object.
|
|
1637 |
||
1638 |
_history is a cache for fast mapping from indexes to version ids.
|
|
1639 |
||
1640 |
The index data format is dictionary compressed when it comes to
|
|
1641 |
parent references; a index entry may only have parents that with a
|
|
1642 |
lover index number. As a result, the index is topological sorted.
|
|
1563.2.11
by Robert Collins
Consolidate reweave and join as we have no separate usage, make reweave tests apply to all versionedfile implementations and deprecate the old reweave apis. |
1643 |
|
1644 |
Duplicate entries may be written to the index for a single version id
|
|
1645 |
if this is done then the latter one completely replaces the former:
|
|
1646 |
this allows updates to correct version and parent information.
|
|
1647 |
Note that the two entries may share the delta, and that successive
|
|
1648 |
annotations and references MUST point to the first entry.
|
|
1641.1.2
by Robert Collins
Change knit index files to be robust in the presence of partial writes. |
1649 |
|
1650 |
The index file on disc contains a header, followed by one line per knit
|
|
1651 |
record. The same revision can be present in an index file more than once.
|
|
1759.2.1
by Jelmer Vernooij
Fix some types (found using aspell). |
1652 |
The first occurrence gets assigned a sequence number starting from 0.
|
1641.1.2
by Robert Collins
Change knit index files to be robust in the presence of partial writes. |
1653 |
|
1654 |
The format of a single line is
|
|
1655 |
REVISION_ID FLAGS BYTE_OFFSET LENGTH( PARENT_ID|PARENT_SEQUENCE_ID)* :\n
|
|
1656 |
REVISION_ID is a utf8-encoded revision id
|
|
1657 |
FLAGS is a comma separated list of flags about the record. Values include
|
|
1658 |
no-eol, line-delta, fulltext.
|
|
1659 |
BYTE_OFFSET is the ascii representation of the byte offset in the data file
|
|
1660 |
that the the compressed data starts at.
|
|
1661 |
LENGTH is the ascii representation of the length of the data file.
|
|
1662 |
PARENT_ID a utf-8 revision id prefixed by a '.' that is a parent of
|
|
1663 |
REVISION_ID.
|
|
1664 |
PARENT_SEQUENCE_ID the ascii representation of the sequence number of a
|
|
1665 |
revision id already in the knit that is a parent of REVISION_ID.
|
|
1666 |
The ' :' marker is the end of record marker.
|
|
1667 |
|
|
1668 |
partial writes:
|
|
2158.3.1
by Dmitry Vasiliev
KnitIndex tests/fixes/optimizations |
1669 |
when a write is interrupted to the index file, it will result in a line
|
1670 |
that does not end in ' :'. If the ' :' is not present at the end of a line,
|
|
1671 |
or at the end of the file, then the record that is missing it will be
|
|
1672 |
ignored by the parser.
|
|
1641.1.2
by Robert Collins
Change knit index files to be robust in the presence of partial writes. |
1673 |
|
1759.2.1
by Jelmer Vernooij
Fix some types (found using aspell). |
1674 |
When writing new records to the index file, the data is preceded by '\n'
|
1641.1.2
by Robert Collins
Change knit index files to be robust in the presence of partial writes. |
1675 |
to ensure that records always start on new lines even if the last write was
|
1676 |
interrupted. As a result its normal for the last line in the index to be
|
|
1677 |
missing a trailing newline. One can be added with no harmful effects.
|
|
3350.6.11
by Martin Pool
Review cleanups and documentation from Robert's mail on 2080618 |
1678 |
|
1679 |
:ivar _kndx_cache: dict from prefix to the old state of KnitIndex objects,
|
|
1680 |
where prefix is e.g. the (fileid,) for .texts instances or () for
|
|
1681 |
constant-mapped things like .revisions, and the old state is
|
|
1682 |
tuple(history_vector, cache_dict). This is used to prevent having an
|
|
1683 |
ABI change with the C extension that reads .kndx files.
|
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
1684 |
"""
|
1685 |
||
1666.1.6
by Robert Collins
Make knit the default format. |
1686 |
HEADER = "# bzr knit index 8\n" |
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
1687 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1688 |
def __init__(self, transport, mapper, get_scope, allow_writes, is_locked): |
1689 |
"""Create a _KndxIndex on transport using mapper."""
|
|
1690 |
self._transport = transport |
|
1691 |
self._mapper = mapper |
|
1692 |
self._get_scope = get_scope |
|
1693 |
self._allow_writes = allow_writes |
|
1694 |
self._is_locked = is_locked |
|
1695 |
self._reset_cache() |
|
1696 |
self.has_graph = True |
|
1697 |
||
1698 |
def add_records(self, records, random_id=False): |
|
1699 |
"""Add multiple records to the index.
|
|
1700 |
|
|
1701 |
:param records: a list of tuples:
|
|
1702 |
(key, options, access_memo, parents).
|
|
1703 |
:param random_id: If True the ids being added were randomly generated
|
|
1704 |
and no check for existence will be performed.
|
|
1705 |
"""
|
|
1706 |
paths = {} |
|
1707 |
for record in records: |
|
1708 |
key = record[0] |
|
1709 |
prefix = key[:-1] |
|
1710 |
path = self._mapper.map(key) + '.kndx' |
|
1711 |
path_keys = paths.setdefault(path, (prefix, [])) |
|
1712 |
path_keys[1].append(record) |
|
1713 |
for path in sorted(paths): |
|
1714 |
prefix, path_keys = paths[path] |
|
1715 |
self._load_prefixes([prefix]) |
|
1716 |
lines = [] |
|
1717 |
orig_history = self._kndx_cache[prefix][1][:] |
|
1718 |
orig_cache = self._kndx_cache[prefix][0].copy() |
|
1719 |
||
1720 |
try: |
|
1721 |
for key, options, (_, pos, size), parents in path_keys: |
|
1722 |
if parents is None: |
|
1723 |
# kndx indices cannot be parentless.
|
|
1724 |
parents = () |
|
1725 |
line = "\n%s %s %s %s %s :" % ( |
|
1726 |
key[-1], ','.join(options), pos, size, |
|
1727 |
self._dictionary_compress(parents)) |
|
1728 |
if type(line) != str: |
|
1729 |
raise AssertionError( |
|
1730 |
'data must be utf8 was %s' % type(line)) |
|
1731 |
lines.append(line) |
|
1732 |
self._cache_key(key, options, pos, size, parents) |
|
1733 |
if len(orig_history): |
|
1734 |
self._transport.append_bytes(path, ''.join(lines)) |
|
1735 |
else: |
|
1736 |
self._init_index(path, lines) |
|
1737 |
except: |
|
1738 |
# If any problems happen, restore the original values and re-raise
|
|
1739 |
self._kndx_cache[prefix] = (orig_cache, orig_history) |
|
1740 |
raise
|
|
1741 |
||
1742 |
def _cache_key(self, key, options, pos, size, parent_keys): |
|
1596.2.18
by Robert Collins
More microopimisations on index reading, now down to 16000 records/seconds. |
1743 |
"""Cache a version record in the history array and index cache.
|
2158.3.1
by Dmitry Vasiliev
KnitIndex tests/fixes/optimizations |
1744 |
|
1745 |
This is inlined into _load_data for performance. KEEP IN SYNC.
|
|
1596.2.18
by Robert Collins
More microopimisations on index reading, now down to 16000 records/seconds. |
1746 |
(It saves 60ms, 25% of the __init__ overhead on local 4000 record
|
1747 |
indexes).
|
|
1748 |
"""
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1749 |
prefix = key[:-1] |
1750 |
version_id = key[-1] |
|
1751 |
# last-element only for compatibilty with the C load_data.
|
|
1752 |
parents = tuple(parent[-1] for parent in parent_keys) |
|
1753 |
for parent in parent_keys: |
|
1754 |
if parent[:-1] != prefix: |
|
1755 |
raise ValueError("mismatched prefixes for %r, %r" % ( |
|
1756 |
key, parent_keys)) |
|
1757 |
cache, history = self._kndx_cache[prefix] |
|
1596.2.14
by Robert Collins
Make knit parsing non quadratic? |
1758 |
# only want the _history index to reference the 1st index entry
|
1759 |
# for version_id
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1760 |
if version_id not in cache: |
1761 |
index = len(history) |
|
1762 |
history.append(version_id) |
|
1628.1.1
by Robert Collins
Cache the index number of versions in the knit index's self._cache so that |
1763 |
else: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1764 |
index = cache[version_id][5] |
1765 |
cache[version_id] = (version_id, |
|
1628.1.1
by Robert Collins
Cache the index number of versions in the knit index's self._cache so that |
1766 |
options, |
1767 |
pos, |
|
1768 |
size, |
|
1769 |
parents, |
|
1770 |
index) |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
1771 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1772 |
def check_header(self, fp): |
1773 |
line = fp.readline() |
|
1774 |
if line == '': |
|
1775 |
# An empty file can actually be treated as though the file doesn't
|
|
1776 |
# exist yet.
|
|
1777 |
raise errors.NoSuchFile(self) |
|
1778 |
if line != self.HEADER: |
|
1779 |
raise KnitHeaderError(badline=line, filename=self) |
|
1780 |
||
1781 |
def _check_read(self): |
|
1782 |
if not self._is_locked(): |
|
1783 |
raise errors.ObjectNotLocked(self) |
|
1784 |
if self._get_scope() != self._scope: |
|
1785 |
self._reset_cache() |
|
1786 |
||
3316.2.3
by Robert Collins
Remove manual notification of transaction finishing on versioned files. |
1787 |
def _check_write_ok(self): |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1788 |
"""Assert if not writes are permitted."""
|
1789 |
if not self._is_locked(): |
|
1790 |
raise errors.ObjectNotLocked(self) |
|
3316.2.5
by Robert Collins
Review feedback. |
1791 |
if self._get_scope() != self._scope: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1792 |
self._reset_cache() |
3316.2.3
by Robert Collins
Remove manual notification of transaction finishing on versioned files. |
1793 |
if self._mode != 'w': |
1794 |
raise errors.ReadOnlyObjectDirtiedError(self) |
|
1795 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1796 |
def get_build_details(self, keys): |
1797 |
"""Get the method, index_memo and compression parent for keys.
|
|
3218.1.1
by Robert Collins
Reduce index query pressure for text construction by batching the individual queries into single batch queries. |
1798 |
|
3224.1.29
by John Arbash Meinel
Properly handle annotating when ghosts are present. |
1799 |
Ghosts are omitted from the result.
|
1800 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1801 |
:param keys: An iterable of keys.
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1802 |
:return: A dict of key:(index_memo, compression_parent, parents,
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1803 |
record_details).
|
3224.1.14
by John Arbash Meinel
Switch to making content_details opaque, step 1 |
1804 |
index_memo
|
1805 |
opaque structure to pass to read_records to extract the raw
|
|
1806 |
data
|
|
1807 |
compression_parent
|
|
1808 |
Content that this record is built upon, may be None
|
|
1809 |
parents
|
|
1810 |
Logical parents of this node
|
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
1811 |
record_details
|
3224.1.14
by John Arbash Meinel
Switch to making content_details opaque, step 1 |
1812 |
extra information about the content which needs to be passed to
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
1813 |
Factory.parse_record
|
3218.1.1
by Robert Collins
Reduce index query pressure for text construction by batching the individual queries into single batch queries. |
1814 |
"""
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1815 |
prefixes = self._partition_keys(keys) |
1816 |
parent_map = self.get_parent_map(keys) |
|
3218.1.1
by Robert Collins
Reduce index query pressure for text construction by batching the individual queries into single batch queries. |
1817 |
result = {} |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1818 |
for key in keys: |
1819 |
if key not in parent_map: |
|
1820 |
continue # Ghost |
|
1821 |
method = self.get_method(key) |
|
1822 |
parents = parent_map[key] |
|
3218.1.1
by Robert Collins
Reduce index query pressure for text construction by batching the individual queries into single batch queries. |
1823 |
if method == 'fulltext': |
1824 |
compression_parent = None |
|
1825 |
else: |
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
1826 |
compression_parent = parents[0] |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1827 |
noeol = 'no-eol' in self.get_options(key) |
1828 |
index_memo = self.get_position(key) |
|
1829 |
result[key] = (index_memo, compression_parent, |
|
3224.1.14
by John Arbash Meinel
Switch to making content_details opaque, step 1 |
1830 |
parents, (method, noeol)) |
3218.1.1
by Robert Collins
Reduce index query pressure for text construction by batching the individual queries into single batch queries. |
1831 |
return result |
1832 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1833 |
def get_method(self, key): |
1834 |
"""Return compression method of specified key."""
|
|
1835 |
options = self.get_options(key) |
|
1836 |
if 'fulltext' in options: |
|
1837 |
return 'fulltext' |
|
1838 |
elif 'line-delta' in options: |
|
1839 |
return 'line-delta' |
|
1840 |
else: |
|
1841 |
raise errors.KnitIndexUnknownMethod(self, options) |
|
1842 |
||
1843 |
def get_options(self, key): |
|
1844 |
"""Return a list representing options.
|
|
1845 |
||
1846 |
e.g. ['foo', 'bar']
|
|
1847 |
"""
|
|
1848 |
prefix, suffix = self._split_key(key) |
|
1849 |
self._load_prefixes([prefix]) |
|
3350.8.9
by Robert Collins
define behaviour for add_lines with stacked storage. |
1850 |
try: |
1851 |
return self._kndx_cache[prefix][0][suffix][1] |
|
1852 |
except KeyError: |
|
1853 |
raise RevisionNotPresent(key, self) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1854 |
|
1855 |
def get_parent_map(self, keys): |
|
1856 |
"""Get a map of the parents of keys.
|
|
1857 |
||
1858 |
:param keys: The keys to look up parents for.
|
|
1859 |
:return: A mapping from keys to parents. Absent keys are absent from
|
|
1860 |
the mapping.
|
|
1861 |
"""
|
|
1862 |
# Parse what we need to up front, this potentially trades off I/O
|
|
1863 |
# locality (.kndx and .knit in the same block group for the same file
|
|
1864 |
# id) for less checking in inner loops.
|
|
3350.6.10
by Martin Pool
VersionedFiles review cleanups |
1865 |
prefixes = set(key[:-1] for key in keys) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1866 |
self._load_prefixes(prefixes) |
1867 |
result = {} |
|
1868 |
for key in keys: |
|
1869 |
prefix = key[:-1] |
|
1870 |
try: |
|
1871 |
suffix_parents = self._kndx_cache[prefix][0][key[-1]][4] |
|
1872 |
except KeyError: |
|
1873 |
pass
|
|
1874 |
else: |
|
1875 |
result[key] = tuple(prefix + (suffix,) for |
|
1876 |
suffix in suffix_parents) |
|
1877 |
return result |
|
1878 |
||
1879 |
def get_position(self, key): |
|
1880 |
"""Return details needed to access the version.
|
|
1881 |
|
|
1882 |
:return: a tuple (key, data position, size) to hand to the access
|
|
1883 |
logic to get the record.
|
|
1884 |
"""
|
|
1885 |
prefix, suffix = self._split_key(key) |
|
1886 |
self._load_prefixes([prefix]) |
|
1887 |
entry = self._kndx_cache[prefix][0][suffix] |
|
1888 |
return key, entry[2], entry[3] |
|
1889 |
||
1890 |
def _init_index(self, path, extra_lines=[]): |
|
1891 |
"""Initialize an index."""
|
|
1892 |
sio = StringIO() |
|
1893 |
sio.write(self.HEADER) |
|
1894 |
sio.writelines(extra_lines) |
|
1895 |
sio.seek(0) |
|
1896 |
self._transport.put_file_non_atomic(path, sio, |
|
1897 |
create_parent_dir=True) |
|
1898 |
# self._create_parent_dir)
|
|
1899 |
# mode=self._file_mode,
|
|
1900 |
# dir_mode=self._dir_mode)
|
|
1901 |
||
1902 |
def keys(self): |
|
1903 |
"""Get all the keys in the collection.
|
|
1904 |
|
|
1905 |
The keys are not ordered.
|
|
1906 |
"""
|
|
1907 |
result = set() |
|
1908 |
# Identify all key prefixes.
|
|
1909 |
# XXX: A bit hacky, needs polish.
|
|
1910 |
if type(self._mapper) == ConstantMapper: |
|
1911 |
prefixes = [()] |
|
1912 |
else: |
|
1913 |
relpaths = set() |
|
1914 |
for quoted_relpath in self._transport.iter_files_recursive(): |
|
1915 |
path, ext = os.path.splitext(quoted_relpath) |
|
1916 |
relpaths.add(path) |
|
1917 |
prefixes = [self._mapper.unmap(path) for path in relpaths] |
|
1918 |
self._load_prefixes(prefixes) |
|
1919 |
for prefix in prefixes: |
|
1920 |
for suffix in self._kndx_cache[prefix][1]: |
|
1921 |
result.add(prefix + (suffix,)) |
|
1922 |
return result |
|
1923 |
||
1924 |
def _load_prefixes(self, prefixes): |
|
1925 |
"""Load the indices for prefixes."""
|
|
1926 |
self._check_read() |
|
1927 |
for prefix in prefixes: |
|
1928 |
if prefix not in self._kndx_cache: |
|
1929 |
# the load_data interface writes to these variables.
|
|
1930 |
self._cache = {} |
|
1931 |
self._history = [] |
|
1932 |
self._filename = prefix |
|
1933 |
try: |
|
1934 |
path = self._mapper.map(prefix) + '.kndx' |
|
1935 |
fp = self._transport.get(path) |
|
1936 |
try: |
|
1937 |
# _load_data may raise NoSuchFile if the target knit is
|
|
1938 |
# completely empty.
|
|
1939 |
_load_data(self, fp) |
|
1940 |
finally: |
|
1941 |
fp.close() |
|
1942 |
self._kndx_cache[prefix] = (self._cache, self._history) |
|
1943 |
del self._cache |
|
1944 |
del self._filename |
|
1945 |
del self._history |
|
1946 |
except NoSuchFile: |
|
1947 |
self._kndx_cache[prefix] = ({}, []) |
|
1948 |
if type(self._mapper) == ConstantMapper: |
|
1949 |
# preserve behaviour for revisions.kndx etc.
|
|
1950 |
self._init_index(path) |
|
1951 |
del self._cache |
|
1952 |
del self._filename |
|
1953 |
del self._history |
|
1954 |
||
1955 |
def _partition_keys(self, keys): |
|
1956 |
"""Turn keys into a dict of prefix:suffix_list."""
|
|
1957 |
result = {} |
|
1958 |
for key in keys: |
|
1959 |
prefix_keys = result.setdefault(key[:-1], []) |
|
1960 |
prefix_keys.append(key[-1]) |
|
1961 |
return result |
|
1962 |
||
1963 |
def _dictionary_compress(self, keys): |
|
1964 |
"""Dictionary compress keys.
|
|
1965 |
|
|
1966 |
:param keys: The keys to generate references to.
|
|
1967 |
:return: A string representation of keys. keys which are present are
|
|
1968 |
dictionary compressed, and others are emitted as fulltext with a
|
|
1969 |
'.' prefix.
|
|
1970 |
"""
|
|
1971 |
if not keys: |
|
1972 |
return '' |
|
1594.2.8
by Robert Collins
add ghost aware apis to knits. |
1973 |
result_list = [] |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1974 |
prefix = keys[0][:-1] |
1975 |
cache = self._kndx_cache[prefix][0] |
|
1976 |
for key in keys: |
|
1977 |
if key[:-1] != prefix: |
|
1978 |
# kndx indices cannot refer across partitioned storage.
|
|
1979 |
raise ValueError("mismatched prefixes for %r" % keys) |
|
1980 |
if key[-1] in cache: |
|
1628.1.1
by Robert Collins
Cache the index number of versions in the knit index's self._cache so that |
1981 |
# -- inlined lookup() --
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1982 |
result_list.append(str(cache[key[-1]][5])) |
1628.1.1
by Robert Collins
Cache the index number of versions in the knit index's self._cache so that |
1983 |
# -- end lookup () --
|
1594.2.8
by Robert Collins
add ghost aware apis to knits. |
1984 |
else: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1985 |
result_list.append('.' + key[-1]) |
1594.2.8
by Robert Collins
add ghost aware apis to knits. |
1986 |
return ' '.join(result_list) |
1987 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1988 |
def _reset_cache(self): |
1989 |
# Possibly this should be a LRU cache. A dictionary from key_prefix to
|
|
1990 |
# (cache_dict, history_vector) for parsed kndx files.
|
|
1991 |
self._kndx_cache = {} |
|
1992 |
self._scope = self._get_scope() |
|
1993 |
allow_writes = self._allow_writes() |
|
1994 |
if allow_writes: |
|
1995 |
self._mode = 'w' |
|
1563.2.4
by Robert Collins
First cut at including the knit implementation of versioned_file. |
1996 |
else: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
1997 |
self._mode = 'r' |
1998 |
||
1999 |
def _split_key(self, key): |
|
2000 |
"""Split key into a prefix and suffix."""
|
|
2001 |
return key[:-1], key[-1] |
|
2002 |
||
2003 |
||
2004 |
class _KnitGraphIndex(object): |
|
2005 |
"""A KnitVersionedFiles index layered on GraphIndex."""
|
|
2006 |
||
2007 |
def __init__(self, graph_index, is_locked, deltas=False, parents=True, |
|
2008 |
add_callback=None): |
|
2592.3.2
by Robert Collins
Implement a get_graph for a new KnitGraphIndex that will implement a KnitIndex on top of the GraphIndex API. |
2009 |
"""Construct a KnitGraphIndex on a graph_index.
|
2010 |
||
2011 |
:param graph_index: An implementation of bzrlib.index.GraphIndex.
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2012 |
:param is_locked: A callback to check whether the object should answer
|
2013 |
queries.
|
|
2592.3.13
by Robert Collins
Implement KnitGraphIndex.get_method. |
2014 |
:param deltas: Allow delta-compressed records.
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2015 |
:param parents: If True, record knits parents, if not do not record
|
2016 |
parents.
|
|
2592.3.19
by Robert Collins
Change KnitGraphIndex from returning data to performing a callback on insertions. |
2017 |
:param add_callback: If not None, allow additions to the index and call
|
2018 |
this callback with a list of added GraphIndex nodes:
|
|
2592.3.33
by Robert Collins
Change the order of index refs and values to make the no-graph knit index easier. |
2019 |
[(node, value, node_refs), ...]
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2020 |
:param is_locked: A callback, returns True if the index is locked and
|
2021 |
thus usable.
|
|
2592.3.2
by Robert Collins
Implement a get_graph for a new KnitGraphIndex that will implement a KnitIndex on top of the GraphIndex API. |
2022 |
"""
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2023 |
self._add_callback = add_callback |
2592.3.2
by Robert Collins
Implement a get_graph for a new KnitGraphIndex that will implement a KnitIndex on top of the GraphIndex API. |
2024 |
self._graph_index = graph_index |
2592.3.13
by Robert Collins
Implement KnitGraphIndex.get_method. |
2025 |
self._deltas = deltas |
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2026 |
self._parents = parents |
2027 |
if deltas and not parents: |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2028 |
# XXX: TODO: Delta tree and parent graph should be conceptually
|
2029 |
# separate.
|
|
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2030 |
raise KnitCorrupt(self, "Cannot do delta compression without " |
2031 |
"parent tracking.") |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2032 |
self.has_graph = parents |
2033 |
self._is_locked = is_locked |
|
2034 |
||
3517.4.13
by Martin Pool
Add repr methods |
2035 |
def __repr__(self): |
2036 |
return "%s(%r)" % (self.__class__.__name__, self._graph_index) |
|
2037 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2038 |
def add_records(self, records, random_id=False): |
2039 |
"""Add multiple records to the index.
|
|
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2040 |
|
2041 |
This function does not insert data into the Immutable GraphIndex
|
|
2042 |
backing the KnitGraphIndex, instead it prepares data for insertion by
|
|
2592.3.19
by Robert Collins
Change KnitGraphIndex from returning data to performing a callback on insertions. |
2043 |
the caller and checks that it is safe to insert then calls
|
2044 |
self._add_callback with the prepared GraphIndex nodes.
|
|
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2045 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2046 |
:param records: a list of tuples:
|
2047 |
(key, options, access_memo, parents).
|
|
2841.2.1
by Robert Collins
* Commit no longer checks for new text keys during insertion when the |
2048 |
:param random_id: If True the ids being added were randomly generated
|
2049 |
and no check for existence will be performed.
|
|
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2050 |
"""
|
2592.3.19
by Robert Collins
Change KnitGraphIndex from returning data to performing a callback on insertions. |
2051 |
if not self._add_callback: |
2052 |
raise errors.ReadOnlyError(self) |
|
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2053 |
# we hope there are no repositories with inconsistent parentage
|
2054 |
# anymore.
|
|
2055 |
||
2056 |
keys = {} |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2057 |
for (key, options, access_memo, parents) in records: |
2058 |
if self._parents: |
|
2059 |
parents = tuple(parents) |
|
2670.2.2
by Robert Collins
* In ``bzrlib.knit`` the internal interface has been altered to use |
2060 |
index, pos, size = access_memo |
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2061 |
if 'no-eol' in options: |
2062 |
value = 'N' |
|
2063 |
else: |
|
2064 |
value = ' ' |
|
2065 |
value += "%d %d" % (pos, size) |
|
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2066 |
if not self._deltas: |
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2067 |
if 'line-delta' in options: |
2068 |
raise KnitCorrupt(self, "attempt to add line-delta in non-delta knit") |
|
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2069 |
if self._parents: |
2070 |
if self._deltas: |
|
2071 |
if 'line-delta' in options: |
|
2624.2.5
by Robert Collins
Change bzrlib.index.Index keys to be 1-tuples, not strings. |
2072 |
node_refs = (parents, (parents[0],)) |
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2073 |
else: |
2624.2.5
by Robert Collins
Change bzrlib.index.Index keys to be 1-tuples, not strings. |
2074 |
node_refs = (parents, ()) |
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2075 |
else: |
2624.2.5
by Robert Collins
Change bzrlib.index.Index keys to be 1-tuples, not strings. |
2076 |
node_refs = (parents, ) |
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2077 |
else: |
2078 |
if parents: |
|
2079 |
raise KnitCorrupt(self, "attempt to add node with parents " |
|
2080 |
"in parentless index.") |
|
2081 |
node_refs = () |
|
2624.2.5
by Robert Collins
Change bzrlib.index.Index keys to be 1-tuples, not strings. |
2082 |
keys[key] = (value, node_refs) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2083 |
# check for dups
|
2841.2.1
by Robert Collins
* Commit no longer checks for new text keys during insertion when the |
2084 |
if not random_id: |
2085 |
present_nodes = self._get_entries(keys) |
|
2086 |
for (index, key, value, node_refs) in present_nodes: |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2087 |
if (value[0] != keys[key][0][0] or |
2088 |
node_refs != keys[key][1]): |
|
2089 |
raise KnitCorrupt(self, "inconsistent details in add_records" |
|
2841.2.1
by Robert Collins
* Commit no longer checks for new text keys during insertion when the |
2090 |
": %s %s" % ((value, node_refs), keys[key])) |
2091 |
del keys[key] |
|
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2092 |
result = [] |
2592.3.34
by Robert Collins
Rough unfactored support for parentless KnitGraphIndexs. |
2093 |
if self._parents: |
2094 |
for key, (value, node_refs) in keys.iteritems(): |
|
2095 |
result.append((key, value, node_refs)) |
|
2096 |
else: |
|
2097 |
for key, (value, node_refs) in keys.iteritems(): |
|
2098 |
result.append((key, value)) |
|
2592.3.19
by Robert Collins
Change KnitGraphIndex from returning data to performing a callback on insertions. |
2099 |
self._add_callback(result) |
2592.3.17
by Robert Collins
Add add_version(s) to KnitGraphIndex, completing the required api for KnitVersionedFile. |
2100 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2101 |
def _check_read(self): |
2102 |
"""raise if reads are not permitted."""
|
|
2103 |
if not self._is_locked(): |
|
2104 |
raise errors.ObjectNotLocked(self) |
|
2105 |
||
2106 |
def _check_write_ok(self): |
|
2107 |
"""Assert if writes are not permitted."""
|
|
2108 |
if not self._is_locked(): |
|
2109 |
raise errors.ObjectNotLocked(self) |
|
2110 |
||
2111 |
def _compression_parent(self, an_entry): |
|
2112 |
# return the key that an_entry is compressed against, or None
|
|
2113 |
# Grab the second parent list (as deltas implies parents currently)
|
|
2114 |
compression_parents = an_entry[3][1] |
|
2115 |
if not compression_parents: |
|
2116 |
return None |
|
2117 |
if len(compression_parents) != 1: |
|
2118 |
raise AssertionError( |
|
2119 |
"Too many compression parents: %r" % compression_parents) |
|
2120 |
return compression_parents[0] |
|
2121 |
||
2122 |
def get_build_details(self, keys): |
|
2123 |
"""Get the method, index_memo and compression parent for version_ids.
|
|
2124 |
||
2125 |
Ghosts are omitted from the result.
|
|
2126 |
||
2127 |
:param keys: An iterable of keys.
|
|
2128 |
:return: A dict of key:
|
|
2129 |
(index_memo, compression_parent, parents, record_details).
|
|
2130 |
index_memo
|
|
2131 |
opaque structure to pass to read_records to extract the raw
|
|
2132 |
data
|
|
2133 |
compression_parent
|
|
2134 |
Content that this record is built upon, may be None
|
|
2135 |
parents
|
|
2136 |
Logical parents of this node
|
|
2137 |
record_details
|
|
2138 |
extra information about the content which needs to be passed to
|
|
2139 |
Factory.parse_record
|
|
2140 |
"""
|
|
2141 |
self._check_read() |
|
2142 |
result = {} |
|
2143 |
entries = self._get_entries(keys, False) |
|
2144 |
for entry in entries: |
|
2145 |
key = entry[1] |
|
2146 |
if not self._parents: |
|
2147 |
parents = () |
|
2148 |
else: |
|
2149 |
parents = entry[3][0] |
|
2150 |
if not self._deltas: |
|
2151 |
compression_parent_key = None |
|
2152 |
else: |
|
2153 |
compression_parent_key = self._compression_parent(entry) |
|
2154 |
noeol = (entry[2][0] == 'N') |
|
2155 |
if compression_parent_key: |
|
2156 |
method = 'line-delta' |
|
2157 |
else: |
|
2158 |
method = 'fulltext' |
|
2159 |
result[key] = (self._node_to_position(entry), |
|
2160 |
compression_parent_key, parents, |
|
2161 |
(method, noeol)) |
|
2162 |
return result |
|
2163 |
||
2164 |
def _get_entries(self, keys, check_present=False): |
|
2165 |
"""Get the entries for keys.
|
|
2166 |
|
|
2167 |
:param keys: An iterable of index key tuples.
|
|
2168 |
"""
|
|
2169 |
keys = set(keys) |
|
2170 |
found_keys = set() |
|
2171 |
if self._parents: |
|
2172 |
for node in self._graph_index.iter_entries(keys): |
|
2173 |
yield node |
|
2174 |
found_keys.add(node[1]) |
|
2175 |
else: |
|
2176 |
# adapt parentless index to the rest of the code.
|
|
2177 |
for node in self._graph_index.iter_entries(keys): |
|
2178 |
yield node[0], node[1], node[2], () |
|
2179 |
found_keys.add(node[1]) |
|
2180 |
if check_present: |
|
2181 |
missing_keys = keys.difference(found_keys) |
|
2182 |
if missing_keys: |
|
2183 |
raise RevisionNotPresent(missing_keys.pop(), self) |
|
2184 |
||
2185 |
def get_method(self, key): |
|
2186 |
"""Return compression method of specified key."""
|
|
2187 |
return self._get_method(self._get_node(key)) |
|
2188 |
||
2189 |
def _get_method(self, node): |
|
2190 |
if not self._deltas: |
|
2191 |
return 'fulltext' |
|
2192 |
if self._compression_parent(node): |
|
2193 |
return 'line-delta' |
|
2194 |
else: |
|
2195 |
return 'fulltext' |
|
2196 |
||
2197 |
def _get_node(self, key): |
|
2198 |
try: |
|
2199 |
return list(self._get_entries([key]))[0] |
|
2200 |
except IndexError: |
|
2201 |
raise RevisionNotPresent(key, self) |
|
2202 |
||
2203 |
def get_options(self, key): |
|
2204 |
"""Return a list representing options.
|
|
2205 |
||
2206 |
e.g. ['foo', 'bar']
|
|
2207 |
"""
|
|
2208 |
node = self._get_node(key) |
|
2209 |
options = [self._get_method(node)] |
|
2210 |
if node[2][0] == 'N': |
|
2211 |
options.append('no-eol') |
|
2212 |
return options |
|
2213 |
||
2214 |
def get_parent_map(self, keys): |
|
2215 |
"""Get a map of the parents of keys.
|
|
2216 |
||
2217 |
:param keys: The keys to look up parents for.
|
|
2218 |
:return: A mapping from keys to parents. Absent keys are absent from
|
|
2219 |
the mapping.
|
|
2220 |
"""
|
|
2221 |
self._check_read() |
|
2222 |
nodes = self._get_entries(keys) |
|
2223 |
result = {} |
|
2224 |
if self._parents: |
|
2225 |
for node in nodes: |
|
2226 |
result[node[1]] = node[3][0] |
|
2227 |
else: |
|
2228 |
for node in nodes: |
|
2229 |
result[node[1]] = None |
|
2230 |
return result |
|
2231 |
||
2232 |
def get_position(self, key): |
|
2233 |
"""Return details needed to access the version.
|
|
2234 |
|
|
2235 |
:return: a tuple (index, data position, size) to hand to the access
|
|
2236 |
logic to get the record.
|
|
2237 |
"""
|
|
2238 |
node = self._get_node(key) |
|
2239 |
return self._node_to_position(node) |
|
2240 |
||
2241 |
def keys(self): |
|
2242 |
"""Get all the keys in the collection.
|
|
2243 |
|
|
2244 |
The keys are not ordered.
|
|
2245 |
"""
|
|
2246 |
self._check_read() |
|
2247 |
return [node[1] for node in self._graph_index.iter_all_entries()] |
|
2248 |
||
2249 |
def _node_to_position(self, node): |
|
2250 |
"""Convert an index value to position details."""
|
|
2251 |
bits = node[2][1:].split(' ') |
|
2252 |
return node[0], int(bits[0]), int(bits[1]) |
|
2253 |
||
2254 |
||
2255 |
class _KnitKeyAccess(object): |
|
2256 |
"""Access to records in .knit files."""
|
|
2257 |
||
2258 |
def __init__(self, transport, mapper): |
|
2259 |
"""Create a _KnitKeyAccess with transport and mapper.
|
|
2260 |
||
2261 |
:param transport: The transport the access object is rooted at.
|
|
2262 |
:param mapper: The mapper used to map keys to .knit files.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2263 |
"""
|
2264 |
self._transport = transport |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2265 |
self._mapper = mapper |
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2266 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2267 |
def add_raw_records(self, key_sizes, raw_data): |
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2268 |
"""Add raw knit bytes to a storage area.
|
2269 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2270 |
The data is spooled to the container writer in one bytes-record per
|
2271 |
raw data item.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2272 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2273 |
:param sizes: An iterable of tuples containing the key and size of each
|
2274 |
raw data segment.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2275 |
:param raw_data: A bytestring containing the data.
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2276 |
:return: A list of memos to retrieve the record later. Each memo is an
|
2277 |
opaque index memo. For _KnitKeyAccess the memo is (key, pos,
|
|
2278 |
length), where the key is the record key.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2279 |
"""
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2280 |
if type(raw_data) != str: |
2281 |
raise AssertionError( |
|
2282 |
'data must be plain bytes was %s' % type(raw_data)) |
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2283 |
result = [] |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2284 |
offset = 0 |
2285 |
# TODO: This can be tuned for writing to sftp and other servers where
|
|
2286 |
# append() is relatively expensive by grouping the writes to each key
|
|
2287 |
# prefix.
|
|
2288 |
for key, size in key_sizes: |
|
2289 |
path = self._mapper.map(key) |
|
2290 |
try: |
|
2291 |
base = self._transport.append_bytes(path + '.knit', |
|
2292 |
raw_data[offset:offset+size]) |
|
2293 |
except errors.NoSuchFile: |
|
2294 |
self._transport.mkdir(osutils.dirname(path)) |
|
2295 |
base = self._transport.append_bytes(path + '.knit', |
|
2296 |
raw_data[offset:offset+size]) |
|
2297 |
# if base == 0:
|
|
2298 |
# chmod.
|
|
2299 |
offset += size |
|
2300 |
result.append((key, base, size)) |
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2301 |
return result |
2302 |
||
2303 |
def get_raw_records(self, memos_for_retrieval): |
|
2304 |
"""Get the raw bytes for a records.
|
|
2305 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2306 |
:param memos_for_retrieval: An iterable containing the access memo for
|
2307 |
retrieving the bytes.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2308 |
:return: An iterator over the bytes of the records.
|
2309 |
"""
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2310 |
# first pass, group into same-index request to minimise readv's issued.
|
2311 |
request_lists = [] |
|
2312 |
current_prefix = None |
|
2313 |
for (key, offset, length) in memos_for_retrieval: |
|
2314 |
if current_prefix == key[:-1]: |
|
2315 |
current_list.append((offset, length)) |
|
2316 |
else: |
|
2317 |
if current_prefix is not None: |
|
2318 |
request_lists.append((current_prefix, current_list)) |
|
2319 |
current_prefix = key[:-1] |
|
2320 |
current_list = [(offset, length)] |
|
2321 |
# handle the last entry
|
|
2322 |
if current_prefix is not None: |
|
2323 |
request_lists.append((current_prefix, current_list)) |
|
2324 |
for prefix, read_vector in request_lists: |
|
2325 |
path = self._mapper.map(prefix) + '.knit' |
|
2326 |
for pos, data in self._transport.readv(path, read_vector): |
|
2327 |
yield data |
|
2328 |
||
2329 |
||
2330 |
class _DirectPackAccess(object): |
|
2331 |
"""Access to data in one or more packs with less translation."""
|
|
2332 |
||
2333 |
def __init__(self, index_to_packs): |
|
2334 |
"""Create a _DirectPackAccess object.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2335 |
|
2336 |
:param index_to_packs: A dict mapping index objects to the transport
|
|
2337 |
and file names for obtaining data.
|
|
2338 |
"""
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2339 |
self._container_writer = None |
2340 |
self._write_index = None |
|
2341 |
self._indices = index_to_packs |
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2342 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2343 |
def add_raw_records(self, key_sizes, raw_data): |
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2344 |
"""Add raw knit bytes to a storage area.
|
2345 |
||
2670.2.3
by Robert Collins
Review feedback. |
2346 |
The data is spooled to the container writer in one bytes-record per
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2347 |
raw data item.
|
2348 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2349 |
:param sizes: An iterable of tuples containing the key and size of each
|
2350 |
raw data segment.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2351 |
:param raw_data: A bytestring containing the data.
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2352 |
:return: A list of memos to retrieve the record later. Each memo is an
|
2353 |
opaque index memo. For _DirectPackAccess the memo is (index, pos,
|
|
2354 |
length), where the index field is the write_index object supplied
|
|
2355 |
to the PackAccess object.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2356 |
"""
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2357 |
if type(raw_data) != str: |
2358 |
raise AssertionError( |
|
2359 |
'data must be plain bytes was %s' % type(raw_data)) |
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2360 |
result = [] |
2361 |
offset = 0 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2362 |
for key, size in key_sizes: |
2363 |
p_offset, p_length = self._container_writer.add_bytes_record( |
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2364 |
raw_data[offset:offset+size], []) |
2365 |
offset += size |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2366 |
result.append((self._write_index, p_offset, p_length)) |
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2367 |
return result |
2368 |
||
2369 |
def get_raw_records(self, memos_for_retrieval): |
|
2370 |
"""Get the raw bytes for a records.
|
|
2371 |
||
2670.2.2
by Robert Collins
* In ``bzrlib.knit`` the internal interface has been altered to use |
2372 |
:param memos_for_retrieval: An iterable containing the (index, pos,
|
2373 |
length) memo for retrieving the bytes. The Pack access method
|
|
2374 |
looks up the pack to use for a given record in its index_to_pack
|
|
2375 |
map.
|
|
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2376 |
:return: An iterator over the bytes of the records.
|
2377 |
"""
|
|
2378 |
# first pass, group into same-index requests
|
|
2379 |
request_lists = [] |
|
2380 |
current_index = None |
|
2381 |
for (index, offset, length) in memos_for_retrieval: |
|
2382 |
if current_index == index: |
|
2383 |
current_list.append((offset, length)) |
|
2384 |
else: |
|
2385 |
if current_index is not None: |
|
2386 |
request_lists.append((current_index, current_list)) |
|
2387 |
current_index = index |
|
2388 |
current_list = [(offset, length)] |
|
2389 |
# handle the last entry
|
|
2390 |
if current_index is not None: |
|
2391 |
request_lists.append((current_index, current_list)) |
|
2392 |
for index, offsets in request_lists: |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2393 |
transport, path = self._indices[index] |
2592.3.66
by Robert Collins
Allow adaption of KnitData to pack files. |
2394 |
reader = pack.make_readv_reader(transport, path, offsets) |
2395 |
for names, read_func in reader.iter_records(): |
|
2396 |
yield read_func(None) |
|
2397 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2398 |
def set_writer(self, writer, index, transport_packname): |
2592.3.70
by Robert Collins
Allow setting a writer after creating a knit._PackAccess object. |
2399 |
"""Set a writer to use for adding data."""
|
2592.3.208
by Robert Collins
Start refactoring the knit-pack thunking to be clearer. |
2400 |
if index is not None: |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2401 |
self._indices[index] = transport_packname |
2402 |
self._container_writer = writer |
|
2403 |
self._write_index = index |
|
1684.3.3
by Robert Collins
Add a special cased weaves to knit converter. |
2404 |
|
2405 |
||
2781.1.1
by Martin Pool
merge cpatiencediff from Lukas |
2406 |
# Deprecated, use PatienceSequenceMatcher instead
|
2407 |
KnitSequenceMatcher = patiencediff.PatienceSequenceMatcher |
|
2484.1.1
by John Arbash Meinel
Add an initial function to read knit indexes in pyrex. |
2408 |
|
2409 |
||
2770.1.2
by Aaron Bentley
Convert to knit-only annotation |
2410 |
def annotate_knit(knit, revision_id): |
2411 |
"""Annotate a knit with no cached annotations.
|
|
2412 |
||
2413 |
This implementation is for knits with no cached annotations.
|
|
2414 |
It will work for knits with cached annotations, but this is not
|
|
2415 |
recommended.
|
|
2416 |
"""
|
|
3224.1.7
by John Arbash Meinel
_StreamIndex also needs to return the proper values for get_build_details. |
2417 |
annotator = _KnitAnnotator(knit) |
3224.1.25
by John Arbash Meinel
Quick change to the _KnitAnnotator api to use .annotate() instead of get_annotated_lines() |
2418 |
return iter(annotator.annotate(revision_id)) |
3224.1.7
by John Arbash Meinel
_StreamIndex also needs to return the proper values for get_build_details. |
2419 |
|
2420 |
||
2421 |
class _KnitAnnotator(object): |
|
3224.1.5
by John Arbash Meinel
Start using a helper class for doing the knit-pack annotations. |
2422 |
"""Build up the annotations for a text."""
|
2423 |
||
2424 |
def __init__(self, knit): |
|
2425 |
self._knit = knit |
|
2426 |
||
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2427 |
# Content objects, differs from fulltexts because of how final newlines
|
2428 |
# are treated by knits. the content objects here will always have a
|
|
2429 |
# final newline
|
|
2430 |
self._fulltext_contents = {} |
|
2431 |
||
2432 |
# Annotated lines of specific revisions
|
|
2433 |
self._annotated_lines = {} |
|
2434 |
||
2435 |
# Track the raw data for nodes that we could not process yet.
|
|
2436 |
# This maps the revision_id of the base to a list of children that will
|
|
2437 |
# annotated from it.
|
|
2438 |
self._pending_children = {} |
|
2439 |
||
3224.1.29
by John Arbash Meinel
Properly handle annotating when ghosts are present. |
2440 |
# Nodes which cannot be extracted
|
2441 |
self._ghosts = set() |
|
2442 |
||
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2443 |
# Track how many children this node has, so we know if we need to keep
|
2444 |
# it
|
|
2445 |
self._annotate_children = {} |
|
2446 |
self._compression_children = {} |
|
2447 |
||
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2448 |
self._all_build_details = {} |
3224.1.10
by John Arbash Meinel
Introduce the heads_provider for reannotate. |
2449 |
# The children => parent revision_id graph
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2450 |
self._revision_id_graph = {} |
2451 |
||
3224.1.10
by John Arbash Meinel
Introduce the heads_provider for reannotate. |
2452 |
self._heads_provider = None |
2453 |
||
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2454 |
self._nodes_to_keep_annotations = set() |
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2455 |
self._generations_until_keep = 100 |
2456 |
||
2457 |
def set_generations_until_keep(self, value): |
|
2458 |
"""Set the number of generations before caching a node.
|
|
2459 |
||
2460 |
Setting this to -1 will cache every merge node, setting this higher
|
|
2461 |
will cache fewer nodes.
|
|
2462 |
"""
|
|
2463 |
self._generations_until_keep = value |
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2464 |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2465 |
def _add_fulltext_content(self, revision_id, content_obj): |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2466 |
self._fulltext_contents[revision_id] = content_obj |
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2467 |
# TODO: jam 20080305 It might be good to check the sha1digest here
|
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2468 |
return content_obj.text() |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2469 |
|
2470 |
def _check_parents(self, child, nodes_to_annotate): |
|
2471 |
"""Check if all parents have been processed.
|
|
2472 |
||
2473 |
:param child: A tuple of (rev_id, parents, raw_content)
|
|
2474 |
:param nodes_to_annotate: If child is ready, add it to
|
|
2475 |
nodes_to_annotate, otherwise put it back in self._pending_children
|
|
2476 |
"""
|
|
2477 |
for parent_id in child[1]: |
|
3224.1.29
by John Arbash Meinel
Properly handle annotating when ghosts are present. |
2478 |
if (parent_id not in self._annotated_lines): |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2479 |
# This parent is present, but another parent is missing
|
2480 |
self._pending_children.setdefault(parent_id, |
|
2481 |
[]).append(child) |
|
2482 |
break
|
|
2483 |
else: |
|
2484 |
# This one is ready to be processed
|
|
2485 |
nodes_to_annotate.append(child) |
|
2486 |
||
2487 |
def _add_annotation(self, revision_id, fulltext, parent_ids, |
|
2488 |
left_matching_blocks=None): |
|
2489 |
"""Add an annotation entry.
|
|
2490 |
||
2491 |
All parents should already have been annotated.
|
|
2492 |
:return: A list of children that now have their parents satisfied.
|
|
2493 |
"""
|
|
2494 |
a = self._annotated_lines |
|
2495 |
annotated_parent_lines = [a[p] for p in parent_ids] |
|
2496 |
annotated_lines = list(annotate.reannotate(annotated_parent_lines, |
|
3224.1.10
by John Arbash Meinel
Introduce the heads_provider for reannotate. |
2497 |
fulltext, revision_id, left_matching_blocks, |
2498 |
heads_provider=self._get_heads_provider())) |
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2499 |
self._annotated_lines[revision_id] = annotated_lines |
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2500 |
for p in parent_ids: |
2501 |
ann_children = self._annotate_children[p] |
|
2502 |
ann_children.remove(revision_id) |
|
2503 |
if (not ann_children |
|
2504 |
and p not in self._nodes_to_keep_annotations): |
|
2505 |
del self._annotated_lines[p] |
|
2506 |
del self._all_build_details[p] |
|
2507 |
if p in self._fulltext_contents: |
|
2508 |
del self._fulltext_contents[p] |
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2509 |
# Now that we've added this one, see if there are any pending
|
2510 |
# deltas to be done, certainly this parent is finished
|
|
2511 |
nodes_to_annotate = [] |
|
2512 |
for child in self._pending_children.pop(revision_id, []): |
|
2513 |
self._check_parents(child, nodes_to_annotate) |
|
2514 |
return nodes_to_annotate |
|
2515 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2516 |
def _get_build_graph(self, key): |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2517 |
"""Get the graphs for building texts and annotations.
|
2518 |
||
2519 |
The data you need for creating a full text may be different than the
|
|
2520 |
data you need to annotate that text. (At a minimum, you need both
|
|
2521 |
parents to create an annotation, but only need 1 parent to generate the
|
|
2522 |
fulltext.)
|
|
2523 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2524 |
:return: A list of (key, index_memo) records, suitable for
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2525 |
passing to read_records_iter to start reading in the raw data fro/
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2526 |
the pack file.
|
2527 |
"""
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2528 |
if key in self._annotated_lines: |
3224.1.10
by John Arbash Meinel
Introduce the heads_provider for reannotate. |
2529 |
# Nothing to do
|
2530 |
return [] |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2531 |
pending = set([key]) |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2532 |
records = [] |
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2533 |
generation = 0 |
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2534 |
kept_generation = 0 |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2535 |
while pending: |
2536 |
# get all pending nodes
|
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2537 |
generation += 1 |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2538 |
this_iteration = pending |
2539 |
build_details = self._knit._index.get_build_details(this_iteration) |
|
2540 |
self._all_build_details.update(build_details) |
|
2541 |
# new_nodes = self._knit._index._get_entries(this_iteration)
|
|
2542 |
pending = set() |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2543 |
for key, details in build_details.iteritems(): |
3224.1.14
by John Arbash Meinel
Switch to making content_details opaque, step 1 |
2544 |
(index_memo, compression_parent, parents, |
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2545 |
record_details) = details |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2546 |
self._revision_id_graph[key] = parents |
2547 |
records.append((key, index_memo)) |
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2548 |
# Do we actually need to check _annotated_lines?
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2549 |
pending.update(p for p in parents |
2550 |
if p not in self._all_build_details) |
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2551 |
if compression_parent: |
2552 |
self._compression_children.setdefault(compression_parent, |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2553 |
[]).append(key) |
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2554 |
if parents: |
2555 |
for parent in parents: |
|
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2556 |
self._annotate_children.setdefault(parent, |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2557 |
[]).append(key) |
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2558 |
num_gens = generation - kept_generation |
2559 |
if ((num_gens >= self._generations_until_keep) |
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2560 |
and len(parents) > 1): |
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2561 |
kept_generation = generation |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2562 |
self._nodes_to_keep_annotations.add(key) |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2563 |
|
2564 |
missing_versions = this_iteration.difference(build_details.keys()) |
|
3224.1.29
by John Arbash Meinel
Properly handle annotating when ghosts are present. |
2565 |
self._ghosts.update(missing_versions) |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2566 |
for missing_version in missing_versions: |
2567 |
# add a key, no parents
|
|
3224.1.29
by John Arbash Meinel
Properly handle annotating when ghosts are present. |
2568 |
self._revision_id_graph[missing_version] = () |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2569 |
pending.discard(missing_version) # don't look for it |
3376.2.4
by Martin Pool
Remove every assert statement from bzrlib! |
2570 |
if self._ghosts.intersection(self._compression_children): |
2571 |
raise KnitCorrupt( |
|
2572 |
"We cannot have nodes which have a ghost compression parent:\n" |
|
2573 |
"ghosts: %r\n" |
|
2574 |
"compression children: %r" |
|
2575 |
% (self._ghosts, self._compression_children)) |
|
3224.1.29
by John Arbash Meinel
Properly handle annotating when ghosts are present. |
2576 |
# Cleanout anything that depends on a ghost so that we don't wait for
|
2577 |
# the ghost to show up
|
|
2578 |
for node in self._ghosts: |
|
2579 |
if node in self._annotate_children: |
|
2580 |
# We won't be building this node
|
|
2581 |
del self._annotate_children[node] |
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2582 |
# Generally we will want to read the records in reverse order, because
|
2583 |
# we find the parent nodes after the children
|
|
2584 |
records.reverse() |
|
2585 |
return records |
|
2586 |
||
2587 |
def _annotate_records(self, records): |
|
2588 |
"""Build the annotations for the listed records."""
|
|
2589 |
# We iterate in the order read, rather than a strict order requested
|
|
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2590 |
# However, process what we can, and put off to the side things that
|
2591 |
# still need parents, cleaning them up when those parents are
|
|
2592 |
# processed.
|
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2593 |
for (rev_id, record, |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2594 |
digest) in self._knit._read_records_iter(records): |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2595 |
if rev_id in self._annotated_lines: |
2596 |
continue
|
|
2597 |
parent_ids = self._revision_id_graph[rev_id] |
|
3224.1.29
by John Arbash Meinel
Properly handle annotating when ghosts are present. |
2598 |
parent_ids = [p for p in parent_ids if p not in self._ghosts] |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2599 |
details = self._all_build_details[rev_id] |
3224.1.14
by John Arbash Meinel
Switch to making content_details opaque, step 1 |
2600 |
(index_memo, compression_parent, parents, |
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2601 |
record_details) = details |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2602 |
nodes_to_annotate = [] |
2603 |
# TODO: Remove the punning between compression parents, and
|
|
2604 |
# parent_ids, we should be able to do this without assuming
|
|
2605 |
# the build order
|
|
2606 |
if len(parent_ids) == 0: |
|
2607 |
# There are no parents for this node, so just add it
|
|
2608 |
# TODO: This probably needs to be decoupled
|
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2609 |
fulltext_content, delta = self._knit._factory.parse_record( |
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2610 |
rev_id, record, record_details, None) |
2611 |
fulltext = self._add_fulltext_content(rev_id, fulltext_content) |
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2612 |
nodes_to_annotate.extend(self._add_annotation(rev_id, fulltext, |
2613 |
parent_ids, left_matching_blocks=None)) |
|
2614 |
else: |
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2615 |
child = (rev_id, parent_ids, record) |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2616 |
# Check if all the parents are present
|
2617 |
self._check_parents(child, nodes_to_annotate) |
|
2618 |
while nodes_to_annotate: |
|
2619 |
# Should we use a queue here instead of a stack?
|
|
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2620 |
(rev_id, parent_ids, record) = nodes_to_annotate.pop() |
3224.1.14
by John Arbash Meinel
Switch to making content_details opaque, step 1 |
2621 |
(index_memo, compression_parent, parents, |
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2622 |
record_details) = self._all_build_details[rev_id] |
3224.1.14
by John Arbash Meinel
Switch to making content_details opaque, step 1 |
2623 |
if compression_parent is not None: |
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2624 |
comp_children = self._compression_children[compression_parent] |
3376.2.4
by Martin Pool
Remove every assert statement from bzrlib! |
2625 |
if rev_id not in comp_children: |
2626 |
raise AssertionError("%r not in compression children %r" |
|
2627 |
% (rev_id, comp_children)) |
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2628 |
# If there is only 1 child, it is safe to reuse this
|
2629 |
# content
|
|
2630 |
reuse_content = (len(comp_children) == 1 |
|
2631 |
and compression_parent not in |
|
2632 |
self._nodes_to_keep_annotations) |
|
2633 |
if reuse_content: |
|
2634 |
# Remove it from the cache since it will be changing
|
|
2635 |
parent_fulltext_content = self._fulltext_contents.pop(compression_parent) |
|
2636 |
# Make sure to copy the fulltext since it might be
|
|
2637 |
# modified
|
|
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2638 |
parent_fulltext = list(parent_fulltext_content.text()) |
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2639 |
else: |
2640 |
parent_fulltext_content = self._fulltext_contents[compression_parent] |
|
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2641 |
parent_fulltext = parent_fulltext_content.text() |
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2642 |
comp_children.remove(rev_id) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2643 |
fulltext_content, delta = self._knit._factory.parse_record( |
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2644 |
rev_id, record, record_details, |
2645 |
parent_fulltext_content, |
|
3224.1.19
by John Arbash Meinel
Work on removing nodes from the working set once they aren't needed. |
2646 |
copy_base_content=(not reuse_content)) |
3224.1.22
by John Arbash Meinel
Cleanup the extra debugging info, and some >80 char lines. |
2647 |
fulltext = self._add_fulltext_content(rev_id, |
2648 |
fulltext_content) |
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2649 |
blocks = KnitContent.get_line_delta_blocks(delta, |
2650 |
parent_fulltext, fulltext) |
|
2651 |
else: |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2652 |
fulltext_content = self._knit._factory.parse_fulltext( |
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2653 |
record, rev_id) |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2654 |
fulltext = self._add_fulltext_content(rev_id, |
3224.1.15
by John Arbash Meinel
Finish removing method and noeol from general knowledge, |
2655 |
fulltext_content) |
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2656 |
blocks = None |
2657 |
nodes_to_annotate.extend( |
|
2658 |
self._add_annotation(rev_id, fulltext, parent_ids, |
|
2659 |
left_matching_blocks=blocks)) |
|
2660 |
||
3224.1.10
by John Arbash Meinel
Introduce the heads_provider for reannotate. |
2661 |
def _get_heads_provider(self): |
2662 |
"""Create a heads provider for resolving ancestry issues."""
|
|
2663 |
if self._heads_provider is not None: |
|
2664 |
return self._heads_provider |
|
2665 |
parent_provider = _mod_graph.DictParentsProvider( |
|
2666 |
self._revision_id_graph) |
|
2667 |
graph_obj = _mod_graph.Graph(parent_provider) |
|
3224.1.20
by John Arbash Meinel
Reduce the number of cache misses by caching known heads answers |
2668 |
head_cache = _mod_graph.FrozenHeadsCache(graph_obj) |
3224.1.10
by John Arbash Meinel
Introduce the heads_provider for reannotate. |
2669 |
self._heads_provider = head_cache |
2670 |
return head_cache |
|
2671 |
||
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2672 |
def annotate(self, key): |
2673 |
"""Return the annotated fulltext at the given key.
|
|
3224.1.5
by John Arbash Meinel
Start using a helper class for doing the knit-pack annotations. |
2674 |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2675 |
:param key: The key to annotate.
|
3224.1.5
by John Arbash Meinel
Start using a helper class for doing the knit-pack annotations. |
2676 |
"""
|
3517.4.1
by Martin Pool
Merge unoptimized annotate code for stacking, and only use it when needed |
2677 |
if True or len(self._knit._fallback_vfs) > 0: |
2678 |
# stacked knits can't use the fast path at present.
|
|
2679 |
return self._simple_annotate(key) |
|
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2680 |
records = self._get_build_graph(key) |
2681 |
if key in self._ghosts: |
|
2682 |
raise errors.RevisionNotPresent(key, self._knit) |
|
3224.1.6
by John Arbash Meinel
Refactor the annotation logic into a helper class. |
2683 |
self._annotate_records(records) |
3350.6.4
by Robert Collins
First cut at pluralised VersionedFiles. Some rather massive API incompatabilities, primarily because of the difficulty of coherence among competing stores. |
2684 |
return self._annotated_lines[key] |
3224.1.5
by John Arbash Meinel
Start using a helper class for doing the knit-pack annotations. |
2685 |
|
3517.4.1
by Martin Pool
Merge unoptimized annotate code for stacking, and only use it when needed |
2686 |
def _simple_annotate(self, key): |
2687 |
"""Return annotated fulltext, rediffing from the full texts.
|
|
2688 |
||
2689 |
This is slow but makes no assumptions about the repository
|
|
2690 |
being able to produce line deltas.
|
|
2691 |
"""
|
|
2692 |
# TODO: this code generates a parent maps of present ancestors; it
|
|
2693 |
# could be split out into a separate method, and probably should use
|
|
2694 |
# iter_ancestry instead. -- mbp and robertc 20080704
|
|
3535.5.1
by John Arbash Meinel
cleanup a few imports to be lazily loaded. |
2695 |
graph = _mod_graph.Graph(self._knit) |
3350.9.1
by Robert Collins
Redo annotate more simply, using just the public interfaces for VersionedFiles. |
2696 |
head_cache = _mod_graph.FrozenHeadsCache(graph) |
2697 |
search = graph._make_breadth_first_searcher([key]) |
|
2698 |
keys = set() |
|
2699 |
while True: |
|
2700 |
try: |
|
2701 |
present, ghosts = search.next_with_ghosts() |
|
2702 |
except StopIteration: |
|
2703 |
break
|
|
2704 |
keys.update(present) |
|
2705 |
parent_map = self._knit.get_parent_map(keys) |
|
2706 |
parent_cache = {} |
|
2707 |
reannotate = annotate.reannotate |
|
2708 |
for record in self._knit.get_record_stream(keys, 'topological', True): |
|
2709 |
key = record.key |
|
2710 |
fulltext = split_lines(record.get_bytes_as('fulltext')) |
|
3517.4.2
by Martin Pool
Make simple-annotation and graph code more tolerant of knits with no graph |
2711 |
parents = parent_map[key] |
2712 |
if parents is not None: |
|
2713 |
parent_lines = [parent_cache[parent] for parent in parent_map[key]] |
|
2714 |
else: |
|
2715 |
parent_lines = [] |
|
3350.9.1
by Robert Collins
Redo annotate more simply, using just the public interfaces for VersionedFiles. |
2716 |
parent_cache[key] = list( |
2717 |
reannotate(parent_lines, fulltext, key, None, head_cache)) |
|
3517.4.2
by Martin Pool
Make simple-annotation and graph code more tolerant of knits with no graph |
2718 |
try: |
2719 |
return parent_cache[key] |
|
2720 |
except KeyError, e: |
|
2721 |
raise errors.RevisionNotPresent(key, self._knit) |
|
3224.1.5
by John Arbash Meinel
Start using a helper class for doing the knit-pack annotations. |
2722 |
|
2723 |
||
2484.1.1
by John Arbash Meinel
Add an initial function to read knit indexes in pyrex. |
2724 |
try: |
2484.1.12
by John Arbash Meinel
Switch the layout to use a matching _knit_load_data_py.py and _knit_load_data_c.pyx |
2725 |
from bzrlib._knit_load_data_c import _load_data_c as _load_data |
2484.1.1
by John Arbash Meinel
Add an initial function to read knit indexes in pyrex. |
2726 |
except ImportError: |
2484.1.12
by John Arbash Meinel
Switch the layout to use a matching _knit_load_data_py.py and _knit_load_data_c.pyx |
2727 |
from bzrlib._knit_load_data_py import _load_data_py as _load_data |