~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 3641.5.5

Rev	Summary	Authors	Date
3641.3.16	Somewhat surprisingly, tracking bytes_out_len makes a Somewhat surprisingly, tracking bytes_out_len makes a large difference in performance. It drops the three_level test time from 9.2s => 8.2s. My best guess is that when adding with Z_SYNC_FLUSH we get a lot of small strings, and we loop over it each time we add another string. Real world tests show improvement, too. For mysql, repack=2,nocopy time 59.3=>57.4 For bzr.dev, repack=2,nocopy time 9.6=>9.3	John Arbash Meinel	16 years ago
3641.3.15	Now that we have real data, remove the copy() code. Now that we have real data, remove the copy() code. It didn't really benefit us over just doing another repack. There was a small benefit when the compressor is genuinely full, but that path is rarely encountered because we stop repacking by effort rather than by space. It also stablizes the tests for all platforms, because the copy() code would alter the packing slightly.	John Arbash Meinel	16 years ago
3641.3.14	Replace time/space benchmarks with real-world testing. Replace time/space benchmarks with real-world testing. Basically, the value was overstated, because the artifical nodes were significantly more compressible than real data. With these results, using .copy() basically is the same time/space trade off as allowing another repack. 1-repack + copy() is mostly equivalent to 2-repack with no copy (in both time and space). They generally seem to be an appropriate 'sweet spot'. The extra pack (copy) avoids pathological behavior of not filling in the last bytes while only adding a small overhead. (approx 10% time cost at 20% space savings.)	John Arbash Meinel	16 years ago
3641.3.13	Some more data points for the time/repack tradeoff. Some more data points for the time/repack tradeoff. Instead of always using copy() we can use a multi-step process. 1) Always add bytes ignoring flush until _default_min_compression 2) Add bytes using Z_SYNC_FLUSH until we hit the page limit, and then repack N times. 3) After repacking N times, switch to using copy() before adding a new node with Z_SYNC_FLUSH. 4) If we re-use the copy() compressor we will get max packing or we can just use the SYNC compressor until it is full, and then return the copy. Current stats are done using the artificial data. We should produce a set using real-world data so we can pick the best time/space tradeoff. Further, we can use this information to find the best space values if someone issues a 'bzr pack' command. Since that is generally a 'spend more time to get better results' command.	John Arbash Meinel	16 years ago
3641.3.12	Collect some info on the space/time tradeoff for _max_repack... Collect some info on the space/time tradeoff for _max_repack. With the test data, we pretty much always hit max_repack before we get everything fully packed.	John Arbash Meinel	16 years ago
3641.3.11	Start working on an alternate way to track compressed_chunk ... Start working on an alternate way to track compressed_chunk state. I tried doing a copy() before flush() but that is quite expensive. Both because it now does copy() but it also does a bigger flush. Next I'll try tracking 2 objects.	John Arbash Meinel	16 years ago
3641.3.10	Flatten a double inner loop into a double list comprehension... Flatten a double inner loop into a double list comprehension.	John Arbash Meinel	16 years ago
3641.3.9	Special case around _iter_smallest when we have only Special case around _iter_smallest when we have only 1 iterator. Shaves another 0.5s off of the test time.	John Arbash Meinel	16 years ago
3641.3.8	Move the add_key helper function into a separate func Move the add_key helper function into a separate func	John Arbash Meinel	16 years ago
3641.3.7	Make it easier to profile the btree writer time Make it easier to profile the btree writer time	John Arbash Meinel	16 years ago
3641.3.6	Dropping the number of nodes to 100k from 200k Dropping the number of nodes to 100k from 200k It is still enough to create a 3-level index, but all tests run 2x faster. We currently spend 1.635s creating and 3.733s adding nodes and 9.842s flushing in the three_level and iter_all tests. The test is 15s down from 68s which is a lot better. It would be nice to have sub second tests, though. For now, we'll live with this.	John Arbash Meinel	16 years ago
3641.3.5	For iter_all and three_level tests adjust spill-at. For iter_all and three_level tests adjust spill-at. It turns out that when adding 200k entries, we spill to disk 2 times. Which adds 20s to the test for each spill to disk. So instead of 20s for 1 flush, we get 60s (for three_level). This does increase memory consumption, but it drops the test time from 64s => 35s.	John Arbash Meinel	16 years ago
3641.3.4	Tweak some 'sum' lines. Tweak some 'sum' lines. change sum(len(b) for b in foo) to sum(map(len, foo)) This is a drastic improvement under lsprof, though it turns out to be a minor to non-improvement in actual time. Under lsprof 272s => 123s (50%) without lsprof 68s => 64s (minor)	John Arbash Meinel	16 years ago
3641.3.3	Change the header to indicate these indexes are Change the header to indicate these indexes are not the same as the index2 plugin.	John Arbash Meinel	16 years ago
3641.3.2	Clean up some variable names, add some documentation. Clean up some variable names, add some documentation.	John Arbash Meinel	16 years ago
3641.3.1	Bring in the btree_index and chunk_writer code and their tes... Bring in the btree_index and chunk_writer code and their tests.	John Arbash Meinel	16 years ago
3641	(jam) Merge in bzr-1.6rc4 including the fix for bug #256757 (jam) Merge in bzr-1.6rc4 including the fix for bug #256757	Canonical.com Patch ...	16 years ago
3640	(robertc) Fix performance regression in status PATH. (Robert... (robertc) Fix performance regression in status PATH. (Robert Collins)	Canonical.com Patch ...	16 years ago
3639	(bialix) remove out-of-date options from Windows standalone ... (bialix) remove out-of-date options from Windows standalone installer.	Canonical.com Patch ...	16 years ago
3638	(jam) bzr uncommit now gives the revision id to restore the ... (jam) bzr uncommit now gives the revision id to restore the branch to the old tip.	Canonical.com Patch ...	16 years ago