~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 3641.5.1

Rev	Summary	Authors	Date
3641.5.1	Update the stats for the current code layout. Update the stats for the current code layout. This shows why I like _max_repack=2 so much. It is the highest value that has 'no waste'. At _max_repack=2, you can always sneak in 1 more line, which avoids triggering an extra repack. Also, updating the timings with the current tuning.	John Arbash Meinel	16 years ago
3641.3.31	NEWS NEWS	John Arbash Meinel	16 years ago
3641.3.30	Rename _parse_btree to _btree_serializer Rename _parse_btree to _btree_serializer	John Arbash Meinel	16 years ago
3641.3.29	Cleanup the copyright headers Cleanup the copyright headers	John Arbash Meinel	16 years ago
3641.3.28	Remove FixedMemoryGraphIndex Remove FixedMemoryGraphIndex	John Arbash Meinel	16 years ago
3641.3.27	Bringing reserved in as a keyword to write() also saves some... Bringing reserved in as a keyword to write() also saves some time. ChunkWriter.write() is called for every key, so having an extra redirection The was a measurable loss. Also, a small tuneup for the compressor that we access often. Unfortunately, there isn't much more tuning that we can do in that function.	John Arbash Meinel	16 years ago
3641.3.26	A couple small tweaks. A couple small tweaks.	John Arbash Meinel	16 years ago
3641.3.25	Shave off some more time by using exact accessors. Shave off some more time by using exact accessors. Basically, the inner access is using PyTuple and PyString functions instead of their PyObject counterparts.	John Arbash Meinel	16 years ago
3641.3.24	Use the compiled flatten function. Use the compiled flatten function. Shows a 3-5% gain in testing and real-world conditions.	John Arbash Meinel	16 years ago
3641.3.23	Flattened all the way down the stack. Flattened all the way down the stack. Next might be to try using something other than plain iteration. Like using PySequenceFast or some variant thereof.	John Arbash Meinel	16 years ago
3641.3.22	flatten the next level flatten the next level	John Arbash Meinel	16 years ago
3641.3.21	Flatten the outermost str.join() into memcpy's Flatten the outermost str.join() into memcpy's	John Arbash Meinel	16 years ago
3641.3.20	We have a single malloc for the final output. We have a single malloc for the final output. We just need to collapse the intermediates.	John Arbash Meinel	16 years ago
3641.3.19	The flatten code now handles the no-ref-list case. The flatten code now handles the no-ref-list case.	John Arbash Meinel	16 years ago
3641.3.18	Start working on a compiled function for transforming Start working on a compiled function for transforming a node => serialized key and line.	John Arbash Meinel	16 years ago
3641.3.17	sum(map(len, foo))) is better than using += for one-off. sum(map(len, foo))) is better than using += for one-off.	John Arbash Meinel	16 years ago
3641.3.16	Somewhat surprisingly, tracking bytes_out_len makes a Somewhat surprisingly, tracking bytes_out_len makes a large difference in performance. It drops the three_level test time from 9.2s => 8.2s. My best guess is that when adding with Z_SYNC_FLUSH we get a lot of small strings, and we loop over it each time we add another string. Real world tests show improvement, too. For mysql, repack=2,nocopy time 59.3=>57.4 For bzr.dev, repack=2,nocopy time 9.6=>9.3	John Arbash Meinel	16 years ago
3641.3.15	Now that we have real data, remove the copy() code. Now that we have real data, remove the copy() code. It didn't really benefit us over just doing another repack. There was a small benefit when the compressor is genuinely full, but that path is rarely encountered because we stop repacking by effort rather than by space. It also stablizes the tests for all platforms, because the copy() code would alter the packing slightly.	John Arbash Meinel	16 years ago
3641.3.14	Replace time/space benchmarks with real-world testing. Replace time/space benchmarks with real-world testing. Basically, the value was overstated, because the artifical nodes were significantly more compressible than real data. With these results, using .copy() basically is the same time/space trade off as allowing another repack. 1-repack + copy() is mostly equivalent to 2-repack with no copy (in both time and space). They generally seem to be an appropriate 'sweet spot'. The extra pack (copy) avoids pathological behavior of not filling in the last bytes while only adding a small overhead. (approx 10% time cost at 20% space savings.)	John Arbash Meinel	16 years ago
3641.3.13	Some more data points for the time/repack tradeoff. Some more data points for the time/repack tradeoff. Instead of always using copy() we can use a multi-step process. 1) Always add bytes ignoring flush until _default_min_compression 2) Add bytes using Z_SYNC_FLUSH until we hit the page limit, and then repack N times. 3) After repacking N times, switch to using copy() before adding a new node with Z_SYNC_FLUSH. 4) If we re-use the copy() compressor we will get max packing or we can just use the SYNC compressor until it is full, and then return the copy. Current stats are done using the artificial data. We should produce a set using real-world data so we can pick the best time/space tradeoff. Further, we can use this information to find the best space values if someone issues a 'bzr pack' command. Since that is generally a 'spend more time to get better results' command.	John Arbash Meinel	16 years ago

Older »