~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 4679.8.1

Rev	Summary	Authors	Date
4679.8.1	Merge in the 2.1-static-tuple-btree branch, and restore the ... Merge in the 2.1-static-tuple-btree branch, and restore the string intern tweaks. The tweaks basically involve not interning 'sha1:' chk strings, which we can be reasonably confident are only accessed as tuples, and thus get interned into the more efficient SimpleSet. Even further, we know that we won't be combining them into larger tuples [like we do with (file_id, revision_id)]. This saves quite a bit of space in the string interned dict, since chk keys actually make up the bulk of our keys.	John Arbash Meinel	15 years ago
4679.5.1	Interning the most common 'value' entries saves about 1-2% o... Interning the most common 'value' entries saves about 1-2% of peak memory. Namely, interning the '0' content values. Note that most values have a common prefix now, and an uncommon suffix...	John Arbash Meinel	15 years ago
4679.3.84	Remove 2 more mallocs per line by using a fixed-width Static... Remove 2 more mallocs per line by using a fixed-width StaticTuple for reflists. Note that 'ref_list' is arbitrary width (think parent pointers), so we can't pre-allocate that. I suppose we could count the occurences of the separator in the memory buffer, though. That may be more efficient...	John Arbash Meinel	15 years ago
4679.3.83	Since we changed the repr() for StaticTuple, go ahead and Since we changed the repr() for StaticTuple, go ahead and wrap it back into a pure 'tuple' object so we get the easier-to-read debug info from _dump_tree.	John Arbash Meinel	15 years ago
4679.3.82	Remove a bogus 'print' statement in the tests Remove a bogus 'print' statement in the tests	John Arbash Meinel	15 years ago
4679.3.81	Fix up _simple_set_pyx.pyx to be compatible with pyrex again... Fix up _simple_set_pyx.pyx to be compatible with pyrex again. Surprisingly not as easy as I expected, but things seem to be working. Found an actual bug in the dealloc code...	John Arbash Meinel	15 years ago
4679.3.80	import_static_tuple_c() is defined as except -1, so the impo... import_static_tuple_c() is defined as except -1, so the import code knows to propogate the exception.	John Arbash Meinel	15 years ago
4679.3.79	Change the repr to print out 'StaticTuple' Change the repr to print out 'StaticTuple'	John Arbash Meinel	15 years ago
4679.3.78	Change the pure-python version of StaticTuple Change the pure-python version of StaticTuple wrather than 'has_a(tuple)' it now 'is_a(tuple)'. Making use of inheritance to avoid wasting memory on having 2 objects.	John Arbash Meinel	15 years ago
4679.3.77	Some code cleanup passes. Some code cleanup passes.	John Arbash Meinel	15 years ago
4679.3.76	Rename StaticTupleInterner => SimpleSet. Rename StaticTupleInterner => SimpleSet. This is a bit more appropriate, because the internal data type is not specialized into StaticTuple objects only. Partially because I didn't see a specific memory/speed tradeoff to caching the hash, and that accessing said hash was siginficantly faster than just calling PyObject_Hash().	John Arbash Meinel	15 years ago
4679.3.75	Minor tweak. Minor tweak.	John Arbash Meinel	15 years ago
4679.3.74	Revert back to before I started trying to move to pyrex/cyth... Revert back to before I started trying to move to pyrex/cython code.	John Arbash Meinel	15 years ago
4679.3.73	start trying to expose everything just from cython. start trying to expose everything just from cython. But I found out the argument is moot. You can't override tp_new (__new__) to have a custom allocator/StaticTuple() be a singleton. So I'm going to go back to the C api, and just make the rest pyrex compatible.	John Arbash Meinel	15 years ago
4679.3.72	Things are ~ working again. Things are ~ working again. Man this is getting messy, I may have to change my mind again. I tried to export the StaticTuple type only through _static_tuple_pyx.pyx by using a little bit of trickery. However, you end up with a circular import issue, and also I forgot to track down one place where I needed to rename '_static_tuple_c' => '_static_tuple_type_c'. The idea was that _static_tuple_type.c would only define the type, and not any extra info. This way the code could be compiled with either cython or pyrex and still get the 'better' StaticTuple object. It ended up, overall, just being a multi-hour mess trying to get the dependencies sorted out. By using a .pxd file, at least the basic circular import problem was sorted out. However at this point, you have to import _static_tuple_pyx before _static_tuple_type_c or you get a segfault, and you have to import the latter if you want to get direct access to the class. So at this point I feel like I either need to: 1) Go back to the way it was, and get rid of the circular import 2) Finish the rest of the steps, bring everything into Cython and say 'if you want the memory improvements, then you have to compile with cython.'	John Arbash Meinel	15 years ago
4679.3.71	Merge bzr.dev 4729 to keep up to date. Merge bzr.dev 4729 to keep up to date. Also brings in the assertion errors while parsing btree indexes.	John Arbash Meinel	15 years ago
4679.3.70	We have to build _static_tuple_interned_pyx first because we... We have to build _static_tuple_interned_pyx first because we use the api in _static_tuple_c	John Arbash Meinel	15 years ago
4679.3.69	Timing 'bzr branch' reveals some code paths that are being s... Timing 'bzr branch' reveals some code paths that are being strict about type. Namely, assuming that a key is a tuple, rather than possibly a StaticTuple. Shame that it is exposed in such a fashion... Anyway, 'bzr branch --no-tree bzr.dev standalone' is: 407320MB @ 2m4.207s => 331876MB @ 1m42s So while 'bzr log' doesn't really show the improvement, it is quite obvious during 'bzr branch'.	John Arbash Meinel	15 years ago
4679.3.68	Some more performance data. Some more performance data. #1 thing is that StaticTupleInterner type is not actually in the GC, because it doesn't have any pure 'object' attributes. As such it doesn't implement tp_traverse either. Which means we don't track the size via Meliae as well as I would like. However, in an interesting result, memory consumption is down and speed is better. On LP: 259460KB => 243936KB (15MB) and 0m30.525s => 0m26.457s My guess is that by not walking the intern dict, gc has a lot less work to do when walking all those keys that we know it will just ignore next. Adding the 'self.hash' attribute brings memory consumption back up a bit: 252724KB and time down barely at all 0m25.756s Probably we could get more time performance out of StaticTupleInterner by making the internal table only point to StaticTuple objects, and then we could access their self->hash directly. However, seeing this means I can move StaticTuple itself into a pyrex function, since I'll only have C attributes and a PyObject* table. I just hope I can figure out how to get a tp_traverse that Meliae can use... Note that as a reference point, w/ bzr.dev: 326984KB, 53.4s Or we now have a 1.34:1 memory savings and 2:1 speed improvement. The other guess on why StaticTupleInterner works well is that the hash functionality really does spread things out evenly, so we rarely get collisions. And if you don't have a collision, then you don't care about the time for hash() because it is only computed for the new key (which you have to do anyway) and not for any of the entries already present. Time for 'bzr log -r -1 -n0 bzr.dev' is at 691ms which is ~= bzr.dev 'bzr log -v bzr.dev' is 19.1 => 18.7s and 169MB=>197MB which is a net loss... :( May have to revisit the CHKMap internals. 'bzr log -n0 -v bzr.dev' is 2m25s=>2m21s and 235MB=>244MB which isn't a great tradeoff. Strange that things don't seem to be a win in 'real-life', I wonder if somewhere we are casting things back into regular tuples that I missed. (tuple(tpl) is tpl, but tuple(static_tpl) is not static_tpl). There are also possibilities that tuples are special cased in more places, or the custom __iter__ functionality or... etc. Next up, mem testing on 64-bit.	John Arbash Meinel	15 years ago
4679.3.67	Minor change to the StaticTupleInterner_Discard api, and sta... Minor change to the StaticTupleInterner_Discard api, and start using the new StaticTupleInterner api in StaticTuple.	John Arbash Meinel	15 years ago

Older »