~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 4679.3.72

Rev	Summary	Authors	Date
4679.3.72	Things are ~ working again. Things are ~ working again. Man this is getting messy, I may have to change my mind again. I tried to export the StaticTuple type only through _static_tuple_pyx.pyx by using a little bit of trickery. However, you end up with a circular import issue, and also I forgot to track down one place where I needed to rename '_static_tuple_c' => '_static_tuple_type_c'. The idea was that _static_tuple_type.c would only define the type, and not any extra info. This way the code could be compiled with either cython or pyrex and still get the 'better' StaticTuple object. It ended up, overall, just being a multi-hour mess trying to get the dependencies sorted out. By using a .pxd file, at least the basic circular import problem was sorted out. However at this point, you have to import _static_tuple_pyx before _static_tuple_type_c or you get a segfault, and you have to import the latter if you want to get direct access to the class. So at this point I feel like I either need to: 1) Go back to the way it was, and get rid of the circular import 2) Finish the rest of the steps, bring everything into Cython and say 'if you want the memory improvements, then you have to compile with cython.'	John Arbash Meinel	15 years ago
4679.3.71	Merge bzr.dev 4729 to keep up to date. Merge bzr.dev 4729 to keep up to date. Also brings in the assertion errors while parsing btree indexes.	John Arbash Meinel	15 years ago
4679.3.70	We have to build _static_tuple_interned_pyx first because we... We have to build _static_tuple_interned_pyx first because we use the api in _static_tuple_c	John Arbash Meinel	15 years ago
4679.3.69	Timing 'bzr branch' reveals some code paths that are being s... Timing 'bzr branch' reveals some code paths that are being strict about type. Namely, assuming that a key is a tuple, rather than possibly a StaticTuple. Shame that it is exposed in such a fashion... Anyway, 'bzr branch --no-tree bzr.dev standalone' is: 407320MB @ 2m4.207s => 331876MB @ 1m42s So while 'bzr log' doesn't really show the improvement, it is quite obvious during 'bzr branch'.	John Arbash Meinel	15 years ago
4679.3.68	Some more performance data. Some more performance data. #1 thing is that StaticTupleInterner type is not actually in the GC, because it doesn't have any pure 'object' attributes. As such it doesn't implement tp_traverse either. Which means we don't track the size via Meliae as well as I would like. However, in an interesting result, memory consumption is down and speed is better. On LP: 259460KB => 243936KB (15MB) and 0m30.525s => 0m26.457s My guess is that by not walking the intern dict, gc has a lot less work to do when walking all those keys that we know it will just ignore next. Adding the 'self.hash' attribute brings memory consumption back up a bit: 252724KB and time down barely at all 0m25.756s Probably we could get more time performance out of StaticTupleInterner by making the internal table only point to StaticTuple objects, and then we could access their self->hash directly. However, seeing this means I can move StaticTuple itself into a pyrex function, since I'll only have C attributes and a PyObject* table. I just hope I can figure out how to get a tp_traverse that Meliae can use... Note that as a reference point, w/ bzr.dev: 326984KB, 53.4s Or we now have a 1.34:1 memory savings and 2:1 speed improvement. The other guess on why StaticTupleInterner works well is that the hash functionality really does spread things out evenly, so we rarely get collisions. And if you don't have a collision, then you don't care about the time for hash() because it is only computed for the new key (which you have to do anyway) and not for any of the entries already present. Time for 'bzr log -r -1 -n0 bzr.dev' is at 691ms which is ~= bzr.dev 'bzr log -v bzr.dev' is 19.1 => 18.7s and 169MB=>197MB which is a net loss... :( May have to revisit the CHKMap internals. 'bzr log -n0 -v bzr.dev' is 2m25s=>2m21s and 235MB=>244MB which isn't a great tradeoff. Strange that things don't seem to be a win in 'real-life', I wonder if somewhere we are casting things back into regular tuples that I missed. (tuple(tpl) is tpl, but tuple(static_tpl) is not static_tpl). There are also possibilities that tuples are special cased in more places, or the custom __iter__ functionality or... etc. Next up, mem testing on 64-bit.	John Arbash Meinel	15 years ago
4679.3.67	Minor change to the StaticTupleInterner_Discard api, and sta... Minor change to the StaticTupleInterner_Discard api, and start using the new StaticTupleInterner api in StaticTuple.	John Arbash Meinel	15 years ago
4679.3.66	Implement the C variant of __iter__ Implement the C variant of __iter__	John Arbash Meinel	15 years ago
4679.3.65	Add __iter__ support. Add __iter__ support.	John Arbash Meinel	15 years ago
4679.3.64	Add functionality for shrinking the table. Add functionality for shrinking the table.	John Arbash Meinel	15 years ago
4679.3.63	Implement resizing. Implement resizing. It can resize up or down, but right now we never shrink.	John Arbash Meinel	15 years ago
4679.3.62	Move some of the information into the pxd header file. Move some of the information into the pxd header file. This will make it easier to directly use attributes of the class in other modules. (More of a case for StaticTuple than StaticTupleInterner, but I figured I'd get practice using it.	John Arbash Meinel	15 years ago
4679.3.61	Invert the logic. Invert the logic. I wanted to add type checking to the C api, however if I do that, I should then create a set of internal functions that don't do type checking. But if I'm going to do that, then I'd rather just hang them off the nice class I already have written. And the C api just becomes thunks to the class attributes.	John Arbash Meinel	15 years ago
4679.3.60	Start working on more of the C api for StaticTupleInterner. Start working on more of the C api for StaticTupleInterner. It is really nice to have the except -1 clauses, as suddenly you don't have to manually check the return values of all of your functions. \o/	John Arbash Meinel	15 years ago
4679.3.59	Capitolize StaticTuple_Intern to conform more to the python ... Capitolize StaticTuple_Intern to conform more to the python C apis.	John Arbash Meinel	15 years ago
4679.3.58	Adding a StaticTupleInterner class. Adding a StaticTupleInterner class. Functions basically like a dict, but with less memory overhead. (well, possibly more like a set that also implements __getitem__). For now, it allows arbitrary objects, though I'm considering only allowing StaticTuple objects for some ease-of-implementation. Especially if I find I want to cache the hash() on StaticTuple for performance, then I may want to expose that in StaticTupleInterner. For now, this only compiles via Cython, but only because I'm using 'inline' in one place. It wouldn't be too hard to factor that out, or make it conditional with an IF and DEF bits.	John Arbash Meinel	15 years ago
4679.3.57	Some ideas about StaticTuple.__richcompare__. Some ideas about StaticTuple.__richcompare__.	John Arbash Meinel	15 years ago
4679.3.56	Fix a test that depended on but didn't 'require' meliae Fix a test that depended on but didn't 'require' meliae	John Arbash Meinel	15 years ago
4679.3.55	A quick note about how I'd like to do things. A quick note about how I'd like to do things. Basically, if I could procedurally declare my API, then all of the boilerplate could be easily generated from that. However, it needs to be 'static' in a C header file, and to get that you'd have to generate it from some other source or have a lot more voodoo in your C pre-processor code. Shame that C sucks so much. Note that Pyrex gets this right, where doing all the boilerplate amounts to adding 'api' to your definition. Aka cdef api object myfunc(int foo) Is enough to generate all the boilerplate C headers, etc you would need. It is unfortunate I can't really use that here, because I want a pure C class to avoid the python garbage collector.	John Arbash Meinel	15 years ago
4679.3.54	Factor out some of the C API code into some helper headers. Factor out some of the C API code into some helper headers. This should make the process a bit easier, if we ever decide to do something similar again.	John Arbash Meinel	15 years ago
4679.3.53	Clean up the fprintf() debugging code. Clean up the fprintf() debugging code.	John Arbash Meinel	15 years ago

Older »