~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 4679.3.69

Rev	Summary	Authors	Date
4679.3.68	Some more performance data. Some more performance data. #1 thing is that StaticTupleInterner type is not actually in the GC, because it doesn't have any pure 'object' attributes. As such it doesn't implement tp_traverse either. Which means we don't track the size via Meliae as well as I would like. However, in an interesting result, memory consumption is down and speed is better. On LP: 259460KB => 243936KB (15MB) and 0m30.525s => 0m26.457s My guess is that by not walking the intern dict, gc has a lot less work to do when walking all those keys that we know it will just ignore next. Adding the 'self.hash' attribute brings memory consumption back up a bit: 252724KB and time down barely at all 0m25.756s Probably we could get more time performance out of StaticTupleInterner by making the internal table only point to StaticTuple objects, and then we could access their self->hash directly. However, seeing this means I can move StaticTuple itself into a pyrex function, since I'll only have C attributes and a PyObject* table. I just hope I can figure out how to get a tp_traverse that Meliae can use... Note that as a reference point, w/ bzr.dev: 326984KB, 53.4s Or we now have a 1.34:1 memory savings and 2:1 speed improvement. The other guess on why StaticTupleInterner works well is that the hash functionality really does spread things out evenly, so we rarely get collisions. And if you don't have a collision, then you don't care about the time for hash() because it is only computed for the new key (which you have to do anyway) and not for any of the entries already present. Time for 'bzr log -r -1 -n0 bzr.dev' is at 691ms which is ~= bzr.dev 'bzr log -v bzr.dev' is 19.1 => 18.7s and 169MB=>197MB which is a net loss... :( May have to revisit the CHKMap internals. 'bzr log -n0 -v bzr.dev' is 2m25s=>2m21s and 235MB=>244MB which isn't a great tradeoff. Strange that things don't seem to be a win in 'real-life', I wonder if somewhere we are casting things back into regular tuples that I missed. (tuple(tpl) is tpl, but tuple(static_tpl) is not static_tpl). There are also possibilities that tuples are special cased in more places, or the custom __iter__ functionality or... etc. Next up, mem testing on 64-bit.	John Arbash Meinel	15 years ago
4679.3.67	Minor change to the StaticTupleInterner_Discard api, and sta... Minor change to the StaticTupleInterner_Discard api, and start using the new StaticTupleInterner api in StaticTuple.	John Arbash Meinel	15 years ago
4679.3.66	Implement the C variant of __iter__ Implement the C variant of __iter__	John Arbash Meinel	15 years ago
4679.3.65	Add __iter__ support. Add __iter__ support.	John Arbash Meinel	15 years ago
4679.3.64	Add functionality for shrinking the table. Add functionality for shrinking the table.	John Arbash Meinel	15 years ago
4679.3.63	Implement resizing. Implement resizing. It can resize up or down, but right now we never shrink.	John Arbash Meinel	15 years ago
4679.3.62	Move some of the information into the pxd header file. Move some of the information into the pxd header file. This will make it easier to directly use attributes of the class in other modules. (More of a case for StaticTuple than StaticTupleInterner, but I figured I'd get practice using it.	John Arbash Meinel	15 years ago
4679.3.61	Invert the logic. Invert the logic. I wanted to add type checking to the C api, however if I do that, I should then create a set of internal functions that don't do type checking. But if I'm going to do that, then I'd rather just hang them off the nice class I already have written. And the C api just becomes thunks to the class attributes.	John Arbash Meinel	15 years ago
4679.3.60	Start working on more of the C api for StaticTupleInterner. Start working on more of the C api for StaticTupleInterner. It is really nice to have the except -1 clauses, as suddenly you don't have to manually check the return values of all of your functions. \o/	John Arbash Meinel	15 years ago
4679.3.59	Capitolize StaticTuple_Intern to conform more to the python ... Capitolize StaticTuple_Intern to conform more to the python C apis.	John Arbash Meinel	15 years ago
4679.3.58	Adding a StaticTupleInterner class. Adding a StaticTupleInterner class. Functions basically like a dict, but with less memory overhead. (well, possibly more like a set that also implements __getitem__). For now, it allows arbitrary objects, though I'm considering only allowing StaticTuple objects for some ease-of-implementation. Especially if I find I want to cache the hash() on StaticTuple for performance, then I may want to expose that in StaticTupleInterner. For now, this only compiles via Cython, but only because I'm using 'inline' in one place. It wouldn't be too hard to factor that out, or make it conditional with an IF and DEF bits.	John Arbash Meinel	15 years ago
4679.3.57	Some ideas about StaticTuple.__richcompare__. Some ideas about StaticTuple.__richcompare__.	John Arbash Meinel	15 years ago
4679.3.56	Fix a test that depended on but didn't 'require' meliae Fix a test that depended on but didn't 'require' meliae	John Arbash Meinel	15 years ago
4679.3.55	A quick note about how I'd like to do things. A quick note about how I'd like to do things. Basically, if I could procedurally declare my API, then all of the boilerplate could be easily generated from that. However, it needs to be 'static' in a C header file, and to get that you'd have to generate it from some other source or have a lot more voodoo in your C pre-processor code. Shame that C sucks so much. Note that Pyrex gets this right, where doing all the boilerplate amounts to adding 'api' to your definition. Aka cdef api object myfunc(int foo) Is enough to generate all the boilerplate C headers, etc you would need. It is unfortunate I can't really use that here, because I want a pure C class to avoid the python garbage collector.	John Arbash Meinel	15 years ago
4679.3.54	Factor out some of the C API code into some helper headers. Factor out some of the C API code into some helper headers. This should make the process a bit easier, if we ever decide to do something similar again.	John Arbash Meinel	15 years ago
4679.3.53	Clean up the fprintf() debugging code. Clean up the fprintf() debugging code.	John Arbash Meinel	15 years ago
4679.3.52	messy but working. messy but working. Start exposing the C API using a nice python dict, rather than offsets into an array. This is much more convenient and allows a small amount of api skew without serious problems. It also allows some safety checks to be performed during import. Mostly we just need to clean up the code a bit.	John Arbash Meinel	15 years ago
4679.3.51	Add a _static_tuple_c.pxd file to define the C api to pyrex ... Add a _static_tuple_c.pxd file to define the C api to pyrex code. I had to drop the notion of void*items[1] and then subtracting it from tp_basicsize because Pyrex directly checks that sizeof(struct) == tp_basicsize. I understand why it was doing so, but it was very hard to track down. The error during import was basically suppressed, and then I just got segfaults because the type wasn't actually available. Anyway, I'm reasonably happy with using a .pxd file, because then I don't have to worry about getting the invocation right in all callers, though I still have to repeat the 'from x cimport a,b,c,d' which is a bit annoying.	John Arbash Meinel	15 years ago
4679.3.50	Add another quick check, since key_length must be 1 for chk ... Add another quick check, since key_length must be 1 for chk keys.	John Arbash Meinel	15 years ago
4679.3.49	Start using StaticTuple as part of the chk_map api. Start using StaticTuple as part of the chk_map api. All tests now pass, because I changed the apis to not use the direct functions. I'm not 100% sure about the performance impact, that needs to be profiled. However, at least everything works, and potentially this will help memory overhead. I'm not 100% sure about speed just yet.	John Arbash Meinel	15 years ago

Older »