~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 4679.3.59

Rev	Summary	Authors	Date
4679.3.59	Capitolize StaticTuple_Intern to conform more to the python ... Capitolize StaticTuple_Intern to conform more to the python C apis.	John Arbash Meinel	15 years ago
4679.3.58	Adding a StaticTupleInterner class. Adding a StaticTupleInterner class. Functions basically like a dict, but with less memory overhead. (well, possibly more like a set that also implements __getitem__). For now, it allows arbitrary objects, though I'm considering only allowing StaticTuple objects for some ease-of-implementation. Especially if I find I want to cache the hash() on StaticTuple for performance, then I may want to expose that in StaticTupleInterner. For now, this only compiles via Cython, but only because I'm using 'inline' in one place. It wouldn't be too hard to factor that out, or make it conditional with an IF and DEF bits.	John Arbash Meinel	15 years ago
4679.3.57	Some ideas about StaticTuple.__richcompare__. Some ideas about StaticTuple.__richcompare__.	John Arbash Meinel	15 years ago
4679.3.56	Fix a test that depended on but didn't 'require' meliae Fix a test that depended on but didn't 'require' meliae	John Arbash Meinel	15 years ago
4679.3.55	A quick note about how I'd like to do things. A quick note about how I'd like to do things. Basically, if I could procedurally declare my API, then all of the boilerplate could be easily generated from that. However, it needs to be 'static' in a C header file, and to get that you'd have to generate it from some other source or have a lot more voodoo in your C pre-processor code. Shame that C sucks so much. Note that Pyrex gets this right, where doing all the boilerplate amounts to adding 'api' to your definition. Aka cdef api object myfunc(int foo) Is enough to generate all the boilerplate C headers, etc you would need. It is unfortunate I can't really use that here, because I want a pure C class to avoid the python garbage collector.	John Arbash Meinel	15 years ago
4679.3.54	Factor out some of the C API code into some helper headers. Factor out some of the C API code into some helper headers. This should make the process a bit easier, if we ever decide to do something similar again.	John Arbash Meinel	15 years ago
4679.3.53	Clean up the fprintf() debugging code. Clean up the fprintf() debugging code.	John Arbash Meinel	15 years ago
4679.3.52	messy but working. messy but working. Start exposing the C API using a nice python dict, rather than offsets into an array. This is much more convenient and allows a small amount of api skew without serious problems. It also allows some safety checks to be performed during import. Mostly we just need to clean up the code a bit.	John Arbash Meinel	15 years ago
4679.3.51	Add a _static_tuple_c.pxd file to define the C api to pyrex ... Add a _static_tuple_c.pxd file to define the C api to pyrex code. I had to drop the notion of void*items[1] and then subtracting it from tp_basicsize because Pyrex directly checks that sizeof(struct) == tp_basicsize. I understand why it was doing so, but it was very hard to track down. The error during import was basically suppressed, and then I just got segfaults because the type wasn't actually available. Anyway, I'm reasonably happy with using a .pxd file, because then I don't have to worry about getting the invocation right in all callers, though I still have to repeat the 'from x cimport a,b,c,d' which is a bit annoying.	John Arbash Meinel	15 years ago
4679.3.50	Add another quick check, since key_length must be 1 for chk ... Add another quick check, since key_length must be 1 for chk keys.	John Arbash Meinel	15 years ago
4679.3.49	Start using StaticTuple as part of the chk_map api. Start using StaticTuple as part of the chk_map api. All tests now pass, because I changed the apis to not use the direct functions. I'm not 100% sure about the performance impact, that needs to be profiled. However, at least everything works, and potentially this will help memory overhead. I'm not 100% sure about speed just yet.	John Arbash Meinel	15 years ago
4679.3.48	Prototype of not interning sha1: strings. Prototype of not interning sha1: strings. This ends up saving 268480KB => 260972KB (7.5MB) of memory for all of launchpad. Most of that is because the string interned dict drops from 12MB to 3MB because it holds 100k keys instead of 500k keys. On the plus side, we can be fairly confident that we don't access sha1 keys as plain strings. If we disable interning for the sha1 keys themselves, we bloat by about 8MB, but save 30s => 27s. We would bloat more, but we are trading a 24MB _interned_keys dict for a 12MB one. So what we really need is to get the custom interning dict, so that we don't waste so much memory there.	John Arbash Meinel	15 years ago
4679.3.47	Work out how to expose the C api using the Python PyCObject ... Work out how to expose the C api using the Python PyCObject interface. At this point, we are at the same speed as bzr.dev at doing 'bzr log -n0 -r-1 bzr.dev' We are faster at parsing all keys, and about 20% lower in memory.	John Arbash Meinel	15 years ago
4679.3.46	Play around a bit with changing the hash function. Play around a bit with changing the hash function. In this form, once hash has been called once, we know we are dealing with only strings, all of which have already computed their hash. As such, we can get rid of most of the internal checks and just iterate over the values. This gets us down to 13.8ms (from 15 at worst case, but 9.7 if we cache self->hash). We may not keep this, but it was an interesting experiment.	John Arbash Meinel	15 years ago
4679.3.45	Do some work to handle comparison to object that aren't tupl... Do some work to handle comparison to object that aren't tuples or strings.	John Arbash Meinel	15 years ago
4679.3.44	Special case the empty tuple as a singleton. Special case the empty tuple as a singleton. tuple() already had this property, we just added it to StaticTuple. After doing that and fixing up one more code path in BtreeSerializer (when there are no ref lists), memory is down to: 98876KB w/ 6.0MB in the intern dict. So it seems we had 4MB of tuples in the 'no-ref-list' cases, and just switching it over to StaticTuple caused no memory decrease, which means there were a lot of empty tuples. Makes sense, since chk_bytes is the bulk of our objects and has no ref lists Note that Launchpad's memory consumption is also down to: 268580KB w/ 24MB intern dict. down from (bzr.dev): 326656KB A 1.22:1 reduction.	John Arbash Meinel	15 years ago
4679.3.43	Using StaticTuple.intern() and recursive structures. Using StaticTuple.intern() and recursive structures. Brings peak bzr.dev memory down to: 102920KB, still with a 6.0MB intern dict. This is down from 121052KB with bzr.dev and roughly 107380KB with the Key version. The big win is that all of the btree data structures can be StaticTuple because we know we are just referencing either strings or tuples of strings, or tuples of mixed tuples and strings, but still all very static data.	John Arbash Meinel	15 years ago
4679.3.42	Implement comparison support when using nested StaticTuple o... Implement comparison support when using nested StaticTuple objects.	John Arbash Meinel	15 years ago
4679.3.41	Finish switching the naming to StaticTuple. Finish switching the naming to StaticTuple. This also means stripping out the Keys type, in favor of a StaticTuple that can reference yet another StaticTuple.	John Arbash Meinel	15 years ago
4679.3.40	Switch away from the name Key instead start branding as Stat... Switch away from the name Key instead start branding as StaticTuple.	John Arbash Meinel	15 years ago

Older »