~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 4679.3.72

Rev	Summary	Authors	Date
4679.3.52	messy but working. messy but working. Start exposing the C API using a nice python dict, rather than offsets into an array. This is much more convenient and allows a small amount of api skew without serious problems. It also allows some safety checks to be performed during import. Mostly we just need to clean up the code a bit.	John Arbash Meinel	15 years ago
4679.3.51	Add a _static_tuple_c.pxd file to define the C api to pyrex ... Add a _static_tuple_c.pxd file to define the C api to pyrex code. I had to drop the notion of void*items[1] and then subtracting it from tp_basicsize because Pyrex directly checks that sizeof(struct) == tp_basicsize. I understand why it was doing so, but it was very hard to track down. The error during import was basically suppressed, and then I just got segfaults because the type wasn't actually available. Anyway, I'm reasonably happy with using a .pxd file, because then I don't have to worry about getting the invocation right in all callers, though I still have to repeat the 'from x cimport a,b,c,d' which is a bit annoying.	John Arbash Meinel	15 years ago
4679.3.50	Add another quick check, since key_length must be 1 for chk ... Add another quick check, since key_length must be 1 for chk keys.	John Arbash Meinel	15 years ago
4679.3.49	Start using StaticTuple as part of the chk_map api. Start using StaticTuple as part of the chk_map api. All tests now pass, because I changed the apis to not use the direct functions. I'm not 100% sure about the performance impact, that needs to be profiled. However, at least everything works, and potentially this will help memory overhead. I'm not 100% sure about speed just yet.	John Arbash Meinel	15 years ago
4679.3.48	Prototype of not interning sha1: strings. Prototype of not interning sha1: strings. This ends up saving 268480KB => 260972KB (7.5MB) of memory for all of launchpad. Most of that is because the string interned dict drops from 12MB to 3MB because it holds 100k keys instead of 500k keys. On the plus side, we can be fairly confident that we don't access sha1 keys as plain strings. If we disable interning for the sha1 keys themselves, we bloat by about 8MB, but save 30s => 27s. We would bloat more, but we are trading a 24MB _interned_keys dict for a 12MB one. So what we really need is to get the custom interning dict, so that we don't waste so much memory there.	John Arbash Meinel	15 years ago
4679.3.47	Work out how to expose the C api using the Python PyCObject ... Work out how to expose the C api using the Python PyCObject interface. At this point, we are at the same speed as bzr.dev at doing 'bzr log -n0 -r-1 bzr.dev' We are faster at parsing all keys, and about 20% lower in memory.	John Arbash Meinel	15 years ago
4679.3.46	Play around a bit with changing the hash function. Play around a bit with changing the hash function. In this form, once hash has been called once, we know we are dealing with only strings, all of which have already computed their hash. As such, we can get rid of most of the internal checks and just iterate over the values. This gets us down to 13.8ms (from 15 at worst case, but 9.7 if we cache self->hash). We may not keep this, but it was an interesting experiment.	John Arbash Meinel	15 years ago
4679.3.45	Do some work to handle comparison to object that aren't tupl... Do some work to handle comparison to object that aren't tuples or strings.	John Arbash Meinel	15 years ago
4679.3.44	Special case the empty tuple as a singleton. Special case the empty tuple as a singleton. tuple() already had this property, we just added it to StaticTuple. After doing that and fixing up one more code path in BtreeSerializer (when there are no ref lists), memory is down to: 98876KB w/ 6.0MB in the intern dict. So it seems we had 4MB of tuples in the 'no-ref-list' cases, and just switching it over to StaticTuple caused no memory decrease, which means there were a lot of empty tuples. Makes sense, since chk_bytes is the bulk of our objects and has no ref lists Note that Launchpad's memory consumption is also down to: 268580KB w/ 24MB intern dict. down from (bzr.dev): 326656KB A 1.22:1 reduction.	John Arbash Meinel	15 years ago
4679.3.43	Using StaticTuple.intern() and recursive structures. Using StaticTuple.intern() and recursive structures. Brings peak bzr.dev memory down to: 102920KB, still with a 6.0MB intern dict. This is down from 121052KB with bzr.dev and roughly 107380KB with the Key version. The big win is that all of the btree data structures can be StaticTuple because we know we are just referencing either strings or tuples of strings, or tuples of mixed tuples and strings, but still all very static data.	John Arbash Meinel	15 years ago
4679.3.42	Implement comparison support when using nested StaticTuple o... Implement comparison support when using nested StaticTuple objects.	John Arbash Meinel	15 years ago
4679.3.41	Finish switching the naming to StaticTuple. Finish switching the naming to StaticTuple. This also means stripping out the Keys type, in favor of a StaticTuple that can reference yet another StaticTuple.	John Arbash Meinel	15 years ago
4679.3.40	Switch away from the name Key instead start branding as Stat... Switch away from the name Key instead start branding as StaticTuple.	John Arbash Meinel	15 years ago
4679.3.39	Switching away from pre-fetching the string_hash function dr... Switching away from pre-fetching the string_hash function drops hash() time from 13.9ms down to 14.5ms. 4% loss. Something we may want to revisit, but if this is an opportunity to make a generic StaticTuple type, rather than a Keys specific one, it is probably worthwile.	John Arbash Meinel	15 years ago
4679.3.38	add a test that Key.intern() doesn't create an immortal obje... add a test that Key.intern() doesn't create an immortal object.	John Arbash Meinel	15 years ago
4679.3.37	Change the Key header a bit. It is simpler if you use unsign... Change the Key header a bit. It is simpler if you use unsigned char for the size field, as it gives you more flexibility.	John Arbash Meinel	15 years ago
4679.3.36	Merge in the babune updates Merge in the babune updates	John Arbash Meinel	15 years ago
4679.3.35	Work on making intern() not generate immortal Key objects. Work on making intern() not generate immortal Key objects.	John Arbash Meinel	15 years ago
4679.3.34	Remove the temp work to create Keys() from lists of strings. Remove the temp work to create Keys() from lists of strings. Instead, we should move the Keys() type to point to Key objects. I think it would be reasonable to allow Keys to point to Key or Keys, which would help with the double-nested ref_lists stuff.	John Arbash Meinel	15 years ago
4679.3.33	Change Key away from being a PyVarObject. Change Key away from being a PyVarObject. This will probably help a bit on 64-bit python, but mostly it is because I plan on using the upper bits of the 'info' field to store stuff like 'is_interned' as a bit field. With 32-bits and a max width of 256, it seems silly to waste 24 (or 56) bits that will always be 0.	John Arbash Meinel	15 years ago