~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 4679.3.55

Rev	Summary	Authors	Date
4679.3.35	Work on making intern() not generate immortal Key objects. Work on making intern() not generate immortal Key objects.	John Arbash Meinel	15 years ago
4679.3.34	Remove the temp work to create Keys() from lists of strings. Remove the temp work to create Keys() from lists of strings. Instead, we should move the Keys() type to point to Key objects. I think it would be reasonable to allow Keys to point to Key or Keys, which would help with the double-nested ref_lists stuff.	John Arbash Meinel	15 years ago
4679.3.33	Change Key away from being a PyVarObject. Change Key away from being a PyVarObject. This will probably help a bit on 64-bit python, but mostly it is because I plan on using the upper bits of the 'info' field to store stuff like 'is_interned' as a bit field. With 32-bits and a max width of 256, it seems silly to waste 24 (or 56) bits that will always be 0.	John Arbash Meinel	15 years ago
4679.3.32	On bzr.dev, Key.intern() saves a little bit of memory On bzr.dev, Key.intern() saves a little bit of memory we get 107.5MB peak vs 110.4MB peak w/ 6MB for the interned dict. However, for lp it is a wash. 293.5MB peak both ways, with 24MB in the intern dict. Then again, if we shrink the intern dict, we can get down to at least 1/3rd the size, and possibly more depending on whether the dict is using 4:1 vs 2:1 packing. It should be using 2:1, though, since there are 792k entries. (vs 522k string entries....) We might also consider only interning the Key if the key width is 1. It makes sense for sha1 keys, though we would lose a little bit for revisions => inventories => file_keys. However, since sha1 keys are 425k out of all 792k keys, it might be worth investigating. Note that if we get the chk_map deserializer et al using Key as well, that will help get some common references in the inventory parsing code.	John Arbash Meinel	15 years ago
4679.3.31	do a bit of profiling of the time it takes to hash(). do a bit of profiling of the time it takes to hash().	John Arbash Meinel	15 years ago
4679.3.30	Interning with a regular 'dict' is a tradeoff for bzr.dev of... Interning with a regular 'dict' is a tradeoff for bzr.dev of: PeakWorking 110268KB final real 0m5.897s to PeakWorking 107380KB final real 0m6.100s So a small memory gain for a small performance loss. Caching the hash gives: PeakWorking 108700KB final real 0m6.193s someone surprisingly, absolutely no performance gain.	John Arbash Meinel	15 years ago
4679.3.29	Start work on implementing a Key.intern() function. Start work on implementing a Key.intern() function. As part of this, I'm upgrading the python type to be a fully fledged class. It will have more overhead than just using tuples (given that we wrap a tuple) but at least it means we can have an interesting class object that does more than just a tuple interface.	John Arbash Meinel	15 years ago
4679.3.28	Add a KEY_HAS_HASH define. Add a KEY_HAS_HASH define. This turns on/off the caching of the Key.hash. For now, performance shows negligible impact of adding it, so it isn't worth the memory consumption. If we get into adding Key to a KeyDict that doesn't cache the hash independently, we should re-evaluate whether it is worthwhile.	John Arbash Meinel	15 years ago
4679.3.27	Shave some time in btree by caching the 'extend' object. Shave some time in btree by caching the 'extend' object.	John Arbash Meinel	15 years ago
4679.3.26	Changing ref_lists back into using tuple([Key1, Key2]) rathe... Changing ref_lists back into using tuple([Key1, Key2]) rather than Keys() brings us down to 648ms. So obviously there is a lot to be gained by improving the Keys constructor, and possibly the on-demand creation of Key objects from Keys is killing performance.	John Arbash Meinel	15 years ago
4679.3.25	Using Key_New() rather than Key(PyTuple_New()) gets us to 72... Using Key_New() rather than Key(PyTuple_New()) gets us to 720ms. It takes some gymnastics to get it to work, because we have to export the symbol and then find the .lib file to use it from the pyrex code. setup.py syntax doesn't seem to support that cleanly :( Going further, we'll have to update _flatten_node to support Key nodes, since right now it has some explicit 'is tuple' checks that fail.	John Arbash Meinel	15 years ago
4679.3.24	Implement Key_New like Tuple_New to create a key from a 3rd-... Implement Key_New like Tuple_New to create a key from a 3rd-party source. Have Keys_item() return Key types rather than tuples. Drops the time to 738ms, probably because we then mostly compare Key <=> Key, rather than comparing Key <=> Tuple.	John Arbash Meinel	15 years ago
4679.3.23	A bit more testing shows that we are comparing Key to tuples... A bit more testing shows that we are comparing Key to tuples fairly often. I wonder if it isn't something about Keys.__getitem__ returning tuples.	John Arbash Meinel	15 years ago
4679.3.22	It seems that optimizing Keys_hash is not the way to go for ... It seems that optimizing Keys_hash is not the way to go for 'bzr log' performance. Baseline bzr.dev time: 702ms Time with Key: 760ms Time with Key_richcompare: 751ms Time with Key_hash: 750ms Time with Key.hash (stashed hash): 750ms. Probably need some profiling, though hard to get accurate profiling at this level.	John Arbash Meinel	15 years ago
4679.3.21	Implement Key_richcompare directly, rather than thunking to ... Implement Key_richcompare directly, rather than thunking to tuples.	John Arbash Meinel	15 years ago
4679.3.20	Implement tp_traverse, and add a soft dependency on meliae t... Implement tp_traverse, and add a soft dependency on meliae to test it. Basically, gc.get_referents doesn't use tp_traverse if the object isn't being tracked. Which is ok, but not what we want here. Key isn't in GC because it is overkill for something that can never have cyclical references. So I implemented a get_referents in meliae that ignores IS_GC.	John Arbash Meinel	15 years ago
4679.3.19	implement slicing as a tuple thunk. implement slicing as a tuple thunk. Clean up a few internals that was causing problems. It seems that btree requires slicing. We should probably implement it directly, depending on what we need. It is also possible now for the internal extract_key to use a Key object, rather than a tuple.	John Arbash Meinel	15 years ago
4679.3.18	Copy the hash and richcompare implementations, and add some ... Copy the hash and richcompare implementations, and add some tests. Again, all of this is bootstrapped by the fact that there is an 'as_tuple' representation.	John Arbash Meinel	15 years ago
4679.3.17	Implement some of the sequence items. Implement some of the sequence items. This also revealed that we need to decref during Key_dealloc.	John Arbash Meinel	15 years ago
4679.3.16	Initial work for a Key class. Initial work for a Key class.	John Arbash Meinel	15 years ago