~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 3735.33.12

Rev	Summary	Authors	Date
3735.33.12	Now we are able to weave 'add_source' into the existing inde... Now we are able to weave 'add_source' into the existing index. This brings 'bzr pack' time down to ~23.6s (with debugging on). According to lsprof time for 'add_delta_source' overall dropped from ~5s down to about 300ms, and now the time for 'add_source' dropped 5s->3.3s->1.6s. Next thing is to probably bump the number of free slots.	John Arbash Meinel	15 years ago
3735.33.11	Shave 5s->3.3s in add_source by copying old entries across d... Shave 5s->3.3s in add_source by copying old entries across directly. This works quite a bit better than copying them over into the hash and then copying them back into the final packed form.	John Arbash Meinel	15 years ago
3735.33.10	Fix a bug when there is more than one entry (increment out_e... Fix a bug when there is more than one entry (increment out_entry). Also make the mini_hsize always the same size as hsize. Otherwise we end up walking the same nodes over and over again. This way, we only walk nodes when we are going to be inserting them.	John Arbash Meinel	15 years ago
3735.33.9	When expanding an index put the entries into a hash. When expanding an index put the entries into a hash. Rather than iterating all entries for every hash index, create another mini hash, and pull them out of there.	John Arbash Meinel	15 years ago
3735.33.8	Reverted back to the same hash width, and bumped EXTRA_NULLS... Reverted back to the same hash width, and bumped EXTRA_NULLS to 3. Most entries in a hash bucket are genuinely random, so they don't trigger extra comparisons. So walking 4-7 nodes is fairly cheap at that level. My guess is that bumping EXTRA_NULL has a bigger effect when you get the occassional non-random data, that forces expansion because it gets a collision. Data with repetition a multiple of 16 (but not 16) will cause this, as you can get a large insertion, with lots of dupes. We filter out when the dupe is exactly a multiple of 16, we may want to do something similar at larger ranges (or use limit_hash_table on the data possibly with a much smaller value than 64.) Most important (next) is to handle the large update case.	John Arbash Meinel	15 years ago
3735.33.7	Different attempt, which I thought would give similar result... Different attempt, which I thought would give similar results but doesn't. Changing it back to EXTRA_NULLS=1 and setting hash map to be twice as wide. This results in 9.2k inserts, 1.3k expands, and overall 3m58s. Also, the assumption of 'not many added' is patently false. The biggest I've seen now is 2.2k insertions, and 2.2k*4k index is going to be stupid-painful.	John Arbash Meinel	15 years ago
3735.33.6	Increasing EXTRA_NULLS to 2 from 1 ups the hit rate Increasing EXTRA_NULLS to 2 from 1 ups the hit rate 9.1k => 10k without expanding, and 1407=>433 expansions. Drops the overall time 3m45s=>3m40s.	John Arbash Meinel	15 years ago
3735.33.5	Restoring the debugging for now. Restoring the debugging for now. Only matching commands that are > RABIN_WINDOW + 3 reduces the total number of resizes from 3.1k to 1.4k. Reduces overall matches from 9.6k => 9.1k. Those match commands were flooding the hash map, because they get repeated and always hit the same hash bucket. That said, this is seems overall slower than the old code, my guess is the O(MN) behavior of the resize loop. Time to put the new data into its own hash. :)	John Arbash Meinel	15 years ago
3735.33.4	The new layout is working. The new layout is working. Commenting out the debug info for now. What I'm finding is a surprising number of repeated strings. Basically, common strings of length < 20, which then end up indexed by the RABIN code, but don't get copied in the output. (because RABIN is a 16-byte match, but the copy command has a minimum size of 20-bytes. Perhaps we need to change the code so that it doesn't try to index <20 character inserts. Or change the copy code so that it allows shorter copies.	John Arbash Meinel	15 years ago
3735.33.3	include_entries_from_hash wasn't properly skipping NULL reco... include_entries_from_hash wasn't properly skipping NULL records. Now the tests pass again, and we can look at bringing back a simpler create_delta_index_from_delta.	John Arbash Meinel	15 years ago
3735.33.2	Revert some of the previous work. Revert some of the previous work. The tests start failing if we insert extra null spaces, so get back to a point where they are passing and work from there.	John Arbash Meinel	15 years ago
3735.33.1	(broken, in progress), change the data structures to allow f... (broken, in progress), change the data structures to allow for inserting small deltas. By adding 2 blank spots per hash spot, we can normally update the structure without having to resize the whole thing. We'll probably want to tune how many extra slots to provide. The general work is probably good, but I need to finish handling the case when we really do need to resize the structure.	John Arbash Meinel	15 years ago
3735.2.152	Merge bzr.dev 4157, this breaks a couple per-repository test... Merge bzr.dev 4157, this breaks a couple per-repository tests. Looks like removing some of the InterRepo optimizers is revealing places that we haven't fully finished the RemoteRepo api.	John Arbash Meinel	15 years ago
3735.2.151	A the source grouping code needs to update current_source A the source grouping code needs to update current_source	John Arbash Meinel	15 years ago
3735.2.150	always repack gc repositories for now, even if only one pack... always repack gc repositories for now, even if only one pack there	Ian Clatworthy	15 years ago
3735.2.149	handle path2id when root not yet known handle path2id when root not yet known	Ian Clatworthy	15 years ago
3735.2.148	Merge the 'trivially correct' things from the flight. Merge the 'trivially correct' things from the flight. Restoring labels because the tests expect them.	John Arbash Meinel	15 years ago
3735.2.147	Track down and fix an unexpected edge case. Track down and fix an unexpected edge case. It is possible for a root node (referenced from an inventory) to not exist in the currently repacking subset of the pack files. We don't want to just use a set(), because we want to preserve the key ordering.	John Arbash Meinel	15 years ago
3735.2.146	Give the right progress message. Give the right progress message.	John Arbash Meinel	15 years ago
3735.2.145	Access the right member variable Access the right member variable	John Arbash Meinel	15 years ago

Older »