~bzr-pqm/bzr/bzr.dev : changes

~bzr-pqm/bzr/bzr.dev » Changes from revision 3735.33.18

Rev	Summary	Authors	Date
3735.33.18	grow the local hmask if it is smaller than expected, don't... grow the local hmask if it is smaller than expected, don't shrink it.	John Arbash Meinel	15 years ago
3735.33.17	Remove an invalid assert. Remove an invalid assert. The assert is only valid if we grow at most by 1 level, with the new limitation of forcing the hsize, we can grow by a lot more than that. (f=>ff). Rather than writing a new assert that fits all cases, just remove it.	John Arbash Meinel	15 years ago
3735.33.16	Handle when our current packing is sub-optimal. Handle when our current packing is sub-optimal. It happens somtimes that our estimated hsize is too big, so that the next estimate tries to shrink it. However the code like pack_delta_index only copes with growing, and that is honestly all we really care about.	John Arbash Meinel	15 years ago
3735.33.15	Remove the noisy debugging code. (down to 23.1s) Remove the noisy debugging code. (down to 23.1s)	John Arbash Meinel	15 years ago
3735.33.14	Simplify the code a bit. We don't repack as often, so go wit... Simplify the code a bit. We don't repack as often, so go with a more obvious code, rather than trying tricks with memcpy() (it didn't seem to really help, anyway).	John Arbash Meinel	15 years ago
3735.33.13	Tweak the number of blank spaces up just a tad. Tweak the number of blank spaces up just a tad. It seems that setting it to 8 doesn't see a net gain. There is a slight improvement in the number of readjustments done, but that is counteracted by the make_delta time.	John Arbash Meinel	15 years ago
3735.33.12	Now we are able to weave 'add_source' into the existing inde... Now we are able to weave 'add_source' into the existing index. This brings 'bzr pack' time down to ~23.6s (with debugging on). According to lsprof time for 'add_delta_source' overall dropped from ~5s down to about 300ms, and now the time for 'add_source' dropped 5s->3.3s->1.6s. Next thing is to probably bump the number of free slots.	John Arbash Meinel	15 years ago
3735.33.11	Shave 5s->3.3s in add_source by copying old entries across d... Shave 5s->3.3s in add_source by copying old entries across directly. This works quite a bit better than copying them over into the hash and then copying them back into the final packed form.	John Arbash Meinel	15 years ago
3735.33.10	Fix a bug when there is more than one entry (increment out_e... Fix a bug when there is more than one entry (increment out_entry). Also make the mini_hsize always the same size as hsize. Otherwise we end up walking the same nodes over and over again. This way, we only walk nodes when we are going to be inserting them.	John Arbash Meinel	15 years ago
3735.33.9	When expanding an index put the entries into a hash. When expanding an index put the entries into a hash. Rather than iterating all entries for every hash index, create another mini hash, and pull them out of there.	John Arbash Meinel	15 years ago
3735.33.8	Reverted back to the same hash width, and bumped EXTRA_NULLS... Reverted back to the same hash width, and bumped EXTRA_NULLS to 3. Most entries in a hash bucket are genuinely random, so they don't trigger extra comparisons. So walking 4-7 nodes is fairly cheap at that level. My guess is that bumping EXTRA_NULL has a bigger effect when you get the occassional non-random data, that forces expansion because it gets a collision. Data with repetition a multiple of 16 (but not 16) will cause this, as you can get a large insertion, with lots of dupes. We filter out when the dupe is exactly a multiple of 16, we may want to do something similar at larger ranges (or use limit_hash_table on the data possibly with a much smaller value than 64.) Most important (next) is to handle the large update case.	John Arbash Meinel	15 years ago
3735.33.7	Different attempt, which I thought would give similar result... Different attempt, which I thought would give similar results but doesn't. Changing it back to EXTRA_NULLS=1 and setting hash map to be twice as wide. This results in 9.2k inserts, 1.3k expands, and overall 3m58s. Also, the assumption of 'not many added' is patently false. The biggest I've seen now is 2.2k insertions, and 2.2k*4k index is going to be stupid-painful.	John Arbash Meinel	15 years ago
3735.33.6	Increasing EXTRA_NULLS to 2 from 1 ups the hit rate Increasing EXTRA_NULLS to 2 from 1 ups the hit rate 9.1k => 10k without expanding, and 1407=>433 expansions. Drops the overall time 3m45s=>3m40s.	John Arbash Meinel	15 years ago
3735.33.5	Restoring the debugging for now. Restoring the debugging for now. Only matching commands that are > RABIN_WINDOW + 3 reduces the total number of resizes from 3.1k to 1.4k. Reduces overall matches from 9.6k => 9.1k. Those match commands were flooding the hash map, because they get repeated and always hit the same hash bucket. That said, this is seems overall slower than the old code, my guess is the O(MN) behavior of the resize loop. Time to put the new data into its own hash. :)	John Arbash Meinel	15 years ago
3735.33.4	The new layout is working. The new layout is working. Commenting out the debug info for now. What I'm finding is a surprising number of repeated strings. Basically, common strings of length < 20, which then end up indexed by the RABIN code, but don't get copied in the output. (because RABIN is a 16-byte match, but the copy command has a minimum size of 20-bytes. Perhaps we need to change the code so that it doesn't try to index <20 character inserts. Or change the copy code so that it allows shorter copies.	John Arbash Meinel	15 years ago
3735.33.3	include_entries_from_hash wasn't properly skipping NULL reco... include_entries_from_hash wasn't properly skipping NULL records. Now the tests pass again, and we can look at bringing back a simpler create_delta_index_from_delta.	John Arbash Meinel	15 years ago
3735.33.2	Revert some of the previous work. Revert some of the previous work. The tests start failing if we insert extra null spaces, so get back to a point where they are passing and work from there.	John Arbash Meinel	15 years ago
3735.33.1	(broken, in progress), change the data structures to allow f... (broken, in progress), change the data structures to allow for inserting small deltas. By adding 2 blank spots per hash spot, we can normally update the structure without having to resize the whole thing. We'll probably want to tune how many extra slots to provide. The general work is probably good, but I need to finish handling the case when we really do need to resize the structure.	John Arbash Meinel	15 years ago
3735.2.152	Merge bzr.dev 4157, this breaks a couple per-repository test... Merge bzr.dev 4157, this breaks a couple per-repository tests. Looks like removing some of the InterRepo optimizers is revealing places that we haven't fully finished the RemoteRepo api.	John Arbash Meinel	15 years ago
3735.2.151	A the source grouping code needs to update current_source A the source grouping code needs to update current_source	John Arbash Meinel	15 years ago

Older »