Changes

final benchmark run
Line 115: Line 115:  
it's comparing very well with the other systems: It's on second place both for total (unweighted) runtime and
 
it's comparing very well with the other systems: It's on second place both for total (unweighted) runtime and
 
final repository size, giving perfect balance between those two goals.
 
final repository size, giving perfect balance between those two goals.
 +
 +
=== 2009-06-08 ===
 +
 +
Finished the planned enhancements to the benchmark. After two runs (with the timings differing by about 20% on average) of about 13h each
 +
(with only the 100 of the Project Gutenberg files) there are a few loosers, but no clear winner. Monotone comes close but doesn't seem
 +
to scale well with branches (both size and runtime) and also has the highest memory consumption.
 +
 +
The results emphasize need to use an abstraction layer so we can switch between different VCSs (depending on what resource is
 +
scarcest) or even to a homegrown backend later. For the prototype git seems to be a good candidate as it combines average
 +
resource consumption with a rich API, including low-level access.
 +
 +
 +
==== Benchmark results ====
 +
 +
[[Image:Vcs-benchmark-run-2-total.png|thumb|Summary of the final run of the VCS benchmark]]
 +
 +
Darcs is fast, but needs over four times the original size to store the data.
 +
 +
[[Image:Vcs-benchmark-run-2-op-vs-size.png|thumb|Repository size summary]]
 +
 +
[[Image:Vcs-benchmark-run-2-op-vs-time.png|thumb|Runtime summary]]
 +
 +
[[Image:Vcs-benchmark-run-2-op-vs-size-monotone.png|thumb|Repository sizes for monotone]]
 +
[[Image:Vcs-benchmark-run-2-op-vs-time-monotone.png|thumb|Timings for monotone]]
 +
 +
Monotone starts out very small, but incurs high costs on branches.
 +
 +
==== Memory usage ====
 +
 +
Determining reliable figures for memory usage, especially peak usage, seems to require using valgrind which is - according to its
 +
documentation - slowing down program execution by a factor of 5 (which would mean 65h for the already reduced sample set).
 +
Instead I've done a few simple measurements using [http://www.gnu.org/software/acct/ BSD process accounting] for a run of the
 +
benchmark with a single file as sample set. This should at least give a very rough idea of memory consumption, especially
 +
relative to the other VCSs in the run.
 +
 +
{|
 +
|-
 +
|arch (incl. tar)
 +
|align="right" | 1340k
 +
|-
 +
|bazaar
 +
|align="right" | 5060k
 +
|-
 +
|cvs
 +
|align="right" | 957k
 +
|-
 +
|darcs
 +
|align="right" | 7385k
 +
|-
 +
|git
 +
|align="right" | 1614k
 +
|-
 +
|mercurial
 +
|align="right" | 3501k
 +
|-
 +
|monotone
 +
|align="right" | 9865k
 +
|-
 +
|subversion
 +
|align="right" | 4121k
 +
|-
 +
|}
344

edits