Bug#722898: benchmarks

Hello Ben,

thanks for your input! I'm attaching a series of patches to wrap up what we've discussed so far, more details are in the commit messages quoted below.

I've tested the patches by running blockdev-wipe, they are looking good. I haven't tried to build the installer with the new block-dev wipe, though, and therefore would appreciate further testing and/or code review.

Even with the latest version of blockdev-wipe, I'm still seeing ~20% improvement by setting min_speed to zero. Yet, I'd suggest to hold back the speedlimit patch because I'd expect similar gains for the package installation phase. Thus I believe that it makes more sense to set speed_limit_min to zero during the startup of the debian installer. Could someone please advise me as to where that would fit in best?

Cheers,

Thiemo

#5    blockdev-wipe: Set blocksize to 512k

    This should be large enough to avoid excessive system call
    overhead and small enough to prevent problems on systems with
    very little RAM.

#4    blockdev-wipe: Sync at most once per second

    Don't open output devices with O_SYNC, instead sync manually
    every time the progress indicator is updated, but not more
    often than once per second. This yields performance gains
    of up to factor 10 in setups with dm-crypt on dm-raid.

Note: Seven years ago, O_SYNC was added to fix OOM issues
    (cf. bug #381135), however it is believed that this problem
    has been addressed in the kernel by now.

#3    blockdev-wipe: Allocate buffer from heap instead of stack

    This allows to use buffers larger than 8M, also it fails more
    gracefully in case the memory can't be allocated.

#2    blockdev-wipe: Reduce progress indicator granularity to 1/1000

On Sun, Sep 15, 2013 at 12:31 AM, Ben Hutchings <ben@decadent.org.uk> wrote:

On Sat, 2013-09-14 at 23:33 +0200, Thiemo Nagel wrote:
[...]
> What I take away from this: For optimal performance, the frequency of
> syncs should be kept low, probably well below 50 Hz, ideally as low as
> possible. I'd be in favour of removing them altogether, but there
> were some OOM issues seven years ago that were fixed by adding them.
> Does anybody know whether bug 381135 still applies with today's
> kernels?

I believe Linux will throttle writing processes so that the size of
buffered writes doesn't keep growing. And I have never seen problems
when wiping disks with dd (I disposed of a whole bunch of old disks
recently) without periodic sync'ing.

However, the size of buffered writes can still grow quite large, which
will limit the accuracy of a progress display. So I would suggest you
sync whenever you're about to indicate progress, but also increase the
granularity of progress (65536 steps is ridiculous) so that doesn't
happen too often.

> In any case, I'd suggest to go at least for 1M block size, even if
> just to reduce the amount of system calls.
[...]

It looks like 1 MB is just about enough to maximise throughput, and this
matches my memory.

Ben.

--
Ben Hutchings
If at first you don't succeed, you're doing about average.