Re: Introducing security hardening features for Lenny

[Kees Cook <kees@outflux.net>]

Hi,

I finally got some time to run some benchmarks.  I checked the results[1]
into the "hardening" svn tree, in case other people want to contribute
more stuff.

On Wed, Jan 30, 2008 at 08:46:55PM +0100, Moritz Muehlenhoff wrote:
> Video:
> mplayer with the -benchmark option in conjunction with -nosound and -vo.

mplayer doesn't compile with PIE due to the various ASM routines.  (I've
noted this failure mode in the wiki[2] now.)  However, with everything
else enabled (including FORTIFY_SOURCE), there was no measurable
difference (it was below the percentage difference between runs):

        runtime in seconds
Mplayer Normal  Hardened    
1        10.87   10.807  
2        10.873  10.824  
3        10.854  10.963  
4        10.809  10.84   
5        10.877  10.838  
avg      10.8566 10.8544   diff: -0.02%
error     0.19%   1.00%   

> HTML rendering:
> Mike Hommey once blogged about benchmarking the ACID test:
> http://web.glandium.org/blog/?cat=17

I followed that to: http://celtickane.com/webdesign/jsspeed2007.php
The differences between runs were too high for me to use, so I skipped
this for now.

> Nexuiz:
> | To run the benchmark: start Nexuiz & open the console (`) issuing:
> | timedemo demos/demo1.dem The results will be stored in:
> | ~/.nexuiz/data/benchmark.log

This one showed a possible difference:

nexuiz  Normal  Hardened    
1       66.68   68.113  
2       66.802  66.93   
3       66.758  67.03   
4       66.728  67.051  
5       66.859  67.037  
avg     66.7654 67.2322  diff: 0.70%
error    0.14%   1.31%   

So, for nexuiz, with all hardening enabled in i386, there was a
less-than-1-percent reduction in speed.  Though the error margin for the
hardened runs were still larger than the measured slow-down.

> Not sure about XML benchmarks.

I did parse/render tests with inkscape on i386.  Some of that is XML, but
I figured it was heavy CPU, which might be worth something.  Note that
inkscape already compiles with all hardening options (excepting PIE),
so the "hardened" time differences are entirely due to PIE.  This one
turned out similar to nexuiz, but with less error.  Again, less than 1
percent slow-down was seen.

inkscape    Normal  Hardened    
1           48.163  48.503  
2           48.227  48.535  
3           48.267  48.647  
4           48.335  48.431  
5           48.199  48.587  
avg         48.2382 48.5406   diff: 0.63%
error        0.20%   0.22%   

I also ran inkscape and nexuiz tests on x86_64, and there was no
measurable difference.  I'm unclear if this was due to the extra
registers, or just that that CPU was much faster and the difference
vanished into the noise.

-Kees


[1] http://svn.debian.org/wsvn/hardening/benchmarks/
[2] http://wiki.debian.org/Hardening

-- 
Kees Cook                                            @outflux.net