Re: Introducing security hardening features for Lenny
On Tue, 04 Mar 2008, Kees Cook wrote:
> mplayer doesn't compile with PIE due to the various ASM routines. (I've
> noted this failure mode in the wiki[2] now.) However, with everything
> else enabled (including FORTIFY_SOURCE), there was no measurable
> difference (it was below the percentage difference between runs):
>
> runtime in seconds
> Mplayer Normal Hardened
> 1 10.87 10.807
> 2 10.873 10.824
> 3 10.854 10.963
> 4 10.809 10.84
> 5 10.877 10.838
Just for future reference, it'd probably be better to run more than 5
tests of each population in the future, as 5 tests means you'll only
detect very large differences in performance at any reasonable level
of signifigance.
FE:
> t.test(x=c(10.87,10.873,10.854,10.809,10.877),y=c(10.807,10.824,10.963,10.84,10.838))
Welch Two Sample t-test
data: c(10.87, 10.873, 10.854, 10.809, 10.877) and c(10.807, 10.824, 10.963, 10.84, 10.838)
t = 0.0722, df = 5.561, p-value = 0.945
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-0.07382831 0.07822831
sample estimates:
mean of x mean of y
10.8566 10.8544
> This one showed a possible difference:
>
> nexuiz Normal Hardened
> 1 66.68 68.113
> 2 66.802 66.93
> 3 66.758 67.03
> 4 66.728 67.051
> 5 66.859 67.037
While there may be a possible difference here, it's not significant,
given the sample size:
> t.test(x=c(66.68,66.802,66.758,66.728,66.859),y=c(68.113,66.93,67.03,67.051,67.037))
Welch Two Sample t-test
data: c(66.68, 66.802, 66.758, 66.728, 66.859) and c(68.113, 66.93, 67.03, 67.051, 67.037)
t = -2.0899, df = 4.154, p-value = 0.1023
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
-1.0779888 0.1443888
sample estimates:
mean of x mean of y
66.7654 67.2322
But useful data nevertheless.[1]
Don Armstrong
1: I won't even begin to discuss how many times I see benchmarks
without SEM or sd reported.
--
I'd sign up in a hot second for any cellular company whose motto was:
"We're less horrible than a root canal with a cold chisel."
-- Cory Doctorow
http://www.donarmstrong.com http://rzlab.ucr.edu
Reply to: