Re: Introducing security hardening features for Lenny
On Wed, Mar 05, 2008 at 01:29:01AM -0800, Don Armstrong wrote:
> Just for future reference, it'd probably be better to run more than 5
> tests of each population in the future, as 5 tests means you'll only
> detect very large differences in performance at any reasonable level
> of signifigance.
I suspected. :) I'm not much of a statistician, so I saved the
raw data, hoping other folks would hop in to help out on this one.
(Which you have!)
Getting larger data sets will be rather time-consuming -- especially for
nexuiz which I didn't figure out how to automate.
> > t.test(x=c(10.87,10.873,10.854,10.809,10.877),y=c(10.807,10.824,10.963,10.84,10.838))
What tool is this you're using?
> Welch Two Sample t-test
>
> data: c(10.87, 10.873, 10.854, 10.809, 10.877) and c(10.807, 10.824, 10.963, 10.84, 10.838)
> t = 0.0722, df = 5.561, p-value = 0.945
> alternative hypothesis: true difference in means is not equal to 0
> 95 percent confidence interval:
> -0.07382831 0.07822831
> sample estimates:
> mean of x mean of y
> 10.8566 10.8544
Which of these outputs should be paid attention to?
> But useful data nevertheless.[1]
>
> 1: I won't even begin to discuss how many times I see benchmarks
> without SEM or sd reported.
Heh, well I know of the ideas, but haven't had any practice actually
calculating them.
Thanks!
-Kees
--
Kees Cook @outflux.net
Reply to: