[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Introducing security hardening features for Lenny

On Wed, Mar 05, 2008 at 01:29:01AM -0800, Don Armstrong wrote:
> Just for future reference, it'd probably be better to run more than 5
> tests of each population in the future, as 5 tests means you'll only
> detect very large differences in performance at any reasonable level
> of signifigance.

I suspected.  :)  I'm not much of a statistician, so I saved the
raw data, hoping other folks would hop in to help out on this one.
(Which you have!)

Getting larger data sets will be rather time-consuming -- especially for
nexuiz which I didn't figure out how to automate.

> > t.test(x=c(10.87,10.873,10.854,10.809,10.877),y=c(10.807,10.824,10.963,10.84,10.838))

What tool is this you're using?

> 	Welch Two Sample t-test
> data:  c(10.87, 10.873, 10.854, 10.809, 10.877) and c(10.807, 10.824, 10.963, 10.84, 10.838) 
> t = 0.0722, df = 5.561, p-value = 0.945
> alternative hypothesis: true difference in means is not equal to 0 
> 95 percent confidence interval:
>  -0.07382831  0.07822831 
> sample estimates:
> mean of x mean of y 
>   10.8566   10.8544 

Which of these outputs should be paid attention to?

> But useful data nevertheless.[1]
> 1: I won't even begin to discuss how many times I see benchmarks
> without SEM or sd reported.

Heh, well I know of the ideas, but haven't had any practice actually
calculating them.



Kees Cook                                            @outflux.net

Reply to: