Re: Hardware compatibility test: draft proposal
On Fri, Aug 22, 2008 at 11:05:49AM +0100, Chris Lamb wrote:
> Wouter Verhelst wrote:
>
> > As I mentioned in my blog[1], I kindof like the suggestion that Bdale
> > came up with during Debconf that we write a hardware compatibility test
> > of sorts that hardware vendors could run on their own hardware
>
> I think this is a great idea, and I generally agree with your decisions and
> assumptions.
>
> > - Scripts should not just test for availability of hardware. Instead,
> > they should test the actual functionality; e.g., tests for a network
> > interface should be done by trying to DHCP off that interface, X.org
> > drivers should try to start X and ask for input using a graphical
> > dialog, and tests for a hard disk should be done by trying to read
> > some data from the disk.
>
> I think this the most important paragraph of all.
>
> What I think is missing is some really concrete info on just how various
> hardware items would be tested.
Well, that's the hard bit, isn't it? ;-)
I didn't go into much detail here yet, simply because I feel this is
something we'll have to figure out as we write everything.
> For example, in the case of ethernet adaptors, I feel that simply
> successfully DHCP-ing on an interface is really not an acceptable
> test.
>
> As an example, Etch's kernel contains various modules (such as sky2) that
> "kinda" work on today's hardware - whilst the driver would probably pass a
> DHCP test, the actual performance or reliability of the device is completely
> inadequate. Wireless devices would pose an even more difficult problem, as
> support for the various encryption modes that the device supports tend to be
> developed at different times, etc.
>
> (I'm fairly confident these comments have parallels with other hardware
> categories, I just give networking examples that most readers might be able
> to relate to)
>
> Whilst I am aware that a testsuite could never be 100% conclusive (and I'm
> sure you are aware too) my underlying enquiry here is trying to work out
> what level of confidence you are aiming for.
Yes, good point.
The main problem is, at this point I'm not actually all that sure of it,
really. On the one hand, I want to test as much as is possible. On the
other hand, that is going to get very hard very quickly, and would
probably take forever to develop.
We'll probably need to find a bit of a compromise here: initially, test
for at least basic functionality, and test additional functionality if
we find out at some point that common modern hardware has problems where
basic functionality isn't available.
> (Hm, this didn't mean to be so negative..)
Negative? What's negative? ;)
> > - A Debian Live image will be provided that will install the
> > 'debian-hct' package plus all packages that say 'Provides:
> > hardware-compatibility-test' plus all their dependencies. This will be
> > the hardware compatibility test that we can give to vendors.
>
> Please let me know if you would like any help with Live image foo.
That's the final step. If/when I get there, I will. If I still need it
then...
--
<Lo-lan-do> Home is where you have to wash the dishes.
-- #debian-devel, Freenode, 2004-09-22
Reply to: