Re: Hardware compatibility test: draft proposal
On Fri, Aug 22, 2008 at 12:55:34PM +0200, Giacomo Catenazzi wrote:
> Wouter Verhelst wrote:
> > Hi,
> >
> > As I mentioned in my blog[1], I kindof like the suggestion that Bdale
>
> Yes, I find the talk very interesting.
>
> > So, after more than twelve hours of boredom on an airplane and half a
> > night of not-being-able-to-sleep-due-to-jetlag, which is certainly
> > enough to think about this problem, I came up with the following things
> > such a system could need:
>
> /me too
>
> > - It should support a notion of what I'll call 'profiles'. A 'server'
> > profile should check for different things than a 'desktop' or 'laptop'
> > profile; e.g., it's usually okay if a server doesn't have graphical
> > support or wireless drivers, while the same isn't true for a laptop.
>
> No. I don't think we should provide profiles.
> The check should be:
> "complete": test all hardware that it is installed on system.
> If the system doesn't have the video-card or the wifi just it doesn't
> test it, but it write a notice.
That will create ambiguity, which I think we should avoid at all cost.
If you tell a user of this test that "it seems to work, but you should
check this and this and this output to make sure we didn't miss
anything", then such a test is worthless; hardware vendors *will* miss
this kind of thing.
> Listening the available hardware is pretty trivial (see i.e. my
> AutoKernConf).
I'm afraid I'm not familiar with this piece of software; and as I'm
offline right now, I can't look it up, either.
However, I don't think checking by listing the hardware is the best way
to go, in any regard. A script that checks for all hardware available in
a system can't know about each and every piece of hardware out there;
especially not if new interfaces are made that the script doesn't yet
know about. Giving a piece of hardware a perfect score because our
script failed to notice there's some unsupported wireless interface
connected is not the way to go.
> Additionally a component test should be furnished, for component
> hardware vendor. (e.g. a network card vendor would test
> only the network card, not the whole system).
I'm not sure that's very useful. A network card vendor has two options:
either they design a card based on an existing chipset that has drivers
already in the kernel (in which case it will probably just work, unless
they use previously-unknown PCI IDs, in which case it won't), or they
will create a totally new chipset for their network card, in which case
it won't work. A vendor is usually aware of this fact, and giving them a
"test" for something they already know about isn't very useful.
If you disagree, of course don't let me stand in your way. Just don't
expect me to work on it ;-)
> > - The vendor should be able to influence the score of a test by
> > explicitly stating that particular hardware isn't available. If the
> > vendor really wants to build a laptop without wireless drivers in this
> > day and age, then it's obviously okay if no wireless drivers were
> > detected. If, however, the vendor is not insane, then the failure to
> > detect a wireless chipset should clearly influence the score.
>
> See above.
> The score should be done only "negatively", i.e. when a
> hardware is found but:
> - doesn't work properly
> - it is buggy (as for the BIOS: linux have some work-around for
> broken BIOS, but the test should alert vendor)
> - it requires manual configuration
> (partly could be a problem of Debian, but it could be also
> a problem in hardware: wrong identification strings/number)
> - requires non-free stuff
> - ...
Yes, I agree; that's the only sane way to do it. If we don't find any
issues at all, a system should get a 100% score; only when we do find
some issue should the score be lower.
> > Now, with the above in mind, and after having considered Holger's
> > proposal to do this with Debian Live[2], I think the following generic
> > spec should cut it, but I'm open to other suggestions at this point.
> > It's also not very detailed yet, but since no code has been written yet,
> > that doesn't really matter at this point.
> >
> > - A base package 'debian-hct' will provide a basic infrastructure for
> > these tests to run in and an initscript that actually runs them. It
> > will also contain some tests that are useful for /any/ system, such as
> > "do we find something that looks like a harddisk controller" etc.
>
> we really want "debian" in package name?
> Should we do a more broad project?
I'm not opposed to making this useful to more distributions than just
Debian; however, I also think that we should provide an official
"Debian" test that will test whether something will work with Debian
Stable, rather than just a random Linux kernel plus some random software
that's available on some random website. The former is something
tangible; the latter isn't.
> > - Additional packages may provide tests. Packages that do so should say
> > 'Provides: hardware-compatibility-test' in their control file.
>
> and Depends it should on 'debian-hct', to have a common interface.
Yes, probably.
> > - Tests are found in /etc/hw-compat-tests. This directory will have
> > subdirectories, one for each of 'hard disk controller', 'wired network
> > interfaces', 'wireless network interfaces', etc. The scripts in this
> > directory will run in asciibetical order, so that, e.g., drivers that
> > need firmware to be loaded can ensure this firmware is actually loaded
> > before allowing the generic test for this class of hardware to be ran.
>
> I don't like to have it in /etc. IMHO it is better to have it in
> /var/lib/ or ... and copied to a root directory on the target system.
Yeah, totally. This is an artifact of me having written this at 4AM-ish.
I wrote /etc because I was thinking this should be something "like"
initscripts; but it's most likely better placed in /usr/share or some
such.
> > - A Debian Live image will be provided that will install the
> > 'debian-hct' package plus all packages that say 'Provides:
> > hardware-compatibility-test' plus all their dependencies. This will be
> > the hardware compatibility test that we can give to vendors.
>
> the BIOS testing and memory testing should be included on the
> basic test.
Yes, probably.
> I'm not so sure that the test should be packages.
> The system is too different on early boot from a Debian system.
In what way? Note that a Debian Live system *is* a normal Debian system;
you build a Debian Live image by giving the debian-live script a bunch
of packages; it then installs them into a chroot, changes a few things
so that the system will actually boot from CD-ROM, and then create an
ISO image from it.
How is that "too different" from a Debian system? Or are you talking
about something else?
> I think it is a lot simpler to do a separate project, using
> building infrastructure as d-i or debian-live.
>
> IMHO the target CD should be very easy to create and it
> should be complete. (but maybe allowing additional external
> kernel and modules).
That's my goal, too.
> The "modular" thing should be put mainly on the testing
> development side.
Yeah, I'm not suggesting we provide vendors with a bunch of packages;
the packages thing is a way for us to make it easier to actually develop
this without having one maintainer who has to write/accept/maintain
*all* the tests in the system. But hey, this is debian-devel, not
debian-announce ;-)
> > - Vendors who pass this test on a particular bit of hardware are allowed
> > to advertise that fact; it might be nice to have provide them with a
> > logo that they may use for this purpose.
>
> This should be discussed after we have comprehensive tests,
Sure.
> and probably to LI. I don't want to see packages with logo of 10
> distributions.
To me, it's not about seeing a logo on a package. It's about seeing
another vendor besides HP make a commitment that "Yes, Debian will be
supported. We won't run away scared to the hills if you say that you're
running Debian." Because that's what most vendors will do currently.
I think you'll also find aving a generic "Linux" hardware compatibility
test isn't very useful; if we provide hardware vendors with such a
thing, they'll go "I ran that test, and it said 'pass'; but my
shiny-new-network-card-chipset-that's-supported-in-the-latest-development-kernel-only
isn't supported in Debian? That's probably a bug in Debian, then".
And it doesn't even have to be "too new" stuff; it may also be that some
particular bit of hardware is supported by some free out-of-tree driver
which is supported by some distributions, but not necessarily by all of
them. Would you test for that in a generic "Linux" compatibility test?
If you don't, you'll fail on an amount of hardware that works perfectly
well in most distributions; and if you do, you'll succeed on hardware
that won't be supported by a number of distributions, possibly the one
the hardware vendor cares about most (because, say, they use it
themselves for their webservers).
No, I think we *do* need a Debian-specific test.
> Marketing people could choose to include on packages only few
> distribution, which could be negative for other distribution (and
> maybe for debian).
I think you'll find that they do that anyway. The more logos, the
merrier, because it sells good. Never mind that you can make those up, of
course.
--
<Lo-lan-do> Home is where you have to wash the dishes.
-- #debian-devel, Freenode, 2004-09-22
Reply to: