Hardware compatibility test: draft proposal
Hi,
As I mentioned in my blog[1], I kindof like the suggestion that Bdale
came up with during Debconf that we write a hardware compatibility test
of sorts that hardware vendors could run on their own hardware to test
whether Debian works on their system. The rationale for such a test:
while most of us know how Debian works and how one should test whether
their hardware actually works with Debian, it is not reasonable to
expect the same of hardware vendors for /all/ GNU/Linux distributions
out there. According to Bdale, currently all other major operating
system vendors, including the commercial Linux distributions, already
provide such a test to the hardware vendors. For this purpose, it is
okay if such a test is interactive to some extent (after all, it is
something that you would run on a hardware prototype in a lab, not on
each and every production machine), although I'm thinking a hardware
compatibility test could be useful in more cases, where it might be
better if it wasn't interactive.
So, after more than twelve hours of boredom on an airplane and half a
night of not-being-able-to-sleep-due-to-jetlag, which is certainly
enough to think about this problem, I came up with the following things
such a system could need:
- It should be modular. People who maintain driver packages for
particular hardware may want to write additional tests that a vendor
may want to run; and if this particular package supports it, the
driver package maintainer may want to provide pointers to a particular
package so that an inexperienced user may be able to configure their
hardware after running the test themselves.
- The different tests should each be able to communicate what type of
hardware they test for, whether that particular piece of hardware has
been found, and whether it actually works. The test of whether
something works may require that network connectivity, hard disks, or
other similar things are available in or to the system, as applicable.
- It should support a notion of what I'll call 'profiles'. A 'server'
profile should check for different things than a 'desktop' or 'laptop'
profile; e.g., it's usually okay if a server doesn't have graphical
support or wireless drivers, while the same isn't true for a laptop.
- The vendor should be able to influence the score of a test by
explicitly stating that particular hardware isn't available. If the
vendor really wants to build a laptop without wireless drivers in this
day and age, then it's obviously okay if no wireless drivers were
detected. If, however, the vendor is not insane, then the failure to
detect a wireless chipset should clearly influence the score.
So, that's probably it at this point. I should have a look at bdale's
talk once it becomes available at meetings-archive.debian.net, but that
doesn't yet seem to be the case. If someone who was at bdale's talk has
anything to add, that'd be welcome. If someone could think of anything
even without being at bdale's talk, that's welcome too, of course.
Now, with the above in mind, and after having considered Holger's
proposal to do this with Debian Live[2], I think the following generic
spec should cut it, but I'm open to other suggestions at this point.
It's also not very detailed yet, but since no code has been written yet,
that doesn't really matter at this point.
- A base package 'debian-hct' will provide a basic infrastructure for
these tests to run in and an initscript that actually runs them. It
will also contain some tests that are useful for /any/ system, such as
"do we find something that looks like a harddisk controller" etc.
- Additional packages may provide tests. Packages that do so should say
'Provides: hardware-compatibility-test' in their control file.
- Tests are found in /etc/hw-compat-tests. This directory will have
subdirectories, one for each of 'hard disk controller', 'wired network
interfaces', 'wireless network interfaces', etc. The scripts in this
directory will run in asciibetical order, so that, e.g., drivers that
need firmware to be loaded can ensure this firmware is actually loaded
before allowing the generic test for this class of hardware to be ran.
- Scripts in the subdirectories, when ran, should end their output with
on a single line the letter 'S', followed by a colon, followed by two
numbers separated by a '/'; the first is the score, the second the
maximum possible score for this script. All output that precedes the
score is redirected to a file for the user to inspect later. Scripts
should make sure to avoid having a line that starts with 'S:' in this
preceding output, perhaps by escaping it with a leading space or some
such (this leading space will not be stripped).
- If no hardware is found that would be supported by the driver this
script checks for, then the second number, the maximum possible
score for this script is, by definition, 0.
- If the hardware being tested for can only be supported by the use of
non-free drivers or firmware, then the script must mention this fact
through the output of the letter 'n' after the actual score/maximum
possible score; e.g., it would say something like 'S:1/1n'. I still
need to flesh out how to best handle this particular bit of
information (it doesn't have to be all the same way; e.g., the
requirement of non-free drivers for the wireless interface is less
problematic when one wants to install than is the requirement of
non-free drivers for a hard disk). Suggestions are welcome.
- Scripts may use debconf to ask the user for input if they need it.
- Scripts should not just test for availability of hardware. Instead,
they should test the actual functionality; e.g., tests for a network
interface should be done by trying to DHCP off that interface, X.org
drivers should try to start X and ask for input using a graphical
dialog, and tests for a hard disk should be done by trying to read
some data from the disk.
- If multiple scripts in a particular subsystem directory offer
maximum possible scores, their values will be added to eachother. If
a script offers 0 as maximum possible score, it will be ignored,
even if it also outputs 'n' to signal non-free usage.
- A script must not assume input to be available from the user, nor
can it expect a controlling terminal. It can only assume that stdout
works and that debconf is available.
- If none of the scripts in a particular directory indicates it sees
some hardware (i.e., all scripts offer 0 as maximum possible score),
the system will ask the user whether this is expected. If this is
expected, the subsystem will be ignored for the final score; if not,
the final score for this subsystem will be 0/1.
- After all the scores have been collected, the framework will multiply
or divide the scores for each of the different subsystem so they fit
within a predefined scheme that ends up giving the system a score on a
scale of 1 to 100, which is then presented to the user along with
details about the tests that were run and the. It is not
necessary that every piece of hardware is given the same amount of
consideration in this final score; e.g., we will want to consider the
proper functioning of a hard disk to be of much more value than the
proper functioning of a fingerprint reader.
- A Debian Live image will be provided that will install the
'debian-hct' package plus all packages that say 'Provides:
hardware-compatibility-test' plus all their dependencies. This will be
the hardware compatibility test that we can give to vendors.
- Vendors who pass this test on a particular bit of hardware are allowed
to advertise that fact; it might be nice to have provide them with a
logo that they may use for this purpose.
Thoughts?
(Back to bed now. Jet lag, gotta love it)
[1] http://grep.be/blog/en/computer/debian/hardware_test and
.../hardware_test_followup
[2] http://debian-live.alioth.debian.org/
--
<Lo-lan-do> Home is where you have to wash the dishes.
-- #debian-devel, Freenode, 2004-09-22
Reply to: