Re: Confusion about where to go in Hurd
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Very long message :-). Anyway, I'll reply to the best of my ability
First, I think its very important to clarify the terms Hurd and mach
Mach - the underlying microkernel used by all versions of Hurd
Hurd - the userland translators which snap into the kernel providing
userland services, and (in theory) are microkernel independent
Hurdng - the project of porting hurd translators to another microkernel
beside mach such as L4.
In addition, I believe this email addresses quite a few things about hurd,
so I CC'ed it to gnu-system-discuss, and bugs-hurd
On Thu, 26 Jul 2007, Anders Breindahl wrote:
Hi,
I'll jump into this, as this is a major source of frustration and
confusion for me. And seemingly also for others. Perhaps this could
clarify.
Firstly, I'm replying to mbanck's post. Then I'm abusing the thread,
Because it bears the right subject for my post anyhow.
I hope you'll bear with me, brittle reader, since I at least felt that I
was making a point when I wrote this. I made some paragraphs for easing
the ingestion of this mail.
On 200707252235, Michael Banck wrote:
Debian will continue the Hurd/Mach port until a viable alternative
exists, which will likely take a few years, if there will ever be one.
At least it exists, although barely-moving and support- and featurewise
horribly behind. But its existence keeps up hopes.
Mach itself is actually pretty much feature complete and fairly stable. An
older version (mach3 vs. mach4) provides the kernel of Mac OS X. The main
reason parts of Hurd are slow and such is that code in the translators
(such as the pager code in ext2fs) haven't been optimized. I've only had
one true kernel panic in serveral months of running Hurd in a VM (on an
experimental kernel); When the system seems to freeze completely, it
usually is a deadlock or translator issue (although I haven't looked into
it THAT much).
>
As nobody knows how Hurd/NG will look like, a transition plan cannot be
made at this point, and whether or not a Hurd/Mach installation can be
migrated over is unknown.
How unknown or unsure is it whether current development of the Hurd will
be usable on a potential Hurdng?
- Will current interfaces suffice in e.g. translators or device
drivers, or will rewrites become necessary (How well did hardware work
in Marcus' L4-attempts? And would that be forewarning for potential
problems in other next generation microkernels?)
- What about Hurd-specific glibc-specials? Are they of
Mach-workarounding nature, or more of a non-Linux-dependency'ish
nature?
Obviously, the work on Mach won't be usable. That includes the device
drivers (and glue code), I suppose.
In regards to translators, by implementing the message interface (and
porting the mig compiler to the kernel), they should be pretty much a drop
in replacement, although I believe most translators do have some mach
specific code (or use the mach IPC directly, but that itself wouldn't be
difficult to fix).
Device drivers on the other hand are a different kettle of fish. Mach's
drivers at the moment are a port of the Linux 2.0 driver code (with ~1000
lines of glue code). That being said, device drivers are handled by the
microkernel under Hurd. The best way to explain this is that Hurd ran on
L4 and mach, and L4 supported device A and mach supported A and B, the
translators would be able to use whatever the underlying kernel supported.
>
Do you plan to help the Hurd development? Unless you are a microkernel
researcher, you can ignore Hurd/NG for the time being.
I have come to understand that key individuals within the Hurd community
are less than satisfied with patching up Mach. And with good reason. But
also that there is a despair of heirs to Mach. Nothing adequate with a
usable license seems available, right?
The problem with mach is that it is a first-generation microkernel. mach3
had problems with translators taking a very long time due to performance
and IPC issues, but mach4 (which was the basis of GNU mach) worked around
the issue with providing co-location (a mechanism where translators go
right into kernel space) and shuffling which, while not completely
solving the problem, they reduce it considerably to the point its more or
less a non-issue in current releases.
The true solution to the problem would be making the kernel even smaller;
L4 defines 7 functions, while mach3 defines 130 (I think). By making
everything be in the userland (which is what L4 does), the overhead can
easily be managed. Wikipedia has a very good article under this under
mach.
While the L4 effort is dropped, Coyotos seemingly is undergoing heavy
development. It has been mentioned as a viable choice for
next-generation Hurd microkernel before. How are current opinions on
that?
I can't tell, but if I am not mistaken, it has been (is?) a design goal
of the Hurd to be microkernel-independent. In these
microkernel-dissatisfaction times, would it be the right time to
consider refactoring towards microkernel-independence, and let Mach live
on as a testbed? (In OO-lingo, a Standard implementation?)
Hurd itself SHOULD be microkernel-independent. I know some progress was
made getting hurd working on L4, but I never looked to see just how far
that got. It would be quite nice if we could achieve this by getting hurd
ported to another microkernel (possibly's Apple's xnu if we wanted to test
on another mach-based kernel, L4, or Coyotos).
Mach itself I doubt will die anytime soon. It's stable, (more or less)
feature complete, and it exists.
By the way, from the name, a microkernel seems to be something easy to
write. Could somebody please outline to me (and others with my knowledge
of the field) why this isn't the case? I suspect that it isn't so much
about actually writing the thing, but more about knowing what it is that
one wants to write?
The problem steams from the fact that very few programmers have the low
level knowledge to write a program that can run from boot to provide a
real userland. I have experience writing assembly for embedded processors,
and writing a kernel is not an easy task (I've never tried it; the most I
did was write single-use programs for use on said board).
> -- end of reply, start of thread-abuse -- >
The Hurd's place in the future
==============================
In recent news, Linus has made a move toward a userspace driver
environment, which someone at coyotos-dev commented was to be ``hitting
the monolithic complexity wall''. In other words, they've come to (or
are at the verge of) preferring maintainability over speed.
(Disregarding Linus' own possible motives, here).
This will -- if it is allowed to evolve -- cause Linux to slow down, as
more and more speed will get lost in Linux' reportedly slow IPC.
Which is where the Hurd comes in, being slow already, but at least
having chosen the prudent (in the ``time has shown'' kind of way) method
of implementing and maintaining drivers and other kernel features. (I'm
guessing that) from this point on, multiprocessing and decoupled
hardware will punish monolithic kernels and lessen the punishment on
decoupled kernels and software. And wishes for maintainability,
scalability and stability will gain momentum, when speed is plentiful.
Which will favor the Hurd.
Having a userspace driver environment and being a microkernel with
translators are two VERY different things (Windows itself has provided
userland drivers for years, although its a hybrid kernel at best). In
addition, Linus would eat crow before turning Linux into a microkernel.
Capability
==========
Coyotos is gunning for being a capability-based system. That is an
interesting take on systems design, and -- seemingly as always --
fine-grainedness of capabilities causes hair loss. I won't venture
deeply into this, since I haven't followed it closely, but one
discussion on coyotos-dev that I did get something out of was such
technicalities as how to toss a capability object around. I say
technicality, since it (from my novice eyesight) seems irrelevant to the
system design. My first thought was to implement the quickest way of
handling it -- typedef'ing a pointer type -- and simply write the
improvement of that on the TODO.
(But those guys and their discussions are a bit too hardcore for me to
fully grasp, so I'll excuse for any misunderstandings on my part).
I don't know anything abotu Coyotos so I won't comment on this.
Zooming further out
===================
Picking up on my ``when speed is plentiful'' remark, I would assume that
the world will (very) soon see a lot of parallel computing.
Locking of resources, moving of tasks between processors and more exotic
things like transparently networked IPC will become more widespread, not
only in the Hurd. And consequently computers ought to get some hardware
support for optimizing these tasks. Just like wireless technologies
prompt for hardware acceleration of cryptographic routines, since it
would otherwise be somewhat of a drawback to take the step to wireless.
This should also lessen fear of performance issues with the Hurd. Linux
already is big enough to have a certain umph in hardware design, and if
stability-, scalability- and maintainability-oriented users start to use
Hurd, support from the chip vendors will probably catch up. (Like it
does with hardware virtualization technologies, these days).
It took many, many years for this to happen to Linux. I doubt we'll be
seeing this for hurd anytime soon. That being said, we use Linux drivers
in mach so our hardware support is equal to the version of Linux used for
our drivers (2.0 is what's in the current mach, but I'm working on porting
2.6)
Continuing existence of the Hurd
================================
Two points that I'm noting:
- What made Linux successful was its mere existence. When GNU was trying
to grasp microkernels, Linux did ``monolithic'' and got through with
reasonable success. It's proprietary competitors were (and are still)
using somewhat the same schemes of kernel design, which doesn't put
Linux in a totally unacceptable competitive position -- allowing Linux
to flourish because of its other advantages.
- The Hurd regularly gains the interest of newcomers. I've been lurking
along for long enough to know that the lists regularly get bumped with
some novice (arnuild being the latest, still fresh in memory) with
lots of enthusiasm over the prospects of the Hurd's goals. Alas, it
soon dawns on us newcomers that it's mostly toughlived vaporware, and
even getting an everyday system running isn't feasible, perhaps not
even doable on modern hardware.
By definition, we are NOT vaporware. We exist, and our work (as well as
source code) are publically available.
These two points tell me one thing, which I also commented above, about
the Debian Hurd port: The Hurd needs to continuously exist to survive.
It doesn't matter whether it sucks performance wise, or that there is no
sound or accelerated graphics. It doesn't matter either, that it's based
on an old-fashioned microkernel, from a user standpoint (albeit it does
clog up on the development pace and -joy).
That's a reason for forcing _some_ decision through, regarding the
future direction of work on the Hurd. It'll be too hard to satisfy my
thought-up existence requirement (assuming you concur it exists) if
development is stuck at Mach and closed development groups. And this
will scare off users and developers alike.
Mach isn't a closed development group. A closed development group would be
like the old FreeBSD and NetBSD groups from many years ago, as well as
nethack, in which case the development efforts are not public and the only
source given is with each point release. Mach accepts patches from anyone
of any size (my first patch for mach ever was a 1 liner which fixed my
network card in Parallels Desktop).
That being said (and I've commented before), we're not the fastest group
ever at getting patches into our projects, which is something that should
be corrected as best as possible, but we're not like other groups which
flat our reject patches. If you ever have submitted a patch to a project
before, you'll find that we're not quite as bad as some other projects.
> DVCS and technical solutions to management problems
===================================================
It was recently suggested that upstream CVS should be semi-transparently
be mapped onto some distributed version control system, so that
experimental development could find a home (or homes, for that matter).
While generally upgrading the VCS is sensible, I can't say that I agree
that this is a needed step. Which Alfred also mentioned in that the
ams-branch -- in his opinion -- didn't get any effect upstream.
That is in regards to glibc; I have no experience with there development
pratices (I work on Hurd and mach alone, as well as porting packages to
debian)
What is however needed in my opinion, is, that development should be
more playful implying that it should be less structural.
While I do agree with this, moving to a DVCS would solve the problem
considerably. If you've ever done development on Linux, you'd know that
Linus's git branch is (for almost all intends and purposes) the offical
branch of Linux. When someone wants a patch added, Linus will ask for the
address of their git repo, and pull from that, and if he likes the work,
merges it into his git branch. With a DVCS, its like giving everyone
commit access, and it also makes the development environment more
friendly.
An example of how this may be done can be seen in Linux' 2.4 vs.
2.5
series. 2.5 was never meant to be used in a production environment, but
was meant to mature into 2.6, which would become usable in a production
environment. While 2.5 was evolving, 2.4 was kept somewhat more stable.
(This scheme has since been abolished, as a sidenote).
This is not a technical issue, and DVCS'ing alone won't solve it. What's
needed is a gust of boldness, or at least, loosening of resistances to
do something not-necessarily-thought-through. Obviously, technical tools
will help out, but we all seem to be computer scientists here, so using
tools is the first thing that comes to mind.
I won't comment about the others, but I failed out of quite a few of my
tech classes (beside programming) because I couldn't do the math.
Everything I've done on mach is self-learned and self-taught.
Development model, human resources
==================================
While of course appreciating the work done, the needed development still
mostly hangs on a few shoulders. Recently, there was a dispute about
reluctance to patching on gnu-system-discuss that shed further light
onto the lack of enthusiasm for development from the senior members of
the community (not necessarily those of Hurd, I should add).
I didn't subscribe to gnu-system-discuss at the time (which is more or
less a dead list it seems), but I did look up the conversation in
question.
Michael Banck (who's one of the debian developers in charge of Debian
GNU/Hurd) and ams were the two major people to respond to that thread. I
find that ams's views of Hurd seem to be somewhat out of touch of how Hurd
is at the moment. I know Thomas S., Michael B., and Marcus are both active
with Hurd development. Ronald works mostly with libc, and helps maintain
the Hurd specific parts (I don't know him that well, so I can't comment).
In this setting, I'd like to point at Ubuntu. While Ubuntu and its
derivatives still seem amateurish to me when I compare them with Debian,
I got to acknowledge how Ubuntu has been able to facilitate much of its
user base, no matter what level of expertise. When web-forums seem
too... How do I put it... ``Fashionable'' for Debian, Ubuntu doesn't
scare off its novice users with mailing lists. And when strict
packaging- and licensing requirements improve the quality of Debian's
packaged software, Ubuntu seemingly has (and allows for) a package for
everything in its ``multiverse''.
The only recent noivce message I got was one that no one has replied to
yet. When I was came to hurd, I had considerable help from people on the
IRC channel to debugging my installation.
The Hurd wiki has been a source of documentation from user to user.
While often its articles have been written by senior members of the
community (since they understood the stuff first) success-stories and
FAQ's may be added by junior members. Openbox has recently moved its
website to MediaWiki for seemingly those reasons. Especially the Hurd
wiki page for using Hurd on qemu has seen some use.
To my knowledge, gnu.org webspace doesn't offer PHP or MySQL that would be
needed for a wiki page. Can someone correct me if wrong?
Also, sources should be more up-front. It's not because I've been
looking for it, but I still haven't got the cvs checkout command
memorized. If I had, I might be more encouraged to check out the source
tree and have a look around -- who knows what I might do?
Most of the time, checking out is done extactly once, and then cvs up is
used to stay up to date. When I need to check out clean, I copy and paste
the command from the website.
(Where ``I'' is someone mildly intertwined with Hurd development).
All words but no action
=======================
It's not the first time I'm posting my frustrations about the Hurd's
state. And the previous times I'd get more-or-less assaulted for having
a big mouth and no coding-creditability. ``If you think it's wrong, you
fix it. But don't blame us, who made actual contributions''.
My up-front answer to that is, that I've made no claims of being able to
solve anything. I -- as all of us -- have no spare time for this project
(which is a weird way of saying that it doesn't really interest me
enough to sacrifice what ever time I have to the project -- because if
enough interest was present, finding time wouldn't be a problem,
right?).
I however, am at the bottom of the hierarchy. I'm the pondering lurker,
whose sole ability is to stir up stuff using arguments that hopefully
will convince others to take action.
What action is to be taken, in my opinion?
It's easy. It's about realizing that problems exist (or, why I'm
wrongfully frustrated).
I completely agree. In one of my last emails, I posted about that, and its
a mentality that you need to be able to code to point out problems and
such. I'm not sure how to help the issues, but on these lists, everyone
are equals to me (which hopefully I prove by spending over 2 hours
drafting this response :-)).
> Good night, and thanks for reading, >
skrewz. >
Michael
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU)
iD8DBQFGp/Y9V/FBeJ6jsZQRAkxhAJ9RV05vVwMlx+VtQUwNOoUadeMMUgCfazlA
nL0i4oSPmM/XGBYhNg8ktY4=
=vCvU
-----END PGP SIGNATURE-----
Reply to: