Re: Cleaning up kernel's headers for userspace

[Tim Yamin <plasmaroo@gentoo.org>]

On Mon, Apr 24, 2006 at 08:51:52PM +0100, David Woodhouse wrote:
> That's true, but the idea is that the 'make headers_install' output
> should be easy to read and process _without_ all that extra noise in it.

Ah, that works :)

> I'm not sure I agree. Keeping stuff like asm/atomic.h might be
> 'complete' but it's broken if people actually try to _use_ it. And you
> get some muppets actually defining __KERNEL__ just to get at stuff they
> shouldn't.

Heh... now that is silly ;)

That raises the question though, how does one pick what headers to ship and
not ship? Not ship headers that are totally __KERNEL__ or some other scheme?

> Those fixes need to go upstream, surely?. Are you _opposed_ to us
> striving for a situation where you can just ship what the kernel spits
> out?

Hell no, in fact that's what I'd want, provided it works :)

> > The other problem is the vastness of our architecture support - we
> > do from x86 to m68k and all those headers need to work -- hence
> > hand-picking really never was seen as the viable option as it means
> > a *lot* of work between upstream releases. Our current way you just
> > fix a few conflicts with the patches have when you move from say 2.6.16
> > to 2.6.17, and run test-cases to see if any new gremlins have
> > appeared.
> 
> The idea is that this all happens upstream; no such merges would be
> required. Once we've got it sorted out, the architecture maintainers are
> responsible for the output of 'make headers_install' on their
> architecture, and 'make headers_check' exists for automated checking of
> the results. If you have more checks which we should apply during the
> 'make headers_check' stage, that would be useful.

Great :)

> I don't think I've communicated the idea correctly then, since this is
> the _opposite_ of where I'm going. No changes would be made except in
> the upstream kernel tree. We'd just use the output of the 'make
> headers_install' step. Never patch the exported files -- only the source
> files.
> 
> We want everything to be upstream, and the results of 'make
> headers_install' from the kernel to be shippable as-is. If the arch
> maintainer adds asm-foo/blah.h which requires asm-foo/baz.h, then they
> add both of them to asm-foo/Kbuild and all is well. Or if they forget,
> and just start including asm-foo/baz.h from the existing asm/blah.h
> without touching Kbuild, then the automatic checks mean they get a
> kicking for it fairly quickly thereafter.

Right, so I probably misunderstood. Thanks for clearing this up and
bringing in the role of arch maintainers: if they're responsible for
maintaining their own little bit rather than an external repo then
and external maintainers that nag then yes, of course, it'll only better
the situation.

> Ideally, I'd like to reach a point where 'headers_install' is nothing
> more than a recursive copy, with no picking of files or unifdef. That's
> a way off yet though.

Yeah, but with some work we can get there... Slowly but surely...

> That hasn't happened, though. And a large part of that is because the
> problems aren't _obvious_. Hence the tools which make the problems
> obvious by removing the __KERNEL__ and the unneeded files.

Well, you get a compile failure either way? Unless your muppet has done
#define __KERNEL__, of course, which is just a silly thing to do. I think
part of the confusion is that you're looking at this from an angle of a
kernel dev (i.e. "is this header kosher or do we have a problem") and
I'm looking at it from a distro point of view ("if there's a header
problem it'll show up regardless and then we can fix it").
 
> Imagine taking a bunch of messy header files, finding the public parts
> of them and splitting them out into separate files -- so that we have
> none of those horrid ifdefs that everyone seems to hate in C code, while
> conveniently overlooking them in headers. Then move the public headers
> into a separate directory instead of leaving them mixed in with the
> private headers.

Yeah, that made sense. The question is do the public bits get shipped by
upstream or generated on-the-fly from the mixed ones with a Makefile
command (once everything is cleaned up and "working")?

> Yes, I agree. And that's what 'headers_install' is designed to provoke.
> It's designed to let us _see_ the result with the __KERNEL__ taken out,
> and designed to help us take diffs from one version to the next. And to
> let Andrew reject patches which introduce regressions. Etc. 

Guess I'm just the weird guy who doesn't hate preprocessor macros. Either
way, whatever helps you fix the problem is the way to go forward...

> Thanks for your feedback. Hopefully, I've managed to persuade you that
> this is just a tool which helps us to get to the same place you seem to
> want to get to -- the upstream header files being cleaned up; with or
> without __KERNEL__. If not, it would be interesting to hear if you have
> a better solution to propose. I'm not particularly wedded to this
> approach -- it just seemed like a good way to get people involved and to
> hold people accountable for user-visible stuff in the future. If you've
> got better ideas, I'm all ears...

Thanks for rephrasing -- and yes, our goal does seem more or less the
same now after sorting the confusion out, which is a good thing. It would
be nice to have a unified set of headers shipped by upstream which just
works again -- I think the major hurdle is to get people to start fixing
the breakages; once most of them are fixed then ensuring new problems
don't crop up isn't so bad.

I assume right now you're just gathering comments -- how do you plan on
proceeding with this afterwards?