Re: speculations to characterize issues for Debian Enterprise
On Tue, Aug 10, 2010 at 09:29:16PM -0700, Russ Allbery wrote:
> CJ Fearnley <cjf@CJFearnley.com> writes:
>
> > - If each package in Debian supported a config-time option for the required
> > functionality, then it might not need to be packaged as a private archive
>
> We haven't done a lot of packaging because we needed to build things with
> different options. I think the main place we've done that is with
> OpenLDAP to build with SSL and we build that for other reasons as well.
> Usually we've had to apply custom patches or other changes beyond just
> building with different options.
These feel to me like bugs: upstream or Debian bugs that prevent stock
software from working properly in specialized environments.
> > - In this class of problems, I just see it as buggy upstream software.
> > Let's identify the issues and either submit bug reports or negotiate
> > with upstream to design better software.
>
> Sometimes it's bugs (our Cyrus SASL issue, for instance, I think is a
> bug), but our weird custom Postfix stuff is just a strange Stanford
> business requirement. We could probably find other ways of doing that.
>
> To take another example, we basically fork Mailman and apply a *ton* of
> changes. To some extent, that's an upstream deficiency, but the amount of
> changes are rather huge and a lot of them involve UI design for Stanford
> and customization to work with our web authentication system.
More from the bugs paradigm: not having a flexible skinning system or
a flexible configuration system is an upstream design bug in my book.
> > - So that's an issue of helping Debian build packages for new
> > upstreams and backporting them more quickly. Maybe we need more help
> > on some of the Debian teams that maintain the package or negotiate
> > to build a team to relieve an overworked DD.
>
> Don't underestimate the importance of having control over exactly what
> version you're running, though. For instance, I can't imagine using stock
> Heimdal packages on our Kerberos KDC; I want to know exactly what version
> I'm running. I can (and do) still start with the Debian packages, but in
> addition to the set of patches that we apply (most of which are now
> integrated upstream), I want to have total control over exactly what
> revision of Heimdal is deployed.
>
> This is our issue for Puppet, for example, because we have to synchronize
> the server and the clients. Over time, Puppet will slow down the degree
> to which it changes and this will be less of an issue.
Synchronizing versions is a tricky issue indeed. But fundamentally,
I'd call it a bug upstream or in Debian. Upstream really should support
interoperability with a reasonably broad subset of versions. And when
there is a major transition, Debian should devise a smooth upgrade path.
A hard problem, but I think we are just dealing with the work of fixing
bugs in software that is not yet well enough engineered.
> > Are there any other classes of issues that have led Stanford to needing
> > to build so many customom packages?
>
> We generally build a package for every class of server that stores our
> local scripts for running that class of server. In theory, we could
> potentially generalize some of that stuff, but it's a lot of work and in a
> lot of cases I'm not sure it's worth it. For instance, we have reporting
> scripts that we run on Kerberos KDCs that do lots of Stanford-specific log
> and database analysis, and making those general enough for everyone to use
> is a somewhat dubious proposition.
>
> We also do a fair bit of local software development. Ideally, all that
> could go into Debian, but some of this is very edge-case software and I'm
> not sure that Debian really wants it. I'm slowly working on getting some
> of that stuff, like all the AFS tools, into a generalized form, but it
> takes a lot of time and effort.
Of course it is often not worth it for Stanford. So as an expediency
you build a custom Debian package. I have a bunch of those too for much
the same reasons. I expect that many people on this list have more of
the same. Some of us are duplicating each other's work. If we worked
together, we could improve the core software and make Debian even better
for the enterprise. Some of the issues are so specialized that upstream
is probably not going to want to add hooks to make their software more
flexible. In that case, we can be grateful that we are building on FOSS
and not unmodifiable closed software. In that spirit, I really agree
with your approach to build Debian packages: it is a great solution to
the problem. But I'd like to take a stand that we should help build
better software so that the number of custom packages we all need to
maintain can be reduced (dare I say eliminated?) over time.
But another issue that I'm reading in between the lines is that there
is a level of configuration management that you do with packages.
I think your "local scripts" and "server classes" really are a level of
configuration management. So thinking about how Debian (and upstreams)
can facilitate the kind of configuration management for which puppet
and revision control on /etc (or even just hacking in vi in /etc) are
inadequate may be a fruitful line of thought.
So far, in summary, I see four classes of enterprise issues:
1. Getting new upstreams into Debian.
* Just (time consuming) work.
2. Packaging tools that are not in Debian (including locally developed
tools which implys becoming an upstream).
* Just (time consuming) work.
3. Bugs. Design flaws upstream. Patches. Sensitivity to versions, etc.
* The only solution that I can imagine for finding and fixing bugs
is just doing the work. With design flaws, it can get really hard.
* If site specific configuration management can't solve the issue,
it is a bug!
* I think much of the work needed is in bug triage and fixing design
issues upstream so the software can "just work" in more use cases.
Fortunately, there are now usually at least two FOSS upstreams
that can do any given thing. So if one upstream doesn't "get it",
we can work with another before we'd need to fork a new codebase
(which is what our local package archives really are).
4. Configuration management.
* I mean at the Debian packaging level primarily. But unless puppet
solves the problem for everyone (and I'm not yet convinced), there
is broader design work needed too.
* Another hard problem. More than "just work", as it requires
creative new ideas too!
Thoughts?
--
We are on a spaceship; a beautiful one. It took billions of years to develop.
We're not going to get another. Now, how do we make this spaceship work?
-- Buckminster Fuller
CJ Fearnley | Explorer in Universe
cjf@CJFearnley.com | "Dare to be Naive" -- Bucky Fuller
http://www.CJFearnley.com | http://blog.remoteresponder.net/
Reply to: