Re: itp: static bins / resolving static debian issues
On Mon, Aug 23, 1999 at 10:51:07PM -0500, Steve Greenland wrote:
> No, Craig continues to point out that statements like "Such-and-so is
> required to use Debian as a stable server system" are opinions, not
> technical arguments, without technical merit.
It is certainly not just an opinion.
I don't want to say that you wrote the above as a lie, you may
honestly have forgotten many of the messages that have come before
this one in this thread; alternately you could have just read a
message from Craig and my response, and thought that there were
no technically meritous arguments.
So let me demolish the above notion, and establish what the
technically meritous arguments behind including statics by
default are.
You will all find it very familiar, since I've said it all before,
but restating it here this way will contrast it with the nonsense
above that claims I haven't done more than state my opinions. You
may find that you want to agree or disagree with some points, and
we may debate them--but for so long as I am able to defend them,
you cannot claim that I am just stating my opinion.
Here we go, here is a complete summary of the logical and technical
arguments for static binaries, and for their being the default:
Proposition #1-- There are kinds of servers for which downtime is
considered unacceptable
Proof: Consider an NFS server with 30+ clients, all of whom are
doing important work, and would lock up.
Proof: Consider an online retailer's website on a busy day
Proof: Consider a database back end sitting behind a line of
web servers, collecting data
Propostition #2-- There are kinds of servers for which a reboot is
considered difficult, due to logistics.
Proof: Consider a server which is remotely administered and it
is difficult or impossible to get to (uncommon case)
Proof: Consider an administrator working from home, where a
reboot would be possible, but time consuming (common case)
Proposition #3-- Debian should, by default, support servers where
downtime is unacceptable or difficult.
Proof: Much of this portion is opinion; but I challenge you to try
adding a policy statement to the contrary. I believe that
most Debian users (maybe not you) think this is important,
if only for the sake of Debian's reputation as a server OS.
Proof: This opinion is implicit even in your quoted statement
above. You are challenging the idea that such-and-so is
required on a stable server system in a way that implies
that if it were required, that would be sufficient to
justify its inclusion as a default.
Proof: Many systems in category #1 and #2 are installed by
people who do not understand the need for statics, but
statics still help them: attempting to log in as root
and fix an easy to diagnose problem is the natural
thing to do; alternately someone else may advise them
on what to do, and statics enable this advice to succeed.
Proposition #4-- Failure of the C library can occur under Debian
Proof: The package manager can delete it as the result of a bug, as
has already happened in the unstable release. Anything
that is possible in the unstable release is also
possible in the stable release, just less likely due
to (imperfect) bug fixing.
Proof: It is trivial to show how an administrator can cause this
by accident, such as with the rm or dpkg commands.
Proof: A disk error can corrupt some files but not others, such
as by duplicating inodes. The C library may be affected
while leaving much of the rest of the system intact.
Proposition #5-- Most servers will survive a C library failure
if the server is already running
Proof: The server is already linked and loaded, and unless it
forks AND execs another binary it will stay that way.
Examples: httpd, nfsd, named, most databases.
Proposition #6-- If a failure of the C library occurs, and the
servers are still running, then on a system where
downtime is considered unacceptable or difficult, you
should fix the problem without a reboot if that is
possible
Proof: Application of propsitions #1, #2, and #5
Proposition #7-- Debian should include by default whatever is
required to fix the server as in #6, providing the
cost of including these services are not too high
Proof: Application of propositions #3 and #6, plus the
assumption that there are other factors to contrast
the cost with.
Proposition #9-- Statics recovery tools are more fault tolerant
than dynamics equivalents
Proof: Dynamics typically depend on a large number of files, and
the failure of any of those files will bring the dynamic
binary down. For example, bash depends on:
-- bash, libncurses.so.4 (symlink), libncurses.so.4.2,
libdl.so.2 (symlink), libdl.so.2.1.2, libc.so.6
(symlink), libc-2.1.2.so, ld-linux.so.2, and
until recently on libreadline as well (2 files)
whereas sash depends on only one file (sash). Thus there
are eight (previously 10) files upon which bash depends,
but only one on which sash depends.
Proposition #10-- There are classes of errors which static binaries
can fix, that dynamic binaries cannot fix, which are
fairly catastrophic to a system, but which leave many
important services running
Proof: Application of propositions #4, #5, and #9, plus
you can repair the C library with the tools we have
proposed be included in statics: sash to log in,
copy, and move files, ar to unpack archives, and
mount to acquire copies of the archives over NFS
Proof: Dynamics require a reboot along with an alternate
root or a rescue disk, which is unacceptable or
difficult on some servers, by propositions #1 and #2.
Proposition #11-- Static Recovery binaries do not use a lot of
system resources, such as memory, disk, or CPU
Proof: Recovery programs are not run very often, and therefore
have negligable impact on CPU or memory overall.
Proof: Recovery programs are relatively small (~300k), and for
example both the memory and disk footprint for sash
is less than bash
Proof: Very few recovery programs are required, given that so
many commands are built in to sash, so they don't
add up to much disk space
Proposition #12-- The presence of static recovery tools will
not pose any difficulties to ordinary use of the system
Proof: We have proposed a mechanism by which roots shell could
include a fallback static shell while transparently
providing access to dynamic bash for ordinary use.
Proof: We have alternately proposed the creation of a second
UID 0: one with a dynamic shell, one without. The default
user would have a static shell, but users who prefer a
static shell can enable the alternate account and use
that.
Subproposition #10.1: This is not a security hole.
Proof: The second account is disabled by default.
Subproposition #10.2: System scripts will not be affected
Proof: Neither crontab nor init nor rc use it, and
all other scripts require #!/bin/sh to execute
Subproposition #10.3: Non-system scripts which access the
root user via rsh or ssh or similar mechanism
may be affected, but this can be resolved.
Proof: Sash can use /bin/sh for the -c option, which
is the only way these services can use the
root shell to execute scripts
Proof: Static binries would be in an alternate directory,
and not ordinarily accessed by users
Proposition #13-- Static binaries can be provided with negligible cost
Proof: Application of propositions #11 and #12
------------------------------------
Conclusion: Debian should provide static binaries
Proof: Application of propositions #7, #10, and #13
> Not only that, they are
> opinions that many of us do not agree with based on our experience with
> *using* Debian as a stable production server.
I think I've more or less dealt with this above; though if you are
going to state your experience, I may as well state mine: statics
have been required to fix Debian systems a few times (and a couple
times required a reboot since I didn't have them installed--I just
expected they would be there already!). These were desktop machines,
but no matter--the experience of needing statics was the same.
Since I can see through the shortcomings in the present release, to
the deeper benefits that Debian can offer, I have not given up on
the idea of using it as a reliable server, but instead have come
here to argue for the things that will make it possible.
I actually started this thread in the first place because I was
so surprised that Debian didn't install statics by default, that I
came here to find out whether stability and reliabiliy were
considered important goals on this list. If not, then I thought
perhaps I should be putting my energy into finding a different
distribution; if so I decided I was willing to wait for the
rough edges to be sorted out and continue building experience
with Debian on desktops and development servers.
The truth is I still don't know whether reliability and stability
are important issues here, since several people have bluntly told
me that live recovery is not what 99% of Debian users need and
therefore I should go away.
> Neither Craig nor anyone
> else that I've noticed has objected to you making optional packages to
> do whatever you want.
However I have repeadely produced arguments that this is not adequate.
All that has been said in response to those arguments is a series of
statements of opinions, and sometimes flames.
#1 -- People who are running important servers often don't know
that statics are important (we live in such a world; even
when that level of experience is present it's frequently
junior people who do installations)
#2 -- The debian policy states that anything an experienced Unix
person expects should be provided by default, and it has
been said on this list by others several times that it's
obvious many experienced Unix admins expect to be able
to recover from a failure of the C library
So there is a two pronged argument against your position:
First, and especially since the cost is so low, it is wise to
include the tools that an admin will need to recover from failure
by default, because many such admins are effectively learning as
they go, and when the server fails, it is too late for them to
learn that statics were important.
Second, your position is in violation of debian policy.
> I think it's the attitude of "we know what's good
> for you, and any objection is crap" that is making Craig crazy
I have never said any objection is crap. I have only said that it
is crap to object without stating any technical reasons why. Several
people here are screaming against my position, apparently until they
are blue in the face--but totally failing to justify their objection
versus the arguments that have been put forward.
> It's particularly annoying when someone
> posts "Don't worry, it'll work for everybody 'cause I tried it on my
> system and it was ok".
I have never done that. On the other hand, your side has repeatedly
done that: "I have never needed it", "My system never failed that way"
are often repeated here.
> (Again, this is *signficantly* different than
> saying "I analyzed the effect of this change on these programs, and here
> is my testing method, and here are the results, so I think it's ok.")
I think that I have done that. Obviously not the testing part, but we
are only in the proposal stage. I'd be happy to talk about what kinds
of testing are required, and certainly willing to accept that this
may not be something that makes it into the current release, due to
the possibly extensive nature of the required tests.
I am happy to wait several releases before putting Debian on my
servers. I have alternate solutions that work fine. Debian's package
manager would make my life easier, and it has several other advantages
(one of them being the Linux kernel).
So, I am not in a hurry, I am arguing for eventual correctness, not
immediate gratification.
> The only benefit you've shown for static binaries is for use on remotely
> administered systems, where the admin has neither physical access nor
> the someone on site to follow simple instructions like "put the CD
> labeled 'Emergency use only' in the tray and hit Ctrl-Alt-Del.", and for
> those situations where /lib is damaged but /bin isn't. I certainly don't
> deny those cases exist, but I strongly doubt they are even a significant
> minority of Debian's user base. Yes, we should accomodate those needs,
> but I see little reason to inflict it on everybody.
That is not the only benefit I've shown. That's only one very large
class of servers (downtime is a logistic difficulty for various
reasons); there is also the additional class of servers where
downtime is considered unacceptable (eg: NFS).
> > I do have Debian running on lots of systems, but not on any servers. It
> > has not yet (and neither have the other Linux's) given me the confidence
> > in practice to want to make it a server. I am trying to address that by
> > discussion my issues on this list.
>
> Opinion, opinion, opinion.
Wrong. The above was written as a refutation of Craig's statement that
I had never used Debian and had no experience with it. That was a
personal attack which required a response.
I did not use the above to argue that statics should be included
in Debian.
Quoting me out of context like this is totally out of line.
> > Kindly stop with the flames; I'm eager to hear your technically
> > meritous arguments, and will henceforce ignore your unsubstantiated
> > opinins and flames. I fear this may be my last post to you.
>
> Ditto. So long as you continue to ignore the core "techinically
> meritous" argument, the rest of us are reduced to trying to shout loud
> enough to gain your attention.
So, exactly what is that core technically meritous argument? I am
dying to hear about it. Several people have raised legitimate
concerns, and we've gone off to contemplate them and come back
with proposals that satisfy those concerns.
I am certainly open to doing that again, and iterating through this
process for a year if necessary, until everyones legitimate concerns
are dealt with.
So please, do raise objections to our proposal by pointing out specific
shortcomings that need to be addressed.
We will either show that the described shortcoming is not in fact
a shortcoming at all, or else go away and work out a resolution. By
doing this many times already in this discussion, we've moved
forward significantly, and now have a much stronger and better
proposal than what we started with.
> In case you missed it, here it is again: Cautious sysadmins don't screw
> with working systems unless there is potential benefit to outweigh the
> risk. I agree that your proposal is not that risky. However, I and
> presumably others see even smaller benefit. Therefore, we object to
> making the sash installation required.
I agree with the sentiment expressed above 100%.
The basic set of statics can be included without any risk, as I
have argued above that they have no significant cost.
The danger involves switching root's shell, and here your concern
must be taken very seriously--we definately should not just rush
in and make this change recklessly. It should, at the very least,
be phased in over two or three releases--to provide adequate
testing, and discover any possible problems, before making it
the unquestioned default.
Here is a possible roadmap:
Step #1: The static root shell is an option, which people can
install and try out. It stays an option until we
establish that it basically works. I think we are
effectively in this stage right now, since many
people (including me) run sash as their root shell.
Step #2: We move to asking a question in the install, over
whether to include it or not. The question would
describe the benefits of a static root shell, as well
as a warning that it was an experimental idea:
"Do you want to use sash as your root shell? This
can help you recover from several kinds of failure
without requiring a reboot or rescue disks; however
it is currently experimental. Yes or no:"
We continue this way until it's been around for awhile,
many people are using it, and there are no problems.
Step #3: We stop asking the above question, and just install it
by default. There would be an optional package that
could be installed to revert to the old behavior, for
people who liked that better
I think at most right now we are proposing to move to Step #2.
Maybe the problem is that I didn't make this clear enough--I think
that a static root shell is extremely important, and will not run
Debian as a production server until it's available by default (or
someone convinces me why I don't need it).
However that doesn't mean I want to be reckless about it; I'm in
no big hurry. I can wait awhile, and we can take our time and
make sure it gets done correctly.
> You, on the other hand, are now saying that you don't want your proposed
> "solution" to be required either. That was *not* clear from many
> previous posts. In which case, I don't know what we're arguing about.
I initially proposed that it be marked important, and later said I
was willing for it to be designated "standard" if there was too
much opposition to the important designation.
My goal has always been to see it go in as a default, so that people
who don't know that they need statics (but do) will get them. It
would also be acceptable if it asked whether you wanted it, with
an explanation of why it was wanted--because then people who needed
it would realize, "Gee I need that" and install it.
I understand that there are systems which may have tight disk space
limitations, or where for whatever other reason the statics aren't
wanted. I don't think we are talking about a whole lot of disk
space, but you still should be able to go out of your way and
deselect the package.
And all of what I have just said is an eventual ideal state, not
something I think needs to be rushed into immediately--if we can
agree that this is where Debian should go, we can talk about exactly
how slow we should go to get there.
Justin
ps: Though we ("my side") seem to have decided that "sash" should
be the root shell, I personally have some reservations about
that and wonder if it shouldn't be "ash" (with sash also provided
due to its many builtins and thus economical use of disk space).
There would then be no '-c' issue, and less need to have a
second UID 0. I am satisfied with having sash as root, though
it wouldn't be my personal first choice. A small, static version
of bash is another option--if there is such a thing as a small
version of bash (I doubt it).
Reply to: