[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: devel files and libraries in /lib

On Tue, Jan 04, 2011 at 07:32:06PM +0100, Olaf van der Spek wrote:
> On Tue, Jan 4, 2011 at 7:29 PM, Steve Langasek <vorlon@debian.org> wrote:
> > On Tue, Jan 04, 2011 at 06:51:13PM +0100, Olaf van der Spek wrote:
> >> On Tue, Jan 4, 2011 at 6:49 PM, Sven Joachim <svenjoac@gmx.de> wrote:
> >> > Maybe we're talking at cross-purposes here; I was speaking about the
> >> > case of turning a directory into a symlink on upgrades, which cannot
> >> > safely be done while there are still files under it.
> >
> >> > Thinking more about it, this cannot be done even if all packages move
> >> > their files away from /usr, since an unknown amount of stuff exists
> >> > under /usr/local, so /usr can ever only be made a symlink to / for new
> >> > installs (or by the sysadmin).
> >
> >> I think it's about /usr/lib being a symlink to /lib.
> >
> > No, it isn't.
> Then what is it about?
> I thought the problem was having both /lib and /usr/lib.

Well, that's the issue at hand.  The reason I mentioned this is
because I believe that the / and /usr separation is a case where we
should stop to consider the "bigger picture" rather than just the
immediate problem. Solving that would solve the /usr/lib and /lib
issue as a natural consequence.

> Linking /usr to / would also solve that but is more complex than
> linking /usr/lib to /lib.

Only minimally; the same issues apply to unifying both.

A bit of background can be found in this thread from 2009:
This is a long thread, but there's a lot of useful detail in it, the
most important bits I've summarised below.  Note that this is
concerning not allowing /usr as a separate filesystem, but eliminating
/usr entirely is a logical consequence of it and so is mostly
relevant here.

Historically, we have always had a separate / and /usr, with the
possibility of /usr being a separately-mounted filesystem.  So, the
points to consider are
• what are the benefits of the separation?
• what are the costs of the separation?
• do the benefits and costs of the separation as originally conceived
  apply today in 2011?

I was previously a proponent of keeping /usr separate, but after
spending some time considering it I've revised my opinion of the
matter.  I previously thought it had a useful purpose in some cases;
I'm now of the opinion that it's essentially pointless.  Most of the
historical reasons for keeping it separate are today no longer
relevant; they are either superseded by better methods, or are not
compatible with modern practices.

To provide some context, I'll run through a few of the use cases and
try to provide some insight into the rationale behind them, and some
opinion on whether it's actually useful.  Most of these were discussed
in the above thread.

1) /usr is shared by many machines over NFS

   This is a "traditional" use of a separate /usr.  It may have even
   been common back when SVR3 was popular and disc space was at a
   premium, and modern package managers were absent.  But, it's
   completely insane to do this today.

   • Saves disc space by sharing one filesystem amongst many hosts
   • Allows centralised management of software updates; updates only
     need doing once for all hosts

   • Debian uses dpkg for package management; you can't separate / and
     /usr and expect it to work since packages need to update both
     locations.  If you have a / per host and a shared /usr, a
     disaster is inevitable.
   • /bin, /sbin, /lib are not shared, so you need an alternative
     mechanism to propagate changes to the hosts; and /etc is even
     more complex.  Also, how do you run dpkg maintainer scripts on
     each host?
   • Debian does not support a *shared* /usr over NFS.  It supports
     mounting a *non-shared* /usr over NFS, but that's not the same

   If you're doing this, you're doing it wrong.  You should be NFS
   mounting the root filesystem in its entirety or else dpkg will
   break.  I use debian-live to manage such images, and NFS mount the
   shared root (containing /usr) with a writable unionfs overlay to
   allow host-specific configuration and customisation.  It's robust
   and supported.

2) /usr is mounted read-only for security and safety

   Mounting /usr read-only is common practice; I even do this myself
   with apt-get configured to remount read-write when changes are made
   and then remount read-only afterwards.

   • You have read-write / and read-only /usr which prevents changes
     to /usr during normal system operation.
   • Allows mounting with nodev

   • Does not protect /bin, /sbin, /lib etc.
   • Can be done on read-write / with read-only bind mounts
   • Can be done with read-only /
   • With udev, nodev can be used on /

   If /usr is eliminated, then we can just mount / read-only instead;
   it's arguably even safer.  Quite a lot of work has gone into having
   a read-only / such as /lib/init/rw.  It works today, though a few
   small issues remain (/etc/mtab, LVM backup/cache/archive).

3) A minimal root filesystem makes recovery easier

   Having / without /usr and other filesystems mounted makes it
   possible to troubleshoot and recover from problems such as
   filesystem corruption.

   • /etc will be recoverable if /usr gets corrupted

   • The system is unusable without /usr mounted; do we really care
     about the small chance that one filesystem will be corrupted over
     the other.  i.e. does having them separate really reduce the
     probability that we will lose /etc in any meaningful way?
   • You also need /var (for dpkg status etc.) for recovery
   • Given that / is typically read-write and /usr might be read-only,
     the chances are that / will be corrupted rather than /usr unless
     you have hardware failure, in which case both will likely be

   I've occasionally used single user mode when LVM problems or
   hardware changes prevented filesystems being mounted etc.  But it's
   been more common to use the initramfs shell or a rescue CD.
   Overall, I can see that this mitigates recovery, but I'm not
   convinced that it's giving that much extra robustness.  I'll be
   keeping backups for recovery/reinstallation rather than playing the
   probability game.

3) Different backup policies for / and /usr

   • Alleged performance gains

   • You don't need them on separate filesystems for separate backup

   I don't buy this one.  While you might have a different backup
   policy for /etc and /var, there's no reason to treat any dpkg-managed
   location such as /bin, /lib and /usr/* differently from each other
   since they exist as a managed whole.

4) / is encrypted and /usr is unencrypted

   • Performance

   • Do you really need the entire / encrypted, or just specific bits?

   This looks like it's for encrypting things in /etc.  I'm not sure
   that this solution is necessarily the best one.  The / vs /usr
   distinction is one of convenience rather than actually encrypting
   only the bits actually required.

5) I have /usr on LVM and / on a partition

   This is a fairly common setup; I used it myself in the past.

   • It is possible to mount the root filesystem with no LVM in case
     there are LVM problems

   • / is of fixed size; you can't online resize it

   Nowadays this is a non-issue.  I've had my root filesystems on LVM
   for years now; with initramfs and separate /boot; there's no need
   for these workarounds and recovery on LVM vs a normal partition is
   just as easy.  A rescue CD can deal with both just as effectively.

   If you're using LVM, you should have your / on LVM, which would
   allow it to be dynamically resized as for all other partitions.

6) Libraries are separated in /lib and /usr/lib

   Programs in /bin and /sbin use /lib; programs in /usr/bin and
   /usr/sbin use /lib and /usr/lib.  Programs on / must work
   prior to /usr being mounted.  Libraries on / must only depend
   upon other libraries in /.

   • It's difficult to draw the line of what should be in / and what
     should be in /usr
   • Special care needs to be taken before /usr is mounted since
     important pieces such as locales are not available

   This is the problem initially brought up here.  As we increase the
   complexity of what is required prior to mounting /usr such as
   having NSS modules with dependencies on other libraries, those
   other libraries need moving to / as well.  The same applies to
   crypto support as mentioned initially.  If / and /usr are unified,
   the problems introduced by having them separate are immediately
   resolved: everything becomes available from / being mounted except

7) A separate /usr is required to boot the system

   Many years ago, a minimal root filesystem was required to bring
   the system up to a certain level of operational readiness in order
   to mount /usr and other filesystems.  Examples included:

   • NFS mounts which required networking and portmap/rpc
   • RAID requiring array reconstruction
   • LVM which required discovery of available volumes
   • Loading additional kernel modules to access hardware
   • And other things

   For quite some time now, Debian uses an initramfs to boot.  This
   does all the setup for you.  This means you can mount / from NFS
   or from LVM or MD (or both stacked) etc.  There's no requirement
   for a separate /usr for booting.

8) Filling up /usr won't fill up /

   Installing software won't use up all the space on the root filesystem

   Nice idea in priciple… but you can still fill up your root
   filesystem with kernel images and initrds if /boot isn't separate.
   This is a preventative measure, but not a particularly effective
   one.  And if filling up / occurs, then it's not exactly hard to
   reverse the procedure.

9) / and /usr are using different filesystem types

   This is a rather hard one to take seriously.  While there are times
   when you would choose one filesystem type over another for various
   reasons, using different types for / and /usr makes little sense
   given their similar requirements.  It provides zero benefit.

I think many instances of confusion and misunderstanding over the / and
/usr separation, particularly with regard to NFS mounts, but also
covering read-only mounts and recovery are due to not fully considering
the implications of a modern package manager on the traditional UNIX
filesystem hierarchy.  We can consider that there are just two
different types of directory in the file system:

• those managed by dpkg
• those containing user data which are not under dpkg control

All locations managed by dpkg must be considered a unified whole; it
does not make any sense to share one part and not another.  They must
be updated together or else the system will be left in a broken and
inconsistent state.  A separate /usr is no longer required to boot the
system now we have initramfs.  In consequence, there's no reason to
have /bin separate from /usr/bin, /lib from /usr/lib etc.  It makes
sense to have /var separate (writable).  It makes no sense to have /usr


  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux             http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?       http://gutenprint.sourceforge.net/
   `-    GPG Public Key: 0x25BFB848   Please GPG sign your mail.

Attachment: signature.asc
Description: Digital signature

Reply to: