[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: Moving to the FHS: not right now!

>>"Wichert" == Wichert Akkerman <- Debian Project Leader <leader@debian.org>> writes:

 Wichert> Luckily the constitution provides us with a way to solve this: the
 Wichert> Technical Committee can be asked to decide on a strategy which people
 Wichert> will have to follow. I hereby ask them to study this and come up with a
 Wichert> strategy for moving to the FHS.

        A draft proposal follows.

Problem statement:

 The FSSTND location for package documentation, /usr/doc, has moved to
 the location /usr/share/doc in the FHS. Now that we have formally
 decided to adopt the FHS, we need to ensure the transition is as
 smooth as we can make it.

     Some of the constraints are:

        * The transition may take a long time, going by previous
          transitions, and not all packages are upgraded anywhere near

          I think that expecting _*all*_ packages to have moved before we
          release potato is futile, unless we do not plan on releasing
          potato for 18 months or so. We *_cannot_* expect to get FHS
          compliance in place by potato.

        * We should not break backwards compatibility during the transition
          period. This is a quality of implementation issue

          During the transition, we need to provide backwards
          compatibility, firstly for programs like `dwww', and `dhelp', and
          also for our users who have gotten used to looking under a single
          dir (`/usr/doc/') for docs (`/usr/doc/`package''). During the
          transition, the documentation could be in in two places, and that
          is not good.

        * There is no elegant way to piece wise move a directory
          spanning multiple packages with dpkg.  Basically any attempt
          to provide 'legacy' symlinks that make /usr/doc/foo ==
          /usr/share/doc/foo will either break upgrading, downgrading
          or both, unless special care is taken

  dpkg has no ability to resolve multiple paths to the same files in
  it's database, so it sees /usr/share/doc/pkg/bar and
  /usr/doc/pkg/bar as -different non overwriting- files and will end
  up doing the wrong thing when it comes time to erase the old
  versions files. Basically it will do:
     create /usr/share/doc/pkg/bar
     rm /usr/doc/pkg/bar
  Which is perfectly sensible -IF- they did not happen to be the same

  What makes this directory move different from the other changes
  mandated by the FHS is the human element: humans are used to
  looking for documentation under one directory, namely,
  /usr/doc/<package>, and having the documentation in two directories
  is breaking that expectation.

  What follows is an updated solution to this issue (the so called
  symlink solution).

  Several other solutions have also been proposed to address this
  situation. I shall present a (hopefully unbiased) selection of
  solutions, with arguments for and against each solution.

 Proposed solution

     I propose that there be a symlink from /usr/doc/package =>
     /usr/share/doc/package, managed by the package itself. Since there is
     some concern that the packaging system does not deal well with
     replacing a directory with a symbolic link, it is suggested that the
     link be manipulated in the maintainer scripts postinst and postrm.

     We create the postinst, prerm now, installing the symlink, Once
 the move is over, we just have a prerm script removing the
 symlink. another release, (potato+2), we stop bothering, since we
 would have handled the most common case (and provide an base-files
 postinst script removing the symlinks for upgrades at that point).

        Yes, this is a long period, but not forever (which was one of
 the major objections to this proposal)

     So potato packages should have: (effectively)

	postinst install upgrade:
	    if [ -d /usr/doc ]; then
	      if [ ! -e /usr/doc/$package -a -d /usr/share/doc/$package ]; then
	        ln -s /usr/share/doc/$package /usr/doc/$package

	prerm remove upgrade:
	    if [ -d /usr/doc ]; then
	      if [ -L /usr/doc/$package ]; then
	        rm -f /usr/doc/$package

	    The prerm removes the symlink, and the postinst reinstates
 it as necessary. The only problem is redundant symlinks when you're
 doing partial upgrades to woody+1 (or later), which no longer need
 any symlinks, from potato/woody. But they still all point at the
 correct places. This is handled by a base-files postinst removing the
 junk symlinks (details below).

     This is how it works:

     1.   Policy 3.X mandates that packages that move the docs to
          /usr/share/doc must provide a compatibility symlink in /usr/doc.
          This shall allow packages to incrementally move to policy 3.X,
          while providing compatibility with older systems.
          (/usr/doc/package symlink is handled by package)

 Thus, potato ships with a full /usr/doc/ (some of which is symlinks),
 and a partial /usr/share/doc.

     2.  Post potato, we continue the transition, with the symlinks in
         place. Before freeze, we file important bugs against any
         package that has not been moved over (in one and a half
         release periods, we may be actually able to accomplish this),
         with NMU-fests to bring over the others.

 Thus, potato+1 (woody) ships with a full /usr/share/doc, and a
 /usr/doc full of symlinks.

     3.   At a later date, another policy (say, 4.X) shall ask for
          packages to no longer provide the link (and possibly remove
          links from /usr/doc). We can also provide a script (possibly
          in base-files postinst) that rm's symlinks from within
          /usr/doc. woody+1 may ship with such a script. (there was a
          proposal as well that potato+2 (woody+1) ships with just the
          prerms, and not the base file script, and potato+3 ships
          with the base-file script, but I am not sure this long a
          reversion period is required).

 No dependency on this base-files shall be required, since they shall
 work with or without a symlink in /usr/doc (and /usr/share/doc shall
 be fully populated by then).

Thus, partial upgrades to potato and woody have a complete /usr/doc,
and full upgrades to woody have a complete /usr/share, and symlinks
throughout /usr/doc. Partial upgrades to anything beyond woody might
have old files left in /usr/doc, but they'll get moved when whoever
finally gets around to run an apt-get dist-upgrade.

     I understand that the forest of symlinks is ugly, but it is
     technically sound, it maintains backwards compatibility, it allows
     incremental compliance to FHS, and caters to a hybrid system. I submit
     that aesthetics takes a back seat to functionality.

 I fail to see what the big problem with fairly obvious symlinks
 existing on a system that's halfway between two significantly
 different file system standards. I also can't see admins being
 particularly bothered by the existence of such symlinks on such

 OTOH, it *is* inconvenient to have documentation scattered
 inconsistently between /usr/doc and /usr/share/doc. We've even had
 user complaints on this list about it.

     To the objection that it shall cause package to be modified twice, I
     say that

        * This is a complex transition, and may require this to meet the
          constraints of the situation

        * One modification of the packages is required anyway for compliance
          with the FHS.

        * The second modification, namely, the removal of the symbolic
          link, shall be well into the future, and added to a future policy
          change. It is conceivable that the packages may need to change
          for policy updates in any case.

  The symlink proposal is far from perfect --- adding postinst's and
  prerm's to every package with the same half dozen lines of code is
  really pretty lame. The *proper*, forward thinking solution would be
  to add a feature to dpkg to run specific items of code before and
  after each dpkg run.

  But modifying dpkg is infeasible, and we've agreed to, among other
  things, keep the needs of our users at the forefront of our
  minds. And from a user's perspective, something that keeps the
  system tidy in the normal case, and works *now*, is much better than
  idealistic fantasies like a working dpkg.

Overview of other solutions proposed.
 * do nothing special
     - means the admin, and all automated tools have to look in both places
       or miss documentation

       This essentially ignores the issues with the split locations
 for the documentation. I think we can live with the problem, but as a
 quality of implementation issue, it would be nice if we had a smoother
 * making a script that works out which of /usr/doc /usr/share/doc to use
     - doesn't work for browsing by hand
     - needs to be written/accessible from lots of different languages
     - is significantly different to any other system on the planet
     + makes it easy to "support" /usr/local/doc or similar ("support"
       in scare-quotes because I'm not really sure of the value of
       such support in this case. YMMV)
  [See Appendix A on details and commentary]

 * symlinks managed by postinst/prerm
     - requires lots of packages to add postinsts/prerms for potato
       and woody, and then to get rid of them for woody+1
     - may leave crufty symlinks about on systems where (a) the admin
       doesn't fix it (b) hasn't had the base packages upgraded to woody+1
     + allows incremental upgrades
     + allows for downgrades
     + does not involves a global change day
     + works with dpkg failings ;-)
  [This was proposed and shot down because 4 people objected;]
  [See Appendix B for details].

I think the symlinks proposal is the best one so far ---
it addresses the issue correctly, and it's drawbacks are both aesthetic
(crufty symlinks that don't damage the system; and extra postinsts)
and temporal (upgrading to woody+1 gets rid of both the symlinks and
the extra postinsts).

 * cronjob that adds/removes symlinks in cron.daily
     - downgrading a package that uses /usr/share/doc to a prior version
       that uses /usr/doc will cause dpkg to move files from
       /usr/share/doc/foo to /usr/share/doc/foo via a symlink, which is
       believed to be dangerous. (*)[proof in appendix C]
     - may leave dangling symlinks / may not have symlinks for up to 24 hours

A cronjob is a bad idea because the links will persist for dpkg
operations and basically cause upgrades/downgrades to fail.  This is
not a working solution.
 * mv /usr/doc/* /usr/share/doc
 * Modify dpkg's internal databases (mainly the .list files in the
   directory /var/lib/dpkg/info) so that they are in sync with the
   previous changes.
 + would make the system to be /usr/share/doc-compliant.
 + would avoid to touch every maintainer script in every package.
 + would support upgrades and downgrades.
 - fiddles with dpkg internals, against all warnings in the docs
 - pointless unless the package is changed, or the next upgrade
   creates the /usr/doc dir all over again

  This isn't trivial, because you cannot be sure that /usr/doc and
  /usr/share/doc are located at the same file system.
  And don't miss the (few) packages which already moved to
  /usr/share/doc (where some of them left back a .dhelp file in
  /usr/doc/<package>). Also, doing it the way mv(1) does means
  failures halfway through leave you with files in /usr/doc/foo and
  /usr/share/doc/foo, which could be hard to deal with correctly.

  If one uses tar/cpio to do the move, one should realize that if you
  *do* have /usr and /usr/share on the same partition, you need to have
  as much free space as /usr/doc takes up.  Unless you do the move on
  a directory by directory basis.

  Also, fiddling with the dpkg database - bugs in the script can cause
  catastrophic system failure by messing up the dpkg database (*) - must
  be done by the admin by hand, rather than automatically as part of the
  upgrade to potato/woody - script is difficult to get right on common

  Messing with dpkg's database is *not* something to be done lightly.
  Getting knowledgeable comment from the author's seems the *least* thing
  that should be done.

     This is not a working solution.

 * add support to dpkg for pre and post processing on a global basis
     - requires dpkg modifications (*)
     + allows update-menus and similar programs to be implemented much more
     This is not a working solution, given the timetable.
* Stick with /usr/doc until potato is released, then begin a massive
  migration, which may or may not involve symlinks.
     -  some people have already moved and may not want to move back.
     - would need a policy amendment partially backing out of the FHS
     - requires all changes to be made in a single release cycle*
        (or we're back in the same position we're in at the moment)
    - delays the move to FHS compliance.
    + gives us a little more time to decide if we want crufty
      symlinks, and if so, what's the best way to handle it.
    + no surprises to the user.
    + no changes to most packages till after potato's release.


I consider the points marked with a (*) to be completely unacceptable.
By "completely unacceptable", I mean that it's unacceptable to just say
"we should do this". If you *can* do it (ie, write the program/make the
modifications), and then can show why the original fears were unfounded
(look, iwj even says this is okay!), then, naturally, there's no problem.



Appendix A

 docdir() {
   dirname $(grep "/doc/$1/copyright\$" /var/lib/dpkg/info/$1.list")
 cddoc() {
   cd $(docdir $1)
 Now we're back to a single probe.  We're just probing the
 database, not the filesystem.

 [This has to be in every environment, and needs to cater to all kinds
 of shells, and shell scripts, and perl scripts, and ...]
> So we make it a script.

It doesn't just have to be in every user's environment, it has to be (or
at least, ought to be) easily available for arbitrary scripts/programs
to be able to access documentation.  It thus needs to work conveniently
with csh, rc, perl, python and C.

It also adds another learning curve to Debian for, what appears to me
to be, very little benefit. Instead of just cd'ing to /usr/src/linux
and running make, we're meant to use kernel-package. Well, that's fine
--- kernel-package makes it a lot easier to deal with upgrading from
one kernel version to another keeping modules in sync and not leaving
cruft lying about on your disk.  Instead of just cd'ing to /usr/doc and
poking around with your favourite utilities, you now have to either know
the name of the program in advance and use some special functions which
you've likely never heard of before.

 Still does not answer less /usr/doc/<package>[TAB]
> Very true.  There are always tradeoffs -- in this case, we gain
> flexibility.

To what end? Is /usr/local/doc particularly common anyway? We certainly
don't have any intention of keeping /usr/doc around, so having more than
one doc directory doesn't seem to help us.

Needless flexibility breeds bugs.

Appendix B
Details of dpkg lossage with symlinks.

All right dammit, here we go...  built a package crap 1.0-1, here is the


Now from /usr/lib/crap:

drwxr-xr-x   2 root     root         1024 Aug  3 02:15 olddir/
-rwxr-xr-x   1 root     root          633 Aug  3 02:14 olddir/file*

And then after I symlink it:

drwxr-xr-x   2 root     root         1024 Aug  3 02:15 newdir/
-rwxr-xr-x   1 root     root          633 Aug  3 02:14 newdir/file*
lrwxrwxrwx   1 root     root            7 Aug  3 02:19 olddir -> newdir//

Install crap 1.0-2 ...


root@icarus2:/usr/lib/crap# ls -lR
total 1
drwxr-xr-x   2 root     root         1024 Aug  3 02:21 newdir/

total 0

Do you believe us yet?  What more proof do you possibly need?

> (Note that I am explicitly avoiding actually installing or deinstalling an
> actual symlink which dpkg thinks is a directory ... with my proposal this
> can, in fact, happen, but only if users deinstall all packages that refer
> to the /usr/doc directory which is rather unlikely.)

Does not matter.  dpkg breaks.  It has been now demonstrated and proven.

> > Your script also has a cow if /usr/doc isn't -> share/doc, which is bad
> > because it may be necessary to use some symlink magic at some point.  Not
> > that it isn't a moot point unless you fix dpkg first.
> This was intentional ... and in fact the script will merely warn you if
> this is the case.  I was merely trying to KISS since this is a rather
> critical script.
> More reactions welcome !

It exits with 1, that's an error condition.
> I am happy to tell you that we agree completely on the behaviour of dpkg on
> your example.  But you are ignoring a very important aspect of my proposal:
> olddir is actually REMOVED by the deinstallation.

This doesn't seem to be the case.

* Create three packages:
	test1 version 1.0 mimicing your average /usr/doc-using package
	test1 version 2.0 mimicing your average /usr/share/doc-using package
	test3 version 1.0 mimicing base-files

test1 1.0 has a file /my_usr/doc/test1/copyright,
          and depends on test3

test1 2.0 has a file /my_usr/share/doc/test1/copyright,
          and depends on test3

test3 1.0 has a file /my_usr/doc/copyright/GPL,
          and a file /my_usr/share/doc/test3/copyright

* dpkg --install test3_1.0_all.deb

* mv /my_usr/doc/copyright /my_usr/share/doc/
* rmdir /my_usr/doc
* ln -s /my_usr/share/doc /my_usr/doc

* dpkg --install test1_1.0_all.deb
* dpkg --install test1_2.0_all.deb

* ls -l /my_usr/doc/test1 -> empty
* ls -l /my_usr/share/doc/test1 -> empty
* dpkg -L test1 | grep my_usr/share/doc -> not empty

The packages are available as:


Possibly I'm just misunderstanding what you're suggesting should be done
though. Can you give a sequence of commands that does whatever you're
suggesting, and still has those three packages survive unscathed?
On Tue, Aug 03, 1999 at 09:02:53PM -0400, Michael Stone wrote:
> (And thus useless to me.) I don't argue that dpkg has some problems with
> symlinks if a package changes the path by which it references one of its
> files. It does not have problems (as far as I have found) if a package
> consistently uses the same path when referencing the file, whether or
> not that path passes through a symlink. That's the case I'm interested
> in and which I've been testing.

*OH*. No, that's completely correct.

> No, what I wrote works fine. The examples were irrelavent. You
> apparantly ignored the part where I said that all packages should put
> their stuff into /usr/doc without regard for whether it's a symlink or
> anything else.

This means packages will have to use /usr/doc instead of /usr/share/doc
until dpkg is fixed. "until dpkg is fixed" could be forever.

This seems like it might also require pre-dependencies on base-files for
every package (so that they don't accidently create /usr/doc as an
actual directory instead of as a symlink).

Hmmm. This also seems like it might require pre-dependencies from every
package against the new version of dpkg that handles following symlinks

>    1. use both /usr/doc and /usr/share/doc; this upsets the partial
>    upgrade people, and worries the UI people.

"upsets" and "worries" aren't really very precise. It results in a
"needlessly" inconsistent user interface for accessing documentation
for potato and partial upgrades from <=potato to >potato.

>    2. use both /usr/doc and /usr/share/doc but provide symlinks from
>    each package in one to each package in the other by means of
>    preinst/rm scripts; this upsets the anti-bloat people and horrifies
>    the elegance people :)

It creates postinst and prerms where there never used to be any, but lets
them go away after woody.

>    3. use both /usr/doc and /usr/share/doc but provide symlinks from
>    each package in one to each package in the other by means of a cron
>    job or somesuch; this upsets the partial upgrade people, horrifies
>    the elegance people, and terrifies the people who don't want cron
>    jobs automatically deleting things on their systems.

It means documentation can be lost when doing a downgrade from a
/usr/share-using package to a prior, /usr/doc-using, version of that

It means dangling symlinks could exist for up to 24 hours (or however
often the job runs). It means symlinks might not exist for the same
amount of time.

>    4. use both /usr/doc and /usr/share/doc but provide symlinks from
>    each package in one to each package in the other by means of an
>    optional script run manually by the admin; this upsets the partial
>    upgrade people, worries the UI people, and horrifies the elegance
>    people.

Same problems as the cron job, and not automatic.

>    Don't explicitly use /usr/share/doc yet, and we can rig up a symlink
>    to effectively use /usr/share/doc until we come up with a better
>    solution; this upsets the policy-says-I-can-therefor-I-must people
>    and dismays the people who have already converted their packages

Requires changes to dpkg, which don't have a particular history of being
made. Possibly requires lots of pre-dependencies that will last for ever.
Appendix C

  Has no one seriously considered the mess that will happen if you
try to follow this path (namely, making each package manage the
transition by itself)? Think about all the typos (like "[-L foo]")
that people are going to make, the number of link-handling scripts
that bomb out or do the wrong thing, the number of unwanted symlinks
that will be lying around once the dust settles, and the crap that
will live in the maintainer scripts forever (well, a long time). Think
about how few maintainers test their scripts exhaustively (for
idempotency etc.)  (when doing upgrade, new install, failed-upgrade,
configure, remove, purge, etc.).

I will oppose the proposal unless it contains code fragments that a)
have been thoroughly tested, b) have been shown to be the only real
solution, and c) will be mandated.

One question, does the FHS permit "/usr/share/doc -> ../doc" ? If so,
there is a very simple solution (forbid the use of /usr/share/doc and
have base-files contain this symlink; this does not address mount and
partition issues).

A better solution (than the proposed one) would be one that has all
the code in only one place and the opportunity to fail at only one
time. Alas, I do not have such a thing.

Unfortunately, various people have pre-empted the policy discussion
and have started using /usr/share/doc already.


 A fool's brain digests philosophy into folly, science into
 superstition, and art into pedantry.  Hence University
 education. Shaw
Manoj Srivastava   <srivasta@debian.org>  <http://www.debian.org/%7Esrivasta/>
Key C7261095 fingerprint = CB D9 F4 12 68 07 E4 05  CC 2D 27 12 1D F5 E8 6E

Reply to: