[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

possible freetype transition; improved library handling needed for all C/C++ packages



Hello cruel world,

Due to upstream ABI changes, it looks very likely that libfreetype is
going to have to undergo a library transition in the near future[0].
The details are still being settled, and it's possible (though unlikely)
that the library will be fixed so that no transition is needed, but in
the meantime I'd like to leave you all with a public service
announcement.

There are currently 583 packages in unstable which depend on the
libfreetype6 package.  That means that if this transition happens today,
it will be bigger than the KDE transition was; it will be bigger than
the OpenSSL 0.9.8 transition was (469 binary packages in unstable depend
on either libssl0.9.8 or libssl0.9.7 today).  *And it shouldn't be*.

The truth is that, although freetype is a widely used library (as the
more or less standard handler in Debian for TrueType fonts), of those
583 packages in unstable which depend on libfreetype6, only *178* of
them come from source packages which build-depend on libfreetype6-dev.
That means that the vast majority of these packages are either missing a
build-dependency, or, more likely, are ending up containing binaries
that link against a library which they do not use directly.  And
freetype is not the only library like this in Debian.

At DebConf 4, I gave a presentation [1][2] which discussed the issue of
brittle library dependency trees in Debian, and the problems they cause
over time for a distribution with a decentralized development model.
For various reasons, this issue never made it to significant discussion
on the mailing lists; this is now threatening to bite us big and bad
with freetype, and it's time to raise awareness of this issue in order
to try to get the freetype transition, and future transitions like it
under control.

I would encourage you to read the presentation in question, but I will
also summarize here: due to accidents of history, the convention when
linking an executable or shared library is to tell ld to recursively add
references to *all* libraries in the dependency tree below it.  This is
wrong for modern GNU/Linux systems when using dynamic linking, because
the dynamic linker recurses the dependency tree on its own; and
following this convention means that executables get not the needed one
reference to the library, but two or more.  Where things *really* go
wrong is when the library goes through an soname change.  If two
different versions of the library are specified in the ELF header, then
either users can't install the package, or they can install it and then
it segfaults.  In either case, this situation causes a lot more work for
maintainers, autobuilders, and the release team (trying to get packages
into testing) than it should.


This email, therefore, is a call-to-arms for maintainers to improve
the library handling in their packages.  The impetus for this request is
the prospective freetype transition, but if you maintain *any* packages
in C and C++, this mail is still addressed to you.

At this point, many of you may be wondering why we don't just keep
around old copies of the libraries in question through the transition,
and save ourselves the trouble of fiddling with individual packages.  It
is actually my intention to keep around a libfreetype6 package through
the transition due to the large number of reverse-dependencies, but this
is only a partial solution, for two reasons:

- there's still a cost to this, because someone has to maintain both
  copies of the lib
- since these applications we're talking about depend on the lib without
  using it, almost all of them got the dependency because some lib they
  *do* use uses it, so loading two versions of the lib into memory at
  once as a result of a partial upgrade will almost always cause
  segfaults... and that's bad!

Moreover, this is a problem that happens over and over again, every time
there's a library transition.  It's not hard to look around and see that
over the course of a release cycle, we spend a lot of time managing
library transitions -- time that could be better spent on enhancements
that actually matter to our users.  Pruning our package dependencies is
(ideally) a one-time fix for each package, with long-term time savings
for each maintainer (and for the release team!).  Think how nice it will
be to have only a third as many dependencies, and only have to rebuild
your package for a third as many transitions!


So what changes should you make to your packages to help?

* Use Debian's libtool.

One of the chief instigators of excess -l options in linker commands is
libtool.  In order to ensure that libtool works for static and dynamic
linking, on both GNU/Linux and on other Unices, libtool keeps track of
all dependencies of all the libs your package uses, and puts them on the
linker line all at once.  There's been discussion with libtool upstream
about fixing this, but in the meantime, you can use the Debian version
of libtool instead, which includes a patch to prune unused libs from the
linker line.

For details on how to update the libtool used in your package, please
see Scott James Remnant's excellent howto at [3].

* Fix your pkg-config .pc files.

If you are a library maintainer, *you* can help make library transitions
easier by making sure you aren't contributing to the problem.  The
second biggest source of bogus lib dependencies is pkg-config.  If you
have a library package that provides a .pc file for use with pkg-config,
please make sure that your upstream is using the Requires.private field.
This field, added by Tollef Fog Heen in pkgconfig 0.18, makes it
possible to separate out the libs everyone needs to link against for a
package from the libs that are only needed for static linking.  If your
upstream hasn't adopted this field yet, please encourage them to do so!

Since support for this field has been recently added, your package
probably needs a build-dependency on pkg-config (>= 0.18) in order to
use it.  For example usage, please see /usr/lib/pkgconfig/cairo.pc in
the libcairo2-dev package.

* Don't use other *-config tools.

While many libraries these days use pkg-config, there are also other
libs which ship their own tools for querying library and header include
paths, libs needed for linking, etc.  The problem is that all of these
tools I've seen are incapable of distinguishing between what's needed
for static linking, and what's needed for dynamic linking.

They're also almost all unnecessary when building Debian packages,
because for any library that's following the FHS, the linker options
needed are very simple: "-lfoo".  So even though it may mean diverging
from upstream (who can't depend on the FHS like we can), I would still
recommend hard-coding the libraries you need instead of using
complicated tools to pick up dependencies you don't.

There are two common ways to hardcode such an override.  The first would
be to edit upstream's build scripts (e.g.,
Makefile/Makefile.in/Makefile.am, configure.ac, acinclude.m4...) to use
a hard-coded value instead of a detected value.  In general, I think
this is a better option even though it means patching upstream code,
because it means upstream changes to lib handling will be detected at
patch time.  The other way is to override a make variable in
debian/rules.  For an example of the first approach, please see [4]; for
an example of the second, [5].

* If all else fails, use -Wl,--as-needed.

If for some reason one part or another of the above doesn't work for
your package, there is one more option available to you.  Since binutils
2.15, it has been possible to pass -Wl,--as-needed as an option to gcc
in order to tell ld to try to figure out which libraries are *really*
needed.  Opinions differ on whether this option is a good idea at all,
and this option was broken on some architectures in the recent past, so
it's good to avoid it if possible; and there are certainly some cases
where it will do the wrong thing.  But it's usually better than getting
to rebuild your package six times a year for nothing!


If you maintain a package that's affected by this issue, you can help
today; there's no need to wait until your package is hit by a library
transition to make the changes described above.  Indeed, for packages
which depend on libfreetype6, it's important that we begin fixing these
as soon as possible to avoid another multi-month transition.  A
complete list of packages that are potentially affected by the freetype
transition can be found at [6].

If you have any questions at all about this (or even objections,
naturally), please ask on debian-devel.

Cheers,
-- 
Steve Langasek                   Give me a lever long enough and a Free OS
Debian Developer                   to set it on, and I can move the world.
vorlon@debian.org                                   http://www.debian.org/


[0] http://bugs.debian.org/314385
[1] http://people.debian.org/~vorlon/dependencies.sxi
[2] http://people.debian.org/~vorlon/dependency-hell/
[3] http://people.debian.org/~keybuk/libtool-updating.html
[4] http://bugs.debian.org/340273
[5] http://bugs.debian.org/340163
[6] http://people.debian.org/~vorlon/freetype-without-builddeps.txt

Attachment: signature.asc
Description: Digital signature


Reply to: