[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: -Wl,--as-needed considered possibly harmful

On Tue, 2007-12-25 at 20:12 -0800, Steve Langasek wrote:
> On Fri, Dec 21, 2007 at 10:55:32AM +0000, Neil Williams wrote:
> > On Thu, 20 Dec 2007 21:44:19 -0800
> > Steve Langasek <vorlon@debian.org> wrote:

> > > But *nothing* here guarantees that the version of libbar the application is
> > > linked against has the same ABI as the one libfoo itself linked against!
> > ... until both the application and libfoo are rebuilt. So the issue
> > here is triggering rebuilds of reverse dependencies of libbar?
> No.  That doesn't cause previously released binaries to blink out of
> existence.
> What's required is to ensure that there's a package rename each time
> libfoo's ABI changes, including when the libfoo ABI change is caused by type
> changes in the underlying lib.  This is essentially the general case of the
> c102, c2, c2a, and ldbl transitions that we've been through for compiler
> changes over the past few years; in theory we should be able to accomplish
> these transitions for libraries with fewer reverse-dependencies than
> libstdc++ with significantly less pain, as long as library maintainers are
> forewarned.
> But once the package name change is handled, the rebuild of reverse-deps
> follows naturally.

That sounds a lot better than the original idea of incorporating the
entire dependency chain into the package name. 

I think I'm doing this already - if libfoo1 implements and exports types
from libbar2 and libbar2 moves to libbar3, I would expect to have to
port libfoo to libbar3 and this would usually cause a SONAME bump in

So could I ask, from an upstream perspective, what kind of changes in
the underlying lib might *not* cause such a port and therefore end up
with libfoo1 still being buildable against libbar3 yet *still* require a
SONAME bump to accommodate the transition?

Have I got these possible scenarios correct:
1. libfoo depends on libbar2 without exporting libbar symbols. libfoo
can migrate to libbar3 with internal code changes and if these do not
require changing any libfoo interfaces, libfoo needs no SONAME bump.
pkgconfig does not export libbar in the libfoo --libs data. 

If an application uses libbar and libfoo (for whatever reason), the
ported version of that application that uses libbar3 simply needs to
depend on the version of libfoo that depends on libbar3. In this case,
an automated rebuild of the application could miss this step, causing
breakage, yet forcing a change in the libfoo package name seems
unnecessary (especially if there are other applications that depend on
libfoo but not directly on libbar). Another problem is if the
application ports to libbar3 before libfoo - currently, there is no way
of reliably picking this up except the application maintainer being
particularly careful with the pbuilder logs and noticing that libbar2 is
still being installed during the build as well as libbar3 due to the
indirect dependency on libfoo. Maybe pbuilder could highlight such
incidences? (or even fail the build if it happens)? Maybe sbuild too?
However, this would again raise the spectre of being unable to fix
critical bugs in the application should libfoo be slow to port to
libbar3. Again, it's a balance of risk - problems are *possible* when
the app is linked against libbar3 and indirectly against libbar2 but in
the absence of real bugs, is it worth causing more delays to other bug

Does libfoo need to use versioned symbols to cope with this or is it
simply down to those applications that use libfoo and libbar to specify
the right version of libfoo once the application itself has ported to
libbar3? (And wait until libfoo has also migrated?) What methods are
needed to ensure that the application does this? (e.g. if left without
changes, an application directly dependent upon libbar3 and libfoo could
be built for unstable and presumably migrate into testing before the
updated libfoo).
There needs to be some check on the application that compares the
dependency chain as well as the dependency list in the package itself.
lintian cannot do that.

2. libfoo depends on libbar2 and exports some libbar symbols in
libfoo-dev. libfoo can migrate to libbar3 with only internal code
changes because it only exports a portion of the libbar API that has not
actually been changed in the libbar2->libbar3 transition. libfoo needs
to export libbar in the libfoo --libs data. Is a SONAME bump needed in
libfoo even though *none* of the libfoo interfaces have changed? As
upstream for libfoo, I would not normally make a SONAME bump in this
situation. Would it be better to rename the Debian package without
bumping the SONAME, e.g. libfoo1-0->libfoo1-1 ? Wouldn't this cause
unnecessary rebuilds of applications that only depend on libfoo and not

3. libfoo depends on libbar2 and needs to make changes to its own
interfaces to cope with the libbar transition or needs to export symbols
from the new libbar API and therefore makes a SONAME bump. No problem
here except ensuring that applications transition to both API's at the
same time. Can be enforced by making libbar-dev conflict with the old
libfoo-dev or making the new libfoo-dev depend on the new libbar-dev

It is only in case 2: that I find confusion and a need for some kind of
Debian-only package name change in libfoo. Is that correct?

Now it may be that for case 2: to happen in real situations, libbar
would actually turn out to be a collection of libraries: libbar-x11.so
and libbar-sql.so. Say the libbar2->3 transition only affects libbar-x11
and it is libbar-x11 that is NEEDED by the application, not libbar-sql:

Old package: libbar-x11.so.2.1.3 libbar-sql.so.2.4.1
New package: libbar-x11.so.3.0.0 libbar-sql.so.2.5.0

A package split in libbar would actually seem to be the better solution,

Have the libbar source build an extra set of packages:
libbar-x113, libbar-x11-dev, libbar-x11-dbg, libbar-doc
libbar-sql2, libbar-sql-dev, libbar-sql-dbg
libbar-dev as a meta package depending on libbar-sql-dev and
pkgconfig data for libbar-x11 in bar-x11.pc
pkgconfig data for libbar-sql in bar-sql.pc
Remove bar.pc or fold bar-x11.pc and bar-sql.pc into a new bar.pc to
support those applications that use both. Get the whole thing done in
one spell in NEW.

To cope with this transition, libfoo could simply swap the pkg-config
call from:
pkg-config --libs bar
pkg-config --libs bar-sql

No SONAME bump, just bump the version of the libbar-dev dependency (or
preferably replace libbar-dev with libbar-sql-dev).

Would this be sufficiently beneficial that the libbar maintainer would
be considered remiss for not implementing such a split?

> > > If
> > > libfoo linked against libbar1, the pkg-config approach only ensures that
> > > when your application is built against the libbar2 version of libbar-dev,
> > > the resulting binary will depend on both libbar1 and libbar2 (despite not
> > > using any symbols from libbar2).  All this does is increase the chances of
> > > segfaults or bad runtime behavior as a result of symbol clobbering.
> > Maybe I've got this wrong but there would appear to be a few methods to
> > fix this:
> > 1. Incorporating the libbar SONAME into the libfoo package SONAME - as
> > Simon Richter recommended - this could quickly end up with
> > libany1-foo2-bar2-baz0-base4-pango0-... etc and I'm not convinced that
> > this would actually help anyone if, for example, libbaz0 migrates to
> > libbaz1 in a way that completely breaks libbar2.
> Then either libbar2 becomes libbar3 at the same time, or libbar becomes
> unsupported (and the stack above it has to be ported or else it's not
> releasable), or libbaz needs to support symbol versioning to permit
> coexistence of the two versions in the same address space.

Even if the actual fix for libbar2 is only internal? Say a change in
libbaz meant that the core functionality of libbar fails. As "core
functionality", it would be used by libbar but not exported so libbar
would not export libbaz symbols; there is no need for libbar2 to become

Yet the problem isn't in libbaz either, so why force libbaz to implement
versioned symbols through (apparently) no fault of its own?

To me, this whole problem comes down to the applications, not the shared
libraries or pkgconfig. Problems only arise if one package is linked
directly and indirectly against different versions of the same object.
The problem should be solved at the point at which it arises - in
whichever package ends up with the double linkage. Changing all the
shared libraries names and causing lots more transitions with
unnecessary SONAME bumps seems the wrong approach, IMHO, because we end
up with more transitions, not easier transitions.

Lintian cannot check beyond the confines of the .changes file for one
source package so we need a supplementary check and one place for this
would appear to be all those scripts that build in a chroot.

What about making the version of libbar-dev that depends on libbar3,
conflict with libbar2? The application maintainer would soon discover
the problem because the application dependency on libfoo could not be
satisfied in a pbuilder environment until libfoo updates to libbar3
itself. This gives time for the application maintainer to consider an
interim release of the current (unported) version of the application
dependent on libfoo and libbar2, if it is possible to fix certain bugs
that way, and seek an upgrade of libfoo in the meantime. i.e. leave
pkgconfig out of it entirely.

This has the advantage that none of the applications that only depend on
libfoo (and libbar indirectly) would be affected - it is limited to
those who need both libfoo-dev (which brings in libfoo and libbar2) and
libbar-dev (which brings in libbar3 and complains about libbar2). It can
also be implemented entirely within the one library that is undergoing
the transition: libbar.

It also allows applications that still use libbar2 to coexist with
libbar3 - only when trying to build against libbar3 does it become a

The biggest problem with that approach is upstream. It complicates
porting the application itself - depending on how many other packages
would be removed by apt when trying to install libbar-dev on a real
system. There would be no point making libgtk+-3.0-dev conflict against

However, I think this is manageable for small transitions and less
common libraries - the very ones that would tend to get missed by the
current methods. Everyone knows when a gtk transition is taking place,
everyone is watching for situations where a package gets linked against
libgtk+2.0.so and libgtk+3.0.so and there is little chance that any
program would work with such duplication.

Maybe a check that can be confined to chroots is needed - a function
within pbuilder that checks the build dependencies being requested and
fails if libbar2 and libbar3 are being installed at the same time during
a build? (The mere fact both are needed in a pbuilder build should be
evidence enough that at least one object in the dependency list is going
to end up linked to both.)

It would seem relatively simple to implement: sort the fully parsed
dependency list before calling 'apt-get' to remove duplicates, strip the
SONAME from the list of dependencies, error if any of the remaining
strings are not unique. 

Something like (although probably not needing tmpfiles):
#!/bin/sh -e

LIST="libfoo1 libbar2 libbar3 myprog libfoo1"
rm -f tmpfile
for PKG in $LIST; do
	echo $PKG >> tmpfile
ULIST=`sort -u < tmpfile`
for PKG in $ULIST; do
	PKG=`echo "$PKG" | sed -e 's/[0-9]$//'`
	echo $PKG >> tmpfile2
ORIG=`grep -c '[a-z]' tmpfile`
SORT=`grep -c '[a-z]' tmpfile2`
if [ $ORIG -ne $SORT ]; then
	echo -n "FAIL: "
	echo $ULIST | sed -e 's/\n//'
	exit 1
echo "OK"

FAIL: libbar2 libbar3 libfoo1 myprog

I'm sure that can be improved to only show the problematic packages.


Neil Williams

Attachment: signature.asc
Description: This is a digitally signed message part

Reply to: