Re: Handling s390 libc ABI change in Debian
On Mon, Jul 14, 2014 at 11:14:42PM -0400, Carlos O'Donell wrote:
> On Mon, Jul 14, 2014 at 4:36 PM, Aurelien Jarno <aurel32@debian.org> wrote:
> > glibc 2.19 has changed the libc ABI on s390, more specifically the
> > setjmp/longjmp functions [1] [2]. Symbol versioning is used to handle
> > some cases, but it doesn't work when a jmp_buf variable is embedded
> > into a structure, as it changes the size of the structure. The result
> > is that mixing programs or libraries built with 2.18 with ones built
> > with 2.19 do not work anymore, usually they end up with a segmentation
> > fault. Some persons from this list have experienced that with perl.
>
> That is not true. This is an over generalization of the problem. You
> can use libraries built with 2.18 and 2.19 and they work just fine.
I agree I probably a bit over exaggerated here, but the problem is real,
breakages do happen, and some persons on this mailing list have already
experienced them.
> The extent of the problem in correct language is listed here:
> https://sourceware.org/glibc/wiki/Release/2.19#Packaging_Changes
This seems to minimize the problem, listing only perl. In practice we
have seen much more breakages, part of them being due to the change of
the __pthread_unwind_buf_t struct.
> > We first thought it was limited to a few packages (even if all perl is
> > already more than that), but as time goes more and more issues are
> > found. libpng and gauche are also affected, the issue with mono is
> > also likely due to this ABI change.
>
> That is new information, and it is important for distributions to
> relay this information back upstream where the decision for a SO bump
> can be made.
I can follow up with a list affected packages, but we are slowly
discovering them one by one, so it might takes time. So far we have:
* Mixing modules/libraries built with pre-2.19 and 2.19 libc
- perl
- libpng
* Using libc 2.19 without rebuilding anything:
- gauche
- mono
> > According to upstream [3], the problem is that Debian doesn't do a mass
> > rebuild, which is the strategy chosen by Red Hat to handle^Wworkaround
> > this issue. This means some programs might segfault during the upgrade,
> > or on partially upgraded systems.
>
> I apologize if you took what I wrote to mean that. I did not mean it
> was Debian's problem, but rather that Debian suffered the most because
> they don't do rebuilds. The two are orthogonal. You face a situation
> that is unique to the framework used to build the distribution.
>
> Please engage upstream to champion a SO name bump for libc for
I think that would be the correct solution. That said as it is not
something trivial and thus not done often, it's an opportunity to push
for more ABI changes if some others are envisaged in the future.
> > Now we have to chose a strategy for Debian. I see multiple options:
> >
> > 1) Ignore the issue and just rebuild (binNMU) the packages that seems
> > affected when we discover them. This means partial upgrades will likely
> > be broken, and that we might discover some broken packages only after
> > the jessie release.
> >
> > 2) Rebuild (binNMU) all packages. This means partial upgrades will
> > likely be broken.
> >
> > 3) Bump the soname of affected packages and rebuild their reverse
> > dependencies. It is the solution that is currently being implemented for
> > perl. It clearly won't scale if more broken packages (and even for
> > libpng) are discovered as it requires a source upload and a transition
> > handled by the release team. It also means breaking the ABI compatibility
> > with other distributions.
> >
> > 4) Bump the libc soname to libc.so.6.1 and do a libc transition. This is
> > probably what upstream should have done instead of breaking the ABI.
> > This is a huge work though, and this also means breaking the ABI
> > compatibility with other distributions.
> >
> > 5) Revert the ABI change. This is likely just postponing the problem as
> > the change is required to support future hardware. This also means
> > breaking the ABI compatibility with other distributions.
> >
> > 6) simply drop the s390x port and tell users to either use an other
> > distribution or use Debian on other hardware.
> >
> > Any opinion? Any other ideas how to handle that?
>
> Option (6) is the nuclear option, and clearly a little excessive given
> the situation. If user's install from an installer they get a
> perfectly working system. Punishing those users because partial
> upgrades don't work seems excessive.
If we are not able to solve this problem by ensuring all the packages
in the next release are using a consistent ABI, I think it is an option
to consider. It's probably better being honest with the users telling
them "we fail" than shipping a half broken release. Consider that we
have limited human resources to maintain this port.
> Option (5) postpones the problem until newer s390 hardware arrives.
>
> Option (4) is likely what upstream should have done. Why do you think
> it's a huge amount of work? If all s390-supporting distributions agree
> then we just change the SO name?
It's a huge work for Debian, maybe not for other distribution, as it
basically means we have to rebootstrap everything. This includes manual
bootstrapping of self-dependent languages (haskell, gnat, ...) and
manual handling of some dependencies loop. In addition it's something
which hasn't been done since the libc5 transition, so we might discover
some unexpected issues.
> Option (3), (2) or (1) all look like reasonable "fire and forget"
> options that require users of s390 to simply move forward with their
> installs to full new versions of Debian. It's an option for RHEL
> because we don't allow partial-upgrades and we are working with users
> to notify them of the breakage during a major RHEL upgrade e.g. RHEL7
> to RHEL8.
Does it mean on RHEL, a full reinstall would be needed? A system being
upgraded from one version to another will transition though partial
ugprades, and we have to ensure that at least the upgrade won't break.
> In my opinion it's not too late to do option (4).
Agreed.
Cheers,
Aurelien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aurelien@aurel32.net http://www.aurel32.net
Reply to: