[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: abi-compliance-checker and library transitions [Re: another attempt at Y2038]



On 2023-02-04 21:42 -0800, Steve Langasek wrote:
> On Thu, Oct 20, 2022 at 09:42:58PM -0700, Steve Langasek wrote:
> > This is good.  One thing that I think has been missing from the discussion
> > about armhf rebootstrap is the fact that we do have experience in Debian
> > doing cross-cutting ABI transitions without having to change an
> > architecture name.  We've done this at least three times in Debian history:
> > the libc5->libc6 transition ('g' suffix - which still remains today in
> > libpam0g!), the GCC 4.0 C++ ABI transition ('c2' suffix), and the 'long
> > double' transition ('ldbl' suffix, and an example of doing this for an ABI
> > transition that didn't affect all architectures).

I feel like I should already know this, but after a bit of thought,
and reading the transitions wiki page:
https://wiki.debian.org/Teams/ReleaseTeam/Transitions I do not
understand how the suffixes are used in these transitions.

That wiki page does not appear to mention adding suffixes to package
names. It appears to only document the process where the version is
just bumped, everything involved is rebuilt in experimental and then
re-uploaded to unstable once we are ready to make the change.

I presume that the suffix is used for transitions involving a large
number of packages, in order to keep both the old-ABI and new-ABI
version of the packages around in parallel (and conflicting with each
other?) for a while which is necessary for larger transitions because
doing it all in one go would take too long, or get entangled with
uploads from the unware that might cause breakage, or something?

Can someone point at an explanation of the process, or add it to the above page?

Just rebuilding and not renaming sounds like a much smoother process,
because as soon as we rename then there is a load of renaming in
reverse dependencies too, and there is presumably a set of rules about
when conflicts are added and other details that need to be got right.

I presume there is a good reason for this distinction between 'minor'
transitions and 'mega' transitions. I'd like to understand it properly.

> > Nowadays, I wonder if abi-compliance-checker would be usable for analyzing
> > the ABIs of C libraries to accurately identify what needs to change for
> > time64_t.  I think it would be interesting to know from this how many shared
> > libraries expose time_t size in their ABIs.
> 
> Well, I didn't see anybody else expressing interest in this, so I had a go.
> 
> I've done a first-pass analysis of Ubuntu lunar/armhf using
> abi-compliance-checker.  The results of my investigation, including scripts
> and detailed reports/logs, are here:

Thanks very much for this. Really helpful. I tried to use abi-dumper
and abi-compliance-checker but wasn't having much success in getting
useful reports out of them. I found abigail-tools to be quite a lot
simpler to use. Probably we should run both and investigat if they
disagree about differences.

I found out about your effort just in time to include it in my FOSDEM talk, which was good.

https://fosdem.org/2023/schedule/event/fixing_2038/ for anyone who is
unaware. (The Q&A has some dropouts and didn't add all that much
useful info so you can stop at about 17:20 without missing much).

>   - abi-compliance-checker was used to analyze headers of 767 library
>     packages in the archive.  Of those, 82 of 558 were successfully analyzed
>     and identified as library packages that would need patched for the
>     switch to 64-bit time_t to mark them as ABI-incompatible with the
>     previous version on armhf.
> 
>   - an additional 209 packages could not be analyzed, because a-c-c failed
>     to compile the headers using its invocation of gcc.  Assuming a roughly
>     equal distribution of ABI-changing vs non-ABI-changing libraries among
>     these that have failed to compile, we'd be looking at around 113 of 767
>     libraries that need changed.
> 
>   - if we decide the level of effort to patch this many libraries is
>     worthwhile, to get an actual actionable list of libraries that need
>     patched for this transition would require getting the compile failures
>     down to zero.  I did dig into the failures alphabetically from "a" to
>     "libe"; most of these failures were resolved by one of three methods.
>     - letting a-c-c know that this is a C library and to not try to compile
>       it with gcc's C++ frontend;
>     - excluding various headers that can't be included directly, either
>       because they are internal headers that are included via other headers
>       in the package, or because they have dependencies on other headers not
>       in the archive and can't be compiled (and therefore can't affect the
>       library ABI);
>     - pulling in additional undeclared package dependencies.
>     Even when applying these fixes, I still had 8 library packages whose
>     headers I wasn't able to get to compile.  So there's still quite a bit
>     of work to do here.
> 
>   - On this preliminary pass, I only included library packages that shipped
>     both headers and .so's, as this supposedly gives the best quality
>     results from a-c-c.  To get a full list of ABI-breaking changes,
>     however, we need to include packages that ship only headers and not
>     .so's, to catch those libraries for which the .so is in a different
>     binary package (boost), and to catch plugin systems for executables
>     (apache).
> 
> While I'm currently working on this using Ubuntu, the results are largely
> applicable to Debian, and the script itself certainly is.  Note however that
> there are a couple of bugs in abi-compliance-checker that I've patched in
> Ubuntu, bug #951076 and bug #1030540, which you would want to have fixed
> before trying to run this analysis in Debian.  Cc:ing the a-c-c maintainer
> for awareness of this.

I tried just running the script on debian and it tried 121 libraries
before dying, and produced compatibility results for 71. So yes it
needs some work to get useful numbers out.

You just ran this on Ubuntu main, so there will be quite a lot more
libraries in debian, I presume?

> Does doing an ABI transition of ~113 libraries seem tractable to folks?

It certainly provides evidence for the idea that this is not
completely intractable, which I think many people (including me)
worried was the case initially.

> If
> there's interest in moving forward on this, I'd say the next step should be
> to take it to debian-devel for broader discussion/signoff.  I would also
> move my scripts into a git repository that folks can contribute to -- adding
> the necessary overrides for a-c-c for each library, so we can get an
> authoritative list of ABI breakages, is very parallelizable.

Yes I think we should proceed with this analysis on debian to get a
better handle on just how many libraries we are looking at.

I would also still like suggestions/test cases for $stuff that we
thing will/might break _other_ than library ABI transitions, which we
at least know how to handle in general. So far there have not been
many things suggested, my list of tests I can run to check has zero
entries. That may be good sign (there just aren't that many), or it
might be a bad sign (no-one knows/is checking).

I have created a releasegoal page to discuss this transition in
general, and collect relevant info:
https://wiki.debian.org/ReleaseGoals/64bit-time

I think it would be useful do a bit more work on the ABI checking and
data-collecting (e.g. raspian input) before going to debian-devel, but
not wait more than another week or two, because people are already
switching, possibly without fully understanding the implications:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1026204


Wookey
-- 
Principal hats:  Debian, Wookware, ARM
http://wookware.org/

Attachment: signature.asc
Description: PGP signature


Reply to: