Re: Symbols/shlibs files for Java
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
On 2011-05-25 15:58, Jonathan Nieder wrote:
> (+cc: debian-java; please cc me on replies if following up there)
:) (assuming you are on debian-dpkg@l.d.o)
> Hi Niels,
>
> Niels Thykier wrote:
>
>> we can still gain a lot
>> from automated ABI/API breakage detection and well defined transitions[1].
>
> This means allowing Java packages to use ${shlibs:Depends} in the same
> way packages with binaries do? Sounds excellent.
>
I think we would use ${java:Depends} (currently used by javahelper
tools), but yes. :)
>> [1] Such being able to tell if reverse dependencies have been rebuilt
>> against the new version of a library or not.
>
> I had thought that Java doesn't require this in the way that DSOs do
> (but I could easily be wrong; see below [*]). According to my
> assumptions, either there is an API break or no ABI break at all; in
> the former case, this means serious (FTBFS) bugs, while in the latter
> case it is nice to rebuild the reverse dependencies in case they run a
> test suite to catch problems but it does not need to affect
> transitions to testing.
>
Turns out I cannot think of an example of an ABI breakage in Java that
is not also an API breakage (in some sense or another). Maybe someone
else on list can think of one (where symbols/shlibs tracking would be
helpful).
Maybe narrowing the return type of a primitive? Or generalizing an
object type paramenter?
> Example: Suppose I rename some important function or method. This
> would be analogous to renaming a function in C, which warrants a
> soname bump. So in C, there would be two versions of the DSO, one
> from before and one after:
>
> libexample.so.1 (in an old libexample1 binary package)
> libexample.so.2 (in a new libexample2 binary package)
>
> Before the updated libexample can transition to testing, we need all
> packages that used to link to the former to start linking to the
> latter, even if they never use the renamed function.
>
> Now on the other hand suppose I decide to add a new function. This
> is also a change in ABI, but it is backward-compatible, so it does not
> warrant a soname bump. The deb-symbols infrastructure means we can
> automatically add versioned dependencies based on functions an
> application uses (which is great[**]), but it doesn't enter into
> transition tracking much --- there is no need to rebuild packages
> before the new version of the library enters testing.
>
> C++ works the same way because constructors, destructors, overloaded
> functions, and so on are implemented as functions with mangled names
> behind the scenes.
>
> In Java, though, I had thought that there was nothing like a soname
> bump when a function gets removed.
>
There is not, but we are trying to make something like it. Basically we
would include a "fake" ABI version in the jar file and rewrite the
classpath to use the ABI version (as I recall at least).
So something like:
foo.jar -> foo2.jar (development symlink)
foo2.jar (real jar)
If foo2.jar breaks backwards compatibility, we would rename it to
foo3.jar (rename its package to make foo2.jar and foo3.jar to be
co-installable) and migrate reverse-dependencies to foo3.jar.
Classpath-wise a reverse-dependency would be built using "foo.jar" and a
helper tool would then re-write it to fooX.jar during build
(post-compilation) to reflect which "ABI" version of foo it was built
against. (So either foo2.jar or foo3.jar in this example)
I vaguely seem to remember that part of the plan here was to try to get
"binNMU" for arch:all packages as well. But I do not recall the details
here very well, so I hope someone can refresh my memory (Matthew?).
> [*] I mentioned above that I am probably missing something. :) It
> might be possible to achieve something like the
> libexample1/libexample2 case above by installing two versions of a
> Java package, setting the classpath appropriately at run-time. If
> there is something similar in common use, that's probably the big
> piece I was confused about.
>
> [**] This is the part that seems exciting to me for Java.
>
Certainly, this is definitely useful as well; currently we do it on a
best effort basis, but automatic tends to do a better job here in my
experience with symbols files in C.
>> As it is now, we manually create a libXY-java package if we suspect
>> that the new version of libX-java is not backwards compatible.
>
> Could you give an example? Does this also change the API or only ABI?
>
In these cases both packages will be co-installable and the jars will be
installed side-by-side with manual migration of reverse-dependencies.
So nothing will be broken by default, but it leaves old packages in the
archive such as libservlet2.3-java (recently removed) and
libservet2.4-java (RM TODO).
Other examples I can think of include libasmX-java and libicu4{,.2}-java.
>> First of all I was hoping that you might have some "Do" or "Don't"
>> pointers from when dpkg added support for these things. Secondly, there
>> might be some code or infrastructure that could be shared.
>> Particularly I am interested in how you handle mapping
>> filenames/SONAMES to a package (especially in cases like libc6, where
>> there more than one lib in the package).
>
> See deb-shlibs(5) and deb-symbols(5). (I know you already have. Just
> listing references for future readers.)
>
Been a while since I read them though. :)
>> We also have cases where two packages provide the same library and it
>> would be optimal for us to end up with libX-java | libY-java in the
>> depends, but I have a feeling that is not entirely trivial to support
>> (in a sane way).
>
> The symbols file supports this. It should be possible as long as
> the build-deps of packages using these libraries have enough
> information to figure it out (so, it can't be "libX-java and libY-java
> install files to the same directory and javahelper figures it out").
>
Okay; I think we have a few spicy cases such as openjdk-6, sun-java6 and
some library package provide some "extra" java API and gcj does not.
Would that imply that openjdk-6, sun-java6 and said library were all
installed during build (or that the symbol file of openjdk-6 "knows"
that its symbols are also available in sun-java6)?
>> I am not too sure that we can recycle the existing formats
>> (maybe the shlibs format with s/SONAME/filename/) as we have to check
>> for things like classes, return-types, inheritance and method
>> overloading as well.
>
> I think checking for ABI breakage is a different and harder problem.
> The symbols infrastructure can notice if a symbol was added or
> removed, but in C and C++ that is not enough to know if something
> (like the return type, as you mention) changed.
>
True, but I believe the symbol infrastructure catches a fair share of
(obvious) broken backwards compatibility. If we can create a format and
a checker that can check slightly less obvious breakages, then all the
better.
> Have you talked to the authors of the "ABI compliance checker" at
> <http://ispras.linux-foundation.org/index.php/ABI_compliance_checker>?
>
No, but thanks for the reminder.
> Thanks and good luck,
> Jonathan
>
>
~Niels
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iQIcBAEBCAAGBQJN3hVYAAoJEAVLu599gGRCaCAP/ibcy459MuLxCxM5bIkwKb9x
CBgjFUbmQFKWAs+kSOf0oi1ILwljJvUrl4ZpLjFdH5+SxuO0uTphhVTjTj5WZe2K
im/wshU/aoZxCRFbgH1I/jqS1O9Z4XrNZ9e0dLyFzzD6AcpEZ3DmlQb+l+jo5Pj3
3JUrMJeh44dI5UemV4tGDdH9Og/kMYDD9jY0YxlraIG5bY97oyBgFlk2cphMkiXc
1QPvH1u/x6j4mmegyRjhTAtj0JojAmVXw8ZZtmvOFitc9VEmIGIMSiFzOKpu92Tp
G7cZ6d3Vv3l/7IDNv18NBd76Vn0IaYBE1D7kFz/By79vwhPTmFvpQIVj6k7ZIE+j
ZtumHmT8hDBY2i9q2MR0XZzNwMIfm6uAME5AhqL0ZIk9g0p983PtgQmx9WtrSbRZ
y5Pu7nIWKopKwHEr/IAza1OhadSv50Z2w5mNtUeYN5Bpa5bJrFIOFAJkBnfxu75I
FuxdOWFmOJ7pzNAfpme4tDlJgxvnRNb1MaEjzlqMJd+R7ZjzDZ9yt5LTt+m+921T
RHDLnW8iPpNHFcTJ7g0YtRIHVE3JqEdhrXZSX0JOJiTfg9y4ocJ3s/K6PWijQLgY
h7Vys7xI43Yl2pTBd9mTlaqiaNGtuwW0eBEZPOG2nEslPafdbAwldeuSBYLmLD2C
LPggAt+FCUtwJmjhRWEo
=2unk
-----END PGP SIGNATURE-----
Reply to: