Re: designing dh_pytorch for PyTorch reverse dependencies

["M. Zhou" <lumin@debian.org>]

On Mon, 2023-09-11 at 11:15 +0300, Andrius Merkys wrote:
> 
> > 
> > Then it is the dh_pytorch processing logic in pseudo code:
> > (dh_pytorch is inserted before dh_gencontrol)
> > 
> > for each binary package $pkg {
> > 
> >    if $pkg.architecture is all {
> >      # does not require specific variant
> >      append the following to python3:Depends
> >      "python3-torch | python3-torch-api-2.0"
> 
> Why not just python3-torch-api-2.0? AFAIU, python3-torch will provide
> python3-torch-api-2.0 as well. Or is this here to indicate
> preference?

If I remember correctly the first package in the Depends field
cannot be a virtual package, as per our policy. At least we do
not use the virtual package libblas.so.3 as the first dependency
in any BLAS/LAPACK reverse dependencies.

And using python3-torch as the first one will implicitly declare
preference over the "free" variant. Say, we have three packages
providing python3-torch-api-2.0:

  python3-torch
  python3-torch-cuda
  python3-torch-rocm

Then which one is preferred to fulfill the virtual api package?

> >    } else {
> >      # for arch-any package
> > 
> >      if $pkg.substvars contains "libtorch*" {
> >        # this variant-specific dependency
> >        do nothing to python3:Depends.
> >        Just use the default dependency.
> >      
> >      } else {
> >        # does not contain "libtorch*" dependency
> >        # that indivates variant-agnostic package.
> >        append the following to python3:Depends
> >        "python3-torch | python3-torch-api-2.0"
> 
> Am I right that src:pytorch-vision would fall in this category? In
> that 
> case python3-torch would fill the dependency requirement, but will
> the 
> binary package get the needed shared libraries in all cases?

That's a good question. src:pytorch-vision is exactly the special
cases in my mind. src:pytorch-vision depends on the libtorch2.0
ABI instead of the python API. So python3-torch-api-2.0 cannot fulfill
its requirement.

You can find from the pytorch official website (https://pytorch.org/)
that they always instruct users to install torch, torchvision,
torchaudio at the same time because the CPU-only build of torchvision
is not compatible with CUDA build of torch. The three package
has to follow the default dependency resolution from dpkg.

The core principal for deciding the API/ABI dependency over
pytorch is simple.

  1. as long as the reverse dependency writes some C/C++/... extension
     using libtorch-dev or pytorch C++ extension utility, it will need
     the ABI compatibility.

  2. as long as the reverse dependency do not contain compiled modules
     that is linked against libtorch2.0, it is agnostic to pytorch
     variant and the api virtual package should be sufficient.
     (even if in this case the package contains some cython modules
      not linked against libtorch2.0)

So, that's the reason why we need to analyse the output of
dpkg-shlibdeps. If there is dependency on libtorch2.0 or
libtorch-cuda-2.0, then it is seen as variant-specific and should
be kept unmodified.
>