Re: designing dh_pytorch for PyTorch reverse dependencies
Hi Mo,
On 2023-09-09 22:29, M. Zhou wrote:
I'm writing down my draft design for dh_pytorch here in order to hear
some comments or feedbacks. If you maintain a package that depends
on python3-torch or relevant pytorch packages, please let me have
your attention.
I maintain a pair of pytorch reverse dependencies.
We need a custom dh module for pytorch because we have multiple
pytorch variants in our repository, which might or might not
fill the reverse dependency requirements of its reverse dependencies.
PyTorch is somewhat similar to numpy, but we have to implement
something different than dh_numpy.
For instance, a reverse dependency might be agnostic to the actual
variant of pytorch. An example is src:pytorch-ignite. It provides
a higher-level abstract to the pytorch API, and contains no arch
specific file. So it can depend on any pytorch variant, e.g.,
Depends: python3-torch | python3-torch-cuda | python3-torch-...
A reverse dependency might be depending on a particular variant
as well. An example is src:pytorch-vision. It compiles binaries
against libtorch2.0 (with B-D: libtorch-dev). In that case,
the cpu variant (python3-torch) will not be able to fill the
dependency requirement.
My proposed solution for automatically filling in the pytorch
dependency is as follows:
Firstly, every pytorch variant will provide a virtual package
python3-torch-api-2.0
to declare that the package provides the high level API of pytorch.
Then it is the dh_pytorch processing logic in pseudo code:
(dh_pytorch is inserted before dh_gencontrol)
for each binary package $pkg {
if $pkg.architecture is all {
# does not require specific variant
append the following to python3:Depends
"python3-torch | python3-torch-api-2.0"
Why not just python3-torch-api-2.0? AFAIU, python3-torch will provide
python3-torch-api-2.0 as well. Or is this here to indicate preference?
} else {
# for arch-any package
if $pkg.substvars contains "libtorch*" {
# this variant-specific dependency
do nothing to python3:Depends.
Just use the default dependency.
} else {
# does not contain "libtorch*" dependency
# that indivates variant-agnostic package.
append the following to python3:Depends
"python3-torch | python3-torch-api-2.0"
Am I right that src:pytorch-vision would fall in this category? In that
case python3-torch would fill the dependency requirement, but will the
binary package get the needed shared libraries in all cases?
}
}
The $pkg.substvars files are generated by dpkg-shlibdeps.
As a result, the src:pytorch-ignite, a variant-agnostic reverse
dependency, will be able to depend on any variant that provides
python3-torch-api-2.0, including python3-torch, python3-torch-cuda,
python3-torch-rocm, etc.
Meanwhile, the src:pytorch-vision, a variant-specific reverse
depenedency, will simply require python3-torch-cuda.
(I will upload the cuda version of torchvision as
python3-torchvision-cuda in the future)
I don't have much experience in writing custom debhelper modules.
Me neither.
Any comments or suggestions are welcome.
The perl implementation is work in progress.
Thank you for working on this.
Best wishes,
Andrius
Reply to: