Re: ImportError: Failed to load PyTorch C extensions

[Mo Zhou <lumin@debian.org>]

There are two factors that makes it difficult to provide support for 
multiple python versions simultaneously. The pytorch upstream sometimes 
slightly lag behind the latest CPython version due to their fixed 
release schedule. BTW, the pytorch build is time consuming, especially 
pytorch-cuda.

With that being said, I'll try to see if there is a good way to avoid 
compiling things twice. Patches are welcome.


On 12/6/23 08:11, Andrius Merkys wrote:
> Hi,
>
> On 2023-12-06 08:58, Andrius Merkys wrote:
> > On 2023-12-05 15:35, Andreas Tille wrote:
> > > when I pushed python-array-api-compat to new two weeks ago this has
> > > built nicely on Salsa CI[1] but now it fails[2] when trying to load
> > > pytorch:
> > >
> > > =========================== short test summary info 
> > > ============================
> > > FAILED 
> > > tests/test_array_namespace.py::test_array_namespace[None-torch] - 
> > > ImportError: Failed to load PyTorch C extensions:
> > >      It appears that PyTorch has loaded the `torch/_C` folder
> > >      of the PyTorch repository rather than the C extensions which
> > >      are expected in the `torch._C` namespace. This can occur when
> > >      using the `install` workflow. e.g.
> > >          $ python setup.py install && python -c "import torch"
> > >      This error can generally be solved using the `develop` workflow
> > >          $ python setup.py develop && python -c "import torch"  # 
> > > This should succeed
> > >      or by running Python from a different directory.
> > > FAILED 
> > > tests/test_array_namespace.py::test_array_namespace[2021.12-torch] - 
> > > ImportError: Failed to load PyTorch C extensions:
> > > ...
> > >
> > > Any idea what might be wrong here?
> >
> > I ran into the same yesterday when building pytorch-geometric 
> > (arch:all). My wild guess is that pytorch needs a binNMU to include 
> > support for Python 3.12.
>
> Simply rebuilding pytorch does not help, alas. Pytorch's source builds 
> for the default Python version only. Thus the appropriate fix would be 
> to build python-array-api-compat for the default Python version only 
> as well. To do so, replace python3-all dependency with just python3.
>
> > > [1] 
> > > https://salsa.debian.org/python-team/packages/python-array-api-compat/-/pipelines/602473
> > > [2] 
> > > https://salsa.debian.org/python-team/packages/python-array-api-compat/-/jobs/5003765
>
> Hope this helps,
> Andrius