Re: Python C-library import paths
On Fri, 2022-04-01 at 02:32 +0100, Wookey wrote:
>
> RuntimeError: Cannot find the files.
> List of candidates:
> /home/wookey/bin/libtvm.so
> /usr/local/bin/libtvm.so
> /usr/bin/libtvm.so
> /bin/libtvm.so
> /usr/local/games/libtvm.so
> /usr/games/libtvm.so
> /usr/lib/python3/dist-packages/tvm/libtvm.so
> /usr/lib/libtvm.so
> /home/wookey/bin/libtvm_runtime.so
> /usr/local/bin/libtvm_runtime.so
> /usr/bin/libtvm_runtime.so
> /bin/libtvm_runtime.so
> /usr/local/games/libtvm_runtime.so
> /usr/games/libtvm_runtime.so
> /usr/lib/python3/dist-packages/tvm/libtvm_runtime.so
> /usr/lib/libtvm_runtime.so
>
> So it tries quite hard to find it, but doesn't know about multiarch
> and thus fails to look in the right place:
> /usr/lib/<triplet>/ (/usr/lib/x86_64-linux-gnu/ on this box)
dlopen should know the multiarch triplet on debian. They have written
their own ffi loader, so I think it is an upstream bug. The upstream
should detect and add multiarch directory to the paths.
>
> I see that /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py
> contains
> a function 'get_dll_directories' which looks promising and adds
> TVM_LIBRARY_PATH to the search list and if I run tvmc like this:
> TVM_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu/ tvmc
> then that path is at the top of the list.
User specified stuff goes first. That makes sense.
> OK, but that mostly reveals a second issue: it's looking for
> libtvm.so, but that unversioned link is only provoded in the dev
> package
> libtvm-dev. The library package has the versioned filenames
> /usr/lib/x86_64-linux-gnu/libtvm.so.0
> /usr/lib/x86_64-linux-gnu/libtvm_runtime.so.0
I think it is fine to let it dlopen the libtvm.so, as it says
itself as some sort of "compiler".
Take pytorch as example, python3-torch has some functionalities
for extending itself with C++. As a result, libtorch-dev is
a dependency of python3-torch.
> So I also have to persuade it to look for libtvm.so.0 not
> libtvm.so. Where does that info live? OK, a bit more research shows
> that that is in /usr/lib/python3/dist-packages/tvm/_ffi/libinfo.py
> which is in the source as python/tvm/_ffi_libinfo.py, in
> find_lib_path
> and that's easy to fix, and probably even the right place to fix it?
If the upstream's own ffi loader does not look at the SOVERSION,
then it should be designed to work without it.
> The paths is harder though. get_dll_directories in
> python/tvm/_ffi_libinfo.py adds $PATH after $LD_LIBRARY_PATH to make
> it's search list. Is searching $PATH for libraries ever right?
>
> What it should actually be adding is what's in /etc/ld.so.conf.d/*
> That can be listed with
> /sbin/ldconfig -v 2>/dev/null | grep -v ^$'\t' | cut -d: -f1
> (yuk? is there really no better way?)
>
> How does one do that in python to get that set of path added in the
> libinfo.py function?
> https://github.com/apache/tvm/blob/main/python/tvm/_ffi/libinfo.py
>
> Am I barking up the right tree here or is there a better way?
>
This really looked like an ffi loader bug.
Reply to: