Hi,
Through some previous discussions I realized that the issue of
threadding implementation of the BLAS can be sometimes cumbersome. For
example, sometimes we can observe severe performance regression from
pthread program + BLAS with gomp, or even observe severe calculation
error from gomp program + BLAS with iomp (aka llvm openmp).
These threadding disasters will be eventually propagated to our end
users, and may possibly harm their scientific computing experience.
This potential disaster has already occurred:
Installing R-4.0.0 on Debian and Ubuntu resulted in systems hanging
on simple matrix operations. This was discussed on the R-sig-debian
list. Quoting a message from Samuele Carcagno:
... after some further investigation I found out that the issue was related
to some calls to the `solve` function in `make.rgb`, and indeed running
`example(solve)` would hang my R session. After I changed the default
sudo update-alternatives --config libblas.so.3-x86_64-linux-gnu
the problem disappeared. The BLAS output of `sessionInfo` now that
BLAS: /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.9.0
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
while before when `solve` was not working it was:
BLAS: /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/liblapack.so.3
so it seems the issue is with the Ubuntu package `libopenblas0-pthread`
and has nothing to do with R. I don't know why openblas-pthread was the
default BLAS on my system.
Fedora uses libRblas private to R. The R documentation notes that many of the faster
implementations may give different results.
------------- Fedora
Recall what Fedora does for the BLAS libraries: they dont do any virtual
package at all. OpenBLAS packages with different threadding support are
given different sonames:
openblas + pthread: libopenblasp.so.*
openblas + openmp: libopenblaso.so.* (IIRC)
openblas + serial: libopenblass.so.* (IIRC)
Although this makes implementations not switchable at runtime, but at
least the users won't have to struggle with threads.
------------- Gentoo
I wrote the alternatives mechanism for gentoo's blas/lapack, which
resembles Debian's. However, gentoo's package management system supports
a "USE flag" feature which allows the user to set global threadding
implementation for the whole system.
This if fine for a system that is dedicated to a single workflow, but
may not be good for a desktop workstation used for a range of
calculations with different packages (including user's own
code) where most software can use a portable BLAS but one
calculation needs optimizations.
------------- what can we do
Maybe we can provide some more virtual shared objects such as
libblasp.so, which has candidates such as openblas-pthread and
blis-pthread?
In that case if a Debian maintainer intentionally choose to link against
a pthread BLAS (libblasp) to avoid issues of an openmp BLAS (libblaso),
we can help the users to avoid threadding issues when they don't read
any documentation (actually I think 99% of the users won't read the doc,
or have enough background to understand the threadding issue).
We can dig deeper into this direction if this turns to be a sensible
direction for making improvement.
The priority for Debian should be for widely used packages like
R to be consistent with other distros, which means R should
provide libRblas so the 95% of users will not encounter
problems using their distro's packages. Some users may
have use cases that need alternative BLAS, and will likely
end up building their own configuration. There are advantages
to keeping libraries consistent with other distros so we don't
end up with distro-specific build instructions for widely used
packages.
Acknowledgement
---------------
This is a part of my GSoC2020 project, although this topic is not on the
original plan.