Re: SIMDebian: Debian Partial Fork with Radical ISA Baseline
Hi Lumin,
thanks for all your work to make Debian a performant platform for
scientific computing.
On Sat, Feb 09, 2019 at 05:24:52AM +0000, Mo Zhou wrote:
> On Sat, Feb 09, 2019 at 01:07:47PM +1100, Drew Parsons wrote:
> > On 2019-02-09 03:25, Mo Zhou wrote:
> >
> > I think it would be more constructive to provide arch-specific packages for
> > eigen/blas etc on amd64 which Conflict/Replace/Provide the standard
> > packages.
> >
> > That way a local administrator can choose to install them if they help
> > improve performance.
>
> Theoretically graceful, but this actually makes packaging much harder
> and more complex, in order to avoid SIGILL. However, in a partial fork
> we could directly assume a haswell baseline and provide rebuilt packages
> from identical source. This way looks unconstructive but it's economic.
>
> https://tracker.debian.org/pkg/isa-support was available since Aug 2017,
> but the highest ISA baseline (SSE4.2) is still too conservative to make
> actual sense.
>
> > Hacking dpkg itself for this purpose and forking an entire linux
> > distribution is way too invasive and distracts from solving the actual
> > problem.
>
> I must emphasize that SIMDebian only cares about packages that would
> definitely benefit from bumped ISA baseline, and hence I call it a
> **partial** fork. The modified dpkg is not used for rebuilding the
> whole archive by brute force, but for (automatically) picking
> low-hanging fruits. If package benefits a lot from bumped ISA baseline,
> the modified dpkg helps you quickly rebuild it without any code change.
I think I've got your idea but what about a different approach. If we
think a bit "Gentoo-ish": What about having some kind of simple
mechanism to re-build those package that are profiting from more
optimisation (= those packages that are included in your SIMDebian)
from their source packages on the target machine?
If we would define some config file mentioning those packages and the
specific options users could build on the local machines. IMHO in the
long term this would reduce your maintenance work to keep SIMDebian
up to date at the expence of local computing cycles on users machines.
It should be pretty easy to add new packages by just adding these to
the said config file.
> Apart from that, reminded by jrtc27 on IRC, I dug a bit on compiler's
> FMV (function multi-versioning) feature[1]. Julia[2] and Clear Linux[3]
> are taking advantage from this feature. And most importantly, packages
> with FMV feature can enter the Debian archive. So FMV is another
> feasible solution to packages need to be accelerated.
>
> After glancing at some benchmarks[4][5] of Clear Linux I started to
> wonder if we could borrow some ideas or even patches from Clear linux
> and introduce them to Debian...
I haven't checked those links but the idea sounds attractive.
Kind regards
Andreas.
> [1] https://lwn.net/Articles/691932/
> [2] https://github.com/JuliaLang/julia/pull/21849
> [3] https://clearlinux.org/documentation/clear-linux/tutorials/fmv
> [4] https://www.phoronix.com/scan.php?page=article&item=arch-antergos-clear&num=1
> [5] https://www.phoronix.com/scan.php?page=article&item=ubuntu-clear-tweaks&num=1
>
>
--
http://fam-tille.de
Reply to: