Re: Fixing sambamba 0.8.6
Am Di., 30. Okt. 2018 um 07:10 Uhr schrieb Pjotr Prins <pjotr2018@thebird.nl>:
>
> Hi Andreas and Matthias,
>
> It appears we are a bit in a fix here. If the D people don't want a
> meson script and I don't think it is a great idea not to inline BioD
> in sambamba, what do we do? I think unDead is only relevant for BioD.
> No other projects really use it and it won't be adopted by new ones.
> Arguably I could make unDead part of BioD.
Then we would have to split it out again in Debian. The
no-bundled-code is pretty much non-negotiable Debian policy,
exceptions for that are very rare.
Fedora has a very good page on why distros won't accept bundled code:
https://fedoraproject.org/wiki/Bundled_Libraries
Static linking is frowned upon, but that's not forbidden (Go uses an
static-linking-like packaging approach).
Meson is used quite a bit in the D world, and with some tooling
support it could even directly recompile included D code, however that
will either require a dirty hack or some work on Meson to implement a
feature like this properly (it's on my todo list, but not with a very
high priority).
> And BioD, currently, is only relevant for sambamba. Though that may
> change in time.
>
> In both cases the source has to be visible for sambamba because it
> acts as include files and is needed for inlining.
The sources are visible. I would actually be very interested in
knowing why LDC wouldn't be able to inline code - it should perfectly
be able to do that, and it feels like an LDC bug if it can't for some
reason.
Maybe worth investigating.
> I strongly suggest we import unDead and BioD as git submodules or dub
> modules without resorting to creating shared libs.
Dub does not work for packaging - some of the necessary changes to
make dub work properly in a distro context have been worked on, but
the features are currently broken and only 50% of the necessary work.
Git submodules won't help either, because Debian will not accept
bundled library code.
> Very much like Ruby
> projects may add Ruby gems. Ruby creates no shared libs because it
> does not compile. D, in many ways, acts like a dynamic language. More
> like Python. It is a feature. You are not telling me that every Ruby
> gem needs to justify linking the source tree in Debian?
Every Ruby gem or Python module lives in a separate package in Debian.
If Ruby code bundles gems, the Debian maintiner will untangle the
bundled code again for packaging. This is a reason why the Gitlab
packaging for Debian took so long.
In Ruby, due to there not being a compiler, there is also no unstable
ABI that we need to track. Whenever a new LDC or GDC release happens,
we need to recompile all D packages in Debian in the right order.
Having shared libraries helps greatly to do these rebuilds
automatically and determine the correct order in which to rebuild the
components.
> Matthias, I like meson and my exposure to it (thanks to you) wants me
> to use it more. No harm done. But we need to find a solution here.
> Sambamba is a popular tool and people resort to bioconda by default
> now. Or even the binary I provide through github (shudder!).
The proper way to do things would likely be to figure out why LDC
applies less optimizations when compiling against the shared library
and whether that is a bug or not. If it's not an issue that can or
should be addressed in the compiler, then we could either implement a
feature in Meson that allows importing a D dependency as source code
(effectively doing what Go does in Debian, recompiling every
dependency when a binary is built) or find another solution. Making
unity builds work for D in Meson would maybe be useful as well.
Alternatively, one could also implement the missing features and
dependency logic in dub, but I think that will be a bit more work.
I could add the missing pieces to Meson, but I might not currently
have the time for that, especially because my daily work involves less
D at the moment. I can maybe take a look though at how complex it
would be to implement very basic support for this feature (as it also
would make D maintenance a lot easier in Debian for some libraries).
There is also the option to write a small Python script that globs the
D files from a shared directory and injects them into the build
process. That's the hackish way I was talking about before, which we
hopefully can avoid.
Cheers,
Matthias
> On Mon, Oct 15, 2018 at 08:53:17AM +0200, Andreas Tille wrote:
> > Hi Pjotr,
> >
> > unfortunately regarding D I have to rely totally on Matthias. I fully
> > agree that a performance loss of this amount is not a good idea.
> > Matthias, can you help?
> >
> > Kind regards
> >
> > Andreas.
> >
> > On Sun, Oct 14, 2018 at 09:42:29AM +0200, Pjotr Prins wrote:
> > > One thing of note: using biod as a library does away with the LLVM
> > > -singleobject optimization step. This has a large negative effect on
> > > performance. Having all D code in memory allows ldc/llvm to inline a
> > > lot of code.
> > >
> > > Just so you are aware. It would be a shame to document that the Debian
> > > package is significantly slower than, say, conda.
> > >
> > > I hit that speed regression myself when I started compiling things
> > > individually.
> > >
> > > I suggest we tell meson not to use libbiod.
> > >
> > > Pj.
> > >
> > > On Sun, Oct 14, 2018 at 08:51:13AM +0200, Pjotr Prins wrote:
> > > > On Sun, Oct 14, 2018 at 08:15:30AM +0200, Andreas Tille wrote:
> > > > > On Sun, Oct 14, 2018 at 07:51:13AM +0200, Pjotr Prins wrote:
> > > > > > Dear Andreas,
> > > > > >
> > > > > > Please mail me when you get stuck. This compile error is because you
> > > > > > need BioD's latest release to match that of sambamba.
> > > > > >
> > > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=907489#33
> > > > > >
> > > > > > bio2/bam/reader.d does not exist in older BioD.
> > > > >
> > > > > You can re-read Matthias' last remark about this at
> > > > >
> > > > > https://alioth-lists.debian.net/pipermail/debian-med-packaging/2018-October/066009.html
> > > > >
> > > > > In the end Matthias is refering to some ldc issue which is currently
> > > > > blocking the sambamba build.
> > > >
> > > > Hmmm. I think it is a different issue. initZ is used for all generated
> > > > object files (part of ldc) and the three modules referred to are part
> > > > of BioD.
> > > >
> > > > Is it possible BioD was built with a differenct ldc?
> > > >
> > > > What version of ldc are you using?
> > > >
> > > > You are doing two things I never tried: using meson and using BioD as
> > > > a lib.
> > > >
> > > > I guess I should try using meson too and a linked in version of
> > > > libbiod - the current build system of sambamba just pulls in the
> > > > sources.
> > > >
> > > > I don't feel like trying Debian builds, so if you can help on your end
> > > > I'll try to help on mine.
> > > >
> > > > Pj.
> > >
> >
> > --
> > http://fam-tille.de
--
I welcome VSRE emails. See http://vsre.info/
Reply to: