Re: Debhelper for R packages
On 09/09/16 14:17, Dirk Eddelbuettel wrote:
> | The plan was to avoid seeing eg, lintian hardening-no-bindnow warnings
> | on packages with compiled extensions. I tried injecting dpkg-buildflags
> | LDFLAGS output into the MAKEFLAGS environment for R CMD INSTALL but it
> | doesn't appear to work. It is possible this is also not particularly
> | important (and/or might interfere with R's own build config) - I don't
> | really see security as a major R issue.
>
> It's an R issue. R "remembers" CC, CFLAGS, CXX, CXXFLAGS, etc pp from its
> configure run and reuses them. When R runs 'R CMD INSTALL' you can add to
> this on a per-package basis via the PKG_* variants from inside the package.
> But we can't override from the outside. Minor annoyance (ie would be nice to
> strip '-g' sometimes, toggle -O2 for -O3, ...). Needs an upstream change.
I understood (from how the CDBS makefile worked) that you could set an
env var MAKEFLAGS encapsulating other build flags and it would be used.
I was trying to use the same mechanism, ie
MAKEFLAGS=LDFLAGS=-z,now R CMD INSTALL ...
Is this mechanism no longer useable (or have I misunderstood what it
should be able to do)?
>
> | > I had once tried to bring some perspective from the R ecosystem to that, but
> | > as I recall I pretty much failed to get my point across. R already does /a
> | > lot/ here via R CMD check, and it may be worthwhile to milk that angle rather
> | > than to shoehorn R (which builds packages somewhat idiosyncratically) into
> | > Debian's framework. Anyway, minor 'day 2' detail.
> |
> | I suspect our testing is most interesting where non-R dependencies are
> | concerned, particularly where we remove bundled copies and force the use
> | of system ones.
> |
> | As with vignette building below, it requires the set of build-deps to be
> | much bigger (and particularly for bioconductor, tends to require big
> | dataset packages available).
>
> CRAN and BioConductor don't seem to know/care about chroot and minimal
> environments. They tend to build on "fat" systems with everything installed.
>
> But they also do it across three or more OSs which we don't so we can take
> advantage of Linux tricks where they can't.
>
> Some of the test philosophy is the same and hence redundant. R cares a lot
> about build "stability" and "possibility" (ie no API changes), we care a lot
> about changes in the toolchain (and so do they). Eg Brian Ripley in Oxford
> tends to also check again as-of-yet unreleased (!!) gcc versions to catch
> things early (but those checks are extra -- at the core where tests are
> automated they are more sensible).
>
> | My workflow is to run autopkgtest (in an schroot) as a postbuild step,
> | which tests the built deb (so no workaround for testing a not-yet
> | installed package is required, and ensuring the binary dependencies are
>
> I fully agree that chroot tests are good, but our build does thosse ...
>
> | correct), and autodep8 ensures that there is at least a `library(RLIB)`
> | test defined for every package.
>
> ... because as I tried to explain last time, _each_ R CMD INSTALL involved
> in _each_ of our builds includes a load of the just built library. From what
> I understood then and understand now you are duplicating that. But I may
> miss sufficient coffee... Anyway
Yes, I've read #752609 and I think I understood your meaning - that
anything coming directly from CRAN/bioconductor should have fairly
strong guarantees of correctness, and re-checking that at build time is
extra unnecessary work for us.
(Although I suspect there must be some edge cases we might nevertheless
catch at build-time, I can't admittedly come up with a concrete one
right now).
I don't think it's quite the same argument as autopkgtest though.
Running autopkgtest immediately post-build checks that the runtime
Depends: are correct. (Hopefully using a substvar instead of a manual
list should mostly make this issue go away, but I've certainly seen
uploaded packages with insufficient Depends: to be loaded).
Running them periodically after (as ci.d.n does) tests for bit rot due
to whatever reason (eg, dependencies which should be versioned but aren't).
>
> | (stdout from R CMD INSTALL includes "checking if the package can be
> | loaded" or something like that, but I'm not sure what that is actually
> | testing in practice)
>
> The very 'library(foo)' for a the package r-cran-foo you are building.
>
> Which checks dynamic linker as well as R aspects (as the R code needs to
> syntactially parseable etc pp). It is a good test. I just fell over it
> yesterday on one or two small packages I am building right now.
>
> | > | > It might also be interesting to explore:
> | > | >
> | > | > * handling binary dependencies between R compiled extensions
> | > |
> | > | What exactly do you mean here? I admit I'd welcome if this dh module
> | > | would be available in sid soon to enable switching packages to it to get
> | > | it fully tested soon. This kind of features might wait for later
> | > | versions.
> |
> | See, eg, r-cran-rprotobuf (LinkingTo: Rcpp) or r-bioc-biostrings
> | (LinkingTo: IRanges).
>
> Being the author of Rcpp which is used by 750+ CRAN packages via LinkingTo
> (whereas we recommended Depends in the day), I am aware of its limitation. It
> is (pretty much) R's version of Build-Depends for the C++ case of header-only
> libraries.
>
> | My understanding is that this means that these packages build a binary
> | extension using header files from another R package, and may link to the
> | so file from the other package at runtime.
>
> Not necessarily a so file.
>
> It is for C++ code where you can get by with headers only. Most prominent
> example is the BH package I maintain -- it has 120mb of Boost but __without
> linking__ as that makes deployment so much easier across OS.
>
> BH, RcppArmadillo, RcppEigen, ... all get used by _their_ client packages
> just via headers. What they _would_ need in standalone use (ie if your built
> a main()) they get from R: LAPACK, BLAS, ...
>
> | I am wondering if this should be interpreted as a tight dependency on
> | the LinkingTo: package, eg
> |
> | Built-Using: r-cran-rcpp (=version) or
> | Depends: r-cran-rcpp (=version)
>
> No.
>
> | or if we need to trigger a transition when a package which can be the
> | subject of LinkingTo: changes.
>
> No.
>
> | I'm speculating a bit here - I don't know how often this is a
> | problem/how stable in practice package ABIs are. This might just be
> | adding more work for a rare problem.
>
> Yes.
That's a nice easy answer to those questions, thanks. No special
treatment of this header needed, then.
Reply to: