Re: Can't find a library that's right there
On Sun, Aug 22, 2004 at 08:41:42AM -0500, Dirk Eddelbuettel wrote:
>
> Hi Ross
>
> I found the thread by accident when browsing a web archive of
> debian-devel, I'm no longer subscribed here as the volume is too high
> for my available time. Feel free to bug me in private on this too
> (given that I'm the R maintainer, that is).
Thanks, Dirk. My initial puzzle was about how the linker was working,
so didn't seem R-specific. By now it's getting a bit more R-specific.
For everyone's benefit, here's the full context. I have avoided
stating it because it's a bit involved:
I am making an R package, which is a different concept from a Debian
package. The source of an R package consists of a directory tree. R
can check this in place, or build it into a tarball. A user then
takes the tarball and runs an R command which builds binaries (if
any--I have them) from it, digests the other files, and puts the
results in a place R can find them. One can then use the package from
R.
The original source includes a src/ directory, containing code to be
compiled. I have a src/test/ directory, which exercises the code in
src/. This is different from the tests/ top level directory for
exercising R code.
My current setup is that I run R CMD check, which builds a .so file
out of the material in src/ (not including the subdirectory). This
.so file is linked to (references?) libR.so.
I want to 'cd src/test/' and then 'make' to run the tests there.
So when I redistribute this, I want people to be able to do the same:
change into the src/test directory, make, and see if the tests pass.
As far as I can tell, R has no built-in way to exercise the src/
code.* The test is intended to be run at installation, rather than
being run from R in standard use of the package.
*(The top level tests/ directory can exercise the whole system from R,
but my tests are unit tests on individual classes in the C++ source.)
The problem arose because the test program, test1, couldn't find
libR.so. It needs to, because the tests exercise the code in
src/libmspath.so, and the latter links to libR. On Debian, at least,
R runs from a wrapper which sets LD_LIBRARY_PATH appropriately, but
since I'm not running R I don't get the benefit of this. Further,
recipients of the package aren't necessarily using Debian.
Since anyone interested in this will be building it on their own
system, and since the test is just a one shot thing, it seems to me
using -rpath $(shell R RHOME)/bin is OK. (Of course, they are out of
luck without GNU make).
>
> On Sat, Aug 21, 2004 at 10:10:07PM +0200, Goswin von Brederlow wrote:
> > Ross Boylan <ross@biostat.ucsf.edu> writes:
> >
> > > I've found several solutions, and have a theory of the problem. If
> > > anyone could comment on the theory, that would be great.
> > >
> > > SOLUTIONS
> > > The best solution seems to be to include
> > > -Wl,-rpath,/usr/lib/R/bin
> > > as an option to gcc.
> >
> > That has all theproblems associated with rpath and is only valid
> > within a package. If you use an rpath to link in another package
> > library you get problems on future updates and version skews.
I'm not sure what problems with rpath you are referring to. And what
do you mean by "only valid within a package." First, is that in the
"R package" or "Debian package" sense? Second, I just don't see the
problem. Say that libR got updated, which is an action outside of my
package. The path to it doesn't change, so I think my approach would
still work. Perhaps because my use, as noted above, is a one-shot
test after building, I don't think the problems with updates or
version skews apply anyway.
>
> Still, I'd recommend this for a _standalone_ binary.
>
> But you said earlier in the thread that you wanted to distribute this as
> part of an R package -- in which case things are different because ...
>
> >
> > > Setting LD_LIBRARY_PATH also worked, when I remembered to export
> > > it. (duh)
> >
> > This can be done by a wraper script and avoids most problems caused by
> > rapth. Worst case you can change the wraper script to fix bugs.
This is running from a makefile, which ideally requires no special
setup. I could write it to invoke the test program via a wrapper
script, and put the path in the wrapper, but that seems just an
additional complexity here.
>
> ... /usr/bin/R is _exactly_ the kind of wrapper script Goswin talks
> about here.
The problem is that the test is intended to run outside of R, even
though the package itself is destined for use within R.
>
> E.g., witness ldd on the .so of my RQuantLib packages:
>
> dd@homebud:~> ldd /usr/lib/R/site-library/RQuantLib/libs/RQuantLib.so
> libQuantLib-0.3.7.so => /usr/lib/libQuantLib-0.3.7.so (0x400bb000)
> libR.so => not found
Yep, that's exactly the problem I was getting with the mspath.so file
I built.
> libstdc++.so.5 => /usr/lib/libstdc++.so.5 (0x404b5000)
> libm.so.6 => /lib/libm.so.6 (0x4056f000)
> libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x40591000)
> libc.so.6 => /lib/libc.so.6 (0x4059a000)
> /lib/ld-linux.so.2 => /lib/ld-linux.so.2 (0x80000000)
> edd@homebud:~>
>
> The morale here is that iff
>
> a) you want to build a regular, CRAN-style R package that will get
> loaded by R using library() or require(), then the normal manual
> 'Writing R Extensions' fromin the various r-doc-* packages applies,
> and everything just works if you build the package using
> R CMD build .... as described --- this will set up the linker
> calls for you etc pp
>
> b) you want to embed R into another application, things are a little
> different but I am not sure if that is the case here.
>
The problem arose from case c), which is a) + wanting to run a test
outside of R. I am not embedding R in my application (the test
program), though I am using part of the R library.
> > But both of those realy are only appropriate for within a single
> > package. If you (R package exporting a library) have libraries that
> > are ment to be linked to by other they should follow the FSH and be in
> > a public place (/usr/lib/). Then you (package linking against R libs)
> > don't have this problem.
The problem I had was not where my library went, but how to find the
standard libR.so. It's in an appropriate place by the FSH, but it is
not found without some help.
> >
> > > Editing ld.so.conf also worked.
> >
> > I think that is a big no-no.
>
Agreed, but it was an interesting proof at one point the library could
be found.
> FWIW I fully agree with the 'no-no' view. There is one ugly-ish fudge
> that also works: Given that /usr/bin/R sets LD_LIBRARY_PATH for you,
> /usr/lib/R/bin will in fact searched. So for one package which expects
> to link R with another private set of .so libs not otherwise know to ldd
> (via ld.so.conf or other means), I simply softlinked from their
> /usr/lib/$foo/lib$foo.so into /usr/lib/R/bin/lib$foo.so.
Again, I'm not running from R. But does this mean that my little
trick for finding libR ( $(shell R RHOME)/bin) is not going to work
outside of Debian systems?
>
> > > Only the first meets my needs, since I want to redistribute the work
> > > (as part of an R package), and it needs to work without extra
> > > environment tweaking.
>
> I think you'd be well advised to follow a) above. Bug me is something is
> unclear; otherwise follow any of those over 350 CRAN packages that
> contain C/C++ code to see how they do. It really is taken care of by R
> itself thanks to its build mechanism.
>
> Dirk
>
Reply to: