Re: Split Packages files based on new section "buildlibs"
On 11/13/20 7:28 PM, Tomas Pospisek wrote:
> Hi Antonio (and anybody else that understands the technical problem involved
> here),
>
> I've been reading the whole thread and it seems to me that the reason, why
> Rust/Go build-time "libraries" need to be handled differently from all the
> other existing stuff in the world and that "no user ever wants to use" the
> Debian-provided build-time Rust/Go libraries has not been spelled out in
> plain, comprehensible english yet.
>
> So since you seem to understand a bit about the technical problem involved
> here and I'd very much appreciate if you could spell it out. I think it would
> benefit the project as then everybody would be able to understand what this
> new section is about.
Hi Tomas,
The simple explanation is probably similar to go, that in the general case the
whole source code of the project at hand and all its dependencies is compiled
into a single static binary. Exceptions are e.g. libc et al which get linked
by default (if you don't build against musl), and linked libraries that are
used through FFI (foreign function interface), e.g. GTK.
> So let me ask a question that could maybe clear things up:
>
> On 11.11.20 14:39, Antonio Terceiro wrote:
>
>> In the Rust world there is no such thing as installing a library
>> globably. A crate that provides a library can only be used to build
>> other stuff, and is never made available globally. "cargo install" only
>> applies to creates that provide binaries:
>>
>> https://doc.rust-lang.org/cargo/commands/cargo-install.html
>
> [I've read the cargo-install.html document in the past but not re-read it now]
>
> So let's say user joe wants to code Go software that depends on Go's third
> party github.com/tazjin/kontemplate/templater package ("package" in Go's
> taxonomy not in Debian's!).
>
> Then he'd `export GOPATH=~/src/go` and `go get github.com/tazjin/kontemplate`.
> Go would then `git clone` everything under
> `~/src/go/src/github.com/tazjin/kontemplate/`.
>
> So far so good and I think Rust has a similar mechanisms with cargo, right?
The mechanism is similar, as you say. In Rust we have the difference that
there is effectively one single platform where the ecosystem lives, crates.io.
When all I want is to install an application that was published there, I
simply use the `cargo install <application>` command, and it gets compiled and
installed into ~/.cargo/bin. Therefore the sources of the dependencies and the
binary crate are downloaded from crates.io into a cache directory that is
managed by cargo, compiled from there and then installed.
> Now given that alice wants to package joe's software. She'll do the above plus
> `go get github.com/joe/joes_app`. All will be under `~/src/go/src/github....`.
>
> The naive thing to do now would be to move `~/src/go/src/` to `/usr/lib/go`
> and package that as `go-tazjin-kontemplate-dev_0.1.deb` or similar.
>
> Debian's automatic build process for "joes_app" would first install
> `go-tazjin-kontemplate-dev_0.1.deb`, then make a symlink from
> `~/src/go/src/github.com/tazjin` (or `~/.local/go` or whereever Go expects its
> stuff by default) to `/usr/lib/go/src/github.com/tazjin` and build and be done.
>
> A user wanting to develop software based on tazjin's stuff would do the same:
> `apt-get install go-tazjin-kontemplate-dev`, symlink, done.
When we package Rust library crates into librust-*-dev binary packages, the
source code is put into the package inside a subdirectory in
/usr/share/cargo/registry which can then be used during build instead of
fetching it from crates.io. In that regard it the procedure is very similar.
However, this is not a standard directory used by cargo, but instead we have
to inject it using the /usr/share/cargo/bin/cargo wrapper during build.
If you were to develop software in Rust, you'd simply add the dependency entry
into your Cargo.toml file and let the whole mechanism of cargo do the magic.
Everything else like using the library installed in a librust-*-dev package
would just complicate and slow down your development workflow. So these
packages are effectively useless for developers that want to write Rust
software, even if they use Debian or a derivative for that.
The additional thing we have in Rust, that doesn't seem to have a strict
equivalent in other languages (yet) is features. These can be provided by
library crates, and a crate depending on it can add a dependency to it. One
popular example that got mentioned frequently. A feature is somewhat a
combination of an entity in the Cargo.toml file, and conditional compilation
in the source.
So let's assume we have a library crate supporting different database
connection types, these would be enabled by features. If a program using that
library would like to use postgres, it would enable the postgres feature of
said library.
This gets mapped to Debian packages by either providing a
librust-*+feature-dev by the default package, or by creating a new empty
metapackage with the same name, depending on whether the required dependencies
are congruent. To make things worse, some very verbose version requirements
need to be added, but I'm not too deep into the details so I can't explain it
here.
That increases the package index size, either by adding to the Provides
fields, or by adding the metapackages. Additional problems come up when a
library crate changes the feature description in such a way that it was
provided by the librust-*-dev package before, and now gets split out into a
separate metapackage, because then it needs to go through NEW again.
Therefore separating these packages out, which are imho of no real use for
anybody other than for building the binary packages, helps to reduce the size
of the index that needs to be downloaded and processed by probably millions of
computers frequently while keeping them available for those that need it.
> This solution seems to be too trivial that nobody would have though of it, so
> what is it that I (and I guess many Debianers) are missing?
If I understand correctly what you wrote, that's exactly what we're doing, but
the problems arise when you take a close look at the details.
Wolfgang.
Reply to: