Re: [Distutils] formencode as .egg in Debian ??
At 09:10 PM 11/24/2005 +0100, Josselin Mouette wrote:
A sane way, first of all, means a consistent way. Having two sorts of
Debian python packages is a no-go. Therefore, if we want to switch to a
new way of distributing packages, there has to be some serious grounds
for it. Currently, the picture I have of eggs doesn't show any
advantages over the plain old distutils we are currently using.
What advantage does the internet have? It lets you be connected to the
internet. Eggs are a network, packaging them lets your packages be part of
that network. That's the only *possible* benefit to eggs from a system
packager's point of view, which is why it has never been my policy to go to
system packagers and ask them to include eggs. Instead, I leave it to
their users to ask, because it's really none of my business whether a
distributor packages eggs. That's between you and your users. I only get
involved if someone asks for help in packaging the eggs, and I help anybody
who asks.
Which is why I find this discussion so frustrating; somehow or other,
people keep spinning this back to some kind of advocacy thing. If you want
to provide eggs, great. You don't want to provide them, great. It really
makes no difference to me; it's an issue between you and your users. I'm
only here to help if you *do* want to provide eggs and have issues with
*how* to do it in a way that works for you.
[snip items already discussed that have technical solutions already arrived at]
A sane way means compliance with standards, especially the filesystem
hierarchy standard. When some people are trying to separate .py
and .pyc/.pyo files to respect it, you're asking to put them in a single
archive.
PEP 304 has been languishing for years, apparently due to lack of
interest. In any case, its motivation was never FHS conformance, as far as
I can tell.
More immediately relevant, I'm 1) not asking anybody to do anything, and 2)
an archive is only one of three already-valid ways to lay out the
installation of an egg.
> 1. Egg-based projects need to install their published metadata, in a
> well-known location relative to the installation location of their
code, so
> that it can be found by searching sys.path, so that it and other projects
> can locate the metadata for currently-importable projects, *without*
> needing to first import the project's code.
>
> 2. Egg-based projects need to be able to identify whether another Python
> project package is installed and what version it is, without requiring
> modification to that other project's code or needing to import it. (And
> this is independent of whether the depended-on project was packaged as an
> egg by its author.)
>
> As far as I'm aware, those are the irreducible technical minimum
> requirements for making an egg-based project work. *How* these
> requirements are met is quite flexible, as there are already three working
> layouts that achieve this. As I said before, I'm quite willing to
> implement a fourth. But nobody has been proposing anything that meets
> these requirements, because they're too busy trying to prove the
> requirements don't exist or are somehow not real.
Alright, let me try to propose something. How about defining a place in
the python distribution, say, /usr/share/python2.x/eggs (but of course
configurable when installing python), where this information should go?
An installed package would look like a directory structure of .py files
in /usr/lib/python2.x/site-packages, and a metadata file
in /usr/share/python2.x/eggs.
See line 2 of item 1 above; we need a "well-known location located relative
to the installation location of their code". Python programs have limited
access to knowledge about platform directory structure, except as expressed
in sys.path. Also, they need to be able to support development
environments, application plugin directories, and other sys.path
manipulation. So, ideally the approach needs to be able to be applied
uniformly to each sys.path entry.
One of the reasons for this is that it allows the runtime to reason about
the contents of sys.path. If the metadata is in a particular directory, it
knows that by adding or removing that directory from sys.path, what
packages will be added or removed from importability, and it also knows
which packages may be shadowing other packages. In the case of namespace
packages, it knows what subdirectories of sys.path directories need to be
added to the package's in-memory __path__ in order to merge
separately-distributed components into a common virtual package.
These goals seem unachievable with an absolute location that's divorced
from any direct connection to a sys.path entry, although there are a couple
of workarounds possible. See below.
In fact, this approach is quite similar to the .egg-info approach, but
it would be better to put the files at the right place from the very
beginning, and, more importantly, to be able to deal with other packages
not having such .egg-info files.
It doesn't matter where the files actually live; it meets the technical
requirements just fine if you symlink the metadata into site-packages, and
would work with the existing runtime. (Or does the FHS disallow symlinking
out of site-packages, too? I'm actually surprisd the FHS would be
dictating anything about Python package internals anyway, but oh well.)
Alternately, you could just put your proposed /usr/share directory on
sys.path (just after site-packages) which would solve the issue of trying
to have some new way of configuring the location, which setuptools would
then need to be able to read. Sure, that would increase the length of
sys.path, but only by one, no matter how many eggs are installed by Debian.
> Since I'm neither a Debian nor Mozilla developer, I have no way to know
> what these problems you speak of are, nor any way to tell whether the
> alleged flaws of the Mozilla packaging relate in any meaningful way to the
> alleged flaws of eggs. It's you who's in the position of giving advocacy
> without providing any real information about the issues.
This is true. In short, my two main concerns about mozilla-related
packaging are total absence of library versioning, and the shipping of
extensions in .xpi files, which are tarballs containing information and
meta-information about the package. You can surely understand that my
concerns about eggs approach those of .xpi files.
Yes - but I fear that you may be making an inaccurate comparison based on a
superficial similarity. Distribution of binary eggs of anything but
pure-Python libraries (i.e., ones with no native code) is understood to be
impractical for anything but well-identified slow-moving platforms like
Windows. If you observe e.g. TurboGears' distribution practices, you'll
see that it relies on Python source packages for "impure" projects on
anything but OS X and Windows. As a practical matter, eggs containing
native code for Linux get built from source on the target machine, unless
some system packager provides them.
It sounds like your Mozilla complaints are directed at Mozilla-provided
pre-packaged binaries that include native code, which is not something that
most Python developers *want* to get involved in. There will undoubtedly
be projects (especially commercial ones) that provide plugin binaries for
various platforms, but even they will prefer to rely on system-packaged
libraries for anything that's not bleeding edge. And the plugins
themselves most likely will live in those apps' plugin directories, not
system directories.
(I'm of course making several assumptions here about what your concerns are
and what kind of issues you've encountered, as you still haven't provided
anything other than a description of similarity in form, and some vague
hints (e.g. "library versioning") about what might actually be at issue.)
Reply to: