Re: python packaging infrastructure
Steve Langasek writes:
> On Wed, Jan 18, 2006 at 01:06:39PM +0100, Matthias Klose wrote:
> > Josselin Mouette writes:
> > > Le lundi 16 janvier 2006 à 15:24 +0100, Matthias Klose a écrit :
> > > > This is the right direction, and adding support for extensions makes
> > > > this complete. Does your proposal allow rebuilding these packages
> > > > without actually changing anything (except the changelog).
>
> > > Being able to rebuild packages for a new python version without changing
> > > anything was the purpose of dh_python from the very beginning, for both
> > > packaging styles:
> > > * for packages with a single python-foo package containing
> > > extension foo, build-depending on python-dev, a rebuild will
> > > generate a new package built against the new version;
> > > * for packages with python2.3-foo and python2.4-foo, a rebuild
> > > will make python-foo depend on the new version. The only case
> > > that isn't handled is when the package isn't maintained much,
> > > and lacks python2.5-foo when the python2.5 transition
> > > approaches.
> > > This is independent of python-support.
>
> > the design decision of putting the binary-all python packages in a
> > separate directory into /var/lib/python2.x/site-packages has some
> > problems when supporting packages with extensions (a proposal beeing
> > made on #irc was to keep the extensions in the standard path).
>
> > suppose you have the following scenario
>
> > /usr/lib/python2.3/site-packages/foo/
> > __init__.py
> > fooext.so (doing a "import foomod")
> > foomod.py
>
> > which is splitted into (by python-support)
>
> > /usr/lib/python2.3/site-packages/foo/
> > __init__.py
> > fooext.so
>
> > /var/lib/python2.3/site-packages/foo/
> > __init__.py
> > foomod.py
>
>
> > Having /var/lib/python2.3/site-packages appended to sys.path let's the
> > import of foomod fail (cannot be found). Using just one package
> > directory inside /usr/lib/python2.3/site-packages does avoid the
> > problem (the way the current python-central works).
>
> I think this objection is misguided for the following reason.
>
> If a package includes a python extension, then any .py files it also
> includes are either in the same directory (or same tree) with those .so
> files, or they are in some other unrelated directory.
>
> If they are in the same directory, then *that directory has the name of the
> python implementation in its path*, because .so's are not shared between
> python implementations. In this case, it is not appropriate or reasonable
> for these .py files to be managed via a symlink farm, because each minor
> version of python has its own separate .py file (which may not vary from
> release to release, but that's beside the point).
correct, we run into problems if a package (python meaning of package)
is hold into two locations.
> If they are in different directories, then foomod.py already doesn't enjoy a
> privileged location in the current path; so whether they are symlink-farmed
> or not, there must already be handling in place to locate foomod.py, so it
> doesn't much matter which particular directory it's located in.
>
> Perhaps you are arguing that a package should be able to ship
> /usr/lib/python2.3/site-packages/foo/fooext.so and
> /usr/lib/python/site-packages/foo/foomod.py, and have the infrastructure
> make foomod.py (or foomod.pyc?) available in
> /usr/lib/python2.3/site-packages/foo. I'm not sure why this would be
> advantageous?
>
> > So how does python-support handle packages, where the extension
> > module is split out in it's own binary package?
>
> If they're split out in separate binary packages, then surely the extension
> package doesn't require any special handling, and the module package can be
> managed as before?
yes, another bunch of packages which then cannot be handled by the
packaging infrastructure.
> (since obviously it's not sensible to split
> /usr/lib/python2.3/site-packages/foo between two separate packages...)
obviously?
- In the past packagers were encouraged to split packages this
way to save disk space (at least on the ftp mirrors), if the size of
the arch independent part exceeds the arch dependent part significantly.
- packages are split this way, if the package does depend on other
libraries or packages, which are not always needed (i.e. have a
dependency on X in a separate package, have a separate package for
each database adaptor, built from the same package source).
- packages are distributed upstream this way, so you have different
sources
sounds rather sensible to split these.
Coming back to:
> Perhaps you are arguing that a package should be able to ship
> /usr/lib/python2.3/site-packages/foo/fooext.so and
> /usr/lib/python/site-packages/foo/foomod.py, and have the infrastructure
> make foomod.py (or foomod.pyc?) available in
> /usr/lib/python2.3/site-packages/foo. I'm not sure why this would be
> advantageous?
I.e. in version 1, a package is implemented as a pure python module,
the package maintainer uses the new packaging infrastructure, version
2 implements some functionality in an extension module to speedup
things. Why does a package maintainer have to change the packaging
infrastructure?
Make it worse, if a pure python module offering some kind of plugin
structure uses the infrastructure, another package (maybe even
maintained by somebody else) chooses to implement the plugin as an
extension enforces the other maintainer to drop using the packaging
infrastructure.
I see this lack of robustness as disadvantages, but probably these
have no impact on release management tasks.
> Also, if you really put all of these files in
> /usr/lib/python2.3/site-packages, doesn't that make it unnecessarily
> difficult to distinguish between symlinks managed by python-support, and
> symlinks managed by the packaging system?
That's another point, but unrelated to the design limitation to not
support arch dependent packages. From a user's point of view it
looks like a normal installation, every step taken to make it better
to distinguish exposes the implementaion to the user.
How do you currently distinguish these symlinks? Is it difficult to
look at the name to which the symlink points? Current Debian policy
doesn't disallow them, is there something planned to explicitely
forbid adding file system objects into /usr?
> (I remember that Joss's description of python-support provided for
> symlinking of other data files that may be included in module
> directories, other than .py files...)
Yes, this is needed, or we do change a user's view on the python
package.
Matthias
Reply to: