Re: Application libraries private, Distutils metadata available for console scripts and introspection

To: debian-python@lists.debian.org
Subject: Re: Application libraries private, Distutils metadata available for console scripts and introspection
From: Ben Finney <ben+debian@benfinney.id.au>
Date: Fri, 27 Nov 2015 09:56:54 +1100
Message-id: <[🔎] 857fl49xop.fsf@benfinney.id.au>
References: <85bndmc0ln.fsf@benfinney.id.au> <85k2s99q1h.fsf@benfinney.id.au>
Piotr Ożarowski <piotr@debian.org> writes:

> As I already suggested in my previous mail, installing scripts into
> private directory and symlinking them in /usr/bin is the cleanest
> solution IMHO

Thank you. I have come to a compromise solution on this which builds on
that suggestion.


Ben Finney <ben+debian@benfinney.id.au> writes:

> * The ‘console_scripts’ entry point is used to specify command line
>   programs, ‘lorem’ and ‘ipsum’, using that Distutils feature. Those are
>   the correct names for the commands, so I don't want to mess with that.

Main point of compromise: Change the installed names of the commands.

Second point of compromise: Make an application namespace package.


Distutils (and Python generally) makes it highly awkward to have an
importable module *also* be an executable program.

Python's import mechanism makes it infeasible to have a module file
named without a suffix; this makes it infeasible to name a command file
by Unix conventions of having no suffix for the implementation language.

There are many approaches to address this, which I won't detail here,
but suffice it to say that none of them work smoothly.

So the compromise is:

* Patch upstream's ‘setup.py’ to rename the command file to something
  like ‘execute-lorem’, so that it can be installed alongside the
  ‘lorem’ package directory.

* Create a symlink in the application-private directory, making an
  application namespace from which to import the private modules.

* Create a public symlink, ‘/usr/bin/lorem’, to the private
  ‘execute-lorem’ command file.

> * The ‘--install-lib’ option is used in ‘PYBUILD_INSTALL_ARGS’ to place
>   the Python packages in an application-private directory,
>   ‘/usr/share/foo-app/’. The point of that is so they won't be on the
>   Python module search path.

This remains the same; the ‘--install-lib’ option is used to place the
application-private packages in a private directory. That's the whole
point of this endeavour, and everything else is a workaround for
Python's awkwardness to deal with this.

The programs will still need to be importable, but we want to protect
against a conflict with some other publicly-installed Python library.

So we create a namespace ‘FooApp’ specifically for the application, with
a symlink internal to the application's own library directory.

> * The ‘--install-scripts’ option is *not* used, because Distutils places
>   the constructed scripts in the correct directory (‘/usr/bin/’) by
>   default.

We must now install the program command files to the same location, in
order that the application-private packages can be imported by those
programs.

So the ‘--install-scripts’ option is used, naming the same
application-private directory as for the ‘--install-lib’ option.

> * The script names are correct, but are identical to names of
>   directories in the ‘/usr/share/foo-app/’ top level. That's
>   another good reason not to use the ‘--install-scripts’ option. It is
>   also normal and expected, and shouldn't be a problem because the
>   scripts will not exist there; they belong in ‘/usr/bin/’.

Because the command files cannot keep their upstream name and live in
the same location as package directories with the same names, we need to
patch upstream's ‘setup.py’.

Patch upstream's ‘setup.py’ so the ‘console_scripts’ names differ from
the package directory names. For clarity of association and purpose, I
chose to name them ‘execute-lorem’ and ‘execute-ipsum’.

Patch the target of the ‘console_scripts’ entry points, so that those
entry points are preceded by the ‘FooApp’ namespace package.

>   According to Robert's earlier message, that means the Distutils
>   metadata file needs to be not in the application's private
>   directory, but in a directory on the Python module search path. That
>   seems odd to me, since this is not amodule import being done.

Thanks to everyone for explaining this to me; it's a messy hack, and I
do wish the Python Packaging Authority patience and strength in solving
this better than the mess we have now.

> Here is the output from the example. I can provide the files if anyone
> wants to experiment.

Here is an updated session to show how this works:

=====
$ pwd
/home/bignose/Projects/debian/foo-app-1.2.3

$ find .
.
./setup.py
./lorem
./lorem/amet.py
./lorem/dolor.py
./lorem/sit.py
./lorem/__init__.py
./ipsum
./ipsum/elit.py
./ipsum/adipiscing.py
./ipsum/consecteur.py
./ipsum/__init__.py
./debian
./debian/compat
./debian/rules
./debian/foo-app.links
./debian/changelog
./debian/control

$ cat ./setup.py
from setuptools import (setup, find_packages)

setup(
        name="FooApp",
        version="1.2.3",
        packages=find_packages(),

        entry_points={
            'console_scripts': [
                "execute-lorem = FooApp.lorem.dolor:main",
                "execute-ipsum = FooApp.ipsum.consecteur:main",
                ],
            },  
        )   


    debhelper (>= 9)

$ cat ./debian/rules 
#! /usr/bin/make -f

PACKAGE_NAME = foo-app

package_share_dir = /usr/share/${PACKAGE_NAME}
export PYBUILD_INSTALL_ARGS ?= \
        --install-lib=${package_share_dir}/ \
        --install-scripts=${package_share_dir}/

%:
        dh $@ --with=python3 --buildsystem=pybuild

$ cat ./debian/foo-app.links 
# foo-app.links
# Symbolic links to create for the ‘foo-app’ package.

# Make an application-private package name for the libraries.
/usr/share/foo-app /usr/share/foo-app/FooApp

# Make public symlinks for the installed commands.
/usr/share/foo-app/execute-lorem /usr/bin/lorem
/usr/share/foo-app/execute-ipsum /usr/bin/ipsum

$ ./debian/rules clean
dh clean --with=python3 --buildsystem=pybuild
   dh_testdir -O--buildsystem=pybuild
   dh_auto_clean -O--buildsystem=pybuild
I: pybuild base:184: python3.4 setup.py clean
running clean
removing '/home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build' (and everything under it)
'build/bdist.linux-x86_64' does not exist -- can't clean it
'build/scripts-3.4' does not exist -- can't clean it
   dh_clean -O--buildsystem=pybuild

$ fakeroot ./debian/rules binary
dh binary --with=python3 --buildsystem=pybuild
   dh_testdir -O--buildsystem=pybuild
   dh_auto_configure -O--buildsystem=pybuild
I: pybuild base:184: python3.4 setup.py config
running config
   dh_auto_build -O--buildsystem=pybuild
I: pybuild base:184: /usr/bin/python3 setup.py build
running build
running build_py
creating /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum
copying ipsum/elit.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum
copying ipsum/adipiscing.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum
copying ipsum/consecteur.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum
copying ipsum/__init__.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum
creating /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem
copying lorem/amet.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem
copying lorem/dolor.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem
copying lorem/sit.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem
copying lorem/__init__.py -> /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem
   dh_auto_test -O--buildsystem=pybuild
I: pybuild base:184: cd /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build; python3.4 -m unittest discover -v

----------------------------------------------------------------------
Ran 0 tests in 0.000s

OK
   dh_testroot -O--buildsystem=pybuild
   dh_prep -O--buildsystem=pybuild
   dh_auto_install -O--buildsystem=pybuild
I: pybuild base:184: /usr/bin/python3 setup.py install --root /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app --install-lib=/usr/share/foo-app/ --install-scripts=/usr/share/foo-app/
running install
running build
running build_py
running install_lib
creating /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr
creating /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share
creating /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app
creating /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum/elit.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum/adipiscing.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum/consecteur.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/ipsum/__init__.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum
creating /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem/amet.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem/dolor.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem/sit.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem
copying /home/bignose/Projects/debian/foo-app-1.2.3/.pybuild/pythonX.Y_3.4/build/lorem/__init__.py -> /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum/elit.py to elit.cpython-34.pyc
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum/adipiscing.py to adipiscing.cpython-34.pyc
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum/consecteur.py to consecteur.cpython-34.pyc
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/ipsum/__init__.py to __init__.cpython-34.pyc
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem/amet.py to amet.cpython-34.pyc
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem/dolor.py to dolor.cpython-34.pyc
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem/sit.py to sit.cpython-34.pyc
byte-compiling /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/lorem/__init__.py to __init__.cpython-34.pyc
running install_egg_info
running egg_info
creating FooApp.egg-info
writing entry points to FooApp.egg-info/entry_points.txt
writing top-level names to FooApp.egg-info/top_level.txt
writing FooApp.egg-info/PKG-INFO
writing dependency_links to FooApp.egg-info/dependency_links.txt
writing manifest file 'FooApp.egg-info/SOURCES.txt'
reading manifest file 'FooApp.egg-info/SOURCES.txt'
writing manifest file 'FooApp.egg-info/SOURCES.txt'
Copying FooApp.egg-info to /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/FooApp-1.2.3.egg-info
Skipping SOURCES.txt
running install_scripts
Installing execute-lorem script to /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/
Installing execute-ipsum script to /home/bignose/Projects/debian/foo-app-1.2.3/debian/foo-app/usr/share/foo-app/
   dh_installdocs -O--buildsystem=pybuild
   dh_installchangelogs -O--buildsystem=pybuild   
   dh_python3 -O--buildsystem=pybuild
   dh_perl -O--buildsystem=pybuild
   dh_link -O--buildsystem=pybuild
   dh_strip_nondeterminism -O--buildsystem=pybuild
   dh_compress -O--buildsystem=pybuild
   dh_fixperms -O--buildsystem=pybuild
   dh_installdeb -O--buildsystem=pybuild
   dh_gencontrol -O--buildsystem=pybuild
   dh_md5sums -O--buildsystem=pybuild
   dh_builddeb -O--buildsystem=pybuild
dpkg-deb: building package 'foo-app' in '../foo-app_1.2.3-2_all.deb'.

$ sudo dpkg -i ../foo-app_1.2.3-2_all.deb
[sudo] password for bignose:
(Reading database ... 759304 files and directories currently installed.)
Preparing to unpack ../foo-app_1.2.3-2_all.deb ...
Unpacking foo-app (1.2.3-2) over (1.2.3-2) ...
Setting up foo-app (1.2.3-2) ...

$ dpkg --listfiles foo-app
/.
/usr
/usr/bin
/usr/share
/usr/share/doc
/usr/share/doc/foo-app
/usr/share/doc/foo-app/changelog.Debian.gz
/usr/share/foo-app
/usr/share/foo-app/FooApp-1.2.3.egg-info
/usr/share/foo-app/FooApp-1.2.3.egg-info/top_level.txt
/usr/share/foo-app/FooApp-1.2.3.egg-info/dependency_links.txt
/usr/share/foo-app/FooApp-1.2.3.egg-info/entry_points.txt
/usr/share/foo-app/FooApp-1.2.3.egg-info/PKG-INFO 
/usr/share/foo-app/ipsum
/usr/share/foo-app/ipsum/elit.py
/usr/share/foo-app/ipsum/adipiscing.py
/usr/share/foo-app/ipsum/consecteur.py
/usr/share/foo-app/ipsum/__init__.py
/usr/share/foo-app/execute-ipsum
/usr/share/foo-app/execute-lorem
/usr/share/foo-app/lorem
/usr/share/foo-app/lorem/amet.py
/usr/share/foo-app/lorem/dolor.py
/usr/share/foo-app/lorem/sit.py
/usr/share/foo-app/lorem/__init__.py
/usr/share/python3
/usr/share/python3/runtime.d
/usr/share/python3/runtime.d/foo-app.rtupdate
/usr/bin/ipsum
/usr/bin/lorem
/usr/share/foo-app/FooApp

$ cat /usr/bin/lorem
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'FooApp==1.2.3','console_scripts','execute-lorem'
__requires__ = 'FooApp==1.2.3'
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.exit(
        load_entry_point('FooApp==1.2.3', 'console_scripts', 'execute-lorem')()
    )

$ /usr/bin/lorem
>From FooApp.lorem.dolor.main
Hello, world!

=====

-- 
 \       “Anyone who puts a small gloss on [a] fundamental technology, |
  `\          calls it proprietary, and then tries to keep others from |
_o__)           building on it, is a thief.” —Tim O'Reilly, 2000-01-25 |
Ben Finney
Reply to:
Follow-Ups:
- Re: Application libraries private, Distutils metadata available for console scripts and introspection
  - From: Piotr Ożarowski <piotr@debian.org>
- Re: Application libraries private, Distutils metadata available for console scripts and introspection
  - From: Barry Warsaw <barry@debian.org>
Prev by Date: Re: Pushing a bunch of packages to jessie-backports
Next by Date: Re: Application libraries private, Distutils metadata available for console scripts and introspection
Previous by thread: Re: How to manually install python scripts and needed modules (Was: How to tweak homebrew install target for Gubbins)
Next by thread: Re: Application libraries private, Distutils metadata available for console scripts and introspection
Index(es):
- Date
- Thread