Hi Antoine, and thanks for the detailed mail!
On Tue, Sep 26, 2017 at 06:29:05PM -0400, Antoine Beaupré wrote:
> We just had a short conversation on the #debian-devel IRC channel
> regarding the upcoming Python 2 EOL, in the context of the Sphinx
> packages.
>
> [...]
>
> So that's the first problem I have. The other problem we identified is
> that the /usr/bin/sphinx-build symlink is not owned by any package.
>
> $ LANG=C dpkg -S /usr/bin/sphinx-build
> dpkg-query: no path found matching pattern /usr/bin/sphinx-build
>
> That's a rather bad practice... We shouldn't install those symlinks in
> postinst. They should be alternatives or conflicts or *something*. In
> 2013, the sphinx maintainer [explained][1] the following:
>
> > True for docutils; however, sphinx doesn't use alternatives, and it
> > doesn't do so for good reasons.
> >
> > The alternatives mechanisms is only suitable if both commands are
> > compatible, i.e. their behavior doesn't vary with Python version. This
> > is the case for rst2html and friends, but no so much for
> > sphinx-build. The problem is that sphinx-build can import 3rd-party
> > Python code, which is not necessarily compatible with both Python 2
> > and 3.
>
> I understand that reasoning, but I don't think I agree with it. Or at
> least, I don't think that, in the long term, it should be something that
> blocks us.
I think the reason why we should not use alternatives still applies. The
current approach is conservative and it guarantees that when a package has
python-sphinx in build-dependencies, /usr/bin/sphinx-build will always use
Python 2. With alternatives there is no such guarantee.
And moving Python 3 packages to /usr/bin/sphinx3-build or something like
that will mean diverging from upstream (see below).
> I am not sure what the solution is, but I would offer the following:
>
> 1. there should be a more easy way to override SPHINXBUILD in
> quickstart-generated Makefiles. IMHO, that means SPHINXBUILD?=
> instead of SPHINXBUILD=
Good suggestion, I have submitted a pull request upstream:
https://github.com/sphinx-doc/sphinx/pull/4092
> 2. similarly, --with=sphinx should do the right thing depending on the
> detected pybuild Python version, that is, call
> /usr/share/sphinx/scripts/python{,3}/sphinx-build directly depending
> on which version we are building for.
I do not understand this. You mentioned pybuild, so you are talking about
setup.py based build systems, not Makefile based ones, right?
If setup.py calls build_sphinx automatically, then there is nothing else
to do. If it does not, then you should call ‘setup.py build_sphinx’ with
explicit interpreter, i.e. python or python3.
The only thing where pybuild may help is when you want to run Sphinx for
all supported Python versions, then you can use its COMMAND syntax.
And there is no such thing as --with=sphinx. There is --with=sphinxdoc,
but it operates on already built and installed documentation, so it cannot
help with building.
> 3. sphinx packages should follow the "python vs python3"
> convention. right now, when you run the "python" binary, you get
> Python 2. if you want Python 3, you run "python3". I don't know if
> there are plans to change this, but that's irrelevant at this stage:
> sphinx should follow convention and call python2 sphinx when you run
> "sphinx-build" and python3 sphinx when you run "sphinx3-build", and
> both should be available in $PATH.
What is the problem with using /usr/share/sphinx/scripts/python3/sphinx-build
explicitly if you want to force Python 3?
If you like shorter command names, then you can also use ‘python3 -m sphinx’.
Using the -m syntax instead of scripts is even endorsed by Sphinx upstream
these days (see your link [2]).
Also there is no such thing as sphinx3-build upstream (unlike other packages
that follow this convention). So all upstream projects will try to use
sphinx-build, not sphinx3-build, even if they are Python 3 only. And if you
override this command anyway, you can use two existing ways, I don’t see a
need for the third one.
> 4. the sphinx package is not following the usual convention of
> providing only libraries in python*. or to be more precise, it's
> rather odd that "sphinx-common" is what installs the python2 or
> python3 binary in $PATH. i would suggest providing new binary
> packages called "sphinx" and "sphinx3" that would provide those
> binaries.
I won’t say that this convention is usual. There are many packages named
python-* or python3-* which provide executable scripts.
Let me summarize what you, in my opinion, should do as a maintainer of
packages using Sphinx:
1) If your project is not using autodoc: just build-depend on python3-sphinx
and use /usr/bin/sphinx-build. There is no need to care about versions.
2) If your project uses autodoc, and is compatible with both Python 2 and 3:
same as in 1), just use /usr/bin/sphinx-build.
3) If your project uses autodoc, and is only compatible with Python 3:
use ‘python3 -m sphinx’ or ‘python3 setup.py build_sphinx’, depending on the
upstream build system.
4) If your project uses autodoc, and is only compatible with Python 2:
Port it to python 3. Or use ‘python2 -m sphinx’ or ‘python2 setup.py
build_sphinx’.
If you think this classification is incomplete, then please let me know
where I am wrong.
> 5. optionally, the sphinx and sphinx3 packages *may* conflict with each
> other to fight over the sphinx-build symlink. then people needing to
> build against *both* packages can use the "python -m sphinx"
> mechanism in their makefiles, since it's what upstream seems to be
> [doing][2]
Let me quote Policy section 5.4:
“adding Conflicts is normally not the best solution when two packages provide
the same files. Depending on the reason for that conflict, using alternatives
or renaming the files is often a better approach.”
> 6. there are currently packages depending on *both* python-sphinx and
> python3-sphinx. for me, that makes no sense at all: there is a
> single set of documentation files built in a package, and it must be
> on either one of those packages, but not both. if the software
> supports building with python3, then it should depend on
> python3-sphinx, if not, it should depend on python-sphinx. by
> switching to sphinx{,3} binary packages, this would make that
> distinction clearer as well.
This may happen when setup.py depends on Sphinx and automatically calls it
during build. Such packages have to build-depend on both Python 2 and 3
versions of Sphinx, even if only one version of documentation actually gets
shipped.
> My pain points would mostly be fixed with th e simple #1 and #3
> fixes. Without breaking anything, we could already provide a
> sphinx3-build binary, and we could file a PR upstream to make
> SPHINXBUILD more easily overridable. The rest is just a brainstorm of
> possible medium-term fixes to try and fix that dangling symlink issue.
>
> I understand this is a complex situation and I would love to have better
> answers, but I'm not sure there are any simple answers here. We
> shouldn't hesitate to make radical changes, however: we have transition
> tools to help us with that stuff, and if we can upgrade libc and gcc
> across all of Debian fairly painlessly these days, i don't see why we
> couldn't articulate such a transition for ~1000 python packages. :)
In my opinion there should be a very strong reason to make changes that
break packages. I have to deal with Sphinx upstream breaking compatibility
which results in lots of FTBFS bugs for each major version. And I don’t
want to introduce even more incompatibility myself.
--
Dmitry Shachnev
Attachment:
signature.asc
Description: PGP signature