Re: Ending/reducing bytecode compilation, loosening dependencies
Joe Wreschnig wrote:
> 1. Stop compiling .pyo files, entirely (I'm hoping for little
> argument on this).
I agree.
> How?: compileall.py:57, - cfile = fullname +
> (__debug__ and 'c' or 'o') + cfile = fullname + 'c'
This is the wrong solution, though. If Python isn't invoked
with -O, the extension will always be 'c'. The right solution
is to invoke compileall only once, without -O.
If you apply the proposed change, the .pyc files will be generated
twice (once for without -O, and once for with -O), you will end
up with the .pyo file named .pyc.
> Disadvantages: * They waste disk space; they use about as much as the
> code itself.
That's not true (or, rather, I disagree): they don't "waste" the
disk space, and the code is not "about" the same as the source code
size. Rather, the size varies: for Python 2.4, htmlentitydefs.py
is 18054 bytes, and htmlentitydefs.pyc is 6405 bytes (which is
roughly a third).
> * It's still far too easy for modules to be regenerated for the wrong
> version of Python; just run the program as root.
I don't understand. What precisely would root do, what precisely
would happen, and why would that be a problem?
Currently, modules are installed with one for each Python
version. So if root runs a program, the Python interpreter picks
up the modules of that Python version, for which .pyc files are
already generated, so nothing is regenerated when root runs
a program.
Even if you have a package which is installed just in a shared
way (e.g. in site-python, not site-packages), root would
generate .pyc files for some Python version, yes. Why would
that be a problem?
Even if generating bytecode files during installation is
suppressed, why would that make it harder for root to
regenerate the modules for the wrong version?
Why
> * .pyc files are not really architecture independent.
Right. Fortunately, they are created installation, instead of being
shipped in the debian package. As you say, they are
platform-dependent only in border-cases.
> * .pyc files result in strange bugs if they are not cleaned up
> properly, since Python will import them regardless of whether or not
> an equivalent .py is present.
That might happen (and I'm sure it did happen). Those bugs would
be strange in various respects, though: you should have a dangling
.pyc file only uninstalled a package which did not remove its
.pyc files on uninstallation, and then only if somebody still
tries to import the module being removed (which also shouldn't
normally happen).
> * If we don't care about byte-compilation, the multi-version support
> suggested in 2.2.3 section 2 becomes much easier -- just add that
> directory to sys.path (or use the existing unversioned
> /usr/lib/site-python). .pyc files are the rationale between tight
> dependencies on Python versions, which is the last of my suggested
> changes.
Not the only rationale, though: Packages with C extension modules
will always have a dependency on the interpreter version.
> Another note: Currently, Python policy is based around the assumption
> that .pyc files are valid within a single minor Python revision. I
> don't find any evidence to support this in the Python documentation.
Nevertheless, this is a promised guarantee. If you wish it documented,
just file a bug report.
Terminology alert: The current "major" release of Python is 2,
the "minor" releases are 2.3, 2.4. The release 2.4.2 is a "micro"
release. The byte code format may change between minor releases,
but will not change between micro releases.
> How?: dh_python should not call compileall.py unless give some
> special flag. Python policy 2.5 should change "should be generated"
> to "may be generated."
What's the point in having a policy, then? The policy should either
mandate it (and specify additional conditions if it's not an
all-or-nothing deal), or forbid it.
> On the other hand, the removal code should be
> a "must" to avoid littering the filesystem if .pyc files do get
> accidentally generated.
That I agree with.
> 3. Python dependencies should be loosened (and here I expect a
> flamewar).
[...]
> How?: Strike the third paragraph from 3.1.1. This would also negate
> the fifth paragraph, which outlines a hypothetical overcomplicated
> solution to the same problem.
So you are talking only about *programs* here, not libraries?
And then only about programs shipping private modules?
That would be fine with me, but
- your proposed change does not work (on its own). You would also
need to mandate that such modules must not be installed
into pythonX.Y/site-packages (else the application breaks
after a Python upgrade), and you should also mandate that
such packages don't create bytecode files during installation
(else those bytecode files become redundant after an upgrade).
- your proposed change does not achieve the affect you want.
What makes the upgrading "suck" is not (primarily) the
programs, but the many library packages.
Regards,
Martin
Reply to: