> On Jan 22, 2016, at 5:30 PM, Barry Warsaw <barry@debian.org> wrote: > > Hey Donald, thanks for starting this conversation. I for one am super > appreciative of all the consideration you give for Debian's little slice of > the world. > > There's a lot to unpack in this thread, and I'm a little under the weather[1], > so hopefully this makes sense. > > Big +1 for recording the files that get installed via the .egg-info/RECORD > file. As you know, I've been working on dirtbike, which is a "rewheeling" > tool to turn an installed package into a .whl. While Debian tightly controls > via policy the set of wheels we'll allow into the archive, dirtbike has code > for parsing the RECORD file. Unfortunately this is never exercised in > practice because we don't have RECORD files - at least not for the packages we > care about[2]. > > Big +1 for using setuptools everywhere. By my count, of the packages that I > happen to randomly have installed on my Ubuntu 16.04 system, I have 67 > .egg-info files and 113 .egg-info directories. I'd rather have .egg-info > directories everywhere. > > +1 for a lintian warning if distutils is used. I guess I'm +0 on forcing that > through pybuild because it'll be unobvious and mysterious, and kind of lets > upstreams off the hook. I'd mildly prefer to patch packages that use > distutils because that's much more discoverable, but I can appreciate that > that's a lot of work we'd be imposing on maintainers, so I won't argue this > too much (other than to say that if pybuild forces it, let's definitely > document this in its manpage!). > > On Jan 22, 2016, at 12:40 PM, Scott Kitterman wrote: > >> Currently --record includes the .pyc files which is both unneeded and bad. >> Before this gets added either in setuptools or by us, this needed to be >> fixed. > > +1 for (I think) another reason than has already been discussed. We won't be > generating .egg-info directories on the end-user's machine, but instead the > machine the package is built on. That could be a maintainer's own system or a > central build machine depending on various conditions. But because the pycs > are generated on the end-user's machine, we actually don't know what pycs will > be generated when the debs are installed, so the egg-info/RECORD file *can't* > contain them, at least not accurately. FWIW It appears that if you do setup.py install —no-compile to tell distutils not to compile the .pyc during .deb build time, I believe that as is it will just omit the .pyc files. Not sure if that’s workable for Debian or not. > > On Jan 22, 2016, at 11:54 AM, Donald Stufft wrote: > >> Regardless of what happens in this thread, pip is going to stop mucking >> around in files that are owned by some other tool without some sort of >> opt-in/--force style flag *and* we're going to be restructuring things to try >> and guide people away from using pip outside of a virtual environment or >> through the user of --user as well. > > Of course, I'd still like --user to be the default[3]. I think that's still > the eventual goal for pip, but isn't yet implemented because $priorities. Yes, still a goal (it’s not as simple as —user will be the default, but that’s the outcome for a non root account in a system Python). > >> A more controversial way that comes with possibly some extra benefits (which >> Debian may not care about) is to use ``pip`` itself as the build tool rather >> than directly invoking setup.py. In this pip would be responsible for mucking >> around with the distutils/setuptools mess instead of that needing to be >> handled by Debian specific tooling. > > I'd like to better understand how this would work. IIUC, the Fedora ecosystem > is making or already has made this switch, but I don't know how it works. > Obviously, we don't want to install wheels into /usr/lib/python3/dist-packages. I haven’t actually personally seen if they’ve done it, but the Fedora maintainer of the python-pip told me they were doing it, and they had a patch to pip to add a feature they needed for it. Forget that pip can fetch files from PyPI and install them for a moment and consider the command ``pip install .``. Fundamentally this is similar to the command ``make install`` right? It builds the files and then installs them into the final location. As I understand it, the standard way for a project like Debian to install a project using a “regular” Makefile install like that is to do ``make DESTDIR=/tmp/some-tempdir install`` so that it invokes the build system, sets all the paths in the resulting build system to the final destination, but then right before it actually copies files to the final paths it just appends a directory in front of them to let you stage the files in a different location instead. We already have an option like this, the —root option which will just append a different prefix to all of the installation paths. So essentially instead of invoking ``python setup.py install —root /tmp/something/`` which is what I think you’re doing now, you’d do ``pip install —root /tmp/something/ —no-deps .`` and it’d just work similarly to what you have now, except pip would be responsible for interacting with the Python build system. Slavek had said there was some bug preventing him from using —root at the time and I’m not entirely sure what that was off the top of my head but the issue he opened was https://github.com/pypa/pip/issues/1351. I’m not 100% sure what the right combination of options are, but if it’s something y’all are interested in I can certainly figure it out (and add anything needed to make it happen). > > FTR, I am still working on updating Debian's pip. I'm slowly shaving the yak, > but there are still a few things to do. If you want to help, get in touch. > >> [3] Import pkg_resources is not the fasest thing in the world, it has to scan >> every directory on sys.path looking for things to activate and it comes with >> a bunch of side effects. This happens implicitly for any project using >> console scripts. > > Which frankly sucks. It's also fragile. Every once in a while a broken > package gets uploaded that breaks pkg_resources, and it's not easy to debug or > repair. I really hope Brett can fix this when/if he builds this functionality > into the stdlib. For console scripts we don’t actually need anything added, this is the console scripts that pip generates when it installs from a Wheel (either explicitly from PyPI or through an implicitly compiled wheel): https://bpaste.net/show/247fe79d91c4 > > Cheers, > -Barry > > [1] double meaning: I have a cold and we're in the early stages of an historic > snow storm! > > [2] dirtbike has two fallbacks, both of which use `dpkg -L` to get the list of > installed files. The first fallback uses `dpkg -S` to find the binary package > that contains the Python package's .egg-info file/directory (doesn't matter > for this purpose). pkg_resources doesn't have one of those. The second > fallback imports the Python package and looks for the binary package > containing the top-level directory. It's all rather ugly, and I'd like to > just use the .egg-info/RECORD file, but I suspect I'll still need the import > fallback for pkg_resources. > > [3] https://github.com/pypa/pip/issues/1668 ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA
Attachment:
signature.asc
Description: Message signed with OpenPGP using GPGMail