At 11:51 PM 11/22/2005 +0000, John J Lee wrote:
1. Does the above affect your concern about reading many zip files? 2. I understand your concern about memory usage (though the above seems to make it a non-issue in practice, if used sensibly), but I must have missed the argument you made for setuptools and/or Python Eggs being problematic for distributors such as Debian and Gentoo. What specifically is/are the problem(s)? It seems at least two distributions are already actively moving towards use of Python Eggs, so it would be good to inform those distributors of any problem you see before they get too far.
Actually, the .egg-info approach that Ian reminded me of should alleviate *all* the concerns raised so far, although it requires more hands-on management of the package definitions. However, if the Debian packagers don't care about this (it's likely no significant change for the better or worse compared to what they have to do already), then they can take a pass altogether on the file/directory issues, at the cost of slowing down tools that actually *do* use eggs.
The tradeoff is that .egg-info files have to have their PKG-INFO files opened and read in order to do dependency processing.... but actually, if they created full ProjectName-Version.egg-info directories, this issue could actually be bypassed as well. Hm. Yes, that's even better. I didn't think of this before because the normal use of .egg-info directories is for "packaging" a source tree as an egg, and you don't want to have to rename the directory every time the version changes. But for a system packager, this isn't an issue.
So, I guess for packaging tools that can't support multiple versions being installed (or don't desire to), the .egg-info approach allows them to precisely preserve the status quo with respect to the implementation tradeoffs, without hurting anybody or breaking anything.
Now that it's clear to me that we can skirt the performance issue for .egg-info directories, it seems reasonable to recommend this approach as a low-risk way for system vendors to support Python eggs, and to provide a way for them to expose metadata for their existing Python packages in a way that setuptools-based packages can use - just make a Project-Version.egg-info directory with a PKG-INFO in it, added to site-packages along with the actual package installation.
Indeed, it seems like it would be reasonable to propose that perhaps the normal distutils install process could provide this additional metadata, which would then eliminate the need for any repackaging or upgrades to any tools that already use "setup.py install" to create their packages. This would address things like ElementTree, which isn't inherently egg-based, but which people would like to depend on without having to rely on a platform-specific packaging tool to resolve the dependency "out of band".