:-( Lookout seems to be mangling my inlines... Apologies if it's hard to read...
From: Theodore Tso [mailto:email@example.com]
Sent: 06 March 2002 14:37
To: Anthony W. Youngman
Cc: 'firstname.lastname@example.org'; email@example.com; firstname.lastname@example.org; George Kraft IV
Subject: Re: [Lsb-CommonPackaging] Re: extension of lsb packages
On Wed, Mar 06, 2002 at 08:10:19AM -0000, Anthony W. Youngman wrote:
> Which is why I'm trying to get a new standard that says "here's an
> api. With this, ANY packaging program can talk to ANY distro package
> database and query it".
> Bingo. All packaging problems solved. You can run any installer on top
> of any distro and check dependencies, log the fact that you've
> installed stuff, etc etc. The trouble with current install mechanisms
> is that they are installer, DBMS, packager, and god knows what else
> all stuffed into one bundle. If we separate the separate functions
> into separate tools like all good linusers do, then the problem gets a
> lot simpler.
If you assume that the packaging system is responsible for installing packages, then it isn't that hard. You just simply have a couple of operations, really:
1) /usr/lsb/lib/install_package <package_file>
2) /usr/lsb/lib/remove_package package_name [version]
3) /usr/lsb/lib/query_package package_name [version]
Of course, then the only reason why you need a fancy installer program is to allow the user to choose which sub-packages the user wishes.
Well, I was thinking possibly along those lines...
If you're thinking more in the Windows model, where each vendor has their own propietary setup.exe program which is responsible for copying files in, then that becomes a horse of a very different color. You need to realize that (a) the packaging system keeps track of which file is owned by which package, and often keeps a checksum information about files, so it can determine if a package has been corrupted,
So when the installer tells the package database what it's installed, it has to include a list of files ... fine.
(b) it's kinda stupid to have each vendor replicating the code to copy a file into a standard location, and
It's also kinda stupid to have loads of static and/or duplicate code where different lsb packages just happen to share functionality that isn't defined in the lsb spec. It would be nice for eg Kylix to say it had installed all the dev libraries, so when the user installs an app written by someone else, the installer realises it doesn't need to install duplicate copies of all the runtimes... maybe several times over if I install several Kylix-based packages from several different sources.
(c) the API between the installer program and the packaging system gets *very* complicated, because of the need of the installer to report where each file is to be installed first.
It also doesn't solve any of the hard problems of packaging, which include:
1) Packaging namespace. Each distribution assumes that it, and only it, controls the namespace of packages. So they will freely create packages without worrying about collisions.
Already dealt with. We've dealt with it in lsb by creating an lsb namespace. The lsb imposes a domain-name-style namespace. Don't comply? You're not lsb-compliant!
2) Version numbers. Each distribution uses a versioning scheme which is roughly related to the original upstream author's versioning scheme, but there are both "epoch" numbers which are more significant than the base version, as well as "release" numbers which are less significant than the base version.
Again, dealt with. We may need exceptions, but in general we ought just to be able to use the original author's numbering scheme. What happens if there are functionality-damaging bugs and the original author doesn't fix it I don't know, but that's hopefully an unlikely occurrence.
3) Incompatible event models for installation and deinstallation and upgrade scripts. The pre- and post- install scripts can be hand-waved away if you assume the propietary, hand-written installation program for each package. (Although this clearly wouldn't scale for the several hundred packages in a base distribution, but we'll ignore that point for now). However, how and when the scripts which get run when the package is deinstalled or upgraded *do* need to be standardized, since the installation program won't be running at that point.
How does lsb do it now? Same problem, and same solution!
4) Detailed configuration information for each base package to be provided by the distribution. The FHS provides general guidance to the distributions, but there are all sorts of details about how Apache is to be configured, or what the precise directory name is used in
/var/lib, which do differ from distribution to distribution.
As above. How do we do it now? If my lsb app needs Apache, then my spec suffers exactly the same difficulties as the lsb suffers now. And it will presumably be able to take advantage of exactly the same solution.
5) Knowledge of how to modify system configuration. If a package needs to add a port to /etc/services, how should it do so such that it will survive an update of the base distibution? How about /etc/inetd.conf? Etc.
And again. As above.
All of these combine to make the problem of dependencies very, very, hard. If my program needs to be able to send mail, what package (or virtual package) do I need to list in my dependencies to guarantee that exim or postfix or sendmail will be installed?
What package do I need to depend upon to guarantee that Apache will be installed. And if my package is providing an Apache plug-in module, where has the distribution decided to place the standard Apache plug-ins? For that matter, where has the distribution decided to place the Apache configuration files? Or log files?
And if my package requires at least Apache 1.3.12, how do I state that? Different distribution may have different epoch numbers, and in some cases, different ways of encoding the version number if the upstream author hasn't been careful about releasing software with consistent version numbers.
With an lsb-defined namespace, we simply say Apache >= 1.3.12. Simple!
So to say, "all we need to do is to specify an API", and like pixie dust, expect that all of the hard problems of packaging will just magically disappear, is a bit naive.
Yes, I know it is incredibly naive. But I fail to see how, using my solution, it's going to be *harder* to solve than using the current mechanism. The solutions we come up with here will carry over.
The LSB 1.0 has a partial solution to these problems, but we do so by significantly constraining what an LSB-compliant package can do. An LSB-complaint distribution can only depend on a single package, named "lsb". We restrict what sort of things the package is allowed to assume (because they're different on various distributions). We require long package names, such as "lsb-quicken.com-turbotax", which are not universally popular.
And I've simply sought ways to *extend* that, with the intention of increasing flexibility. Just like with lsb 1 we restrict heavily what the package is allowed to assume, I've done the same, with my "necessary and sufficient" set (which pretty much means "if you require more than current lsb, then you're broken"). But it also allows for the addition of a lot of *OPTIONAL* extras, which allows packagers to innovate, try things out, and then ask for them to be added to the standard.
> Of course, the fly in the ointment is this becomes a "standard
> creating" exercise, rather than the current "codifying existing
> standards" exercise and there'll be horrendous politics. But if you
> think about it, the proposed approach will work perfectly, even with
> an ISV who wishes to create a totally proprietary, obscured to the
> hilt, 100% totally self owned code, setup.exe to run on a
> lsb-compliant system, AND IT WILL WORK. If we can address and allow
> for that setup, it then makes it easy for every one else to be far
> more sharing and co-operative.
Well, ask yourself this. What parts of a propietary, obscured to the hilt, hand-rolled installer would the ISV really want to keep secret? And why? Copying files around? That's not secret. The list of what files are installed? We can compare the system before and after the installation, and know what files where installed/modified.
That comment was not meant seriously. It's just a genuine worst-case scenario, and as I said, if we can deal with that then it'll be dead easy to deal with developers working in friendly co-opetition.
The only thing which really needs to be kept secret/propietary is the initial configuration of the package, and arguably that shouldn't be part of the installation process in the first place. You *want* to separate that out, and perhaps the most you should do is to be able to call the initial configuration tool from the post-installation script.
The bottom line is that I don't think there's any real value in having a windows-style SETUP.EXE custom installer. People may be accustomed to it because it's what Windows used to do, but even in the newer Windows system, Microsoft has actually been moving to the packaging model used by Linux distribution systems, where the vendor provides a package, and the packaging system takes care of installing it. Indeed, with Windows 2000, Microsoft was boasting about how for the first time, they can verify whether or not a package has been corrupted or not --- of course, they conveniently failed to mention that Linux and its packaging systems (especially RPM) has been able to do this for years....
Which is why my spec is based on (a) separating the front and back ends of the installer, (b) abstracting the interface, and (c) trying to spec that interface such that it's compatible with rpm, apt, and lsb.
Once we've got that, and a mechanism for extending it, then co-opetition means the whole shebang is going to evolve rapidly.
As I've said before, "monolithic" is NOT the linux way. You wouldn't expect a database client to directly modify the files on disk, would you? It goes through a SQL interface or similar and lets a DBMS look after it. Why not apply *exactly* the same approach to an install tool. After all, the front end currently DOES modify the back end database, doesn't it? How stupid is that!
At the end of the day, all the objections you've brought up are either (a) obvious, and with obvious solutions, or (b) ones the lsb is already falling over, and we can reuse the same solution.
And all I'm trying to do is abstract the interface between the installer and the package database, which means we can work out a common subset between the main packagers, and seek to build on it. At present, I feel all this debate is having a seriously stifling effect on package management development, and if we start codifying things the way they are, things are going to get worse.
I'm quite happy with the way things are going for lsb 1. We need the current standard. But things need a total rethink for lsb 2. When this came up before I got the impression Jeff and Wichart grok'd what I was getting at. Albert most certainly did. And I would not be in the slightest bit surprised if my ideas didn't get taken up in some form or other. I'm a database professional. I make my living programming the damn things. And the way the package databases are managed just feels horribly "inelegant". As you ought to know, "if it's inelegant, it's wrong!". Maybe I'm wrong, but we are talking about my personal sphere of expertise...
This transmission is intended for the named recipient only. It may contain private and confidential information. If this has come to you in error you must not act on anything disclosed in it, nor must you copy it, modify it, disseminate it in any way, or show it to anyone. Please e-mail the sender to inform us of the transmission error or telephone ECA International immediately and delete the e-mail from your information system.
Telephone numbers for ECA International offices are: Sydney +61 (0)2 9911 7799, Hong Kong + 852 2121 2388, London +44 (0)20 7351 5000 and New York +1 212 582 2333.