LCC and Debian: summary and next steps
This message is intended to continue the discussion started on
debian-devel about how Debian can engage the Linux Core Consortium and
vice versa. I've created a mailing list for this discussion, which you
can subscribe to here:
http://lists.progeny.com/listinfo/lsb-workers . I'd like to focus
primarily on the technical issues involved and to begin doing some
experimentation as well, since we won't fully know the extent
of the technical issues until we've actually tried to do some things.
I'm CC'ing debian-devel on this initial message but would ask
that followups go to lsb-workers (and have set Reply-To accordingly).
(Note: I wanted to summarize the debian-devel discussion and next steps
in one place. If you want to respond to multiple points, please try
to do so in different subthreads so we can keep the discussion focused.)
DISCUSSION SUMMARY, AND MY RESPONSE TO EACH POINT
I'll start by summarizing the main technical points of the LCC
debian-devel thread, which fall into a few major categories:
1. How can Debian reconcile its development processes with that of the
other distros involved in the LCC effort? The main issues here are: 1.
Debian policy is to generate all binary packages from source, which
would appear to be at odds with the LCC's goal of producing a
common set of binary packages; and 2. Debian values its ability to
make decisions independently of external forces, which would appear to
be at odds with the LCC's goal of uniform decisions across distros.
My response: We've said from the beginning we want the LCC to be an
open, collaborative effort. Among other things, this means that while
binary packages will be available, source packages will be available as
well, and also that the binary packages can be regenerated from
the source packages as necessary (i.e., the build process will be open).
So, there's nothing to say that Debian *must* redistribute the
binary packages the LCC provides. There's also nothing to say
that Debian couldn't change the source packages LCC provides either,
or that Debian couldn't use its own build process (i.e.,
treat the LCC source packages as "upstream", as Bruce has suggested).
As with everything, there will be tradeoffs involved--not using
the LCC binary packages may result in the loss of
some certifications, because many of the larger ISVs require
a common binary core. However, there's absolutely nothing
preventing Debian from making those tradeoffs as it sees fit.
There's also nothing saying Debian couldn't use the result of the LCC's
work in a piecemeal fashion--perhaps some Debian developers that
maintain core packages in Debian may choose to adopt the LCC source
packages as the upstream while others do not. Even if participation is
not total, there's still an enormous benefit here--25% the same at the
source level, say, is still better than the situation today. It doesn't
have to be all or nothing, i.e., it doesn't have to be 100% the
same at the binary level, modulo the above comment about the tradeoffs.
2. How can Debian reconcile divergence in priorities with those of the
other distros? The main issues here appear to revolve around
architecture support--the LCC will initially support only IA32, IA64,
AMD64, and Intel EM64T, whereas Debian supports far more.
The canonical example of a problem that might arise is: What
happens if there's a critical bug fix to an LCC package
that is only relevant to an architecture LCC doesn't support?
My response: Having an open process helps mitigate this. For example, as
per my response to #1, Debian might add its own patches to the LCC
source package, as it does now with other upstream packages, then submit
the patch to the LCC for inclusion in future versions. Assuming the
LCC sources are under version control, the patch could be applied on
a branch and merged with the trunk the next time the trunk is modified
(i.e., when there is a bug fix to an architecture that is supported).
Because the patch doesn't affect the "official" LCC core (i.e., it only
affects an architecture that's not supported), the core itself wouldn't
be affected (i.e., the trunk would remain unmodified, and there would be
no new versions of the source or binary packages as a result of the
patch), but that doesn't mean there can't be close coordination to make
sure the patch is included the next time the trunk *is* modified, so
any divergence between the LCC and Debian cores would remain temporary.
This is just one possible solution, and it's clearly one area that needs
further discussion; but, with an open process, there *are* solutions out
there.
3. What are the technical hurdles to building a LCC core that has both
RPM and Debian packaged versions, i.e. that can form the basis of both
RPM-based and Debian-based distros?
The key to producing RPMs and .debs that are bit-identical is to build
both from a common source package, and to fully understand and deal with
any differences in the system that results when the two sets of binary
packages are unpacked.
The former isn't that hard to do. We've already built RPMs from Debian
source packages by writing an RPM .spec file that takes the binaries
built using the standard Debian build process and puts them in an RPM
package; one could imagine going in the opposite direction (building
a .deb package from an SRPM), but because Debian policy is largely
provided by the source package, the possibility of using the Debian
source package as the common source package is an interesting one.
In the latter category, two issues that jump out at me initially, as
I've already described on debian-devel, are 1. package namespace
differences (i.e., Debian and the RPM-based distros name their
packages differently--one example that has already come up
is "libacl1" vs. "acl"); and 2. file system differences, mostly
having to do with configuration (e.g., /etc/network
vs. /etc/sysconfig/network, which has already come up as well).
The package namespace issue is probably not that difficult to deal
with. For example, one simple solution might be to supply dummy
packages for Debian that provide the RPM names and depend on
the appropriate Debian packages and vice versa. The difficultly
here lies in fully understanding where the differences exist.
The file system/configuration differences will be harder. A
"compatibility layer" that maps from one mechanism to the other and vice
versa is the right way to go here, but it's not entirely clear how to
build such a thing. One potential solution is to attack the problem
piecemeal, including tools that, say, populate /etc/sysconfig/network on
a Debian system using what is found in /etc/network and vice versa on an
RPM-based system. A more ambitious solution is to work with one of the
distribution-neutral configuration framework projects (such as
Config4GNU) that aims to mask the differences on a more global scale.
>From my point of view, this is going to be the hardest of the tasks in
front of us, and I'm not sure the right way forward, except that
understanding the scope of the problem is the first step to solving it.
Since understanding the scope of such problems is a key first step in
both cases (and in discovering the other cases which no doubt exist
that haven't occurred to us), experimentation is a critical next step.
4. Assuming Debian does not participate fully in the LCC, what options
exsit to producing a Debian version of the LCC core, so other distros
that derive from Debian can benefit from the ISV/IHV certifications to
the LCC core? And how would Debian benefit from partial participation?
There are two options here: One option is to simply
produce a Debian-packaged version of the LCC core independently,
and to make sure that core is 100% compatible with Debian (i.e., you
can take any Debian package and install it on the LCC Debian core
and get the same results as if you'd installed it on Debian itself).
Another option is to produce a set of optional add-on packages for
Debian that transform the Debian core in the LCC core, similar to the
way LSB-compliance is currently achieved. This probably isn't enough
for the ISVs that want "bit-identical", but it could very well be enough
for the smaller ISVs, and that would be a significant step in itself.
As for what's in it for Debian, perhaps there could be multiple levels
of being "LCC-based". Level 1 might be to use the same binary packages.
Level 2 might be to use the same source packages, rebuilt according
using the member distro's mechanisms and/or policies. Level 3 might be
to use most or all of the LCC source packages, perhaps in a modified
state, while still remaining LSB-compliant. Each level probably implies
the loss of some certifications, but it provides more flexibility for a
member distro to make its own decisions without diluting what
it means for a distro to be "LCC-based", since that's clearly specified.
Note that I'm thinking out loud in that last paragraph. Any decision
like this will have to be made by the LCC itself. I suggest it as a
possibility primarily to make the point that nothing is set in stone.
Bottom line, even if Debian participation ends up only meaning 25% the
same at the source level, that's still a step. The smaller the
differences, the smaller the likelihood of multiple groups solving the
same problems independently of each other in potentially incompatible
ways. That can only benefit all of us. Less divergence is always good.
NEXT STEPS
>From my vantage point, the next steps fall into three categories: 1.
building an initial version of the core, and using that initial core to
better understand the compatibility issues we will face as regards
package namespace, configuration, and other differences; 2.
experimenting with mechanisms that will allow us to build RPM and .deb
binary packages from a single source package; and 3. understanding what
mechanisms and policies would need to exist in the LCC
to facilitate Debian involvement (primarily addressing #1 and #2 above).
1. We're in the process of using a debootstrap'ed sarge with the lsb
package installed to assemble the minimal Debian set of packages that
need to be installed to constitute an LSB-compliant system. We
are then building a database of the files each package installs,
resulting in a mapping from package names to file names, and building
a similar database of Fedora Core 3 package names to file names.
By cross-checking the databases, we will come up with two lists: 1. a
mapping of Debian to Fedora Core 3 package names (where the files each
installs are substantially the same); and 2. a list of files installed
by one or the other of Debian or Fedora Core 3 that are not present in
the other distro. This will give us an initial idea of both
the package namespace problem and areas where the two distros diverge.
2. We're in the process of experimenting with mechanisms to build RPMs
from Debian source packages; our first attempt, as I said above, is to
create an RPM .spec file that creates an RPM package using the binaries
generated by the standard Debian build process. Once we've refined this
technique, we'll build an RPM version of the Debian core built in step
#1 and use that RPM core for a more direct comparison of the differences
explored in the database comparison step. We'll also use the common
source package from step #1 to begin the process of dealing with the
differences as we better understand them (e.g., the creation of dummy
packages to solve the package namespace problem) and
to test our solutions against a real RPM-based distro, Fedora Core 3.
3. Meanwhile, the LCC will continue to flesh out its development
processes. The continuation of the discussion started on
debian-devel will be enormously helpful here. For example, Debian
policy can and should play a big role in shaping LCC policy, and
we can also ensure that whatever mechanisms are needed to enable
at least partial Debian participation are in place (e.g., the
ability to branch the source packages in some coordinated
fashion to include unsupported-architecture-specific changes).
That's enough for now. Again, please direct follow-ups to
lsb-workers@lists.progeny.com . Thanks.
--
Ian Murdock
317-578-8882 (office)
http://www.progeny.com/
http://ianmurdock.com/
"All men dream: but not equally. Those who dream by night in
the dusty recesses of their minds wake in the day to find that it was
vanity: but the dreamers of the day are dangerous men, for they may
act their dreams with open eyes, to make it possible." -T.E. Lawrence
Reply to: