Bug#682010: #682010 re celt and mumble referred to the TC
On Thu, Jul 19, 2012 at 12:55:59PM +0100, Ian Jackson wrote:
> Ron writes ("Re: #682010 re celt and mumble referred to the TC"):
> > You understand this is a fairly arbitrary 'daily' snapshot of an
> > experimental research codec, from ~2.5 years ago, that nobody has
> > looked at since that day, that nobody has committed to maintaining,
> > that its author has explicitly said he has no interest in maintaining,
> > that has had large parts completely rewritten since for all manner of
> > reasons, and is bitstream incompatible with every other snapshot that
> > was ever released, right?
>
> However, it appears that some other distros have chosen to ship this
> particular version. For example at least Ubuntu are still
> distributing it.
I don't want to niggle over words here, but "chosen" would imply that
somebody actually exercised some conscious judgement in that decision.
Which there doesn't really seem to be a whole lot of evidence for.
Nobody from Ubuntu has ever spoken to me or upstream about this, and
the version they are shipping appears to simply have been auto-imported
from Debian with no changes or human intervention. There are open bugs
in launchpad like:
- LibCelt Package Breaking Apt-Get
- package libcelt0-0 0.7.1-1 failed to install/upgrade:
Which nobody has commented on or triaged.
So the only rational conclusion there is that Ubuntu will do whatever
we do - and naturally they will then lag somewhat. If we are to insist
on not changing things until they do, then we're going to be deadlocked
shipping obsolete, unmaintained, code for a long time ...
If they *had* asked, they would have got the same advice on offer here.
(and yes, those are almost surely bogus bugs - but that just reinforces
the point that not even minimal attention is being paid to this there)
However if this list is any indication:
https://bugs.launchpad.net/ubuntu/+source/mumble
Then the old version of mumble that Ubuntu is shipping looks long overdue
for an update anyway, since there's a long list of reasons there why its
users can't communicate with anyone at all, none of which have anything
to do with celt ...
> > > - This is a serious problem for mumble at least and is arguably RC.
> >
> > Yes, mumble has a serious problem, that is arguably RC.
> > In fact it has several of them aside from this corner that people
> > have painted themselves into with it ...
> >
> > - Its primary author, who is also a DD and comaint of the package,
> > is currently MIA with a new job that has consumed pretty much
> > every minute of his time for the last 2 years or so now.
> >
> > The people who remain that are hacking on it, can't even do a
> > formal new release at present, because he is the only one with
> > access to the systems they need to do that, and is not available
> > to do so.
>
> I'm not convinced by this complaint. The whole point of free software
> is that it is possible to carry on without needing the cooperation or
> involvement of one's upstreams.
Sure, but in order for that possibility to be real, someone has to
collapse the waveform and step up to do the work. So far nobody has
stepped in to fill Thorvald's shoes. I only stepped up to help with
the packages because I consider him to be a friend (and indeed I also
advocated him to NM because I consider him a prolific and talented
programmer - they aren't easy shoes to fill by a part time dabbler).
And I don't want to sound like I'm taking potshots at Ubuntu here
(because absolutely I am not) but this seems directly relevant to
your first point above, because maintenance of mumble there appears
to have completely stopped dead in its tracks when Thorvald vanished
off there scene there too, and they haven't yet replaced him either.
The important question for the release is reality, not theoretical
possibilities. The simple reality is, he's not easy to replace, and
so far hasn't been. And the result of that is clearly showing.
> Also, are you saying you think mumble should be removed ? Now is a
> bit late to be deciding this.
I'm saying that what I've seen in the short time since I stepped into
this tarpit, along with the rush of RC bugs like #675955 and #676816
and things like #628847, #615492, #673602 -- all of which have nothing
to do with anything I changed -- don't exactly inspire confidence in
its release readiness to me.
I'm not saying they can't be fixed. But given that #676816 was a
guaranteed crasher since the day it was introduced in March 2010 and
only just noticed in the last month - I'm not terribly confident that
we've seen the last of this sort of thing.
I talked through this with people from the release team as the freeze
deadline loomed -- deciding to remove it earlier would have been
premature. But you're quite correct about the 'lateness' in the cycle
limiting our choices.
> > If I'd known that Thorvald was not going to be here to manage this
> > transition for Wheezy, I'd have never agreed to shipping libcelt in
> > the Squeeze release either, and would have instead kept it in sid
> > only. If I'd known that his plan to have all other distros ship
> > the 0.7.1 release as a temporary interoperability snapshot would
> > fail as dismally as it did (no other distro shipped this version
> > except debian derivatives who took it from us), I'd have similarly
> > vetoed the idea of shipping this as a public library in the last
> > stable release too.
>
> I think interoperability with the ecosystem of our derivatives is
> sufficiently important that it's worth preserving.
We don't need an 'exclusive to us' version of celt to have that.
> > Mumble already ships this as an embedded private library on every
> > system other than direct Debian derivatives.
>
> I'm not sure exactly what you mean. Do you mean they support some
> other version of the celt codec ? Or are you just talking about how
> they manage the packaging ?
I mean mumble embeds something like seven mutually incompatible versions
of celt, all of which it provides as private implementation libraries
on every platform except Debian - because nobody else provides the version
that we do. And we only provide that version because upstream tagged the
current head on a random day when Thorvald and I said, "if we're going to
do this, we need it today".
The idea was, he was going to try to get everyone else to use it too.
But that failed completely, so all we have now is this random Debian-
specific snapshot, that nothing and nobody else uses or interoperates
with, and which mumble has to embed anyway for every other user.
It really is time to just admit that was a mistake, and correct it.
> I certainly think it's better to have the celt codec, even if it's an
> unreleased and now-unmaintained-upstream snapshot, in a separate
> package, than in an embedded library.
If there were other things that should be using it, I'd agree.
But there are not, and they very much should not.
I was shocked at the number of people who had no idea what celt was
or what it did, or that it broke bitstream (and usually API) on every
release, and was purely a research codec -- who had blindly added
support for it "just because it was there". They were all genuinely
stunned when I told them there were no compatibility guarantees being
made about it until the research phase was over. The first I'd ever
hear from them was when it did break their code and they reported
that as a bug ...
As a general rule, I'm all for letting people shoot themselves in the
feet as a path to learning -- but there are very good reasons why
libraries hide private symbols, and applications hide private libraries,
and this is exactly one of them.
Exposing this for another whole release when we don't need to, and gain
nothing useful from doing so, doesn't make any sense now.
If I ever upload another experimental codec, it will definitely be
handled very differently to how this one was. There are lessons to
be learned here that won't be forgotten easily.
> > I see that this proposal has already resulted in Chris skating down
> > the slippery slope of "let's re-enable it for everything else too".
> > And we'll get people with no experience or prior involvement to
> > maintain it, and we'll enable multi-arch too, and ...
>
> Your message has too much personal animosity in it.
I don't see where you read that here? (though I don't deny my frustration
at being called upon to repeat the same things over and over leaked out
badly in a few places in the other bug) Or had you not actually read his
message at the time you replied to mine?
The above seems like a fairly simple summary of fact, and indeed the
release team independently quickly converged on consensus horror at
precisely those three elements shortly after I hit send, and before
anyone there had the chance to see my message bemoaning them ...
I'll not niggle over words if you have a more PC way you'd like to say
that too, but the facts as stated still remain ... and those were bad
ideas layered on bad ideas. Sorry. But this is exactly the sort of
thing that I expect to ensure
I can put my hand on my heart and say all I care about is getting to
the bottom of the technical questions here, as quickly as possible.
We have a release to get out. There is lots of Real Work still to do.
> > So I really hope that it's actually clear to the people presiding
> > over the tech-ctte, that the *whole point* of a codec is that it
> > lets you *interoperate* with others. Which this snapshot does not.
>
> It lets you interoperate with Ubuntu, and with users running squeeze.
We don't need an 'exclusive to us' version of celt to have that.
> > If people really want to do this for mumble, then it really ought
> > to be done as a private implementation detail, like it is for every
> > other platform - not by setting traps in public for the unsuspecting
> > and otherwise uninformed. If we ship celt publicly, we are sending
> > the message that people should use it. That's the wrong message
> > for an obsolete, unmaintained codec, and there is no reason to tie
> > 'being pragmatic' about the mumble screwup to making an even bigger
> > mistake. That's not 'avoiding risk', it's "not avoiding risk".
> > And one that is really, really trivial to avoid.
>
> Just to be clear, you seem to be suggesting that while reintroducing
> libcelt as a package is a bad thing, it would be fine to reintroduce
> it as an embedded library in mumble ?
No, I'm not saying that would be 'fine', otherwise I'd have done it
already. I think it's a terrible idea.
All I'm saying is that if you are going to use the big hammer and
insist on celt being reintroduced, despite all of the clouds hanging
over it - this would be the least stupid way of doing something stupid,
that exposes the least number of people to it. (no personal animosity
intended here either, but it seemed like your first instinct in your
first reply was to lean that way, so this was simply a damage mitigation
suggestion if you weren't going to be swayed from that line of thought)
I know I definitely won't be exposing any system I own to celt 0.7.1
traffic - and I would consider it quite irresponsible for Debian to
expose its users in that way too for the life of another release.
What people choose to do for themselves, is their choice and they
are free to make it as they please. But if we ship this, we are
saying "we believe it's ok". And I don't believe that is true.
I've been lurking among the people who developed celt for many many
years now -- so if somebody wants to try to convince me that they have
both the technical expertise and the time to audit and maintain this,
then that would be most wonderful - we have a small mountain of other
challenging codec work we'd love to put you to work at too!
But I don't honestly believe that is actually going to happen beyond
lip service. All the people I know who I would trust to do this are
already very very busy working on things that actually do have some
sort of real future still ... All the people with an actual vested
interest in maintaining this for mumble I've asked, and they've all
been very explicit in their refusal to take on the responsibility of
being its maintainer beyond "we'll apply a patch if someone else does
all of the work".
This isn't something you can just fuzz test and go "oh it's fine".
We were only a few bits in the bitstream away from being able to
produce the test vectors with a PRNG. Almost any sequence is a
'valid' bitstream, but the permutations of code path and state are
vast. The crashers that have been found have all been found through
intimate knowledge of how to set up a very precise chain of events.
Nobody has gone back to analyse the 0.7.1 code with them in mind,
because nobody actually cares to maintain it, and we were racing
forward to get a frozen bitstream out. When the people who did that
work tell me they think there are unfixed issues in that version,
I'm not going to be fool enough to handwave them away. I've seen
first hand how rarely they tend to be wrong about these things.
Given how easy this really is to fix without creating that kind of
exposure, I'm a bit lost for words at the push back it seems to be
getting from some quarters ...
> > My general feeling is that mumble is currently in an awkward state
> > which really doesn't make it a good release candidate for Wheezy,
> > and we'd probably be best served by simply dropping it from testing,
> > fixing this in sid as the fixes come or are needed, and then pushing
> > snapshots to bpo as we have reasonable candidates for that.
>
> I don't think this is a good approach. But I'm pleased to see that
> you agree that this is probably an RC problem.
It doesn't sound ideal to me either, but if mumble is going to need
updates more often than we make point releases, and with code changes
larger than can reasonably reviewed for that - then I'm not really
sure what other realistic options we have.
I'm open to ideas which might actually fly though.
Cheers,
Ron
Reply to: