Re: Red Hat 5.0 Release date
On 20 Nov 1997, Jim Pick wrote:
>
> Dale Scheetz <dwarf@polaris.net> writes:
>
> > Aside from the fact that you have ignored the testing team in your
> > schedule (I don't really have a problem with this if you can put a better
> > team in its place)
>
> It was just a first stab at things. I propose renaming the Testing team
> to the "Release Team", and they would be responsible for producing the
> final release and doing the final release testing. They take over 100%
> control of the distribution after the developer testing phase ends. Of
> course, they can be active beforehand.
>
> We need an organized developer testing phase. A small testing team, like
> what we used last time, doesn't have any reasonable way of testing 1500+
> packages. However, it does make sense to have a small close knit team to
> do final release installation/upgrade testing.
>
It was always my hope that the testing team would grow and eventually be
able to address all of the packages in the distribution. We felt pretty
lucky last time to have enough manpower to do "core" package testing.
We did very well at new installation testing. Many different machines, and
everyone on the same page. It was only when we tried to spread ourselves
thinly over the rest of the packages that things sort of fell appart.
> > This schedule has far too many milestones that are far
> > to close together. Several of your important milestones are less than 5
> > days appart. Experience with the testing team suggests that this is enough
> > time to do one simply round of boot disk testing and little else.
>
> That's why I called it "fast track". Maybe we aren't capable of doing
> that. But if we let the release slide a few more months, Debian is going
> to suffer from the lack of momentum. We're already open to lots of
> criticism - just try raising some capital for a Debian-based business,
> and you'll know what I'm talking about.
>
> I think the tight schedule might work if everybody is working at it
> together in a coordinated manner. For our last release, the left hand
> didn't know what the right hand was doing. The looser release schedule
> just meant things were allowed to slide around more. Overall, we did
> a good job - but we ought to try to do better.
>
Always! The trick is to realize what can and can't be done in a volunteer
anarchy.
> > As to the overall schedule, it seems to me that if we can get to code
> > freeze by the first of the year (you have a similar milestone here) that
> > we can release by Feb 1.
>
> OK - just a bit of clarificition. I proposed a "new packages" code-freeze
> for Dec. 15, so that organized testing by developers would start from there.
> Somebody (probably a team) would organize who tests what before that - so
> that "developer" testing could start immediately.
>
> That would continue until Jan. 10th, when the core release team would
> "take possession". Hopefully, all the major bugs will have been identified,
> and mostly fixed by then.
>
Dream, dream, dream ;-)
Even though we all work at our best pace, the bug list grows faster than
we seem to be able to smash them. While bug free is an ideal we should
strive for, we should be realistic enough to realise that failure to reach
that goal is not failure.
> That is a pretty short time scale for developer testing. Perhaps we should
> add another week. The dates I picked were pretty arbitrary, and should
> probably be aligned with the weekends.
>
> I'm arguing for a compressed testing period, because it's not good to expect
> the developers to hold all their systems in a frozen, limbo state for several
> months while we do unorganized testing.
>
> > My feeling about the last release and its vast improvement over 1.2 was
> > that it was primarily the result of our first attempt at a Managed testing
> > proceedure. While we have a long way to go with the organization of the
> > testing team, pulling a global reorganization of the release proceedure at
> > this point will not be helpful to this effort.
>
> I agree with your first point - disagree with the second. I'm going to offer
> some constructive criticism. Please don't take it the wrong way.
>
> Although I thought your testing team did improve the release, I don't think
> it accomplished the goal of a bug-free release. I think the problem was
> that the testing team was a small, closed group that accepted responsibility
> for the entire release. It's just too large of a testing job to not
> involve everybody. We've got 1500+ packages - if each package is tested
> 5 times - that's ~8000 tests, or roughly 40 tests each (based on 200
> developers). A core group of 10 QA testers would have to do 800 tests
> each in order to accomplish the same thing. That's impossible for volunteers
> on a tight release schedule.
>
See above for "bug-free" discussion.
Testing is not closed! I actively recruited both from developers and users
for members of the testing team and ended up with what we got.
My problem with your proposal is that it assumes that you (retorical you,
meaning someone) can get all of the developers on board with all the
controles you would need to manage verification of tests.
Yes, we need to encourage all developers to do extensive testing of their
individual packages, but the reason for centralized testing is that none
of the developers have the same machine. Last time, we all built base
systems and then installed all of "standard". This gave us identical test
benches for testing X and the other "ancillary" programs that we were able
to focus on. Expecting developers to do this is probably just as hard as
holding them to stable after a release. (which causes untold problems with
post-release fixes) It is my intention to provide every tester for 2.0
with a 1.3.1 CD. This will allow us all to do our "upgrade" testing on
identical systems. While this doesn't deal with all the problems we are
certain to see from systems that are "partially" hamm, it will be far more
informative about integration problems within hamm than the scatershot
testing by developers.
> My impression was the core testing group basically did run-throughs on the
> basic installation and upgrades. This is very essential and we still
> need to do that.
>
> My concern was that major bugs (such as the xdm bug) slipped through - and
> afterwards you said you never tested that.
While that is not a completely accurate statement (the never tested part
in particular), it is true, and will always be true as long as humans do
this work, that something will always slip through. (Most likely to be
found by the first user to install the software, and in the first 5
minutes of opperation ;-)
> Basically, the QA team only
> tested a subset of the Debian product - which is good, but I think we
> need to do more if we are going to claim a fully tested product.
>
Basically the QA team did very little of the testing. They did a great job
(considering the help that Vincent didn't get) of fixing bugs when
maintainers were unavailable to do so.
Yes, testing only tested the "standard" system and several of the X
servers with any degree of completeness...
> I'd rename the QA team to the "Release team". They'd do release testing,
> and decide which packages go into the release, and which stay out. Last
> time, all the packages went into the release - which shouldn't happen,
> because many of the packages were noticeably broken.
>
> The Release Team's job should be easy, if we have a round of developer testing
> first, which uses the bug system to identify broken packages. The release
> team can monitor this, and prioritize what needs to be fixed.
>
I think that this is the fundamental part where your idea breaks down.
We are supposedly doing "developer testing" every time a package gets
released. The only thing that the testing team did last release that was
complete was "integration testing". If all the developers could do
complete regression testing on their packages individually, the your
statement about the easy job the release team has would be correct.
> There is a lot of work in organizing the developer testing - I'll volunteer
> for that job, if this plan goes ahead.
>
I have absolutely no objection to your doing this!
In fact I encourage you to do so. You will then get some first hand
experience with the difficulties I have been (somewhat poorly) trying to
communicate to you, and we would get some kind of improvement in the
testing process.
YES! Go for it!
> We should write a "Release/Testing Strategy Manual" to codify everything so
> there aren't any misunderstandings this time around.
>
While this is not a bad idea (outside of who writes the damn thing) it
doesn't come close to eliminating misunderstandings (see current DFSG
threads ;-)
While mistakes were made in the last release, most of the ones I can think
of had to do with mistakes in judgement rather than misunderstanding of
proceedure.
> > While your schedule would probably work in a tightly controlled effort, I
> > don't think it will work here.
>
> Our releases should be a tightly controlled effort.
>
Only in a corporation or the military. We don't come close to either of
these. (Maybe churches ...)
> So, when are we going to release 2.0, if ever? We should announce a date,
> and stick to it. Frankly, I think we've been in good shape since August
> to make a release.
I see no reason that we can't make code freeze by the first of the year
and release by 1 Feb. This seems to fit your schedule, if not in the
details. With Thanksgiving comming up and Christmas close behind I see no
way to organize anything at all tight before then.
>
> We still need to be very organized, even if we extend the release date a
> few more months. I don't really want to keep my system in a code freeze
> for 2-3 months.
This is a problem for may people, specialy folks who rely on others for
their hardware use. I have several partitions (655meg) that I use for
building CDs on, so I only need to clear one off to be able to set up a
"test" system. I certainly don't do testing on my development partition,
and neither do any of my testers (that I know of). This puts special
conditions on testers that all developers aren't going to be able to deal
with.
>
> If we can't pull off a quick release turnaround - that's basically saying the
> Debian open development model is irretrievably broken, and we all might as
> well switch to Red Hat.
>
I've heard this before every release I have worked on for Debian. I don't
really know how to speak to this except to work to get my packages in
order and do what I can to help get things right. That's what gets the
beast out the door.
Personally I don't see what a quick release turnaround has to do with the
development model, but I'm known for being dense at times ;-)
As an aside, does anyone know what the package count is currently in hamm?
Luck,
Dwarf
--
_-_-_-_-_-_- _-_-_-_-_-_-_-
aka Dale Scheetz Phone: 1 (904) 656-9769
Flexible Software 11000 McCrackin Road
e-mail: dwarf@polaris.net Tallahassee, FL 32308
_-_-_-_-_-_- If you don't see what you want, just ask _-_-_-_-_-_-_-
--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-devel-request@lists.debian.org .
Trouble? e-mail to templin@bucknell.edu .
Reply to: