Re: Red Hat 5.0 Release date
On 20 Nov 1997, Jim Pick wrote:
>
> Dale Scheetz <dwarf@polaris.net> writes:
>
> > Testing is not closed! I actively recruited both from developers and users
> > for members of the testing team and ended up with what we got.
>
> Oops. I mostly meant that it didn't involve everyone. Poor choice of
> words.
This speaks to your expectations, and I'm afraid you will be disappointed
with the amount of involvement you can elicit from the developers.
>
> > My problem with your proposal is that it assumes that you (retorical you,
> > meaning someone) can get all of the developers on board with all the
> > controles you would need to manage verification of tests.
>
> I don't think the "organized developer testing" has to be too controlled.
But you seem to think that the current developer testing practices are not
sufficient. This implies the need for additional controles.
> But I think we could probably find and prioritize a huge bunch of bugs if
> we could organize things so that every package is looked at. Granted, the
> developers are always doing this - but I believe there are still plenty of
> bugs that go unnoticed, or unreported. I guess I want to use the developers
> to decide which packages pass/fail our criteria for going into stable -
> which is too big of a job for a small testing job.
>
These are the same developers who are currently doing all they can to find
the time to deal with their individual packages?
> > Yes, we need to encourage all developers to do extensive testing of their
> > individual packages, but the reason for centralized testing is that none
> > of the developers have the same machine. Last time, we all built base
> > systems and then installed all of "standard". This gave us identical test
> > benches for testing X and the other "ancillary" programs that we were able
> > to focus on. Expecting developers to do this is probably just as hard as
> > holding them to stable after a release. (which causes untold problems with
> > post-release fixes) It is my intention to provide every tester for 2.0
> > with a 1.3.1 CD. This will allow us all to do our "upgrade" testing on
> > identical systems. While this doesn't deal with all the problems we are
> > certain to see from systems that are "partially" hamm, it will be far more
> > informative about integration problems within hamm than the scatershot
> > testing by developers.
>
> I don't think the "scattershot testing" (good description) is going to be
> comprehensive at all - but I think it would definitely lighten the testing
> load on the release team. It won't catch all the bugs, but it would catch
> the really obvious ones, which unstable is full of. I'd estimate that
> 20-30% of the packages in unstable aren't ready to go into a stable release.
> In the past, we released them anyways. We shouldn't this time around.
>
That is perfectly reasonable, but quite frankly, finding unknown bugs in
current packages is no where near as important as fixing the bugs we
already know about.
> > Basically the QA team did very little of the testing.
>
> I must have been confused, I thought the QA team was the testing team.
>
Well Vincent R. did a heroic job with the limited resources that collected
around the QA team. They took critical bugs in hand and did non-maintainer
uploads that fixed them. This was a division of labor. The testing team
found problems that the QA team then attempted to resolve. From reports I
got from Vincent, he was a little jealous (read disapointed) of the
number of folks who joined testing compared to the number he had helping
with QA. Your plan could go a long way toward helping this effor if it can
focus more effort on fixing those bugs that are on critical paths for the
integration process.
> > > The Release Team's job should be easy, if we have a round of developer testing
> > > first, which uses the bug system to identify broken packages. The release
> > > team can monitor this, and prioritize what needs to be fixed.
>
> > I think that this is the fundamental part where your idea breaks down.
> >
> > We are supposedly doing "developer testing" every time a package gets
> > released. The only thing that the testing team did last release that was
> > complete was "integration testing". If all the developers could do
> > complete regression testing on their packages individually, the your
> > statement about the easy job the release team has would be correct.
>
> I don't think we can rely on each individual maintainer to do a good job
> of testing. Some will, some won't, and some are on holidays...
>
Keep in mind it is these same maintainers that you wish to enroll in your
"organized testing". Don't these same deficiencies apply to your plan as
well?
> The "integration testing" stuff is tricky, since that has to mostly come
> at the end of the release process. That has to be done by a small team,
> like what you organized last release, for sure.
>
Well, I would say that integration testing is "everything". I don't mean
to imply that we don't have excellent programmers, writing needed and
useful code for the project, but constructing a distribution is much more
about package integration than it is about coding those packages and
making them work within their own limited context. One of the great
beauties of the packaging system is that it focuses on these integration
problems. The part that needs testing comes from the human frailties of
the various packagers. The whole concept of a code freeze speakes to these
issues.
> > > There is a lot of work in organizing the developer testing - I'll volunteer
> > > for that job, if this plan goes ahead.
> > >
> > I have absolutely no objection to your doing this!
> >
> > In fact I encourage you to do so. You will then get some first hand
> > experience with the difficulties I have been (somewhat poorly) trying to
> > communicate to you, and we would get some kind of improvement in the
> > testing process.
> >
> > YES! Go for it!
>
> Ok, I will, boss. :-)
>
Boss? If you meant what you said, that is your title now ;-)
You can count on me for whatever help the "old" testing team can provide.
> > > We should write a "Release/Testing Strategy Manual" to codify everything so
> > > there aren't any misunderstandings this time around.
>
> > While this is not a bad idea (outside of who writes the damn thing) it
> > doesn't come close to eliminating misunderstandings (see current DFSG
> > threads ;-)
> >
> > While mistakes were made in the last release, most of the ones I can think
> > of had to do with mistakes in judgement rather than misunderstanding of
> > proceedure.
>
> I found that most of the misunderstandings I saw were based on differing
> interpretations of testing terminology. That's why I think a manual might
> help.
>
No problem ;-)
> > > So, when are we going to release 2.0, if ever? We should announce a date,
> > > and stick to it. Frankly, I think we've been in good shape since August
> > > to make a release.
> >
> > I see no reason that we can't make code freeze by the first of the year
> > and release by 1 Feb. This seems to fit your schedule, if not in the
> > details. With Thanksgiving comming up and Christmas close behind I see no
> > way to organize anything at all tight before then.
>
> > Personally I don't see what a quick release turnaround has to do with the
> > development model, but I'm known for being dense at times ;-)
>
> Less time testing/releasing (no fun) = more time developing (fun)
>
If that is truely how you see this, then you are in for some (no fun).
Personally, I took on the testing project because I have some skill in
that area, but more because I _do_ find testing a fun occupation. I think
most of the folks on the testing team would have to say that they had fun
doing their jobs as well.
The other problem I have with your equation is that it ignores the fact
that the same group is doing the work on both sides of the equal sign, so
the RHS is only true if total available time is unlimited, which it isn't.
(at least not for me)
> Maybe we should aim for Feb. 15?
>
I would say "aim" for 1 Feb. so we have a good chance of making a 15 Feb.
release. Don't forget all those last minute failures that push the release
date. (down servers and broken mirrors...seven years bad luck?)
I feel compelled to say that I have enjoyed our conversation on this issue
and look forward to helping you impliment your ideas.
Luck,
Dwarf
--
_-_-_-_-_-_- _-_-_-_-_-_-_-
aka Dale Scheetz Phone: 1 (904) 656-9769
Flexible Software 11000 McCrackin Road
e-mail: dwarf@polaris.net Tallahassee, FL 32308
_-_-_-_-_-_- If you don't see what you want, just ask _-_-_-_-_-_-_-
--
TO UNSUBSCRIBE FROM THIS MAILING LIST: e-mail the word "unsubscribe" to
debian-devel-request@lists.debian.org .
Trouble? e-mail to templin@bucknell.edu .
Reply to: