Re: [Debconf-team] penta ranking calculation metrics
(for tracking purposes, forwarded to pentabarf@debconf.org, which goes
to the RT instance)
On Thu, May 27, 2010 at 12:40:20PM -0400, Daniel Kahn Gillmor wrote:
> On 05/22/2010 06:41 PM, Daniel Kahn Gillmor wrote:
> > I'm concerned that the rating metric is wrong in penta when reviewers
> > leave one of the three categories in a "don't know" state.
> [...]
> > Any thoughts on this? Am i misunderstanding something?
>
> OK, i've looked into this further. edrz pointed me toward the SQL that
> calculates the scores: sql/views/report/view_report_review.sql
>
> The old wiki page [0] description of the overall score was:
>
> >> The total rating is the average (arithmetic mean) of these three
> >> numbers. Of course, this isn't a perfect rating system, but that's why
> >> we don't use it directly.
>
> But that is untrue.
>
> I've updated the wiki page [0] to describe the actual calculation:
>
> >> We compute a talk's score in each category by taking the average
> >> (arithmetic mean) of all ratings of the talk in that category. A talk's
> >> total score is equal to (2*acceptance + actuality + relevance)/4. (so
> >> acceptance counts twice as much as the other categories).
> >>
> >> If no one has rated a talk in a given category (e.g. if everyone has
> >> left "actuality" unrated), that category's contribution to the total
> >> score is 0.
>
> I actually think this is a reasonable approach, i just didn't understand
> what it was doing. So i withdraw my earlier objection.
>
> To be clear, the nice features of this approach are:
>
> * the proportional contributions of the three categories to the overall
> score are independent of the number of ratings in each category.
>
> * if a category has no ratings at all, it is as though everyone rated
> it zero. If a category has one rating, that is the score used for that
> category. So if you don't know, you can let people who do know provide
> information without tainting their ratings. And if no one knows, then
> there is real ambivalence which is best represented for that category
> contributing 0 to the final score.
>
> * i like that acceptance is rated as much as relevance and actuality
> put together.
>
> hope this makes sense,
>
> --dkg
>
> [0] http://wiki.debconf.org/wiki/Pentabarf
>
> _______________________________________________
> Debconf-team mailing list
> Debconf-team@lists.debconf.org
> http://lists.debconf.org/mailman/listinfo/debconf-team
Reply to: