On 05/22/2010 06:41 PM, Daniel Kahn Gillmor wrote: > I'm concerned that the rating metric is wrong in penta when reviewers > leave one of the three categories in a "don't know" state. [...] > Any thoughts on this? Am i misunderstanding something? OK, i've looked into this further. edrz pointed me toward the SQL that calculates the scores: sql/views/report/view_report_review.sql The old wiki page [0] description of the overall score was: >> The total rating is the average (arithmetic mean) of these three >> numbers. Of course, this isn't a perfect rating system, but that's why >> we don't use it directly. But that is untrue. I've updated the wiki page [0] to describe the actual calculation: >> We compute a talk's score in each category by taking the average >> (arithmetic mean) of all ratings of the talk in that category. A talk's >> total score is equal to (2*acceptance + actuality + relevance)/4. (so >> acceptance counts twice as much as the other categories). >> >> If no one has rated a talk in a given category (e.g. if everyone has >> left "actuality" unrated), that category's contribution to the total >> score is 0. I actually think this is a reasonable approach, i just didn't understand what it was doing. So i withdraw my earlier objection. To be clear, the nice features of this approach are: * the proportional contributions of the three categories to the overall score are independent of the number of ratings in each category. * if a category has no ratings at all, it is as though everyone rated it zero. If a category has one rating, that is the score used for that category. So if you don't know, you can let people who do know provide information without tainting their ratings. And if no one knows, then there is real ambivalence which is best represented for that category contributing 0 to the final score. * i like that acceptance is rated as much as relevance and actuality put together. hope this makes sense, --dkg [0] http://wiki.debconf.org/wiki/Pentabarf
Attachment:
signature.asc
Description: OpenPGP digital signature