On Sat, 2010-05-22 at 18:41 -0400, Daniel Kahn Gillmor wrote: > I'm concerned that the rating metric is wrong in penta when reviewers > leave one of the three categories in a "don't know" state. > > for example, if i say "+" for all three categories (relevance, > actuality, acceptance), the score will be 50. (50 + 50 + 50)/3 > > if i say "+" for relevance and acceptance, but "0", the score shows 37. > This seems slightly wrong -- i'd have expected 33. but meh, not a very > big deal. > > A bigger deal is if i say "+" for relevance and acceptance, but leave > actuality as unspecified (because i don't know anything about the > presenter), the score also drops to 37. I'd have expected it to stay at > 50: (50 + 50)/2 > > In particular, i'm proposing that unspecified axes should be removed > from both the numerator and the denominator, rather than just being > treated as a 0 in the numerator. > > Any thoughts on this? Am i misunderstanding something? Yeah, I agree with your assessment. It would definitely be better if the ranking differentiated between the 'ambivalent' and 'not ranked' cases. I know there were a few I deliberately left unranked because I felt others were likely to be better qualified than I for those cases. Cheers, Andrew. -- ------------------------------------------------------------------------ andrew (AT) morphoss (DOT) com +64(272)DEBIAN You will give someone a piece of your mind, which you can ill afford. ------------------------------------------------------------------------
Attachment:
signature.asc
Description: This is a digitally signed message part