Re: [Debconf-team] Rethinking the way travel sponsorship works
Hi
Philip Hands <phil@hands.com> writes:
> Moray Allan <moray@sermisy.org> writes:
>
>> On 2013-05-07 22:38, Gunnar Wolf wrote:
>>> tiago_: h01ger, we from team were kind of accused of not being clear
>>> about money allocation for travel sponsorship last year, so
>>> I've been thinking that the team should take care of
>>> procedures, but rating should be done by (semi)random people,
>>> in short.
>>
>> From subsequent discussion, I think we mostly agree: we should look at
>> doing something along the lines of what Phil has proposed in the past,
>> i.e. getting a more diverse group (not just debconf-team or our direct
>> friends) to vote on this.
>>
>> I don't think that having truly random (with equal probability for each
>> person) selection is the best idea, as I fear this could reduce to
>> finding the people who are most popular out of those who are most
>> visible -- many Debian people don't have a good idea of who is involved
>> beyond their own area of interest. (This danger would be especially
>> present if we, as has been mentioned as an idea, only asked each person
>> to vote for a few people who they think should be sponsored, rather than
>> asking them to rate almost everyone.)
>>
>> Similarly if we ask too many people to vote, I think they won't take
>> the task as such a serious responsibility, and won't perform it as well,
>> as if it's the team is kept fairly compact -- it's not rational to spend
>> a lot of time looking up information about people if your own rating
>> won't count for much.
>
> I think that's probably true if you were asking them to rate everyone,
> and would probably give results that were reliable at the extremes, and
> fuzzy in the middle. Such a process would effectively pick out the easy
> cases and not help with the hard decision of what the order in the
> middle of the pack should be, and where to draw the line in that pack
> between sponsored and non-sponsored.
>
> How about giving people a short list of people to rate, on a (largely)
> random basis, and then combining the results of these ratings?
>
> Something like the slashdot moderation system, say.
>
> If we know roughly what proportion of funded to unfunded folk we're
> aiming at, I'd think that rather than simply asking people to put
> applicants in an order we could instead say to them something like:
>
> We would like you to assume that you are responsible for allocation of
> $10k in travel sponsorship. Here are applicants asking for a total of
> $15k [or whatever] in sponsorship. Please allocate the sponsorship as
> you see fit, but treating the task as though the money were you're
> own. Do not feel that you must allocate all the funds if you think
> that the candidates are too weak to deserve sponsorship.
>
> choosing the numbers so that each judge only sees about 5 applications, say.
>
> Perhaps we could also allow people to indicate a difference between
> thinking that someone should be left unfunded and having no opinion
> either way.
>
> Given that each judge would only get to see a short list of
> applications, I'd think they should see full details of the applications
> (rather than anonomised versions) but only after having agreed to keep
> the detains private.
>
> We could do this in multiple rounds, focusing in on comparisons between
> people where the decision is still unclear.
>
> We could perhaps give people who get a lot of don't knows an opportunity
> to provide more information, but that's probably unfair on those that
> get roundly rejected and so don't get a second chance.
>
> I've no idea how one combines the results from such a collection of
> partial orderings, but I presume that methods exist.
>
> I think if we make it clear that there is no obligation to perform the
> selection, if you don't like the idea of an NDA, or just don't have
> time, we should be left with people that have sufficient motivation for
> the task.
IMHO the travel sponsorship rating process should stay as lightweight as
it is now. The process you propose just makes things much more
heavyweight. I'm quite convinced that the process will always be fuzzy
in some parts and the problem should not be solved by just mathematicaly
combining as many ratings as possible.
I also think that instead of selecting people that did not really
volunteer to do the job, we will get better results if the rating is
done by people that are ready to invest a significant amount of time.
This makes it less likely that rating is mostly done on popularity.
I agree that a diverse team of raters is important. And I also think
that we should avoid the case were the raters just self-nominate among
the core debconf-team. To avoid this we could launch a broder call for
help with rating to debian-devel-announce or debian-project where we
describe the task, teh requirements, the time investment it needs and
ask people to nominate themself.
I'm also not sure if we would be solving the right problem with this
process. Your underlying assumption seems to be that the current rating
system is not fair. After serving on the team last year I don't think
that unfairness or bias of the raters was a big problem. The team
already consisted of very different views and raters applied quite
different criteria. I have no knowledge about other years though.
To me the biggest problems last year were:
- Very different amount of information provided by the applicants. Some
provided quite detailed reasoning while others just wrote 1-2
sentences and only focused on parts of what I considered important for
my decision.
- Unclear rating criteria. The rating was done on two scales (amount
requested and benefit/involvement to Debian), but the team did not
have a clear view what each of these scales mean. This is not a
problem per se if all possible views are evenly represented in the
rating team (and this was mostly the case), but it made it quite
difficult for me to make my own ratings consistent.
Making the whle process fully public was already rejected in previous
discussions. I agree that this might be too much. But to balance
accountability towards the project with the need for privacy for those
requesting sponsorship I think we could at least announce the list of
persons who were granted travel sponsorship (without amounts). This
would allow for at least basic oversight of the process by outsiders.
Gaudenz
--
Ever tried. Ever failed. No matter.
Try again. Fail again. Fail better.
~ Samuel Beckett ~
Reply to: