[Debconf-discuss] Talk selection for Debconf 7
On 8 Jun 2006, Alexander Schmehl outgrape:
> Dear Manoj,
>
> [ written during a long "discussion" in #debian-women, which made me
> angry; rewritten, deleted and again rewritten the next morning; This
> mail is intended as constructive critique, not as personal attack
> ]
>
> * Manoj Srivastava <srivasta@acm.org> [060607 21:49]:
>
>> An interesting idea came up during debconf; now that we have
>> software for people to vote on talks, and indicate which talks they
>> would be interested in attending. With such a mechanism in place,
>> I think we can replace the academic committee, except perhaps to
>> fill a few slots left open for the organizers to give to less
>> popular but desrving talks a chance.
>
> I think you miss the "big picture behind that". The committees job
> is not only to select "good" from "bad" talks; is also to ensure a
> well balanced conference programm.
While a fine goal, it is a difficult, subjective one. And the
execution depends on experience, and judgement, and, as such,
leaving it to any one who volunteers may not be the best way of
selecting people who make such a decision. This is too much of a
burden to impose on the few who do stand up to be counted, and
volunteer their time.
> And as usual done on community projects, it's done in a doocratic
> way: Whoever feels capable and willing do the job, decissions are
> made after public discussion. That's it.
I am afraid that this might not be adequate criteria for peer
reviews and sophisticated juggling that determines what "balanced"
means. This is a huge, complex, task, and is quite quintessentially a
subjective decision; the more input that is considered, the better it
represents the views of the people attending the conference, more on
why it is important to do so below. I think expecting people who are
volunteering to be able to shoulder this burden is unreasonable.
> If you - or someone else - thinks we did a bad job, feel free to
> join the team for the next conference.
Err, I think the entire process of having a "team" which makes
such a judgement is the wrong mechanism, so just perpetuating the old
way of doing business is not good enough. Allow me to elaborate.
The selection of a set of talks is perhaps the most critical
single decision for a technical conference; the quality of the
conference, and the benefits to the community, stem from this single
decision.
Let us go to the basic rationale for having talks and why hte
project, and the sponsors, should care (the benefit of a number of
people meeting for rapid discussions, for creating and refreshing
personal relationships, and the innovations and ideas stemming from
off the cuff shooting the breeze sessions are beyond the scope of
this analysis).
Talks are essentially information transfer. To an individual
developer, talks help them in doing things for Debian. It is either
instructive, teaches them new things, it is in an area they are
interested in, and perhaps gives them an impetus to do something
new, something that perhaps had been on the back burner before they
heard the talk. Or it explains critical infrastructure for the
project, or critical processes, that help future contributors to the
infrastructure, or future assistants (for example, looking up build
log failures and fixing rc bugs).
In this case, no one knows better than the attendee what kind
of talks would help them improve debian; so collectively, voting for
talks gets us stuff that motivates/energizes more people; this is
something a small group of people is less likely to do well.
So how do we go about implementing this selection, and why not
select a cathedral select few that have the authority to decide and
exercise the judgement about what goes in the conference?
One of the interesting new processes about how to make
decisions based on incomplete data, and which involves subjective
judgement of the information that is available, is the market
mechanism. The collective judgement of the market, in which each
investor is trying to maximize their own return on investment,
outperforms the judgement of experts in the field most of the time.
As long as the "investors" are involved in the field (investing, or
security experts, or any domain involvement at all), and have some
incentive, the collective judgement is often better.
The people attending the conference are not uninformed -- we
are spending a considerable effort to attend the conference, and we
all are vested in improving Debian. We all have domain expertise in
what it takes to work on some part of Debian. This part of the
criteria fits.
We all have a personal incentive to not have our time wasted
by talks that one is uninterested in, and select talks that,
individually, would help us in our tasks for Debian; so the market
incentive exists. All attendees would be trying to maximize the
benefit of the conference for themselves (return on their investment
of time and effort). This criteria fits as well.
We can open up talk selection to the attendees, and improve on
the results, since collectively the group has a better judgement than
_any_ group of experts (not that we have a process of selecting for
expertise in the academic ctte -- it is mostly self selecting, small,
group, the common criteria being that one stands up to be counted,
and putting on their shoulder such a requirement is not good).
The concrete proposal is this: We already have software
in place that allows registered attendees to vote on proposed
talks. Every individual is allocated, say, $1000 in funny money (or,
1000 points, if you will), to allocate as they please between the
talks proposed. The amount of funny money (or points) that a talk
garners would determine a rough ranking. Timing issues can be
discussed (as in, deadline for abstracts, deadline for selection of
talks, deadline for full paper submission, etc).
The benefit of this process is that it is transparent, far
more democratic than open membership in a small group, and leverages
the distributed domain knowledge about which talk would serve the
attendees the best.
We can also reserve a few slots for the organizers to
"balance" the talk selection, and to allow for a few talks that are
deserving, but failed to meet the group selection thresholds.
There are a few potential lacunae in this approach: People may
vote for "popular" speakers. I am not sure that this is really an
issue: firstly, most people would want to minimize the number of
"boring" talks, so there is a self correcting incentive in place.
This is similar to some people having "favourite" stocks in the
market analogy -- because they work for the company, or Daddy worked
for so-and-so -- but overall, the market still outperforms experts.
Secondly, this is cathedral think: that a select few people, by
volunteering, have better judgement than the unwashed masses about
what is good for them. If the attendees collectively want to hear
Joey Hess speak, why should a small group thwart the wishes of the
many?
Secondly, it was noted that sponsorships are preferentially
awarded to speakers, and people may be able to sway enough others to
vote for their talk in order to attend the conference. Again, I feel
this is a low likely-hood corner case; I certainly am not going to
vote for a talk just so someone can bilk Debian of money. I strongly
suspect most of us would not, so this is not something I think we
really need to guard against. I would rather not presume that the
people voting are going to defraud the conference.
So, this is my proposal, and rationale, for improving the
selection of talks presented at debconf 7, and how to make the
process more transparent, go from the cathedral to the bazaar model,
involve more diverse viewpoints, and, by the nature of the problem
being solved, come up with more optimal results, though of course
that last part is hard to quantify.
Most people would want to skip the next part of this email;
but I do feel I have to respond to the personal attacks, in as
non-inflammatory manner that I can.
> But - now a bit more personal note - let me give you a small tip:
> IMHO this was a bad start, before you start to critisize our
> practise you should at least listen how things are done and learn
> why they are done this way.
A) I was there. I saw the talk selection, and read the abstracts,
including the talks thatwere rejected.
B) This is not rocket science. Submission of papers to peer
reviewed journals and professional conferences is not an obscure,
arcane process (unless you have never participated in one)
The proof of the pudding is in the eating of it. No one
expects a small team of volunteers to perform like the vast numbers
of reviewers that professional journals utilize; no small group has
the breadth of expertise required for this task. Expecting such from
a small group of volunteers is unsustainable. Nobody, but nobody,
does that; journal editors farm out that task to domain experts in
the field for each paper submitted to them.
> <irony> You know, just because we are younger than you, doesn't mean
> we do the thinks on a gut level. We have reasons to do things, the
> way we do. </irony> Which leads to the next point: Don't play the "i
> have more experience, because I'm older, so shut up" card. It
> doesn't work; even worse: It might even be seen as arogant by the
> people you try to "dicsuss" with.
My age has nothing to do with it. Neither does
yours. You really do not want to go there. Get the chip off your
shoulder, stop obsessing about your age, and look at the substance
of what is being proposed.
>
> During last nights chat, you barly gave me the chance to explain the
> reasons of our doings, after you asked for them! So I rather think
> you are not interested in our position, but mostly on discussing /
> trolling. Sorry, but I have other things to do than to discuss
> stuff that way.
I am interested in improving the future conferences, and the
the benefit to the project. The talk selection, and the importance
given to streaming video over hacking and collaborative development
are the two major flaws I, and others, felt existed at this
conference. I think we can do better, unless we take this all
personally and refuse to hear any proposals for improvement, and the
accompanying critiques of what could have been done better.
manoj
--
"Big Brother is hallucinating." Elizabeth D Zwicky
(zwicky@cis.ohio-state.edu), title of a comp.risks article
Manoj Srivastava <srivasta@acm.org> <http://www.datasync.com/%7Esrivasta/>
1024D/BF24424C print 4966 F272 D093 B493 410B 924B 21BA DABB BF24 424C
Reply to: