Re: Reforming the NM process
Marc 'HE' Brockschmidt <marc@marcbrockschmidt.de>
> 1. Current situation
Thank you for summarising.
> [...] and the excellent introduction given in the
> talk Hanna Wallach, Moray Allan and Dafydd Harries held last year [2].
> [2] http://www.srcf.ucam.org/~hmw26/publications/debconf5.pdf
I'll refer to this when outlining a "Suggestion:" below.
> 1.1 Known problems [...]
I'll stick to my experiences here:
> 1.1.1 Applicants [...]
Time delays and poor communication were most annoying.
> 1.1.2 Application Managers
>
> The lack of free Application Managers that led to the accumulation of
> applicants waiting for an AM is mostly based on the fact that many
> developers don't care about the NM process, so only a few people are
> actually helping out. [...]
I think it's jumping to conclusions to claim that developers don't
*care* about the NM process. The often-firey email discussions
suggest to me that some care, but could be frustrated or confused
by the current process and expressing that badly.
I started sponsoring with the intention of helping NMs and
reducing the long waits they will encounter during the NM
process before they can help easily. I've not asked to become
an AM for various reasons, including:
1. little idea that NM had a bad AM bottleneck as well as DAMnation;
2. seeming lack of interest from NM in sponsored work;
3. lack of belief in the NM process as a fit assessment.
Perhaps these could be avoided/overcome by:
1. NM committee communicating status to DDs in general more;
2. clearer information and relationship between sponsoring and NM;
3. making NM use a more common summative assessment model.
> 1.2 Already proposed solutions [...]
In general, I don't see any of these as a full solution,
but I feel that some are dismissed too lightly.
> 1.2.1 Add more people [...]
This problem may be a consequence of other ones. If NM is no
fun, only those with either a strong sense of duty or masochism
will join it. A correlation with sense of duty could explain
why AMs are "quite active developers in other areas". Fix the
other problems, then do a little volunteer recruitment (yes,
it helps to recruit clearly) and this might happen anyway.
> 1.2.2 Fewer checks [...]
I'd suggest different and less time-consuming, not fewer.
> 1.2.3 Drop Front Desk/merge with DAM [... no opinion ...]
>
> 1.2.4 Task-based checks
> ~~~~~~~~~~~~~~~~~~~~~~~
>
> Some people, including me, have discussed the possibility to use a
> task-based approach to the NM process. As far as I know, I'm the only AM
> who has actually finished this with an applicant. It was an interesting
> experience, more challenging for applicant and AM than the usual
> templates, but the amount of time needed for it is enormous. Also, it
> misses the best feature of the NM templates - comparability. Each
> applicant takes on other tasks, with other demands.=20
> After doing this once, I'd not recommend it as a regular replacement for
> the checks based NM templates we use at the moment, mostly because of
> its time needs.
Other qualifications use task-based assessment and yet are still
comparable and reviewable. What model did you use for your
task-based assessment?
> 1.2.5 More than one AM per applicant [...]
Most of the problems with this could be overcome by simple
measures, such as designating a "lead AM" in the team for
each applicant to keep responsibility. I'm not sure whether
it has enough benefit to be worthwhile, though.
> 1.2.6 Web-based checks
> ~~~~~~~~~~~~~~~~~~~~~~
>
> It was proposed to change the NM process to be based on simple HTML
> pages with some forms, checking for some things. This makes it quite
> easy to "cheat".
It's already quite easy to cheat the templates, isn't it?
Applicants with fast/free internet connections, local mirrors
and so on can do the bookwork needed for the template questions
without much pain. A lot of the questions repeatedly confirm
an ability to look stuff up and restate it, testing research
skills and language skills, rather than knowledge.
> Also, our current checks include a lot of free writing,
> discussing matters of philosophy, which won't be possible in a fully
> automatic system. The current questions also allow to educate NMs in
> areas they don't know much about.
Should NM itself be a mentoring system, though? Does it have
the resources to carry that function out properly? Is performing
that function delaying admission of ready-to-help DDs?
Now, onto the suggested solutions:
> 2. Possible solutions [...]
> 2.1 Multiple advocates [...]
Advocates seem pretty useless in the current system. The
history in Wallach Allan and Harries suggests it is partly a
simple filter and time-delay, while this suggestion seeks to
"encourage prospective advocates no to advocate too fast".
This does not seem a scalable solution for any of the problems
outlined so far, makes the time-delay for applicants worse and
may discriminate more on a social than a technical basis.
Can we reform advocates into something more useful?
****
Suggestion: Ask advocates to take on the formative/educational
part of the current AM role and prepare a summary in a given
format about the applicant. The summary could then be used as
the basis for simpler summative testing by an AM, with swift
referral back to advocate and applicant with direction, if the
AM or FD is not satisfied. The aims are:
1. to reduce the amount of context that AMs and FD need to keep
referencing and so speed up their work,
2. give a concise answer to "why was this person accepted as
a debian developer" for posterity (not the 25-80 pages that
Joerg Jaspert mentioned), and
3. to give DDs a low-entry-barrier way to help out and scale
the system up to meet demand, while maybe learning how to become
an AM themselves as a consequence.
****
I'm modelling that suggestion on my understanding of England's
Key Skills (KS) system, which uses task-based assessment
and splits the formative and summative educator roles. I
roughly translate applicant=student, advocate=tutor,
AM=assessor, FD=moderator and DAM=awarding body, or similar.
I worked as a KS tutor a few years ago. More info might be
http://www.qca.org.uk/6445_6485.html but it's not easy reading.
The article at http://www.tcarden.com/coursework/testassess.htm
isn't a bad introduction to testing jargon, at first glance.
> 2.2 Requiring (more) work before applying [...]
This would be a consequence of an AM role resplit. Only when
the advocate/mentor is happy should the application be made.
This has the benefit that no-one gets depressed at the size
of the NM queue, but maybe it merely makes the problem harder
to track. Would that cause problems?
After implementing the suggestion, any applicants without an
AM could be returned to their advocate and AMs could consider
either referring assigned applicants back, or becoming their
advocate if there's a part-completed report for the current
process.
> 2.3 Separate upload permissions, system accounts and voting rights
I agree this should happen, but I'd still carry other reforms.
This seems like a big and far-reaching project and doesn't
interfere with the Advocate/AM suggestion either way.
> 3. Conclusions
>
> Discussions about NM stuff in the past often happened without relevant
> people taking part. This has lead to proposals for solutions that often
> ignored the actual situation. I hope to avoid this problem with this
> mail, describing the problem from a position that gives a quite good
> overview.
Thank you for taking an active lead in this discussion.
Some short comments on other responses:
Pierre Habouzit <madcoder@debian.org>
> =46or 2.2, I'd recommend that NM's maintain a page about them on=20
> wiki.d.org (my current applicant did that, and I found that rather=20
> useful). In a glance you can see applicants that are not comited=20
> enough.
wiki.d.o should not be recommended until basic usability bugs
like "ImmutablePage" and non-WCAG CSS are fixed and there's
some generally-accepted way to track bugs about it.
Raphael Hertzog <hertzog@debian.org>
> Indeed, but in fact, when a DD sponsors someone, he carefully checks the
> 2-3 first uploads and then only does a superficial review... and it's not
> a big deal as long as the applicant is responsive and correct bugs as they
> get submitted.
Which DD are you describing there?
"Sponsoring merely by signing the upload or just recompiling
is definitely not recommended." -- s7.5.1 developers-reference
I check the changes between versions as carefully as any
new submission. I assume that my previous check was correct,
which is a weakness, which I address by re-examining the
whole package when time permits.
I think we should insist that sponsors are named in Uploaders
and debian/changelog, and probably Changed-By too.
[other email, same author]
> > 1.2.1 Add more people
> > ~~~~~~~~~~~~~~~~~~~~~
>
> Change that into "recruiting more people" and then it becomes a solution.
> People don't come alone ... :-)
>
> (cf my previous point of asking for new AM while doing a nm.d.o report on
> d-d-a)
I agree. Comments about the process would be a useful addition
to those reports.
So, are there drawbacks to moving some AM functions to the
advocates that I'm not seeing? On balance, would it be better
or worse than the current situation?
Good luck,
--
MJR/slef
Laux nur mia opinio: vidu http://people.debian.org/~mjr/
Bv sekvu http://www.uk.debian.org/MailingLists/#codeofconduct
Reply to: