Bug#668000: Scalability problem with large numbers of registered users
Package: python-moinmoin
Version: 1.9.3-3~bpo60+1
Severity: important
Tags: upstream
We've been experiencing performance problems for a while on
wiki.debian.org, which uses moin. Specifically, page saves take a
*very* long time, on the order of 90 seconds on a system that is
otherwise performing well. I've debugged into this and I've found the
problem: the code that works out which users need notifying about page
changes.
We've currently got 10,686 registered users on wiki.debian.org. strace
shows that moin is opening the user data file for every one of those
users in turn to check the subscribed_pages array. That takes a long
time! I've verified this is the problem - hacking saveText to disable
the notification code reduces the page save time from ~90 seconds to
~3 seconds.
This is clearly an upstream issue - it's a scalability bug in moin's
design. This area needs fixing in some way - maybe add a cache in
front of the user lookup here, or store the subscribed_pages
information differently. I might be able to help with coding this, but
I'd want to see what upstream plans might be first.
-- System Information:
Debian Release: 6.0.4
APT prefers stable
APT policy: (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.32-5-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages python-moinmoin depends on:
ii python 2.6.6-3+squeeze6 interactive high-level object-orie
ii python-parsedatetime 0.8.7-2 Python module to parse human-reada
ii python-pygments 1.3.1+dfsg-1 syntax highlighting package writte
ii python-recaptcha 1.0.5-1+squeeze1 client library for reCAPTCHA and M
ii python-werkzeug 0.6.2-1 collection of utilities for WSGI a
ii python2.5 2.5.5-11 An interactive high-level object-o
ii python2.6 2.6.6-8+b1 An interactive high-level object-o
Versions of packages python-moinmoin recommends:
ii apache2-mpm-worker [ht 2.2.16-6+squeeze6 Apache HTTP Server - high speed th
ii exim4-daemon-light [ma 4.72-6+squeeze2 lightweight Exim MTA (v4) daemon
ii fckeditor 1:2.6.6-1 rich text format javascript web ed
ii python-xapian 1.2.3-3 Xapian search engine interface for
ii python-xappy 0.5-4 easy-to-use interface to the Xapia
Versions of packages python-moinmoin suggests:
ii antiword 0.37-6 Converts MS Word files to text, PS
pn catdoc <none> (no description available)
pn cifs-utils <none> (no description available)
pn docbook-dsssl <none> (no description available)
ii miscfiles [wordlist] 1.4.2.dfsg.1-9 Dictionaries and other interesting
ii poppler-utils [xpdf-utils 0.12.4-1.2 PDF utilitites (based on libpopple
pn python-4suite-xml <none> (no description available)
ii python-docutils 0.7-2 utilities for the documentation of
pn python-flup <none> (no description available)
pn python-gdchart <none> (no description available)
pn python-ldap <none> (no description available)
pn python-mysqldb <none> (no description available)
pn python-openid <none> (no description available)
pn python-pyxmpp <none> (no description available)
ii python-tz 2010b-1 Python version of the Olson timezo
pn python-xml <none> (no description available)
ii wamerican [wordlist] 6-3 American English dictionary words
ii wbritish [wordlist] 6-3 British English dictionary words f
-- Configuration Files:
/etc/moin/mywiki.py changed [not included]
-- no debconf information
Reply to: