[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Bug#668000: Scalability problem with large numbers of registered users

Package: python-moinmoin
Version: 1.9.3-3~bpo60+1
Severity: important
Tags: upstream

We've been experiencing performance problems for a while on
wiki.debian.org, which uses moin. Specifically, page saves take a
*very* long time, on the order of 90 seconds on a system that is
otherwise performing well. I've debugged into this and I've found the
problem: the code that works out which users need notifying about page

We've currently got 10,686 registered users on wiki.debian.org. strace
shows that moin is opening the user data file for every one of those
users in turn to check the subscribed_pages array. That takes a long
time! I've verified this is the problem - hacking saveText to disable
the notification code reduces the page save time from ~90 seconds to
~3 seconds.

This is clearly an upstream issue - it's a scalability bug in moin's
design. This area needs fixing in some way - maybe add a cache in
front of the user lookup here, or store the subscribed_pages
information differently. I might be able to help with coding this, but
I'd want to see what upstream plans might be first.

-- System Information:
Debian Release: 6.0.4
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.32-5-amd64 (SMP w/2 CPU cores)
Locale: LANG=en_GB.UTF-8, LC_CTYPE=en_GB.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash

Versions of packages python-moinmoin depends on:
ii  python                  2.6.6-3+squeeze6 interactive high-level object-orie
ii  python-parsedatetime    0.8.7-2          Python module to parse human-reada
ii  python-pygments         1.3.1+dfsg-1     syntax highlighting package writte
ii  python-recaptcha        1.0.5-1+squeeze1 client library for reCAPTCHA and M
ii  python-werkzeug         0.6.2-1          collection of utilities for WSGI a
ii  python2.5               2.5.5-11         An interactive high-level object-o
ii  python2.6               2.6.6-8+b1       An interactive high-level object-o

Versions of packages python-moinmoin recommends:
ii  apache2-mpm-worker [ht 2.2.16-6+squeeze6 Apache HTTP Server - high speed th
ii  exim4-daemon-light [ma 4.72-6+squeeze2   lightweight Exim MTA (v4) daemon
ii  fckeditor              1:2.6.6-1         rich text format javascript web ed
ii  python-xapian          1.2.3-3           Xapian search engine interface for
ii  python-xappy           0.5-4             easy-to-use interface to the Xapia

Versions of packages python-moinmoin suggests:
ii  antiword                  0.37-6         Converts MS Word files to text, PS
pn  catdoc                    <none>         (no description available)
pn  cifs-utils                <none>         (no description available)
pn  docbook-dsssl             <none>         (no description available)
ii  miscfiles [wordlist]      1.4.2.dfsg.1-9 Dictionaries and other interesting
ii  poppler-utils [xpdf-utils 0.12.4-1.2     PDF utilitites (based on libpopple
pn  python-4suite-xml         <none>         (no description available)
ii  python-docutils           0.7-2          utilities for the documentation of
pn  python-flup               <none>         (no description available)
pn  python-gdchart            <none>         (no description available)
pn  python-ldap               <none>         (no description available)
pn  python-mysqldb            <none>         (no description available)
pn  python-openid             <none>         (no description available)
pn  python-pyxmpp             <none>         (no description available)
ii  python-tz                 2010b-1        Python version of the Olson timezo
pn  python-xml                <none>         (no description available)
ii  wamerican [wordlist]      6-3            American English dictionary words 
ii  wbritish [wordlist]       6-3            British English dictionary words f

-- Configuration Files:
/etc/moin/mywiki.py changed [not included]

-- no debconf information

Reply to: