Re: mailman3 hyperkitty_import reports sqlite3.OperationalError
Hi,
On Thu, Jul 21, 2022 at 02:39:53PM -0500, kjohnson@eclypse.org wrote:
> re: 4. mailman 'import21' appears to work:
>
> $ mailman import21 tl2@example.com /var/lib/mailman/lists/tl2/config.pck
> root@example.com is already imported with role MemberRole.owner
>
> re: 4. 'hyperkitty_import' fails:
>
> /usr/share/doc/python3-django-hyperkitty/examples$ python3 \
> /usr/share/doc/python3-django-hyperkitty/examples/manage.py \
> hyperkitty_import -l tl2@example.com \
> /var/lib/mailman/archives/private/tl2.mbox/tl2.mbox
It is absolutely imperative that you run all "mailman" commands as
the mailman user (on Debian this is "list") and all django commands
as the web server user (on Debian this is www-data). Doing otherwise
(as you have done) will probably not work and will possibly break
things, as it may create files as the wrong user, which mailman may
later not be able to manipulate.
Also, on my bullseye mailman3 install, manage.py is at
/usr/share/mailman3-web/manage.py, NOT in a document examples
directory.
My process looked like this:
$ sudo -u list mailman import21 ${listname}@${mailman_host} /path/to/${listname}/config.pck
$ sudo -u www-data /usr/share/mailman3-web/manage.py hyperkitty_import -l ${listname}@${mailman_host} /path/to/archives/private/${listname}.mbox/${listname}.mbox
$ sudo -u www-data /usr/share/mailman3-web/manage.py update_index_one_list -v3 ${listname}@${mailman_host}
Your problem is likely to be that your regular user does not have
permission to connect to (and modify) your sqlite DB file.
Furthermore, sqlite is the default database backend only for the
purpose of getting a proof of concept working without installing
and configuring a big database server. It isn't recommended for
production use. I can assure you that if your mailman setup is going
to see real use, you do NOT want to be using sqlite. You want to be
using MariaDB or PostgreSQL. I recommend just setting one of those
up from the start.
While you're at it you'll want to reconfigure from using the default
full text search indexing of Whoosh, to using Xapian, because Whoosh
is just completely unworkable. With only 6,000 or so messages a
Whoosh reindex (done monthly, and after every list import) was
running for over 16 hours at 100% CPU usage and using 3½GB RAM the
whole time. Xapian finishes the same task in 23.13 seconds.
As I understand it, mailman3 (a GPLv3+ codebase) is unable to come
out of the box configured to use Xapian because
python3-xapian-haystack is GPLv2-only. As soon as you bring up the
issue of Whoosh's slowness the advice from upstream is to switch to
Xapian or Elasticsearch. So may as well do that.
I think that should probably be in the README.Debian for mailman3
and I've been meaning to submit a bug to that effect. All of the
required things are packaged in Debian, so it's trivial to do.
The mailman-users mailing list is probably better placed to answer
any questions you have about mailman3 rather than specifically
Debian's packaging of it. I've found it very helpful, though you do
have to bear in mind that they can only really talk about installs
of mailman3 done in a venv from pip, not through Debian packages.
https://lists.mailman3.org/mailman3/lists/mailman-users.mailman3.org/
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Reply to: