Re: Possible akonadi problem?
On Donnerstag, 22. Januar 2015 22:11:37 CEST, Martin Steigerwald wrote:
Am Freitag, 23. Januar 2015, 07:17:13 schrieb Dmitry Smirnov:
Hi Brad,
On Fri, 2 Jan 2015 11:28:53 Brad Alexander wrote: ...
Thank you very much, I will try this. And see whether it helps with these
upstream bugs:
[Akonadi] [Bug 338402] File system cache is inneficient : too many file per
directory
Bug 332013 - NFS with NetApp FAS: please split payload files in file_db_data
into several directories to avoid reaching maxdirsize limit on Ontap / WAFL
filesystem
Bug 341884 - dozens of duplicate mails in
~/.local/share/akonadi/file_db_data
I bet it may help with the first two, but third one might be another bug.
Right now locally after the last akonadictl fsck I only have
from 4600 after
the fsck to 4900 files now in my private setup with still is
mostly POP3 just a
30 day limited IMAP for the Fairphone and as a backup when I
accidentally mess
up with something locally. It really seems Akonadi is more snappy since the
fsck. I have lots more files in there.
I also bumped innodb_buffer_pool_size but didn´t see that much of a change,
what helped most with MySQL load is to use Akonadi git 1.13 branch with
database performance improvement.
I now implemented the treshold size change you suggested and
did another fsck
and vacuum.
*beware this is partly a rant, but I think a well founded one*
In the end it I had mysqld excessively writing 150-300 MiB for about 15
minutes or more, sure it wrote the *complete* size of the database several
times. I do have an mysqld where it complains about not being able to get
locks and suggesting that a second instance might be running. I found this
later one.
I don´t think it had to do with the change you suggested, but with another
problem. I found it often enough that after akonadictl stop and akonadictl
status telling it is actually stopped, mysqld was still running. I usually
check this and do SIGTERM on it waiting for it to end. But I didn´t
yesterday.
After a long time of writes I sigkilled the mysqld process in order to end
the excessive writes.
I tried to recover from backup, but it didn´t work out. At one time I
created a second maildir resource and even after making sure I had
everything of ~/.config/akonadi from backup and also making sure there are
no other files in it with rsync --del and same for ~/.local/share/akonadi
and even after deleting akonadi_maildir_resource_1rc from
~/.kde/share/config it insisted on having a maildir resource 1 today and it
filled database and file_db_data with the results of scanning my million of
mails again.
So I suggest anyone trying this change:
On akonatictl stop make *perfectly* sure that mysqld process has ended.
After my attempts on getting back from backup failed *twice* I am now doing
it *all* from scratch. Including adapting a ton of filter rules.
I really hope at some day Akonadi as it is today is gone and replaced by a
better designed Akonadi Next, just as with Baloo replacing Nepomuk. Akonadi
Next I think needs to meet the following requirements:
1) It is a *cache* and *nothing* else. It never stores any information
inside the cache that isn´t stored elsewhere or that it isn´t able to
rebuild. That said, in case of issues with the cache, it is possible to
remove it and rebuild it from scratch *without* *any* data loss involved
and *without* having to recreate filter rules. If it needs an ID for a
folder, fine, store it in an xattr or otherwise with the original data
store. I really suggest a functionality to recalibrate filter rules by the
*name* of the folder otherwise.
2) Make it *just plain* obvious *where* Akonadi stores the configuration
and the data. For now its at least ~/.config/akonadi one or two files per
resource (the changes.dat there), ~/.kde/share/config/akonadi*,
~/.kde/share/config/kmail2rc (it contains references to Akonadi resources),
~/.kde/share/config/ which contains the local filter rules.
3) Make the filter rules more robust, make them survive a complete deletion
of the cache, separate config from cache *completely*. Never ever *mix*
these.
So replace:
[Filter #25]
Applicability=2
AutomaticName=true
ConfigureShortcut=false
ConfigureToolbar=false
Enabled=true
Icon=system-run
StopProcessingHere=true
ToolbarName=<List-Id>: <debian-backports.lists.debian.org>
accounts-set=akonadi_pop3_resource_0
action-args-0=128
action-name-0=transfer
actions=1
apply-on=manual-filtering
contentsA=<debian-backports.lists.debian.org>
fieldA=List-Id
funcA=contains
identifier=yNhsmyF7PrdhRuTL
name=<List-Id>: <debian-backports.lists.debian.org>
operator=and
rules=1
with something sane that can recover from the folder name in or an ID that
is stored *with* the folder on cache loss.
4) I suggest a completely obvious structure:
<configdir>/<resource>/<allconfigfromresource-mostly-in-cleartext>
<configdir>/<generalconfig>
<datadir>/<resource>/<allcachedataforresource>
with that model it should be able to wipe out the cache data in case of any
problem. Really make this work, make the software assume that the data may
get corruption, it is not under your sole control to avoid any hardware
failures. It is *just a cache*.
5) If you use a database, make perfectly sure that there *never ever* can
be two database processes trying to access the same database. I have seen
this several times with Akonadi MYSQL backend that I had two mysqld
processes. Thread the database as *part* of Akonadi and make an akonadictl
*stop* it *or* report a failure when it cannot stop it. And make akonadictl
never start it, if there is still one running.
Reply to: