Bug#942078: [high quality bug] baloo crash, can't recover; forced reindex does not fix; unusable via dolphin C-f
Dear Nicholas,
Le jeudi 4 février 2021, 04:23:36 CET Nicholas D Steeves a écrit :
> Norbert Preining <norbert@preining.info> writes:
>
> > Control: tag -1 +unreproducible
> >
> > Hi Nicholas,
> >
> > to be clear, tagging this bug as "unreproducible" means that **we** the
> > developer cannot reproduce it. So please don't change it. Thanks.
> >
> > Also, moreinfo is still needed, and you didn't provide it.
> >
>
> I did. The minimum dataset needed to trigger was with an extensive set
> of exclusion patterns that result in 436767 indexed files, with the
> following results returned for balooctl indexSize
>
> Actual Size: 3.27 GiB
> Expected Size: 2.12 GiB
>
> PostingDB: 424.52 MiB 19.516 %
> PositionDB: 845.84 MiB 38.884 %
> DocTerms: 344.65 MiB 15.844 %
> DocFilenameTerms: 47.45 MiB 2.181 %
> DocXattrTerms: 0 B 0.000 %
> IdTree: 7.64 MiB 0.351 %
> IdFileName: 35.77 MiB 1.645 %
> DocTime: 22.29 MiB 1.024 %
> DocData: 23.21 MiB 1.067 %
> ContentIndexingDB: 0 B 0.000 %
> FailedIdsDB: 0 B 0.000 %
> MTimeDB: 2.59 MiB 0.119 %
I beat you although by a small margin ;)
Baloo File Indexer is not running
Total files indexed: 438,883
Files waiting for content indexing: 0
Files failed to index: 0
Current size of index is 4.26 GiB
File Size: 4,26 Gio
Used: 2,17 Gio
PostingDB: 584,37 Mio 26.301 %
PositionDB: 946,16 Mio 42.585 %
DocTerms: 499,82 Mio 22.496 %
DocFilenameTerms: 57,64 Mio 2.594 %
DocXattrTerms: 0 o 0.000 %
IdTree: 9,84 Mio 0.443 %
IdFileName: 45,09 Mio 2.029 %
DocTime: 28,65 Mio 1.290 %
DocData: 43,68 Mio 1.966 %
ContentIndexingDB: 0 o 0.000 %
FailedIdsDB: 0 o 0.000 %
MTimeDB: 6,57 Mio 0.296 %
… and I can’t reproduce the bug either… :(
I’m using it for searches on a nearly daily basis and it’s been working reliably for as long as I can remember. Both with file names and file contents.
[…]
> >> Has anyone on the team tried testing baloo with a large $HOME? Mine is
> >> 131GB, in 142460 mixed files and directories. At the moment my
> >> baloo_file process is consuming over a terabyte (!) of memory. After
> >> resetting baloo and forcing a full reindex it takes a couple of weeks
> >> before it gets out of control again and crashes.
> >
> > Well, maybe you should disable the service since it obviously is not
> > able to cope with the amount of data/files/size you are asking it to
> > index.
> >
>
> Right, that's why I filed this bug. With a non-toy dataset, baloo is
> more often than not unusable. Typical lore says to disable baloo and
> use the more robust recoll. I'm willing to conceded that it's possible
> that there's something special about my data/files, and that they're
> causing baloo to fail in a corner-case. If that's the case, what data
> do you need? Other than all my files ;-)
As said above, this has not been my experience on a definitely non-toy dataset.
Maybe there’s something more specific to the crash you experienced that would need investigating before it can be fixed.
I had an issue on a a secondary machine where it would crash / stop due to starvation of inotify watches. This was clearly visible in .xsession-errors with the advices to increase fs.inotify.max_user_watches plainy written, and baloo was not the only piece of software having issues in this situation. Probably not your case but I’m mentioning it here just in case.
> >> Please don't sweep baloo bugs under the rug by tagging bugs containing
> >> crashes with obvious misbehaviour as severity "normal", eg:
> >
> > Again, since it is not reproducible and nobody else has either reported
> > this problem and none of us see this problem, it is normal, would you
> > please kindly trust the decision of the maintainers?
> > And thanks, you don't have to educate me about severities.
> >
>
> Important: "a bug which has a major effect on the usability of a
> package, without rendering it completely unusable to everyone." It is
> clear that this bug meets Debian criteria for "important". If no one
> else on the team (I'm a team member too) is testing baloo with a medium
> size set of diverse file types than no one else one the team will be
> able to reproduce it.
>
> A politic of "normal + unreproducible" is a contributing factor to
> why baloo is never fixed. In other words, it's a politic of ignoring
> problems in baloo.
Given that we failed to reproduce the issue so far, with little indication as to what could actually help reproducing it, this looks really overkill to me.
Happy hacking !
--
Aurélien
Reply to: