Bug#710813: lintian: Restructure the laboratory/data storage

[Niels Thykier <niels@thykier.net>]

On 2013-06-10 21:16, Niels Thykier wrote:
> On 2013-06-08 19:23, Jakub Wilk wrote:
>> * Niels Thykier <niels@thykier.net>, 2013-06-02, 18:49:
>>> During the current full run, lintian.d.o ran out of inodes (1.9M). At
>>> the moment, I have disabled experimental which I hope will work around
>>> the problem for now.
>>
>> Today on #debian-qa:
>> 19:17 < jwilk> Wouldn't it be a matter of auto-removing more collections
>> than only unpacked?
>> 19:17 < nthykier> we could do that, but then we might as well just not
>> have a lab on lintian.d.o
>> 19:18 < jwilk> \o/
>> 19:19 < nthykier> in which case we could go for tmpfs processing
> 
> I have enabled auto-removal for several collections including debfiles
> and strings.  At the moment, lintian is currently still processing sid
> and has reached "phpbb3".  At the same time, lintian.d.o has dropped its
> inode use from 1.9 M (100%) to 1.3M (68%).
> 
> ~Niels
> 
> 

At the end of the run (plus the follow up incremental run) we were down
to about 56% used.  I have re-added experimental (~1100 "groups atm.).

It would seem that the some of five collections auto-removed collections
are consuming most of the inodes.  AFAICT:

 * ar-info
   - 1 per binary package.
   - most binary packages probably won't need this collection
 * debfiles
   - At least 3 per source package.  In pratise probably a bit more.
   - (Extreme?) Example: linux has about 1.9k entries in its
     debian.tar.xz
 * debian-readme
   - 1 per binary package.
   - probably more used than ar-info
 * menu-files
   - 4 + N inodes per binary package. (N being the number of
     "menu-files")
 * strings
   - 1 + L + N inodes per binary package.
     - N is the number of ELF files in the binary package.
     - L is the number of dirs needed to replicate the dir structure
       containing ELF files.
   - Also here, the linux package(s) makes an "interesting" example.

So, probably debfiles + strings took the majority of the load.  That
said, menu-files is currently costing us at least 0.18 million inodes or
about 1% of all inodes on our FS on lintian.d.o[1].

~Niels

[1] That figure is assuming no menu-files in any package, i.e. it is
simply the number of binary packages times 4.