Bug#641468: lintian: update the lab layout (i.e. use pools)

[Niels Thykier <niels@thykier.net>]

On 2011-09-14 12:34, Niels Thykier wrote:
> On 2011-09-13 19:21, Jakub Wilk wrote:
>> * Niels Thykier <niels@thykier.net>, 2011-09-13, 18:04:
>>> Jakub realized the source of a lot of our errors on lintian.d.o are
>>> caused by limitations in the file-system.  We should probably use a
>>> pool or something similar to reduce the amount of elements in each dirs.
>>
>> Just to shed more light on what the problem is:
>>
>> $ stat /srv/lintian.debian.org/laboratory/binary/ | grep Links
>> Device: 807h/2055d    Inode: 7512069     Links: 32000
>>
>> On ext3 filesysytem, at least in squeeze, 32K is hard limit on number of
>> hard links, so we can't create more directories in binary/.
>>
>> ext4 doesn't have this limitation, so a work-around would be to convert
>> the filesystem.
>>
> 
> Upgrading to ext4 might be a solution, but I personally think that
> changing the Lab layout is the "right thing(tm)" to do in this case.
> Considering we want derivatives to do archive-wide Lintian runs, it may
> be prudent to be file-system agnostic.
> 
> Also, we can (ab)use this oppertunity to enable "multi-version" +
> "multi-architectures" in static labs as well.  Hopefully we can also
> clean up the Lab API while we are at it. XD
> 
> ~Niels
> 
> 
> 
> 

Okay, so before diving into this - can anyone elaborate on the current
Lab design?  The User Manual does not give me a lot here.  I am asking
because I want to know if there is something we should pay attention to
when working on this.


Beyond this I have been spending some time looking at the situation and
possible solutions and extensions.  My basic does not affect the layout
of entries/unpacked packages (i.e. collections should be unaffected).
  It appears that we have been bumping the LAB_FORMAT about once a year
the last two years (in 01-2010 for changes and 03-2003 due to "many
recent changes"[LFB]).  Hopefully format 11 can last far longer than a
year. :)

So, simple solution is to use a "mirror-like" pool, so something like:

 $LINTIAN_LAB/pool/l/lintian/lintian_2.5.3_all_binary/
 $LINTIAN_LAB/pool/l/lintian/lintian_2.5.3_source/

The last entry would be "${name}_${version}(_${arch})_${type}", where
the ${arch} part is not relevant for source.  Not sure what to do about
changes and architecture though (as they may have multiple architectures).
  The above have the advantage of trivially allowing multiple versions
(and architectures) of the same package in the pool.

I am thinking this would be a good time to make the Lab maintain its own
state files (info/$type-packages).  This implies updating the state
files when adding or removing a package from the lab.  If the lab
maintains these, we will most likely have an easier time providing a
sane API for accessing packages in the Lab.
  If we go down this route I would probably use the oppertunity to empty
the "standards-version" and the "architecture" field in
info/source-packages.
  Furthermore, each entry in these files have enough information to
provide create a "Lintian::Processable".  If we extend L::Processable we
could be looking at a "standard" way of requesting and submitting
packages to the Lab.

I also consider the option of creating a "migration utility" that could
upgrade a LAB_FORMAT 10 to 11.  Since we have gotten rid of the unpack
scripts and "cross-package" symlinks, this change will basically just be
a bunch of "mkdir" followed by a tons of "mv" (and updating the
Lab-Format in the ".lintian-status" file).
  Admittedly lintian.d.o probably has relatively little to gain from it
as we will need to doing a full run once this bug is fixed anyway.


~Niels

[LFB] Commits:
fc10b608
8b1f0cfe