Update on infra-513663

[Niels Thykier <niels@thykier.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi

As of 91dc917 in infra-513663, lintian is now running with the new
"package grouping" back-end.  For now it merely emulates the behaviour
of the old lintian with slightly different output and possibly different
scheduling (but this is what we want).

Finishing #513663 soon is definitely a possibility now.  Basically all
we have to is to rewrite this:


foreach my $gname (sort $pool->get_group_names()) {
    my $group = $pool->get_group($gname);
    foreach my $proc ($group->get_processables()){
	process_package($proc, $action);
    }
}

to something like

foreach my $gname (sort $pool->get_group_names()) {
    my $group = $pool->get_group($gname);
    process_pkg_group($group, $action);
}

Although this probably requires quite a bit of refactoring of
process_package (formerly known as "the big PACKAGE loop").

This also means that we might soon want to merge infra-513663 into
master.  So if you have any concerns to the "interface" or way things
have been done, please voice them now or in the near future!

Generally checks and collections are unaffected by this so far.  A few
new collections have been introduced to remove the deprecated "unpack
level 1" for all package types except src packages.
  So as far as writing/updating checks/collections goes, it is pretty
much the same.  What will much likely change is the order in which these
are run.

To exploit the grouping, we may have to unpack part of package "A" in
order to do a check in package "B".  As I see it there are "basically"
two ways of doing this: 1st) unpack all of "A" and "B" before running a
single check on either.  2nd) introduce more sophisticated collections
dependencies.
  Personally I prefer to start with the 1st solution since it is far
simpler to implement and easier to test.  Right now we concurrently run
checks and collections so the user will now see a speed
decrease/increased latency. The first check is stalled till all
collections have been run on all packages.
  As the code matures, we can improve this to stalling the check only
until all the collections it actually depends on have been run on all
packages in the group (which is more like the 2nd solution).  This
requires more book-keeping and that we can support unpacking/collecting
multiple packages at the same time (with a limit to avoid "fork-bombing"
the system if there are a lot of packages in the group + a lot of
schedulable collections per package).


~Niels

[1] So far the worst speed offending thing tends to be the "unpacked"
collection for "large" packages.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQIcBAEBCAAGBQJNkmB2AAoJEAVLu599gGRCDoEP/2doQGvgGANHbvJwQE8i6T9k
OpOR2AN4+AltYmZuYiuxIhctLi55JzLPi/EAaW2dUM5DLbzoxrpxWjAKkuGE50dT
WawRzDwIVHJBsCVMBo4WxzLsQ/Knusu1c5G3IOIRgjMX3TOy+rXzJP9bJGpr1k46
FLLq66HGwp1BQH8LnpP0WJpn2BEGSkoCkKZgLNzWgS4CUl3ZHrDklXpnkwbSi1MV
hELEtjas2OPceAb0/Crn8zfv1wvfS25DX7hqyaovuOzjDaH8Xr8Zc0tFG6qIeITG
P46gAPTtPpm3SW5Dasd51HZMVs4TU/SU7NjmZjmyrmrrM8f1aimtKcPT0ns2vCZW
OM2voAxNW5/OkqN/kIRa0ayuIW2xG1dKW5w5XuWY/Y7FtZeGm2Me/bnSLtU5YbCK
+ro5wygcHaDVRVEZR9Z0UrjVdx4QTOx1ZnZ3SZjRAIFhDTcwk6Lj5ZLL3o7k3wkp
qgb0VrwMsnsdqQ+qBtNMrgGrcYyYa3DbpmxwMIQ6p0umV5Kxwi1yaAvF+ZtTY7V/
9XP3WvruFcDsB1zqEPC/OU2mQT0zww26vh5IeMLByFcoKRhJGuRmsDKo6Dsyp24v
ZNXcTwksz1Kib611K1XIA1H+TZxsFKqLgMYO4tyxHHnpMq6CV+Y8rue2WX5FqANT
2SrqGYnU/uEPrH0G6edq
=kXyD
-----END PGP SIGNATURE-----