Re: Piuparts state-dependency-failed-testing analysis

[Dave Steele <dsteele@gmail.com>]

In the past week, the number of state-dependency-failed-testing has increased by 510, to 3409. Of those, about 80% are blocked by the first package in the list, and 75% are blocked only by that package.

On Sat, Oct 22, 2011 at 11:03 PM, Dave Steele <dsteele@gmail.com> wrote:

I find myself frequently called back to the Piuparts waiting-to-be-tested package list at http://piuparts.debian.org/sid/state-dependency-failed-testing.html, and got curious about what is keeping that list as long as it is. Following is an analysis.

There are currently close to 3000 packages in state-dependency-failed-testing that can be traced to one or more state-failed-testing packages. That number actually went down quite a bit in the last week, when libgtk2.0-bin passed, clearing hundreds of packages for testing. 

There are less than 300 packages in state-failed-testing holding up those tests. The blocking distribution is typically very skewed. At any given time, half of the packages in state-dependency-failed-testing may be held there, directly or indirectly, by 3 to 5 packages. Eliminating libgtk2.0-bin last week exposed libreadline6 as the new top blocker.

Here's the output of a script that scrapes http://piuparts.debian.org/sid/state-dependency-failed-testing.html and analyzes blocking packages:

# ./piublocker 

dependency failed -  2899

failed testing -  274

blocking free cum  package

1588    1005  1894 libreadline6

 451      35  1854 sgml-data

 429       1  1466 docbook-xsl

 191     153  1286 texlive-base

 155      79  1182 ca-certificates-java

 125     106  1071 php5-cli

  89      29  1042 gnustep-base-common

  60       0   982 gnustep-back0.20

  49       6   973 menu

  49       4   957 libcommons-httpclient-java

  42       0   915 python-ibus

  40      15   897 libcommons-beanutils-java

  35      34   862 gosa

  35       1   827 blends-common

  33      33   794 liquidsoap

  30       3   791 libspring-core-java

  28      20   771 libwebkit1.1-cil

  28       1   750 libatinject-jsr330-api-java

  26      26   724 drupal6

  26       7   703 antlr

  24      24   679 cyrus-common

  21      13   664 libmx-1.0-2

  20      15   645 openssh-client

  17       0   630 libhttpclient-java

  17      17   613 drizzle

  16      16   597 dovecot-core

  14       0   597 sat4j

  14       6   591 proftpd-basic

  14       0   577 libjasper-java

  14      11   566 libhttp-date-perl

  14       7   559 libgcj-bc

  13       4   546 libmono-addins-gui0.2-cil

  12      10   535 postfix

  12       3   523 libusrp0

  12       9   512 libgtkhtml-4.0-common

  11       6   501 libncurses5-dev

  11       6   490 libgtkglext1

...

This output says that there are currently 2899 packages in state-dependency-failed-testing traceable to a state-failed-testing package (that doesn't exactly match Piupart's count of 2920). 274 packages are responsible for that blocking. More than half of them (1588) are blocked by a single package, libreadline6. 1005 of those packages would be cleared for testing by removing only libreadline6 from the list of blockers. Possibly, at least some of those exposed packages may have blocking numbers in the 1000 range (e.g. 'python' is in that list).

The "cum" shows how many packages are left as each subsequent top blocker is cleared. From that number, you can see that the dependency list could be cut in half by removing the top 4 packages on the list. 80% would be cleared by passing the top 29 packages.

It seems worth the effort to identify and focus fixing the top packages on this list, over time. There should be opportunities for low-hanging fruit whenever the state-dependency-failed-testing list gets over about 1000-1500 entries.

The source for the script piublocker is at https://github.com/davesteele/piublocker