Re: [GSoC] blends-gen-control hints (Was: blends-dev, gsoc 2013)

[Andreas Tille <andreas@an3as.eu>]

Hi Emmanouil,

On Fri, Jul 12, 2013 at 07:07:01PM +0300, Emmanouil Kiagias wrote:
> I just committed the new version of {sec-}blend-gen-control which now uses
> the {contain-}provides column.

Great to see the code developing!

> I run again the tester and a surprising, I
> have to say, bug(?) in sec-blend-gen-control came out( I will explain it
> below ). The control/task-description files for almost all blends seem to
> be ok(also the differences we had with debian-imaging with "djvu-viewer"
> virtual package are now solved), only debian-edu control files still have
> differences with packages which are not found in any of the
> tables(packages, new_packages, ubuntu_packages, blends_prospectivepackages)
> that the blends_metadata_gathener.py checks. For example:
> 
> diff control/control.amd64 control-sec/control.amd64
> 130c130
> <  dpt-i2o-raidutils | raidutils,
> ---
> >  dpt-i2o-raidutils | raidutils | dpt-raidutil,
> 138c138
> <  foomatic-filters-ppds,
> ---
> >  foomatic-filters-ppds | cupsomatic-ppd,
> 440a441
> >  flashplugin-nonfree-pulse | libflashsupport,
> 524c525
> <  kig,
> ---
> >  kig | keuklid | kgeo,

Up to this I admit to say that rather control-sec is correct (see below).

> 553d553
> <  libflashsupport,
> 603a604
> >  sodipodi,

I do not understand this difference, thought.

> 706c707
> <  gpsim-led,
> ---
> >  gpsim-led | gpsim-lded,

Here also control-sec seems to be correct.

> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
> - - - - -
> 
> Except from debian-edu other Blends seem to be ok(and virtual packages are
> handled properly, except from the case which I explain below).

That's most probably due to the extensive use of alternatives.  I simply
injected a new task "alternatives" into the fun Blend which was based on
a copy of debian-edu common task leaving only those Depends + Recommends
that are containing a '|'.  (I removed lines matching "Suggests: .*|.*"
because at this moment I assumed suggests are harmless for us - perhaps
it should be reinjected and observed closely as well.)  I added a field
"X-Expected-amd64" to each line which is ignored by the Blends tools
(as any field starting with "^X-") but could be helpful for our test suite
(or not - just to bring in an idea).

> Now about the interesting bug. For debian-edu:
> For the packages:  nfs-kernel-server | nfs-server , we have:
> 
> 1)In blends_dependencies_alternatives for debian-edu we have:
> 
>    blend    |    task     |          alternatives          |
> contains_provides | dependency
> ------------+-------------+--------------------------------+-------------------+------------
>  debian-edu | main-server | nfs-kernel-server | nfs-server | t
>     | d
> 
> 2)and in blends_dependencies we have:
> 
>    blend    |    task     |      package      | provides | dependency
> ------------+-------------+-------------------+----------+------------
>  debian-edu | main-server | nfs-kernel-server | f        | d
>  debian-edu | main-server | nfs-server        | t        | d
> 
> Let's take the control file for architecture kfreebsd-amd64
> For kfreebsd-amd64, testing release virtual package "nfs-server" is
> available but "nfs-kernel-server" package is not. So blend-gen-control
> correctly places "nfs-server" in Depends: and "nfs-kernel-server" in
> Suggests:.

Well, this is actually *incorrect*!  If a control file contains a
virtual package you should make sure that there is a really existing
alternative given.  In other words: you would trigger the following
lintian warning

  http://lintian.debian.org/tags/virtual-package-depends-without-real-package-depends.html

when delivering a control file that only mentions a virtual package.  I
think I remember the usage was described in policy or developers
reference but failed to find this with a very quick search - you will
definitely find something when seeking more deeply.  The problematic
phrase in your arguing is: 'virtual package "nfs-server" is available'.
By definition virtual packages are *not* available - they are just
virtual and need to be provided by a real package.  If there is no such
real package the whole set of alternatives should be moved to Suggests.

> In the blends_dependencies table where each package is single I know
> exactly which packages are virtual or not. So for this example I will check
> if nfs-kernel-server exists by outer joining the blends_dependencies
> table(like I do for all the packages) but for the virtual
> nfs-server(provides=true so I handle this packages in different way than
> the other  ) I will do to this(in order to check if it exists):
> 
> SELECT DISTINCT provides, p.distribution, component FROM packages p
>    JOIN releases r ON p.release = r.release
>      WHERE provides ~
> ('((\s|,)'||'nfs-server'||'(\s+|,|$)|^'||'nfs-server'||'$)')
>      AND r.role='testing' and p.architecture='kfreebsd-amd64' AND
> component='main'
> 
> output:
>            provides            | distribution | component
> -------------------------------+--------------+-----------
>  nfs-kernel-server, nfs-server | debian       | main
> 
> So here I get a result which contain the "nfs-server" so I know it's
> available.

As I said above you do not know that a *package* "nfs-server" exists -
you only know that it might be provided by some other package.  When
thinking about this I need to make a mental note that I need to handle
the blends_dependencies table with care and need to exclude the
"provides=t" cases in other applications I intend to base on this table
explicitly.  For instance virtual packages do not have features like a
Homepage or a package Description.  "Virtual packages" are just
something else than packages and need to be handled with care.

> Notice here that nfs-kernel-server also appears as a virtual package
> 
> In the case of blends_dependencies_alternatives where I have:
> 
> blend    |    task     |          alternatives          | contains_provides
> | dependency
> ------------+-------------+--------------------------------+-------------------+------------
>  debian-edu | main-server | nfs-kernel-server | nfs-server | t
>     | d
> 
> I do not know which package is virtual or not so I will check for both of
> them like in a common result set:
> 
>  SELECT DISTINCT package, p.distribution, component FROM packages p
>    JOIN releases r ON p.release = r.release
>      WHERE package in ('nfs-server', 'nfs-kernel-server')
>      AND r.role='testing' and p.architecture='kfreebsd-amd64' AND
> component='main'
> UNION
>  SELECT DISTINCT provides, p.distribution, component FROM packages p
>    JOIN releases r ON p.release = r.release
>      WHERE provides ~
> ('((\s|,)'||'(nfs-kernel-server|nfs-server)'||'(\s+|,|$)|^'||'(nfs-kernel-server|nfs-server)'||'$)')
>      AND r.role='testing' and p.architecture='kfreebsd-amd64' AND
> component='main'
> 
> output:
> 
>             package            | distribution | component
> -------------------------------+--------------+-----------
>  nfs-kernel-server, nfs-server | debian       | main
> 
> Although  nfs-kernel-server does not appear in "packages" table for
> kfreebsd-amd64, testing release(so blend-gen-control place it in suggests),
> it appears in a provides column(as a virtual package) so
> sec-blend-gen-control place them both in Depends: nfs-kernel-server |
> nfs-server (it does not know which of them is a package or a virtual
> package so it checks for each of them in a common result set). Anyway this
> bug(I do not know if actually it's a bug) will appear in
> sec-blend-gen-control ONLY in cases where a package appears in  packages
> UDD table both in "package" and "provides" columns.
> 
> So due to way the different {sec-}blend-gen-controls work, they handle it
> these packages in a different way. I will try to think another way of
> handling the virtual packages, without changing current UDD state, in order
> to avoid that.

As a consequence of my arguing I thing sec-control is right and control
is wrong ... at least from what I'm reading in your mail.  I'd suggest
to put all problematic / interesting cases into the alternatives task of
the fun Blend and by doing so reducing the signal noise ratio around our
problematic cases.  Than we can finally discuss these with other people
lurking on this list.

> PS: I hope I used proper wording to explain the issue, in case you don't
> understand let me know so I will try to rewrite it a better way.

I think I perfectly understood what you mean and I hope I was precise
enough to make my point clear. ;-)

Kind regards

      Andreas.

-- 
http://fam-tille.de