Vendor specific data files

[Niels Thykier <niels@thykier.net>]

Hi,

In light of #648777, #652114 and #649852 I am considering to add "Vendor
specific" data files.

Personally I think the checks/tags should generally be oblivious to the
current profile.  I think it would be harmful for us if
"$profile->derives_from ('ubtuntu')" found its way into our check code[1].
  My resistance towards this is that I am afraid it will (eventually)
clutter our checks with "if vendor X"-cases.  I feel a much better
solution would be for us to (eventually) fix #359059 - hopefully sooner
than later, actually.  :)

I am proposing the "Vendor Data Files" as an extension to the current
"Vendor Profiles".  It is not a "Silver Bullet" (tm), but it should
solve #648777 and #649852 just nicely.

A possible alternative to my solution could "Vendor filters" that
filters tags between Lintian::Tags and Lintian::Output based on tag name
and extra.
  Personally I think the filters would be a great extension to the
"Vendor Profiles", but I believe it is (somewhat) ortogonal to data
files.  I think data/fields/obsolete-packages is a perfect example of
where a filter could not replace a vendor specific data file.

The second alternative would be to implement Vendor specific checks
(#359059).  However, I think it would be a terrible solution to #649852
(effectively, suppress the debian tag and re-implement with an exception
for the particular field).


Proposed solution:
------------------

Technically I suggest we create a "vendor" directory in data for this
purpose.  Each subdir maps to a vendor and inside the vendor subdir is a
"vendor-specific" mirror of the data directory.  As an example with
#649852, this would be:

 data/fields/binary-fields
 ...
 data/vendor/ubuntu/fields/binary-fields


If a file is missing form the vendor data dir, we look for a data file
in its "parent vendor" directory until we run out of "parent vendors".
At this point we look for it in data/, where our default file will be[2].


Extension:
----------

The above is the "simple" solution to the problem.  Though I am thinking
we could do with extending it a bit.  The problem I want to address is
that changes to the Debian file will not automatically apply to
derivative files.  Depending on the data file in question this may (or
may not) be a good default.
  My proposal here would be to include some simple "processing"
statements to our data files.  I believe the following will do:

  @include-parent
  @delete <key>


The semantics of them are:

  @include-parent
     - Include the contents of the same data file from the first parent
       vendor that explicitly defines it.
       - see the question below on "is parent vendor determined by
         dpkg-vendor or Lintian profiles's Extends"?
     - The included data "appears" at the place of @include-parent.
     - Fails if no parent has this data file.
       - Thus, not allowed in the data files of the "root" vendor(s).

  @delete <key>
     - drops a previously read key (and its value)
     - does nothing if the key is not known
     - does not affect a later appearance of <key>


If a key is seen twice (possibly via include) the old definition/value
is silently replaced by the new one.  This is the current behaviour; we
merely need to document this as a "feature".
  We may need a "@set <key> [value]" in case we need keys starting with
@ (or whatever we choose instead of @).


With the extension, I believe we should be able to handle most common
cases, avoid some copy-waste and (more importantly) "out-of-date" data
files for derivatives.



Open Questions:
---------------

Open question: Is parent vendor defined in terms of "dpkg-vendor" or in
terms of Lintian profiles's "Extends:" relation?  Which is less
confusing/most useful?
  - I admit thinking dpkg-vendor for the first hour of writing this
    email.
  - But would the "Extends" relation be more useful?

Open question: would it be better to have a "Data-directory" field in
the Lintian vendor for vendors that need this?
  - Also see below.

How do we handle "sub-vendors" (e.g. --profile vendor/group)?
 - Trivially solved if we use "Data-directory" in the Profile (see
   above)
 - It is not unreasonable to think that if #359059 is fixed there will
   appear some "debian/pkg-$group" profiles/checks.
 - Should a sub-vendor be allowed to have its own data-files
   - obsolete-packages and deprecated-makefiles comes to mind as
     possible use cases.




Comments, alternatives, suggestions etc. much appreciated.  If people
are okay with us going down this path, I will bounce this to
d-derivatives@l.d.o for further comments from derivatives.

~Niels

[1] Admittedly, the "derives_from" is less evil than "$profile eq
'vendor'", but still.

[2] A variant here is to only have vendor dirs and move all current data
files into "data/debian/".