[Summary]: Another take on package relationship substvars

[Niels Thykier <niels@thykier.net>]

Hi

It seems the discussion on this topic has settled, so I am now doing a 
summary of the discussion as I understand it.

 * Generally, the original proposal seems to have been received
   favorably at a conceptual level.

 * Several people requested the scope to be expanded to extra fields.
   So far, no one seems to have questioned any of the extra fields
   proposed. Accordingly, I will expand the scope to all mentioned
   extra fields (such as Built-Using/Static-Built-Using and negative
   relationship).

 * My alternative proposal of making relational substvars mandatory
   did not have anyone supporting. Additionally, Simon McVittie provided
   a strong argument for why the alternative would scale poorly. Given
   the argument from Simon and no one openly supporting that proposal,
   I am considering it a dead-end with no support.

 * All the concerns raised related to promotion and demotion of
   substvars were related to the shlib subvars (such as
   ${shlib:Depends} vs Pre-Depends/Recommends). I have yet to see anyone
   concerned about a non-shlib related variable, which is great since
   dpkg-shlibdeps as the only substvars provider has infrastructure for
   promotion/demotion. No case presented seems to have been problematic
   nor controversial. I have included an extended section below on this
   topic for the details, should you be interested in those.

 * Guillem proposed some concrete ideas for moving parts of the
   implementation into the dpkg layer, which seems fine with me.
   I consider these implementation detail and will handle that
   bilaterally with Guillem.

In other words, it seems like there is consensus that this proposal at a 
conceptual level is acceptable. I will look into the implementation 
details with Guillem (as mentioned, in a smaller forum).

I fully anticipate that there will be a transition for this feature 
(such as a debhelper compat bump) to ensure we have a controlled 
migration. Notably substvar demotion does not work out of the box (see 
below) as one of my arguments to ensure it happens in a controlled manner.




                   = Detailed explanations =

From here on, I will expand on the substvars cases and how they work. 
If you are not interested in those, you can stop reading as the 
remainder of the email is dedicated only to this topic.


# Promotion and demotion of substvars

For those, who are curious or concerned about promotion or demotion of a 
substvar, here is an extended coverage of cases presented in the 
discussion and how they work.

First, a bit of terminology as people might not have read the full 
thread. I use the word "promotion" when the substvar is used in a 
**stronger** field that the one it is named for. Example:

    # The substvar in this example is promoted to Pre-Depends
    Pre-Depends: ${shlib:Depends}

Demotion is the opposite. Here the substvars implies a strong field than 
the one it is used in:

    # The substvar in this example is demoted to Recommends
    Recommends: ${shlib:Depends}


For the case, where the tool can split the substvar into multiple 
substvars, I will use the phrase selective promotion or selective 
demotion of the content. The only known tool that supports this is 
`dpkg-shlibdeps` and the only known usage involves the ${shlib:*} namespace.


There are multiple cases to cover and how they would interact with this 
proposal:

 * Selective promotion/demotion of content (Unaffected)

 * Full substvar promotion                 (Unaffected)

 * Full substvar demotion                  (Breaks)

 * Manual fiddling with substvars          (Unaffected)

Each will be expanded in their own subsection below. The `(Unaffected)` 
and `(Breaks)`-marker represents what happens in a rebuild of an 
unchanged source package with with this proposal suddenly active. As 
mentioned, I expect we will do this as an a opt-in style transition to 
avoid unexpectedly triggering any cases that might break.


## Selective promotion/demotion of content


Selective promotion and demotion of the content works out of the box 
with this proposal.

 * The logic for splitting a substvars will generally be in debian/rules
   via manual calls to dpkg-shlibdeps or dh_shlibdeps. This part remains
   unaltered.

 * The main difference is that you no longer have to manually any of the
   substvars in debian/control.

This method generally only works with dpkg-shlibdeps, since that is the 
only tool with infrastructure to do the splitting. At the same time, it 
is also the only substvar provider mentioned in the discussion so far, 
where anyone had an example of needing promotion or demotion.


Note in this case, the split off substvars will be named are the field 
they go in. Strictly speaking, none of the substvars have been promoted 
nor demoted in this particular case. This is also why it is promotion / 
demotion of **content** rather than of the substvar itself.


## Full substvar promotion

This is the case of doing `Pre-Depends: ${shlib:Depends}`. This example 
is also the only discussed use-case so far and is relevant for 
`Essential: yes` packages.

This would work correctly out of the box. A naive implementation of the 
proposal would lead to the following pseudo example:

    Pre-Depends: ${shlib:Depends}
    # The Depends field would be implicit and is only shown
    # to assist the reader.
    Depends: ${shlib:Depends}


The `dpkg-gencontrol` tool will de-duplicate relationship fields by 
pruning relations from weaker fields that are implied by stronger fields 
per second paragraph of man:dpkg-gencontrol(1). So there is generally no 
loss of functionality nor extra bloat in the final output.

Final remark: Guillem suggested that it would make sense for 
`dpkg-shlibdeps` to special case `Essential: yes` packages such that it 
puts dependencies into `${shlib:Pre-Depends}` by default for essential 
packages. This would reduce the boilerplate needed for essential 
packages. Details about how to transition to this will be handled in a 
smaller forum.


## Full substvar demotion

This is the only case that does not work out of the box with the new 
world. This is the case of doing `Recommends: ${shlib:Depends}`.

The reason why this does not work, is exactly the reason why the 
promotion case works. Namely, the effective behavior becomes:


    Recommends: ${shlib:Depends}
    # The Depends field would be implicit and is only shown
    # to assist the reader.
    Depends: ${shlib:Depends}

And as mentioned for the promotion case, the weaker field will be pruned 
by `dpkg-gencontrol`. Even if you could magically skip the pruning logic 
in `dpkg-gencontrol`, the Depends field would still be there and 
overrule in practice.


When looking at how to handle this case, we have two solutions. 
Specifically for ${shlib:Depends}, selective demotion is preferable.
  Particularly because a full field demotion only works if **all** ELF 
binaries are optional features of the packages. The rationale here is 
that any ELF binary would need a libc dependency. Any non-optional ELF 
binaries implies that the package must have a `Depends: 
${shlib:Depends}` (or stronger).
  A full field demotion with a non-optional ELF binary would 
technically be an RC bug for the shlib case. Nonetheless, I am still 
going with the "no surprises" road here. It is not for me to judge 
whether a concrete package has all its ELF binaries as optional features 
even when I assume such cases to be exceedingly rare.

I have not seen any other substvar being mentioned for a "full substvar" 
demotion and I am not going to invest a lot of effort on them for this 
reason. Even if they appear, the following hack presented in the next 
section would work for them. The main question is whether we need this 
hack enough to provide a proper solution.


## Manual fiddling with substvars

As mentioned by Colin Watson, we do have one "get out of jail" card that 
can be played in our current packaging stack. Namely, manually fiddling 
with debian/foo.substvars to rename or tweak the substvar as needed.

This solution already existed and is unaffected by this proposal. 
Basically, roll your own substvar "fixer" and throw it at the right spot 
in `debian/rules`.

If you find yourself in a corner, it will solve your problem.


However, if we find ourselves repeatedly in weird corners, we should 
provide better tooling to solve this problem. I do not think we want 
every debhelper tool to grow its own "selective promotion/demotion" 
feature as they are not trivial to do.
   While I am open to providing better support at a central level, I 
will expect multiple concrete real world use-cases before providing any 
supporting logic. This is because debhelper is not built around having 
logic to support every special-case in the world. Instead, it covers 
common cases[1] and provides its hook target cop-out for truly 
special-cases.
  If it turns out full-field demotion is a common case, then debhelper 
will support while if it is an unique snowflake case, the manual 
substvar fiddling will have to do for that handful of packages.


Best regards,
Niels

[1]: The original debhelper maintainer used "~10 cases in the archive" 
as a rule of thumb for whether it made sense to consider if a build 
system should be added into debhelper itself.
  I have largely adopted this rule of thumb for adding features for 
where I am unsure if Debian at large needs them. I fully anticipate to 
apply that rule of thumb to whether debhelper should be involved in 
reducing the number of "fiddling with substvars" cases.