[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Another take on package relationship substvars

Our current way of dealing with package relationship substvars such as ${misc:Depends} has been annoying me for a while. As it is, we are stuck in this way setup where the "Depends" field in debian/control is de facto mandatory. It is an RC bug (waiting to happen) if you omit ${misc:Depends} or, for arch:any packages, ${shlib:Depends} from the dependency field.
Personally, I feel we have this weird setup where we have tooling that will scream at you if you forget. But I lack a compelling reason for why we need tooling to remind us when we could just as well have tooling solve this problem for us. Additionally, our tooling for reminding us rarely knows about anything but the 2-3 standard substvars everyone uses even though we should get reminders for other substvars like ${python3:Depends}, ${gir:Depends}, etc. Particularly the rules for when a substvars is applicable turns out to be quite non-trivial.
I have given this some thought and I think I have a conceptually idea for how to solve this, which I will go over in this email. 

# Scope

Just to ensure we are on the same page.
When I am talking about package relationship substvars, I mean basically any substvar of the format ${*:<Field>} where Field is a relationship field such as Depends, Pre-Depends, etc.
Other subvars like ${binary:Version} are out of scope for this proposal. This also includes other variables such as ${gnome:Version}. 


# The proposal
I think our package helper tooling should just automatically aggregate all provided substvars of the format ${*:Depends} and append it the Depends field. Rinse and repeat for other relationship fields.
The list of fields where this is applied would be curated, so it only applies to known relationship fields where we feel it makes sense. My starting list would be: 

 * Any dependency field, that is: Pre-Depends, Depends, Recommends, and
   Suggests

 * The Provides field.
I am omitting Breaks, Conflicts, and Replaces because I am not aware of any users of these at the moment. I am open to adding them, if there is a strong use-case. 

# The advantages
Thanks for humoring me so far. Obviously, this is a change and there should be a benefit to changing the status quo that outweighs the cost of the change itself. 

I see the following advantages:

 1) As a consumer of a tool, I would no longer have to remember if and
    when to add a custom substvars. Tools providing dependencies or
    provides will provide them automatically without requiring me to
    do anything. This means I spend less time maintaining d/control
    files where tool can and should just solve this problem for me.
    (see the next section for how many substvars we are sporting)

 2) As a provider of a substvar generating tool, I would be able to
    focus my documentation on other things than "Please, pretty please,
    remember this subtvars!". Instead, I can focus on what problems my
    tool will solve for you and then push the substvar into a footnote.

 3) As a distribution, it becomes a lot easier to do archive-wide
    changes. Remember the "multiarch-support" a decade ago where all
    shared libraries had to be updated to add
    `Pre-Depends: ${misc:Pre-Depends}`? How about all the
    `Provides: ${t64:Provides}` that are added now that people have to
    remember to remove on their next soname bump? Would have been an
    non-issue if these substvars were applied automatically.

 4) As a provider of linting tool, I no longer have to remind people
    about idiosyncratic rules of Depends being pseudo-mandatory. Nor
    spend hours figuring out whether I should mind people to add
    ${foo:Depends} plus ${bar:Provides}. Nor do I have to learn when
    those substvars are applicable to a package. Instead, I can
    recommend users to migrate to the new tool stack that gives them
    the advantages listed above. Much better use of my time and gives
    more benefits for everyone.
The first advantage would apply immediately per package for consumers that has migrated to this flow. For the other items, as adoption increases the pain point decreases. 

# Is there really a problem?
You might be of the belief that substvars are easy to manage. There are just the two, ${misc:Depends} and ${shlib:Depends}, right? No, we have a lot of substvars that we have to remember and they different rules for when they apply. 

 * ${misc:Depends} and ${misc:Pre-Depends} should *always* be added to
   Depends and Pre-Depends respectively.

 * ${shlib:Depends} should be added to Depends for arch:any but not
   arch:all. Additionally, if you use dpkg-shlibdeps with -d you have
   to remember to add a new shlib sustvar to other fields as well.

 * ${t64:Provides} should be added to shared library packages that are
   a part of the ongoing time_t transition.

 * The ${gir:Depends} and ${gir:Provides} should be used for packages
   that use dh_girepository. Though I think it only provides them for
   packages that actually contain gir files, so you have to manually
   select only those. Admittedly, ideally those packages would be
   ideally match the pattern "gir1.2-*", but not all do.

 * Then we have ${python3:Depends}, which is provided by dh_python3.
   I think it is generally only present for packages named python3-*,
   but I never fully understood the code, so I might be wrong.
   (If you want to correct me here, do it off-list. It is not relevant
   to the point I am making)

 * We have ${perl:Depends} from dh_perl which only appears if your
   package contains perl code or libraries.
And these are just the ones *I* know about (which sums to 8 if you are not counting ${shlib:Recommends}). I would not be surprised if other language stacks had their own substvars like a ${haskell:Depends} or a ${nodejs:Depends}, etc. 

For all of these substvars, the following rules tend to apply.
If you forget to add a susbtvars that you should added, it is a latent RC bug with only a warning from dpkg-gencontrol that you might miss if you grab a coffee while waiting for the build to complete. If you add one that is not provided, you get a warning from dpkg-gencontrol that will nag you when you *don't* go for coffee while waiting for the build.
In other words, if you like to provide "correct" packaging with "no warnings" you are navigating a minefield of rules. A minefield of rules that our packaging tools could have solved for you. 


# How do we get here?
High level, any helper framework above dpkg can provide this feature if it wants to. It is basically a question of parsing d/*.substvars to figure out which substvars are relevant, compute a new field from substvars file + the static provided data from d/control, and then passing -DField=<...> to dpkg-gencontrol.
There are probably a bunch of nitty gritty implementation details in practice, but the overall approach stands.
I think each helper stack (debhelper, cdbs, debputy, etc.) would define their own rule set for how to get there. As an example, I could see this be the new default in the next debhelper compat level, possibly with a way to be early adopter on existing compat levels. 


## Alternative solutions
We could also make unused substvars a hard failure (FTBFS). Personally, I feel auto-managing them will be less painful for users. But if the consensus goes down this direction, then I would be behind it as it is still better than the status quo in my book.
That is my proposal for how Debian contributors can spend less mental effort tracking relationship substvars and use their new spare mental capacity providing value for our users. Thanks for reviewing the proposal and any feedback you might have for making relationship substvars be less annoying for users and Debian as a whole. 


Best regards,
Niels
Reply to: