Re: Guidance for CVE triage and listing packages in dla-needed.txt
Hi Roberto
Please read also the very end of my email since I think maybe the most
important thing is there at the very end.
I think I need to clarify myself a little. Partially because your good
and lengthy email made me think more.
The reason why I pointed to the unimportant, low, medium and high was
that it was a definition of what is an issue worth fixing and which
ones are not in a Debian context. I have noted the same thing as you,
that the classification is not really in use in practice.
In fact <low> do not seem to have been used since 2019 for (unfixed
issues). I can only see that tag for things that have in fact been
corrected with (low) tag.
But the definition gives you some guidance on what we should fix and
what we should not fix.
I'm not sure myself how we handle things so therefore I'll try to dig
out statistics from our CVE database.
STATISTICS
==========
Let me use some data from CVEs for last year 2023.
I used the following method to extract the data
grep -B 5 '\[buster\]' list | grep -A 5 "^CVE-2023-" | grep '\[buster\]'
and then grepped for the end-of-life, not-affected (and so on to count further).
It is not a perfect method, but I think it is good enough to crunch the numbers.
1260 CVEs with tag [buster]
382 end-of-life
410 not-affected
281 no-dsa
71 postponed
58 ignored
58 fixed (most in the linux kernel)
Wrote a script to crunch the data for several years, here is the result:
YEAR CVES EOL NA NODSA POST IGN FIX
2023 1260 382 410 281 71 58 58
2022 1328 473 395 209 49 76 126
2021 1521 284 424 329 24 93 367
2020 974 5 80 294 38 55 402
2019 765 0 71 199 11 75 409
Please note that this is CVE-2023-... this do not mean that we have
only fixed 58 CVEs last year, but we have only fixed 58 CVEs tagged
with CVE-2023-...
We do fix older CVEs as well.
I did not try to extract any <unfixed> (low) or similar entries
because a simple search in emacs did not show anything for those years
(except 2019).
COMMENT
=========
What do this show? I'm not 100% sure myself but I think this shows
that out of the 1260 CVEs that we got for last year, with a buster
tag, we have not fixed that many. Just 58 of them.
You wrote the following:
"The high/medium/low scheme would be potentially useful if we were
overwhelemed with CVEs to the point where we had to routinely decide
to
fix only some subset of outstanding fixable CVEs because we lacked a
sufficient number of contributors to fix all 100%. However, in
practice
we have enough contributors that we regularly fix all, or most, of the
fixable CVEs"
I think the above data shows that we do routinely decide to fix only
some subset. With that said, please read also what I write below
inline.
I do that, routinely following what the regular Debian Security team
have decided already, when I'm the front desk.
I do not seem to be the only one doing that as well. I looked at quite
a few git log items and it is general practice to simply follow Debian
Security team.
A large portion of the CVEs are marked as no-dsa for a long time. We
still have 200 of them from year 2019 for example.
And I think this is a good practice. Security is always a balance
between a lot of things. Fixing everything will likely result in a lot
of tools breaking.
Also following the regular Debian Security team is also a good
practice. They do a good job of analyzing things and we should
normally not fix issues in buster when it will not be fixed in later
releases. So this makes a lot of sense.
What I think we should be better at is to define and communicate what
we think are worth fixing. This is an area where we can improve. I
realize this since I cannot even define this myself and I have been
involved in Debian for 24 years now and in LTS team for quite a few
years as well.
DEFINITION
==========
If I then go back to the original question. What is an example of a
minor issue, or better yet a definition of it.
I do not have a good definition because I'm not 100% sure myself.
1) One way to view it is that a minor issue (or at least what we
should postpone or ignore) is something that will not be fixed in
later releases.
This way we rely on the Debian Security team and the regular package
maintainers for their judgement without thinking too much ourselves.
It will in fact work quite well even though we are not really defining
what a minor issue is.
2) Another way is to use the definition we have of
unimportant/low/medium/high classification and use that as judgement.
3) A third way is to in fact define what we mean by doing
classifications similar to what we can find in the low/medium/high
definition.
(2) and (3) above are good, but they share one common problem and that
is that the definition may deviate from what the regular Security team
decides and therefore we will fix things in LTS that will not be fixed
in later releases.
If we go for (2) or (3) we should probably synchronize the definition with them.
We should also not forget that Debian is a community project where we
have package owners. Some package owners have very high security
standards while others have much more relaxed. At a fist glance this
is a problem, but I do not think it is, really. The packages have very
different properties.
It is much more important to fix problems in server tools like exim,
sshd, apache compared to for example a command line debugger.
I'm also giving some comments inline below. At the very end of the
email there is an important thing to read.
On Mon, 8 Apr 2024 at 22:06, Roberto C. Sánchez <roberto@debian.org> wrote:
>
> Hi Ola,
>
> Thank you for putting thought into the matter of issue severity.
>
> On Sun, Apr 07, 2024 at 11:19:08PM +0200, Ola Lundqvist wrote:
> > Hi Roberto
> > After first some thinking on what "constitutes a minor issue?" I did some
> > research and realized that there is in fact a good classification in the
> > Debian Security team list here:
> > [1]https://security-team.debian.org/security_tracker.html#severity-levels
> > We have "unimportant", "low", "medium" and "high".
>
> While I agree that the security tracker documentation does give four
> possible priority levels, in practice it seems that only 'unimportant'
> is currently in use.
Agree. This seems to be the case.
I was primarily referring to this as this was examples on what is
important and what is not.
For example it tells that some classes has "mild" impact, like "local
DoS, /tmp file races and so on"
> Using grep to search data/CVE/list, the most recent CVEs carrying tag
> priorities of high/medium/low are from the following years.
>
> high (371 occurences) - 2016
> medium (1223 occurences) - 2017
> low (4509 occurences) - 2021
Yes and as far as I can tell the "low" classification is only on
already fixed packages for that year. Maybe I misunderstood the
tagging.
> The counts are slightly over the actual numbers, because in some
> instances the issue title or one of the notes may contain a string of
> the form "(high/medium/low ....)" but which is not meant to be a
> priority indicator. I did not put the effort into perfecting the
> expression to filter these out and the numbers seem "close enough".
>
> The 'unimportant' tag is still in regular use and there are quite a
> number of current CVEs with that tag.
Seems so, yes.
> For context, there are a total of 728,466 lines in the file, tracking a
> total of 295,240 distinct CVEs.
>
> > The ones classified as "high" are clearly things that warrant a DSA/DLA,
> > so they should never be postponed in LTS.
> > We can discuss medium, I think they should in general give a DSA/DLA but
> > with lower prio.
> > Then we need to discuss how to handle all the "low" severity issues we
> > have. We are fixing a lot of them. I think these should in general be
> > postponed or ignored. But that is my view. Sure fixing such can be good
> > but I'm not sure the risk of regression outweighs the security gain by
> > fixing them and also we can discuss whether it is worth fixing.
> > Related to this I think we should in fact start to use this severity
> > classification more because the severity level will definitely help with
> > the decision on which packages to fix first.
>
> All of that said, I think that the 'unimportant' tag may continue to be
> useful for us for LTS work, but I am not convinced that high/medium/low
> are still useful. And the contents of data/CVE/list quite convincingly
> show that the security team is not really using high/medium/low any
> longer.
Could very well be so.
> The high/medium/low scheme would be potentially useful if we were
> overwhelemed with CVEs to the point where we had to routinely decide to
> fix only some subset of outstanding fixable CVEs because we lacked a
> sufficient number of contributors to fix all 100%. However, in practice
> we have enough contributors that we regularly fix all, or most, of the
I do not think this is correct. See above in the beginning of the email.
> fixable CVEs (i.e., those which can be backported and for which the risk
> is not too high, etc). It seems to me that we are in a state where if we
> elect to not fix a CVE, it is a technical matter (i.e., fix is too
> intrusive, risk of regression is too high, etc.) and in those cases we
> would simply mark the CVE as ignored in the applicable suites.
I do not think this is the case. I think we routinely mark things as
no-dsa/postponed simply because the regular Debian Security team have
done so.
> The remaining CVEs are either fixed right away, or we defer them (either
> explicitly by marking the CVEs as 'postponed', or implicitly becuase
> they were already marked 'no-dsa' by secteam and we choose to treat that
> the same as 'postponed' for LTS purposes).
This part is true.
> It seems to me that in such a
> case, trying to then apply high/medium/low priorities will require
> additional work but not produce any significant additional benefit. I
That is probably so.
> also don't think that the "Minor issue" comment seen on many CVEs is
> particularly relevant for LTS.
So far I have seen it as quite relevant, but this is a part we can discuss.
> That is, the tracker requires a comment. To tag a CVE as simply
> '<no-dsa>' but not put a comment there is a syntax error. I can
> certainly imagine that many occurences of "Minor issue" are simply there
> to make the tracker happy and that all the relevant meaning is conveyed
> by '<no-dsa>'. I.e., this should get fixed at some point, but it is not
> especially urgent and doesn't warrant a DSA all on its own.
This may very well be true.
> ========================================
>
> So, breaking from that let me try to return for a moment to the original
> topic of the thread, "Guidance for CVE triage and listing packages in
> dla-needed.txt".
>
> In essence, there are two basic scenarios that we have:
>
> 1. a particular Debian release passes from the responsibility of secteam
> to the LTS team; with it the LTS team inherits a load of CVEs with
> previous triage decisions from secteam
>
> 2. for the active LTS release(s) some new CVEs are assigned which
> (potentially) have to be fixed
>
> In neither of those scenarios does there seem to be a compelling reason
> to try to grade CVEs high/medium/low.
Agree. It was not my intention either. I was not clear on that. I was
merely trying to use an already existing definition on when an issue
is considered minor.
> In the first scenario, a whole
> bunch of packages will get added to dla-needed.txt and contributors will
> pick them, perform more in depth triage of the open CVES (which are most
> probably already marked 'no-dsa' and 'postponed' by secteam), and then
> make appropriate decisions. In some cases the CVEs will be fixed, and in
> other cases they will re-triaged (as either 'postponed'* or 'ignored').
Yes and most likely postponed since you need to fix later releases as well.
That is of course allowed but it is likely not the most common output.
> * The re-triage of 'postponed' may be a 'no-dsa' that is not fixed right
> away, but which we simply leave as 'no-dsa' to avoid unnecessary churn
> in the security tracker. If there is another change that is needed,
> e.g., adding or updatign a comment, then changing the 'no-dsa' to
> 'postponed' would be OK.
Yes.
> The second scenario works in much the same way, but just deals with CVEs
> as they arrive. The key difference is that we are more likely to be
> making a first pass decision on the CVE as it pertains to LTS, rather
> than inheriting a previous decision from secteam when the release was
> not yet under LTS.
Yes. When triaging it is very common that secteam decision is followed.
> So, unless I miss something, we can accomplish everything we need, by
> simply discontinuing new assignments of the type 'no-dsa'. This is what
> was described in last month's LTS meeting. Here is the propsal that I
> made:
Yes and no. :-)
> Proposal: discontinue use of "no-dsa" in LTS (and ELTS) triage
> - Instead, any CVE markings added by FD or another (E)LTS contributor
> must either be "postponed" or "ignored"
Agree to this, BUT there is an important aspect that is not mentioned.
I'll describe it below.
> - Contributors are encouraged to review "no-dsa" entries when beginning
> new work on a package and deciding if those CVEs can be fixed as well
> (which is something that many contributors, perhaps all, already do)
Agree, but it is a rather difficult judgement.
> - To avoid unnecessary churn in the security tracker, we will not batch
> update (e.g., marking all LTS "no-dsa" as "postponed" instead); rather
> individual CVEs will be reviewed and either fixed or (if appropriate)
> updated to "ignored", "not-affected", etc.
Ok.
> Additionally, our practice is to make comments that explain the triage
> decision and those comments are always more verbose than "Minor issue".
This was new to me.
> So, if we are applying a 'postponed' or 'ignored' tag and making an
> explanatory comment with it, that seems to me like just what we need to
> be doing.
>
> Do you think that this would be sufficient? Or did I miss something that
> would make the use of high/medium/low priorities potentially beneficial?
Yes there is one thing that we miss. If we take the case when we have a
no-dsa entry for a later release.
There are three possible outcomes for a person re-traging it.
ignored
postponed
add to dla-needed
Ignored has already been discussed so we skip that from this discussion.
You know I think we should ignore more issues (like all unimportant
and maybe even all low), but that is a separate thing.
The question is when to decide to postpone and when to add to dla-needed.
This is where we need to have some kind of guidance.
If we say that we simply follow what the regular Debian Security team
have decided, then the principle is simple because that team have
already decided that it is not worth fixing right now and therefore
all (with possible exceptions in rare cases) no-dsa will become
postponed.
Possible exceptions would then be if it is explicitly mentioned in the
tracker that we should wait for a point release, or when the issue has
already been fixed in a later release. The latter will in fact be a
re-triaging at a later point when the lts-cve-triage tool will mention
that the CVE is in fact fixed in a later release.
Other exceptions appear when the package content is significantly
different between releases.
One proposal is that we do the following as a general rule. But with
all general rules there are exceptions. Exceptions will occur when the
package content vary between the releases for example.
1) All issues secteam have ignored, will be ignored in LTS as well.
2) All no-dsa (minor issue) will be postponed (minor issue) in LTS as well.
3) When a package is fixed in a later release we will then again
re-triage it to judge if we should change the postpone decision to a
"add to dla-needed".
If we should not go for this proposal (which I think is what we in
practice follow today to a large extent) we need some guidance on what
kind of issues we should not postpone. Without guidance the triaging
judgement may become quite random.
Such guidance could for example be to look at the
unimportant/low/medium/high severity definition for judgement, or we
can invent something new.
If we want to go for something new I can make a proposal as a start.
Hope this helps. :-)
Cheers
// Ola
> This proposal came out of the discussion from when I first started this
> thread nearly a month ago. That is to say, it was after a substantial
> discussion had already developed. I am inclined to prefer a simpler
> approach and the fewer triage states that we have to manage, the better.
>
> Regards,
>
> -Roberto
>
> --
> Roberto C. Sánchez
>
--
--- Inguza Technology AB --- MSc in Information Technology ----
| ola@inguza.com opal@debian.org |
| http://inguza.com/ Mobile: +46 (0)70-332 1551 |
---------------------------------------------------------------
Reply to: