Re: Guidance for CVE triage and listing packages in dla-needed.txt
Hi Cyrille
Yes CVSS is a good starting point. A question there is how accurate
that score is, especially for CVEs on obscure packages.
I think it is valuable to have a guideline so we can evaluate if the
CVSS is reasonable.
It is sometimes a little dangerous to only focus on a number. :-)
But yes I think you have a good idea.
// Ola
On Thu, 11 Apr 2024 at 09:15, Cyrille Bollu <cyrille@bollu.be> wrote:
>
> Why not using CVSS as a base calculation for assigning severity levels?
>
> IIRC, something like:
>
> CVSS>=8 => High
> 4<=CVSS<8 => Medium
> CVSS<4 => Low
>
> was a good guidance in my previous job.
>
> FYI, I've attached the table that drove us to these score.
>
> Cyrille
>
> Le mercredi 10 avril 2024 à 23:30 +0200, Ola Lundqvist a écrit :
> > Hi again
> >
> > I have started with a document that clarify the severity levels. I
> > also introduced the level "critial" but I'm not sure it adds any
> > value.
> >
> > https://inguza.com/document/debian-security-severity-levels
> >
> > This is just a first draft. It is not final. But comments are
> > welcome.
> >
> > // Ola
> >
> > On Wed, 10 Apr 2024 at 22:03, Ola Lundqvist <ola@inguza.com> wrote:
> > >
> > > Hi Raphael
> > >
> > > You can see corrected statistics in a separate email.
> > >
> > > Now to comment a few things below.
> > >
> > > On Wed, 10 Apr 2024 at 10:49, Raphael Hertzog <hertzog@debian.org>
> > > wrote:
> > > >
> > > > Hello,
> > > >
> > > > On Tue, 09 Apr 2024, Ola Lundqvist wrote:
> > > > > Let me use some data from CVEs for last year 2023.
> > > > > I used the following method to extract the data
> > > > > grep -B 5 '\[buster\]' list | grep -A 5 "^CVE-2023-" | grep
> > > > > '\[buster\]'
> > > > > and then grepped for the end-of-life, not-affected (and so on
> > > > > to count further).
> > > > > It is not a perfect method, but I think it is good enough to
> > > > > crunch the numbers.
> > > > >
> > > > > 1260 CVEs with tag [buster]
> > > > > 382 end-of-life
> > > > > 410 not-affected
> > > > > 281 no-dsa
> > > > > 71 postponed
> > > > > 58 ignored
> > > > > 58 fixed (most in the linux kernel)
> > > >
> > > > Those numbers are quite surprising. I hope there's some error
> > > > somewhere
> > > > otherwise I wonder what has been done in the 2400+ hours paid
> > > > each year to
> > > > work on LTS... I'm pretty sure we have fixed more than 58 CVE.
> > > > The average
> > > > month has 20 to 30 updates (see
> > > > https://lists.debian.org/debian-lts-announce/2024/03/threads.html
> > > > for
> > > > example).
> > >
> > > Yes I was surprised as well. When I checked yesterday I could not
> > > find
> > > the fault but today with fresh eyes it was rather obvious. :-)
> > > Sorry for the confusion. See the other email about that.
> > >
> > > > > A large portion of the CVEs are marked as no-dsa for a long
> > > > > time. We
> > > > > still have 200 of them from year 2019 for example.
> > > > >
> > > > > And I think this is a good practice. Security is always a
> > > > > balance
> > > > > between a lot of things. Fixing everything will likely result
> > > > > in a lot
> > > > > of tools breaking.
> > > > > Also following the regular Debian Security team is also a good
> > > > > practice. They do a good job of analyzing things and we should
> > > > > normally not fix issues in buster when it will not be fixed in
> > > > > later
> > > > > releases. So this makes a lot of sense.
> > > >
> > > > This comment still conflates "no-dsa" with "doesn't need to be
> > > > fixed". We
> > > > clearly told you many times that it doesn't mean that at all.
> > >
> > > Did I write that? I cannot see it it what you comment. But after
> > > reading the whole email I think I understand why you think so.
> > > If you read my proposal I suggest we postpone all no-dsa and then
> > > wait
> > > for point release updates as one way to handle it.
> > > That is basically the same thing as what you write below, that no-
> > > dsa
> > > means no need for an urgent fix.
> > >
> > > But then you commented that we should do our own judgement. Yes it
> > > would be good if we have a guilde. More about that below.
> > >
> > > > "no-dsa" means "doesn't need to be urgently fixed, and will not
> > > > be fixed
> > > > by the security team". It basically defers the question to the
> > > > package
> > > > maintainer. It's up to the package maintainer to decides between
> > > > "to fix via spu", "postponed" or "ignored".
> > >
> > > I agree to this.
> > >
> > > > Some package maintainers will typically decide to fix it via a
> > > > point
> > > > release. But they rarely update the triaging to document
> > > > "postponed" or
> > > > "ignored". So that's why it's up to the LTS team to make that
> > > > call
> > > > when we are (alone) in charge of a given release.
> > >
> > > This is a good point. I had missed that package maintainers do not
> > > update the security tracker. Can we have tools to help us with
> > > this?
> > >
> > > > And regarding "we should normally not fix issues in buster when
> > > > it will
> > > > not be fixed in later releases", we have already explained that
> > > > it goes
> > > > the other way around: if we decide that something is worth to be
> > > > fixed
> > > > in buster, then we should help to get it fixed in later releases.
> > >
> > > I do agree with that approach. The question then is when we should
> > > do that.
> > > I think we need some type of guidance. It is very easy to fall into
> > > extremes.
> > > One extreme is to only follow secteam.
> > > The other extreme is to try to fix all no-dsa no matter how
> > > unimportant it is.
> > >
> > > > Obviously if the sec-team triages it as unimportant/ignored, then
> > > > we are
> > > > unlikely to fix it. But when the sec team opts for no-dsa, then
> > > > we are
> > > > entitled to decide to fix it and we can also prepare the SPU to
> > > > fix newer
> > > > releases.
> > >
> > > Yes, then the question is what rule/guide to follow. We can't fix
> > > all.
> > > That is clear from the statistics.
> > >
> > > > > 2) Another way is to use the definition we have of
> > > > > unimportant/low/medium/high classification and use that as
> > > > > judgement.
> > > >
> > > > I wanted to comment about this classification. While it's not in
> > > > active
> > > > use by the security team, I think it would be nice if we could
> > > > provide
> > > > such a classification:
> > >
> > > Yes, I think so. Or some other method we choose to follow.
> > > But I think we need to clarify it, because it is a little fuźzy at
> > > the moment.
> > >
> > > > 1/ it can help to drive attention to the the most pressing CVE
> > > > 2/ it lets us monitor how well we are performing for different
> > > > severities
> > > > (in terms of delay to provide a fix)
> > > >
> > > > Clearly the LTS sponsors who are funding security work would like
> > > > to see
> > > > some things like "90% of the high-priority CVE are fixed within
> > > > one week".
> > > > But we are not able to provide any data.
> > >
> > > Precisely.
> > >
> > > > Also we are regularly telling customers/sponsors that we are
> > > > doing on own
> > > > assessment of the importance of CVE and that it may diverge from
> > > > the
> > > > standard CVSS score that they use as reference. In the ideal
> > > > world, we
> > > > would provide our own CVSS score with explanations but we are
> > > > quite far
> > > > from this. In the mean time, it would already be great if we had
> > > > more
> > > > reliable low/medium/high scoring...
> > >
> > > Yes it would be really good. If we add this as part of the triaging
> > > work it would be good.
> > > In a way we need to do that already because we say that we should
> > > not
> > > simply tag no-dsa, but rather postpone or not.
> > >
> > > There are a few things we need to decide on:
> > > - What severity levels warrants the package to be in dla-needed? Is
> > > low one of them?
> > > - Should all medium be in dla-needed?
> > >
> > > If you ask me I think the line is somewhere in medium, but I'm not
> > > sure of what definition to use yet.
> > >
> > > Cheers
> > >
> > > // Ola
> > >
> > >
> > > >
> > > > Cheers,
> > > > --
> > > > ⢀⣴⠾⠻⢶⣦⠀ Raphaël Hertzog <hertzog@debian.org>
> > > > ⣾⠁⢠⠒⠀⣿⡁
> > > > ⢿⡄⠘⠷⠚⠋ The Debian Handbook:
> > > > https://debian-handbook.info/get/
> > > > ⠈⠳⣄⠀⠀⠀⠀ Debian Long Term Support: https://deb.li/LTS
> > >
> > >
> > >
> > > --
> > > --- Inguza Technology AB --- MSc in Information Technology ----
> > > > ola@inguza.com opal@debian.org |
> > > > http://inguza.com/ Mobile: +46 (0)70-332 1551 |
> > > ---------------------------------------------------------------
> >
> >
> >
--
--- Inguza Technology AB --- MSc in Information Technology ----
| ola@inguza.com opal@debian.org |
| http://inguza.com/ Mobile: +46 (0)70-332 1551 |
---------------------------------------------------------------
Reply to: