Than you for the thoughtful reply.
Yes the apt upgrade submit ends there with no further error messages. When moving that source list it proceeds and asks me if i want to install the upgrades.
Please help me to understand the MITM scenario. So you're saying an attacker could be able to redirect all sources except one? Is that even technically likely? Has that ever happened?
I appreciate you explaining the risk, as I'm trying to understand why. Please understand that this also prevents unattended upgrades from proceeding, which aside from a real indication of MITM attack seems like unnecessarily putting many systems at a security risk. (Also, shouldn't https be used by default in the main sources list that includes security repos? Doesn't https prevent MITM?)
Re space checking: I understand estimates are inaccurate. But it seems better to fail on the side of caution in this instance. To be clear, I'm not expecting to make waves for so many other users, this could be rolled out with a flag option like (--check-space). I do think opt out should be the default though.
I know there are separate partitions, but doesn't the estimation look at say /boot when a kernel is queued for install? If it knows where to install packages, it can check those exact locations for free space. (But still imho better to fail by default)
I would take the conservative side of the estimate, as I've said the uninstall process can be very complicated, especially when inside a VM with no other space available. Usually means manually deleting files that upsets dpkg and then having to reconfigure all that first before making apt happy again.
I think 1.5GB is reasonable. Doesn't the filesystem not have to be uncompressed to install? Wouldn't it already be uncompressed during apt use?
I understand about rolling out in a way to see what possible problems arise before implementing defaults.
Thank you
On Sun, Oct 09, 2022 at 01:21:45PM -0700, xaq xaq wrote:
> To simplify a frequently seen halt of `apt upgrade` I have this example.. I
> had to remove keepsolidinc.list
> from /etc/apt/sources.list.d. Then apt update proceeds. Here is the error I
> get with that source list in place:
>
> *Hit:1 http://archive.ubuntu.com/ubuntu
> <http://archive.ubuntu.com/ubuntu> - InRelease
> Hit:2 http://security.ubuntu.com/ubuntu
> <http://security.ubuntu.com/ubuntu> --security InRelease
> Hit:3 http://archive.ubuntu.com/ubuntu
> <http://archive.ubuntu.com/ubuntu> --updates InRelease
> Hit:4 http://archive.ubuntu.com/ubuntu
> <http://archive.ubuntu.com/ubuntu> --backports InRelease
> Ign:5 http://apt.keepsolid.com/ubuntu
> <http://apt.keepsolid.com/ubuntu> - InRelease
> Err:6 http://apt.keepsolid.com/ubuntu
> <http://apt.keepsolid.com/ubuntu> - Release
> 404 Not Found [IP: 144.217.71.199 80]
> Reading package lists... Done
> E: The repository 'http://apt.keepsolid.com/ubuntu
> <http://apt.keepsolid.com/ubuntu> - Release' does not have a Release
> file.
> N: Updating from such a repository can't be done securely, and is
> therefore disabled by default.
> N: See apt-secure(8) manpage for repository creation and user
> configuration details.*
What do you mean with "halt of `apt upgrade`"? You have shown us output
of 'apt update'? And that output suggests that the other archives were
updated (just that they were already current: cache "Hit").
I suppose if the old data still on disk for keepsolid is indicating
newer versions of packages than you have installed an upgrade would try
to download them potentially running also into 404 errors (assuming this
repo is entirely offline). That would indeed "block" upgrades of other
packages not from that repo, but what is apt supposed to do instead…
It is asked to upgrade all packages and can't: Partial upgrades are
technically supported, but hardly tested in practice as it is
a combinatorial explosion if you tried. So in theory you upgrade from
a perfectly supported system with updates available to a system with
unknown support status and still more updates available. That doesn't
look like a good default move.
You can tell apt to carry on with --fix-missing if that is what you
desire, but really, you should either remove that repository (and the
packages coming from it) for good or work with its provider on fixing
it/making it available again. After all, --fix-missing lets you carry
on in a pinch, but it remains that the repository fails to serve updates
to you, which in the worst case means public known security issues remain
exploitable on your system.
(This could be part of an attack. It usually isn't, but it is
indistinguishable from a MITM blocking access to a repository so
that you can not get the security updates the attacker wants to
exploit. There are better ways, hence its unlikely, but it is just
a less good way as it shows errors grabbing attention that something
is wrong. If apt wouldn't, this would be really easy to use.)
> For the size estimation, I understand it is an estimation, and there could
> be guesswork, but I would posit that if the estimate is even roughly
> correct, a user most likely will not want to proceed, because the
> consequences greatly outweigh the benefits of proceeding.
As I tried to explain already apt downloads itself at the expense of 1.5
MB. It will take up installed 4.5 MB. It hardly changes size from
version to version, so more space freed/used after the upgrade amounts
to more or less 0 MB. Running and using apt requires additional data
which easily takes up 200 MB. Lets assume every package is similar to
apt (they aren't, you e.g. have things like texlive which comes in a
couple hundred MBs of download already). A typical upgrade can contain
more than 1000 packages, so which value between 0 and 200 GB are we
gonna use as a rough estimate for that upgrade? And again, which
partitions are we gonna talk about if you have multiple, like e.g.
a separate /boot ?
APT in this scenario picks 1.5 GB (the size it will need to download all
the packages) on the partition housing its download location:
apt's free space check for the download is done against free blocks
available for everyone. In a default configuration root has an additional
5% of a partition as free space available for their exclusive use.
If your idea is that lets say apt should not continue if you don't have
100 MB free on /boot and 5 GB on / (should be enough, right?) that is
fine for your system and pretty easy to implement with e.g. a
dpkg::pre-invoke hook (no need for the heavier dpkg::pre-install-pkgs
hook). That wont work in the general case though… there are systems
which don't even have 5 GBs of disk space to begin with.
(Did you think apts 1.5 GB is a reasonable low bound? For some people
that is already way too high! Like if your filesystem supports
compression)
That script you could package up and make available for everyone who
needs it. Probably with config knobs to specific sizes. Probably with
more iterations as people come up with pretty interesting partition
schemes if you give them the option to… perhaps indeed moving to a
pre-install-pkgs hook later on to make better guesses at what a given
upgrade might need (install of two packages, probably fine… upgrading
2000 package ~ now we are talking!) …
If a few months down the line we discover that there is indeed a setup
which works for – lets say – 95% of Debian users we can always set it
to prio:standard and install it by default or integrate it into apt
so (basically) everyone gets it everywhere, but until then it is more
beneficial for you and us if we are not burdening ourselves with
making it work for everyone from day 1.
Best regards
David Kalnischkies