Re: Reducing apt's memory footprint (on small boxes)
On Sun, 14 Feb 2021 14:42:08 +0100
Christoph Biedl <debian.axhn@manchmal.in-ulm.de> wrote:
>Hello,
>
>this story isn't new: Older boxes with rather small memory. In my case,
>DockStar with 128 Mbyte RAM. They still serve a job as e.g. a router,
>but that limitation becomes more and more a problem. The biggest issue,
>at least for me, is apt although it's just the bringer of the bad news:
>The packages indexes became that big they no longer fit into memory. In
>September 2015, there was a suggestion to create subsets of a release,
>but I objected it will be more or less impossible to create them
>without breaking some builds or installations for unresolved
>dependencies.
>
>Time has passed, the problem became worse, and my dockstars are still
>on stretch for that very problem - after a buster upgrade, they would
>often crash during an automated "apt update".
>
>However, I figured the above dependency problem isn't one I really have
>to care about. If there is an unresolvable dependency, I still could
>switch to the full packages indexes if I really have to. Also, it's
>only about installing packages, building happens on machines with
>sufficient ressources.
>
>And so I started a small project:
>
>* Take the list of binary packages that are actually installed, less
> than 500 in my case.
>* Have copies of `dists/stretch/main/binary-{all,armel}/Packages`.
>* Drop all stanzas that are *not* in the above list.
>* gzip, xz. Create a Release file and sign it.
>* Place the file structure in an appropriate webroot.
>* Have a rewrite rule, so /pool/(.*) is redirected to the next Debian
> mirror.
>
>As a result: Execution time of "apt update" dropped from 52 seconds to
>some 12, and the memory pressure is gone.
>
>Next steps were to repeat that for buster, do a dist-upgrade, and
>observe how the box makes it through the next hours. Looking good so
>far, therefore I'll keep maintaining this, at least for myself.
>
>Now, from my experience: Every good idea that I believe to have found
>turned out had been prosoped earlier by somebody else and/or was not as
>good as it initially seemed. So, where's the actual catch in that
>setup? Or is that something that might be of broader interest and
>possibly even become part of a Debian release?
>
>Of course there is the problem of defining which packages should be
>included in such a "debian-narrowed"¹. I have some ideas about this but
>it's a bit too early to share them. Still, even if the list contains
>2000 binary packages, it should have everything for the tasks such a
>box would do while still being *way* smaller than the ~75k that I see
>in stretch currently,
>
> Christoph
>
>¹ Initially, I was thinking of "narrow-gauge" but dropped that because
> it abbreviates to "ng" ...
>
what are your thoughts about network booting? A lot can be accomplished
using this often forgotten method of running *nix. You can build an
image in a vm with a lot more memory, pare it down to fit your real
hardware, and boot it from storage. There are all kinds of cool things
you can do in this world, like mount the main OS read only, with only
tmp and var dirs writable, and with logs sent to a remote syslog.
'union type file systems' are a great thing in this world for allowing
multiple overlaid file systems to behave as one with per overlay
features available. Maybe keep your configurations in git that get
pulled into place after boot. So much to be creative about in this
space, that fat package managers will never even enter your mind.
For routers, this way is a good additional layer of security, as a
simple reboot 're-installs' everything clean.
Other than that Mr. Biedl, buy yourself a shiny new gizmo with more
memory, you deserve it, and good luck! Hmm... /maybe/ that's the real
reason it will no longer fit or run or ... :)
food for thought...
Christopher
Reply to: