System unusably slow after Debian upgrade.
Hi there,
Long time Linux user here, very familiar with tools for system
administration but somewhat stumped by the behaviour of a system
installed by me about six years ago at a local farm. It's an old
Intel 'NUC' like this one:
https://ark.intel.com/content/www/us/en/ark/products/78577/intel-nuc-kit-de3815tykhe.html
It has an Intel E3815 CPU (~1.5GHz), 8GBytes of RAM, 250Gbyte SSD. As
luck would have it there's a second system on the site, identical save
for a 1Tbyte laptop-style spinning disc, but I'll come to that later.
The 'problem' box is used for all the usual stuff (email and document
processing, scanning, printing etc.) plus banking, online orders from
suppliers, security monitoring (some custom Perl scripts and 'motion',
using a variety of cameras), and United Kingdom VAT returns through a
combination of a Windows 7 VM under VirtualBox (fired up quarterly -
to power some Microsoft Office scripts and Microsoft's browser - for
what should be ten minutes but what in reality, through no fault of
the computer, is usually nearer a day). Remote management is by VPN,
I use Nagios/Icinga and Smokeping for routine monitoring, plus shell
access for any ad-hoc stuff. The users at the farm don't have root;
they don't even know what it is and don't want to.
Running Debian 'Jessie', until now it has performed very acceptably.
Not of course the quickest box you've ever seen but perfectly capable
of carrying the load. I left moving to a later edition of Debian for
as long as I felt comfortable because I don't like change for the sake
of change and because of misgivings shortly to be more than justified
about the likely result. Last week, because Buster's now been out for
a year, I thought it's time. Big mistake.
As the box was still running Jessie I had to do the move in two bites.
First move from Jessie to Stretch, then from Stretch to Buster. The
entire process was a bit long-winded and spread over a couple of days,
but seemed to go smoothly enough. Immediately, the users started to
complain about performance. Not just a small reduction, but the sort
of thing that makes the whole system completely unusable. My estimate
after looking at the response on the desktop is several hundred times
slower than normal for this box. If you click on a message in the
list of messages presented by the mail client, instead of seeing the
message in under a second you can go and have a coffee break and still
get back before it's shown on the screen.
The usual admin tools like top, atop, sysstat, sar, free, don't give
me much to go on. The system load averages are elevated to an extent,
but 'top' doesn't show any particular processes hogging CPU. IO wait
times are if anything less on the problem machine than they are on the
box which is still running Jessie and disc accesses aren't excessive.
Many hours of searching later I've drawn a blank. One suggestion was
that the xserver-xorg-video-intel package might be a problem but I've
uninstalled that with no obvious effect. I first disabled, and later
removed, AppArmor - same result. It's as if the box has just decided
to put its feet up.
To get the farm office back up and running I took the office manager's
home directory from the problem machine, dropped it onto the other NUC
which still runs Jessie, and replaced the problem box with that. The
office could at least then pay the wages. It might be worth noting
that it took about two and a quarter hours to gzip that home directory
(about four gigabytes) on the problem machine (Buster), and about five
minutes to gunzip it on the Jessie box.
What can possibly cause such a drastic reduction in performance?
--
73,
Ged.
Reply to: