[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: buildd reliability, was: Is an ARM computer a good choice? Which one?



Hi,

Let me answer that with both my wanna-build and DSA hats.

On 2023-03-25 12:19, Wookey wrote:
> On 2023-03-25 13:46 +0800, Paul Wise wrote:
> > On Thu, 2023-03-23 at 23:33 +0000, Wookey wrote:
> > 
> > > The arm64 servers debian uses for buildds are fine.
> > 
> > DSA often complain on #debian-admin about how flaky they are and often
> > have to reset them, there were a few jokes about rebooting from cron,
> > also the release team arch qualifications have a note about this:
> > 
> >  * concerns-dsa: arm64/armhf/armel: yes: unstable and ageing hardware
> 
> I still think that's referring to the 32-bit machines. I'm a buildd

The 32-bit buildds are running quite stable besides a few disks dying,
given their age.

However they are difficult to manage remotely (they don't start
automatically after power cycle) and more importantly they are slow,
only have 4GB of RAM and are limited to 3GB address space.

We were using them until recently, but given that the kernel has gotten
support for building armel and armhf in a chroot, they are currently
disabled, although they are not yet decommissioned.

> admin for those ports and the machines on this site and I'm not aware
> of problems with the 64-bit machines, and I don't get Nagios messages
> about them having gone down/back up, like I do about the armhf/armel
> hardware. Perhaps they are happenning but I'm not on the right
> alias/list so don't see them?

The 3 arm64 boards running at ARM are pretty fine, we do not have any
issues with them, however they start to be old.

On the other hand we have many issues with the Ampere servers hosted at
UBC and the Applied Micro servers hosted at Conova. All of them crash
regularly (a few times per week in total) and need a powercycle. In
addition the bullseye kernel does not work on Applied Micro servers, so
we are currently stuck with buster on them :(.

> I thought I was getting all monitoring
> messages for machines on this site

My guess is that you don't receive any notification as the buildds
running on arm64 hardware are VMs on the above hosts. From that point of
view, the buildds are running fine, however the underlying hosts are
not.

> So if there is an issue there we have a communications problem as well
> as a hardware problem. I'm sure we can get new arm64 buildds if we
> need them.

Yes please. It's becoming urgent to get new ARM64 hardware to overcome
all those issues, and we (DSA) failed to find new hardware to buy at a
decent price.

Aurelien

-- 
Aurelien Jarno                          GPG: 4096R/1DDD8C9B
aurelien@aurel32.net                 http://www.aurel32.net

Attachment: signature.asc
Description: PGP signature


Reply to: