Hi all,
Here is some food for thought for the minidebconf that starts tomorrow
in Cambridge [1]. Unfortunately I will not make it there, though I wish
I did, but I'm in the process of job searching at the moment[2] and
could not afford the expense.
Anyway, since I got nothing close to a proper reply to my suggestions
over IRC or email before, I thought of doing a proper proposal. Keeping
the iMX53 locos just doesn't cut it anymore, they're slow and have
started failing builds due to lack of RAM (OOM in linking stage
for iceweasel [3]), 2 out of 7 are offline, 1 has a failing disk (hoiby,
hasse had its own replaced already). And taking ~2 days to see that the
build fails is a bit irritating.
By contrast, I've built the same package, iceweasel, on my chromebook
(dual-core A15) in 11h. I'm using a USB3 external 2.5" case with an SSD
disk and the result is a very well performing system. I guess the XU
would do the same compile in about 4-6h due to its being 8-core
(4xCortex A15 + 4xCortex A7). So I propose something similar: Upgrade 5
of the iMX53s with 5 Odroid-XU and be done with armhf builders for a
long time.
This is a counter-proposal to what Hector Oron (zumbi) suggested: that
we buy or get donations for real server class hardware (arm64). In
theory that would be best as it would build all 3 ports, but it has
several drawbacks:
1. Terribly expensive: a single board costs ~7k USD. And it needs the
chassis which is also expensive. So if you take redundancy in mind, you
need 2 setups in 2 locations, so at least 14k just for the boards and
I'd not be surprised if the chassis costs more than 5k itself. So in
total ~25k. Now if we get this thing donated, even half of it, by all
means, that would be great, but the impression I got is that this is
unlikely. And I would never propose that Debian spends $25k of its
budget for arm* builders.
2. It causes a SPOF (Single Point Of Failure) risk if we go the route
of building for all 3 ports on it. If one board breaks, all 3 ports
would have to wait for a repair/replacement. Huge downtime which no
port can afford.
3. Liability. Right now armhf builders (iMX53 Loco boards) are in ARM HQ
and York University, if I'm not mistaken. If something happens and that
equipment breaks due to eg. some fault in the electrical wiring and a
board burns, the boards are very cheap to replace and there is no
issue there. However with $10k+ worth of equipment I'm very doubtful if
ARM or York or anyone really would accept to host these systems and
accept liability in case of fault.
4. Availability. In short we don't know when those arm64 boards will be
available. Most likely 2014Q1 but it's not concrete.
So, my counter-proposal to that is that we get instead some cheap easy
to replace boards like the Arndale [4] or the Odroid-XU [5]. Personally
I'd prefer the XU as it's better equipped, but I'd go with either
choice if people think it's better. This way, all 4 points above are
solved, we can even buy a couple of spares in each site, in case one
breaks down so we can replace it easily.
I've done some short list of items, a BOM if you like, converted in GBP
since that's where the boards will most likely end up (used amazon for
some parts, feel free to look elsewhere, this is just a rough cost
estimate):
Qty Item Price Total
5 Odroid-XU 105.00 525.00
5 Odroid-XU case 5.61 28.05
5 Odroid-XU serial cable 9.35 46.75
5 Odroid-XU PSU 6.23 31.15
5 2.5" USB3 SATA case [6] 28.50 142.50
5 Kingston 2.5" SSD V300 60GB 41.96 209.80
Total 983.25 GBP
So for less than 1k GBP, we can upgrade our complete buildd farm with
just 3 builders and keep 2 spares at each site, we could even keep our
current iMX53s as spares of course, but seriously why bother? Just
keep them for a while and send them to interested developers afterwards
(no I'm not interested myself, have enough MX5x hardware already as it
is).
One issue I didn't mention is how do you solve remote power and serial
console management with these boards. This is actually an advantage in
favour of the arm64 server class hardware. Well, with a simple google
search, I found several products to solve this case but I'm not an
expert so I'll let you find the best solution for this. BUT in case
anyone is interested I have a PCI Acceleport 8i cart (one PCI
multi-serial cart that extends to 8x DB25 ports, not DB9
unfortunately), which I'd be willing to donate if there is a need for
this cause -it was actually used in the original armhf EfikaMX
buildfarm to monitor each efika.
Finally, last but not least, there is the issue of mainline kernel
support for exynos5 and the odroid/arndale boards. Right now 3.11
doesn't give you much, but I've heard that the situation with 3.13 is
much better -still not completely operational, but it should get better
in time. My point with regards to mainline kernel is that it really
doesn't matter that much at the moment -yes, I know some will say it
matters, but no honestly it doesn't. In a few months time, mainline
will get to a working point just like it did with the Locos -for anyone
that remembers those used an FSL kernel initially and only moved to
mainline a few months afterwards, as soon as the parts we cared about
worked well (serial, SATA). I don't see why the same won't be valid for
the exynos5 boards.
If people *really* don't want non-mainline kernels, then we could go
for the utilite boards, I would suggest the Pro version:
http://utilite-computer.com/web/utilite-pro-specifications
It wouldn't be as fast as the exynos5, but it would be better in terms
of networking/SATA as it has dual-gigabit and 32GB SSD, though I guess
32GB SSD might just be enough for a builder. Still, better than the
current solution by far.
Anyway, apologies for my long email, I really hope you people discuss
this issue in the minidebconf the following days and I also hope we
will solve this soon, before we get more builders breaking on us.
Enjoy and have fun
Konstantinos
[1] https://wiki.debconf.org/wiki/Miniconf-UK/2013
[2] there are some talks going on but nothing really concrete yet, so
if you have an interesting project and looking for remote developers,
send me an email :)
[3]
https://buildd.debian.org/status/fetch.php?pkg=iceweasel&arch=armhf&ver=24.1.0esr-1&stamp=1383943405
[4] http://www.pyrustek.com/
[5]
http://www.hardkernel.com/renewal_2011/products/prdt_info.php?g_code=G137510300620
[6] I'm using this one personally, but feel free to choose whichever
you feel like
http://www.amazon.co.uk/gp/offer-listing/B008PDD6LE/ref=dp_olp_0?ie=UTF8&condition=all
[7] Again this is what I use, or its 120GB version, this one would work
fine for a builder:
http://www.amazon.co.uk/Kingston-Technology-Solid-2-5-inch-Adapter/dp/B00A35X6GM
Attachment:
pgpn2mjnDuBjD.pgp
Description: PGP signature