[Date Prev][Date Next] [Thread Prev][Thread Next] [Date Index] [Thread Index]

Re: too many CD ISOs

Raphael Hertzog wrote:
Reason: it's much more effective for them when users download an ISO than
when they use jigdo. For example the ftp.fr.debian.org admin explained me
that they were limited by the disk seek and not by the bandwith used by
the users.

Right and wrong. I am not the admin of ftp.fr.debian.org (Aurélien is, I am mostly a backup) which does not host (AFAIK) CD ISOs but the admin of ftp.free.fr (AKA ftp.proxad.net) which hosts them (and many other things). Both of them are hosted on a french ISP network. As one of the purpose is to deliver the contents as fast as possible (we have around 2 millions of ADSL users with uncapped connections), we try to bypass most of the bottlenecks.

The server is a dual Opteron X2 with 16 GB of RAM, 24 SATA disks on 3x3Ware (8 disks are setup in RAID5, the rest of them in JBOD and then soft mirrored) and two aggregated Gbps links. On a regular day, there is between 700 to 1000 simultaneous connexions for a daily traffic from 4 to 6 TB, evening peak is around 800Mbps to 1 Gbps. Historic peak was with around 3K connexions, 2 GBps for a few hours and 9.5 TB delivered in a day.

Richard Atterer wrote:
> That's interesting - I hadn't heard of this kind of problem before!
> :-/

Sooner or later, you will hear more and more often about this kind of problem. Have a look to http://lwn.net/Articles/190222/ to get a brief introduction (just read "the problem" part) about disks limitations. Roughly, the problem is disks have less and less IOs/s per storage size (ie between 2006 and 2013, disks capacity should grow by 16 times, their bandwidth by 5 times and seek time by 1.2 times). Unless you are ready to scale with the number of disks, you will try to reduce the number of disks seeks (and thus try to read more and more data per seek).

16 GB of RAM is huge but with 2.5 TB of hosted data, disk IO load was huge too (some high traffic mirrors like Mandriva or Fedora were on 4 disks RAID1 volume to keep up with disk load).

So, in our case, we had to :
- force pre-reading in a smart way (cf fadvise patch in ), using 1 MB chunk allowed us to move some high-traffic mirror from 4 disks RAID1 volume to 2 disks with a lower IO load. - stop doing RAID5 volumes (excepted for low traffic mirrors) to reduce disks seeks induced by stripping (64 KB on 3Ware cards).

But we also had to disabled resume ability (some "download accelerators" are using very small chunks and most of them are mis-using server ressources).

So, as far as I am concerned, unless one has to download less than 650 packages to build up his 650 MB CD ISO, I prefer ISOs downloads to jidgo... :-)


Reply to: