Re: how can we help DSA save money

[Steven Chamberlain <steven@pyro.eu.org>]

Hi!

On 2013-07-21 08:09, Tollef Fog Heen wrote:
> Backups is 8 x 4T Seagate Constellation drives.  Bytemark is 24 x 4T 
> Seagate Constellation drives.  We get setup, hosting, power, etc 
> donated, so that is not part of the cost there.

Thanks;  was this just a purchase of drives, or also a new storage
appliance / enclosure for them?

Having these numbers allows for some very rough estimates of cost saving
from different ideas.

> It's not really possible to code ourselves out of this one.

It's a challenge of course.  I think these are fun and educational;  I
like to keep things like this in mind over a period of months, while
working on unrelated projects.

This problem is unlimited in scope, and the benefit of any change could
possibly extend beyond snapshot.d.o, e.g. to mirrors or end-users.

* if maintainers knew the true cost of snapshot.d.o it may affect their
upload habits

* we can measure impact of xz and other compression of .debs

* dedup.d.n seems it could help reduce unnecessary growth of the archive
in future

* de-duplicating between *versions* of a package is another area of
interest;  just one method of that is:-

On 2013-07-22 09:31, Philipp Kern wrote:
> Well, we could make snapshot store binary deltas.
> [...] Just sayin', not suggesting that we should do that.

Exactly, and with some numbers and costs we can evaluate if it's
worthwhile or practical.  That may change over time depending on
compute/storage cost tradeoff or through different techniques.

At a lower level we can consider the capabilities of the storage system,
which is really software.  Depending on that, it may have been practical
to manage with just 6 disks initially and add more later (when they may
be half the price or less).  OTOH its performance may then be
insufficient, but then it could be potentially fixed in software of the
application.

So yes, I think this certainly could be seen as a software challenge.
Thanks again for all the details and your thoughts.

Regards,
-- 
Steven Chamberlain
steven@pyro.eu.org